At Hirize, we’ve spent years building and stress-testing document extraction systems across hundreds of millions of pages. One pattern is consistent across every benchmark we run: document extraction is not a single-metric problem. It’s a multi-dimensional engineering challenge where simple averages hide catastrophic failure modes.<\/p>\n
This post is a field guide for engineers who have lived through the grind of parsing messy PDFs, spreadsheets, scanned faxes, and multi-column filings. We’ll break down a rigorous evaluation methodology that goes beyond surface-level accuracy scores.<\/p>\n
Most document AI discussions rely on a single number\u2014usually accuracy or F1 on a small corpus of clean PDFs. That’s comfortable for comparison, but it ignores four independent axes of failure that each make extraction outputs unusable in production:<\/p>\n
If any one of these fails, the entire document pipeline fails.<\/p>\n
A page can have perfect OCR but broken reading order that renders text incoherent to any LLM in a retrieval system. A table can have correct wire detection but incorrect structure that destroys header-to-data relationships. A contract can have accurate paragraphs but orphaned cross-references that make it impossible to answer basic questions.<\/p>\n
This is why Hirize evaluates document extraction along multiple orthogonal axes\u2014not just a single accuracy score.<\/p>\n
Our document intelligence evaluation framework scores extraction models along orthogonal axes, with dataset slices and decision rules per axis. The goal is to measure not only average performance, but variance under real operating conditions.<\/p>\n
We evaluate extraction at multiple nodes in the pipeline. For example, Hirize computes text similarity per region immediately after OCR and again after structure mapping to catch ordering or structure regressions.<\/p>\n
Reading order breaks whenever models process pages in raster order. Enterprise documents treat pages as physical constraints, not logical boundaries. The result is semantic fragmentation unless the pipeline reconstructs continuity explicitly.<\/p>\n
Hirize uses a four-stage reading order subsystem:<\/p>\n
1. Pre-segmentation<\/strong> identifies blocks, figures, tables, headers, and footers. We learn a coarse taxonomy of block types to feed different downstream logic.<\/p>\n 2. Column graph construction<\/strong> converts spatial relations into a directed acyclic graph encoding containment and adjacency. This prevents naive left-to-right ordering from crossing columns or mixing headers\/footers with main content.<\/p>\n 3. Cross-page linking<\/strong> carries context\u2014section titles, table headers, figure references\u2014across page boundaries. We maintain state for open sections and tables, similar to a streaming parser.<\/p>\n 4. Semantic stitching<\/strong> merges block sequences into paragraphs and sections, preserving hierarchy and reference links.<\/p>\n We evaluate reading order as minimum edit distance over block sequences. Let \u03c0* be the ground truth ordering and \u03c0\u0302 the predicted ordering<\/p>\n We also report Kendall tau rank correlation to capture pairwise inversions.<\/p>\n Hirize slices ROAcc by layout pattern: single column, two column, multi-column with sidebars, and newspaper-style with interstitial artifacts. We report separate scores for cross-page continuity by computing the metric only on edges that cross page boundaries.<\/p>\n Hirize runs explicit tests for continuity failure modes:<\/p>\n Tables fail in ways that don’t show up in simple text metrics like CER or WER. Without structure, correct wire boundaries are useless. Hirize treats table evaluation as both a tree structure and a set of alignment constraints.<\/p>\n We compute Tree Edit Distance-based Similarity (TEDS) between predicted and ground truth table trees. Trees are derived from HTML or Markdown with explicit rowspan and colspan markers.<\/p>\n Hirize adds alignment tests reflecting how downstream systems actually use tables:<\/p>\n Table structure brittleness is the top source of silent data corruption in document extraction. Numeric columns that shift by one position pass text similarity checks yet break analytics pipelines. Alignment and header association metrics catch this failure class.<\/p>\n Hirize computes ANLS (Average Normalized Levenshtein Similarity) at the region level rather than page level. Regions are blocks produced by layout segmentation. This prevents high ANLS from masking incorrect ordering or boundary errors.<\/p>\n We also compute Section Boundary Detection F1 from labeled boundaries\u2014headings, list starts, table starts, figure captions, and appendix markers.<\/p>\n Per-region ANLS prevents layout errors from hiding under high page averages. The region view correlates with human review speed and reduces approval time in QA loops.<\/p>\n Hirize reports uncertainty and significance explicitly:<\/p>\n A model that wins on average but collapses on multi-column scans is not production-ready. Hirize’s evaluation framework surfaces these failure modes before they reach production.<\/p>\n Latency and throughput matter because enterprise document processing runs at scale. Hirize measures:<\/p>\n Robustness curves are more actionable than single-point scores. Noise sensitivity tests often reveal steep drops at realistic perturbation sizes\u2014engineering targets for preprocessing or model fine-tuning.<\/p>\n After processing 500M+ pages, these patterns repeat across corpora and inform engineering priorities:<\/p>\n Reading order dominates perceived quality.<\/strong> Users tolerate light OCR noise but reject outputs that read out of order.<\/p>\n Table structure brittleness causes silent data corruption.<\/strong> Numeric columns shifting by one position pass text checks but break analytics. Alignment and header association metrics catch this.<\/p>\n Per-region metrics beat per-page metrics.<\/strong> They correlate better with human review speed and QA effort.<\/p>\n Continuity checks reduce error reports in long documents.<\/strong> Persisting table headers and section state across pages reduces hallucinated headers and orphaned rows.<\/p>\n Robustness curves reveal engineering targets.<\/strong> Sensitivity tests surface where preprocessing or fine-tuning will have the highest impact.<\/p>\n Based on Hirize’s evaluation methodology, here’s what we recommend:<\/p>\n Open problems requiring new evaluation methods:<\/p>\n Evaluating document extraction isn’t about chasing a single leaderboard score. It’s about proving your system will keep working on the next million documents you haven’t seen yet.<\/p>\n The only way to do that is to measure what actually breaks pipelines in production. Reading order and table structure are the twin pillars of that effort. Treat them as first-class citizens in your evaluation, and your document extraction models will improve where it matters.<\/p>\n At Hirize, this evaluation methodology is built into our document intelligence infrastructure. Every page we process is held to these standards\u2014which is why our customers trust us with their most critical document workflows.<\/p>\nReading Order Accuracy Metric<\/h3>\n
ROAcc = 1 - EditDistance(\u03c0\u0302, \u03c0*) \/ max(|\u03c0\u0302|, |\u03c0*|)<\/code><\/pre>\n<\/div>\n<\/div>\nContinuity Checks<\/h3>\n
\n
Table Extraction: The Hardest Problem in Document AI<\/h2>\n
Structure Metric: TEDS<\/h3>\n
Alignment Metrics<\/h3>\n
\n
Text and Layout Metrics<\/h2>\n
Statistical Rigor in Document Extraction Evaluation<\/h2>\n
\n
Performance and Robustness Testing<\/h2>\n
\n
Key Insights From Evaluating Document Extraction at Scale<\/h2>\n
Practical Guidance for Document Extraction Teams<\/h2>\n
\n
The Future of Document Extraction Evaluation<\/h2>\n
\n
Building Production-Ready Document Intelligence<\/h2>\n
\n