Tables: Detection & Structure Recognition

TL;DR

ComFinTab is a 10,000-image benchmark of financial document tables, of which over 70% are compound tables: tables that integrate multiple basic sub-tables with complex, non-uniform header structures. Alongside the dataset, the authors propose CTUNet, a multi-modal graph-based framework that jointly handles table structure recognition and a new unified “table item extraction” task defined over the dataset.

What kind of paper is this?

Dominant: $\Psi_{\text{Resource}}$: The headline contribution is the ComFinTab dataset and its associated annotation scheme. The paper devotes substantial space to collection methodology, annotation pipeline, and the formal task definition that the dataset enables. No existing public dataset provided compound-table coverage with both structure and understanding annotations for image-based tables.

Secondary: $\Psi_{\text{Method}}$: CTUNet is a non-trivial multi-modal framework combining Faster R-CNN cell detection, RoI-Align visual features, BERT text features, positional embeddings, and a graph attention network for joint cell classification and relation linking. It serves primarily as a strong baseline to validate the new task formulation rather than as a standalone architectural contribution.

What is the motivation?

Real-world financial documents routinely contain tables that go beyond the three basic forms (relational, entity, matrix) described in prior taxonomies. These “compound tables” integrate multiple basic sub-tables into a single grid, so cells in the same row or column may belong to entirely different semantic groups and cannot be correctly linked by simple header-chasing heuristics.

Existing image-based datasets such as PubTabNet (568k), FinTabNet (113k), and SciTSR (15k) contain fewer than 2% compound tables. Digital-format table understanding datasets (WebSheet, SAUS, DeEX, Spider, WikiSQL) use spreadsheet or database representations that carry color, font, and structural metadata unavailable in scanned images. The result is a gap: no public benchmark tests visual table understanding on the challenging compound tables that appear routinely in financial filings.

The paper identifies a secondary motivation: the proliferation of incompatible task formulations across prior table understanding work (table type classification, cell type classification, column type identification, entity linking, Table QA). The authors argue these tasks can largely be unified into a single “table item extraction” formulation, and use ComFinTab to instantiate and evaluate that formulation.

What is the novelty?

Dataset. ComFinTab contains 10,000 table images from public annual reports of companies listed on the Shanghai and Shenzhen Stock Exchanges: 6,000 Chinese-language and 4,000 English-language tables. Over 70% are compound tables. Annotations cover cell bounding boxes, row and column indices, text bounding boxes, text content, cell type (TH, LH, DA, OT), and cell linking (which data cells are linked to which header cells). This combination of structure and understanding annotations in an image-based, majority-compound-table dataset is the first of its kind.

Task formulation. The paper proposes a “table item extraction” task. Each data (DA) cell is the root of a tree: the left subtree stores the left-header (LH) hierarchy and the right subtree stores the top-header (TH) hierarchy, with virtual header nodes added when a sub-table lacks one axis of headers. This representation encodes cell type and cell linking jointly, and can losslessly express the outputs of prior cell-type classification and cell-linking tasks for basic tables.

Evaluation metric. The authors introduce Tree-F1-Score, which extends standard F1 with Tree-Edit-Distance-Based Similarity (TEDS) as a soft matching credit. Formally:

$$ \begin{aligned} \text{Tree-R} &= \frac{\sum_{t_i \in T_G} \text{TEDS}(t_i, t_i’)}{|T_G|} \\ \text{Tree-P} &= \frac{\sum_{t_i \in T_P} \text{TEDS}(t_i, t_i’)}{|T_P|} \end{aligned} $$

$$ \text{Tree-F1} = \frac{2 \times \text{Tree-R} \times \text{Tree-P}}{\text{Tree-R} + \text{Tree-P}} $$

where $T_P$ and $T_G$ are predicted and ground-truth tree sets, and $t’_i$ is the predicted tree sharing the same root DA cell as $t_i$. Items with no corresponding root in the counterpart set score TEDS = 0.

CTUNet. The framework fuses three feature modalities per detected cell:

• Position feature: $PE_i = \text{embedding}(b_i)$ from the bounding box coordinates $b_i = (x_1, y_1, x_2, y_2)$, $PE_i \in \mathbb{R}^{d_F}$.
• Visual feature: RoI-Align crop from the Faster R-CNN feature pyramid, followed by convolutions and a linear projection, $V_i \in \mathbb{R}^{d_F}$.
• Textual feature: BERT (or BERT-Chinese) encoding of cell text content, $T_i \in \mathbb{R}^{d_F}$.

The combined cell representation is:

$$F_i = \text{LayerNorm}(PE_i + V_i + T_i)$$

A graph attention network (GAT) with a masked self-attention mechanism propagates features within each cell’s structural neighborhood (all cells in the same row or column). Enhanced node features $F’_i$ are computed as:

$$ \begin{aligned} e_{ij} &= \text{LeakyReLU}(w^T [WF_i | WF_j]) \\ \alpha_{ij} &= \frac{\exp(e_{ij})}{\sum_{k \in N_i} \exp(e_{ik})} \\ F’_i &= \sigma!\left(\sum_{j \in N_i} \alpha_{ij} W F_j\right) \end{aligned} $$

Edge features are formed by pairwise concatenation $E_{i,j} = [F’_i | F’_j]$. Three branches are trained simultaneously: node classification (cell type, 4 classes), row relation linking (sigmoid binary), and column relation linking (sigmoid binary). The combined loss is:

$$L = L_{\text{det}} + \lambda_1 L_{\text{node}} + \lambda_2 (L_{\text{rlink}} + L_{\text{clink}})$$

where $L_{\text{det}}$ is the Faster R-CNN detection loss and the remaining terms are cross-entropy losses.

What experiments were performed?

The paper evaluates on ComFinTab-Chinese (4,500 train / 1,500 test) and ComFinTab-English (3,200 train / 800 test), split at company level to prevent format leakage between splits.

Because no prior method directly addresses the table item extraction task, the authors construct two comparison settings:

1. Cell Classification + Rules. TUTA (a table-pretrained language model) and LayoutLMv2 (a multi-modal document model) are fine-tuned for cell type classification, then handcrafted adjacency rules link headers to data cells. TUTA is English-only; LayoutLMv2 is tested on both splits.

2. End-to-End Relation Extraction. The task is recast as triple extraction: (‘DA 1’, ‘Left-Linking’, ‘LH 1’) etc. AGCCN (an NLP relation extraction GCN) is adapted and augmented with the same multi-modal features used in CTUNet.

All baselines and CTUNet use the same OCR source: PaddleOCR for Chinese, Tesseract for English. A ground-truth oracle variant (GT) is also reported to isolate table understanding performance from OCR errors.

Table structure recognition quality (TEDS) is reported as a reference: CTUNet achieves 98.99 on English and 98.83 on Chinese, reflecting that bordered financial tables are relatively straightforward structurally once cell detection is accurate.

Ablations test (a) the contribution of each modality (position, visual, semantic) and (b) neighborhood definitions in the GAT (none, 1-step, all-pairs, row/column). The row/column neighborhood setting outperforms all others on both splits.

What are the outcomes/conclusions?

On the table item extraction task (Tree-F1), CTUNet substantially outperforms rule-based approaches and modestly outperforms the end-to-end AGCCN baseline:

Rule-based methods struggle specifically on compound tables: cell classification accuracy is adequate (91-92%), but heuristic header-to-data matching fails when the same row or column spans multiple logically independent sub-tables.

Ablation results confirm that all three modalities contribute. Removing position features causes the largest drop (Tree-F1 falls to roughly 31-38%), reflecting the centrality of spatial layout in table reasoning. Visual and textual features each add 5-6 points independently; together they further improve over any single-modality baseline.

The GAT neighborhood ablation shows that using row/column neighbors (the structural prior) outperforms both 1-step spatial neighbors and all-pairs connections, suggesting that long-range row/column context is genuinely useful and that the structural graph provides meaningful inductive bias.

Limitations noted or apparent: The model relies on an offline OCR engine, so OCR errors propagate into the understanding stage (the GT oracle consistently scores 1-2 points higher). The dataset is restricted to bordered, horizontally-oriented tables extracted from PDF annual reports, which is a narrower distribution than general document tables. The annotation pipeline used heuristic pseudo-labels for cell types (subsequently manually corrected), and heuristic linking rules (also manually cleaned), leaving some residual annotation noise. The authors do not report inter-annotator agreement statistics.

Reproducibility

Models

CTUNet uses a ResNet-50 backbone with an FPN neck, instantiated within a Faster R-CNN cell detector. RoI-Align crops from the FPN map feed a small CNN stack followed by a linear projection to produce visual features. BERT-base (multilingual or Chinese variant) produces textual features. A single-layer GAT with masked self-attention constitutes the relational graph construction module. The paper does not specify the feature dimension $d_F$ explicitly; the re-implementation in DAVAR-Lab-OCR targets the same architecture. Pre-trained model weights (gated download; access codes in the DAVAR-Lab-OCR README) are provided separately for Chinese and English variants; re-implementation numbers differ slightly from paper results (Chinese: 93.59 vs. 91.78 Cell-F1; English: 92.75 vs. 92.27).

Algorithms

Training uses SGD with momentum 0.9, weight decay $1 \times 10^{-4}$, initial learning rate $1 \times 10^{-3}$, divided by 10 every 20 epochs, batch size 4. OCR is handled offline by PaddleOCR (Chinese) or Tesseract (English) and is not part of the end-to-end training graph. Loss weights $\lambda_1$ and $\lambda_2$ are not stated explicitly in the paper. All experiments run on 8 Tesla V100 GPUs.

Data

ComFinTab contains 10,000 table images from public Chinese-listed company annual reports (Shanghai and Shenzhen Stock Exchanges). Split by language: 6,000 Chinese, 4,000 English. Train/test splits are 4,500/1,500 (Chinese) and 3,200/800 (English), partitioned at the company level.

Annotation pipeline: (1) bordered table detection and cell-line crossing to extract cells; (2) LGPMA cell-matching strategy for row/column indices; (3) PDFPlumber for text locations and content; (4) color-based auto-labeling for cell types, refined by a text classifier for pseudo-labels, then manual correction; (5) heuristic linking rules, manually cleaned.

The dataset is publicly available via a gated application process (application form required; email to the corresponding author). License: CC-BY-NC-SA-4.0 (non-commercial only). See ComFinTab dataset page.

Evaluation

Primary metric: Tree-F1-Score (TEDS-weighted precision and recall over tree representations of table items). Secondary metric: Macro-F1 for cell type classification. Table structure quality is also reported via TEDS (the standard TSR metric from PubTabNet). All baselines and CTUNet use the same OCR pipelines. Ground-truth oracle variants are reported to isolate understanding from OCR quality. No error bars, confidence intervals, or multi-seed averages are reported.

Hardware

All experiments use 8 Tesla V100 GPUs. Total training time and GPU-hours are not reported.

BibTeX

@inproceedings{li2022comfintab,
  author    = {Zaisheng Li and Pengfei Li and Shiliang Pu and Yi Li and Zhanzhan Cheng and Xi Li and Qiao Liang and Yi Niu},
  title     = {End-to-End Compound Table Understanding with Multi-Modal Modeling},
  booktitle = {Proceedings of the 30th ACM International Conference on Multimedia (MM '22)},
  year      = {2022},
  pages     = {4112--4121},
  doi       = {10.1145/3503161.3547885},
  publisher = {ACM}
}

TL;DR

ICDAR 2021 SLP Task B is the competition track for full table recognition (structure plus cell content) on PubTabNet. With 30 teams and 30 final-phase submissions, the top systems reached 96.36 TEDS overall (97.88 on simple tables, 94.78 on complex), well above the 91 TEDS baseline reported with PubTabNet’s original EDD model. Two-stage pipelines (cell detection then structure inference) dominated over end-to-end sequence methods.

What kind of paper is this?

Dominant: $\Psi_{\text{Resource}}$: The primary contribution is a competition benchmark event. The paper documents the evaluation protocol, participation statistics, leaderboard results, and system descriptions. No new model or dataset is introduced; PubTabNet (v2.0.0) provides the underlying data.

Secondary: none.

What is the motivation?

Prior table recognition competitions (ICDAR 2013, ICDAR 2019) were small (156 and 3,600 samples respectively) and did not require systems to recover cell content, only cell bounding boxes or adjacency relations. Task B raised the bar: participants must produce a complete HTML representation of each table image, including both structure (row/column spans) and the text content of every cell. No intermediate cell-position annotations are provided during evaluation, pushing systems toward truly end-to-end solutions.

The competition is part of a two-task event (ICDAR2021-SLP). Task A, which addresses document layout recognition on PubLayNet, is covered in the layout notes.

What is the novelty?

The competition establishes a documented, large-scale snapshot of the state of full table recognition in 2021, with TEDS as the community-standard metric. The system descriptions from the top teams reveal the architectural patterns that worked best at scale, serving as a useful reference for understanding the field’s trajectory toward later models like TableFormer, TRUST, and UniTable.

What experiments were performed?

Data

Task B used PubTabNet v2.0.0. The final evaluation set (9,064 images) was released three days before the competition deadline. Participants had access to training (500,777 samples) and development (9,115 samples) sets throughout.

Metric

TEDS (Tree-Edit-Distance-based Similarity), the metric introduced with PubTabNet. It computes normalized tree edit distance between the predicted and ground-truth HTML table trees, including cell content:

$$\text{TEDS}(T_a, T_b) = 1 - \frac{\text{EditDist}(T_a, T_b)}{\max(|T_a|, |T_b|)}$$

Results are decomposed into Simple and Complex subsets following PubTabNet’s original split.

Known evaluation bug: Bold tags (<b>) were inadvertently excluded from scoring in the final evaluation phase due to a data preparation issue. All reported numbers are therefore slightly inflated for tables containing bold text. This affects all teams equally, so relative rankings hold, but absolute TEDS numbers are not directly comparable to evaluations that include bold tags.

Results

Top 9 teams on the Final Evaluation Phase leaderboard:

Prior state of the art (EDD model, reported with PubTabNet): 91 TEDS.

What are the outcomes/conclusions?

Key findings:

• Complex tables are consistently 3-4 points harder than simple ones across all teams. The gap is stable, suggesting it reflects genuine structural difficulty rather than a calibration artifact.
• Two-stage pipelines dominated: detect cells (Mask R-CNN variants, often HRNet backbone with pyramid mask supervision), then infer structure by horizontally/vertically aligning bounding boxes via maximum clique search or similar combinatorial post-processing. This approach cleanly separates cell localization from topology recovery.
• One team (VCGroup) decomposed the task into four sub-tasks: table structure recognition, text-line detection, text-line recognition, and box assignment. Their structure recognizer was based on MASTER (a scene text recognition model), demonstrating that sequence models from adjacent domains transferred well.
• A pure sequence approach (Kaen Context, Kakao Enterprise) using a 12-layer linear-attention transformer operating on flattened image patches also competed, anticipating the later EDD/TableFormer direction.
• Performance improvement over EDD (91 TEDS) is substantial (~5 points), driven by larger backbones, ensemble strategies, and better OCR modules.

Limitations:

• The <b> bold tag bug means all reported TEDS numbers are slightly inflated. The magnitude of the effect depends on how bold-heavy the evaluation tables are; the authors acknowledge it but do not quantify the impact.
• Only 30 teams participated in the final phase, far fewer than Task A (78 teams). Table recognition attracted less participation, likely due to higher implementation complexity.
• The competition uses PubTabNet, which is scientific literature only. Transferability to financial tables (FinTabNet) or wild tables (WTW) is not evaluated.
• No source code was released by competition organizers; most system descriptions are high-level.

Reproducibility

Models

No trained weights were released by the competition organizers or most participating teams. Davar-Lab-OCR referenced their lab website and later released code via DAVAR-Lab-OCR GitHub; VCGroup released MASTER-pytorch (the backbone of their structure recognizer).

Algorithms

Top two-stage systems: Cascade Mask R-CNN with HRNet-W48 backbone for cell detection (with pyramid mask supervision for row/column-aligned bounding boxes), followed by structure inference via alignment overlap analysis and maximum clique search for row/column index assignment. OCR was handled by a separate single-line text detection and recognition model. Optimizer, learning rate schedule, batch size, and other training hyperparameters are not reported in the competition overview paper; individual teams did not provide full training details in their system descriptions.

Data

PubTabNet v2.0.0 is publicly available. Annotations are under CDLA-Permissive-1.0; underlying PMCOA images have mixed per-article licenses. See PubTabNet notes for full dataset details.

Evaluation

TEDS computation code is available in the PubTabNet repository. Competition evaluation harness at ICDAR2021-SLP GitHub (Apache-2.0). The online leaderboard is no longer active.

Statistical rigor: single-run competition results; no error bars, confidence intervals, or multi-seed averages are reported. Rankings should be interpreted with caution given the small score differences between top teams (e.g., rank 1 vs. rank 2 differ by 0.04 TEDS overall).

Hardware

Not reported. Individual teams used V100-class GPUs; no aggregate compute figures available.

BibTeX

  Copy
  

@inproceedings{yepes2021icdar,
  title={ICDAR 2021 Competition on Scientific Literature Parsing},
  author={Jimeno Yepes, Antonio and Zhong, Peter and Burdick, Douglas},
  booktitle={Document Analysis and Recognition -- ICDAR 2021},
  year={2021},
  publisher={Springer},
  doi={10.1007/978-3-030-86337-1_40}
}

TL;DR

Horn and Keuper evaluate 21 contemporary PDF parsers on table extraction using 100 synthetically generated documents with exact LaTeX ground truth. Their central finding is that LLM-as-a-judge evaluation correlates substantially better with human judgment (Pearson $r = 0.93$ for Gemini-3-Flash-Preview) than rule-based metrics such as TEDS ($r = 0.68$) or GriTS ($r = 0.70$). Among the 21 parsers tested, the Gemini 3 family achieves the highest scores, while rule-based tools lag far behind all learning-based approaches, and parser choice can mean the difference between near-perfect and nearly unusable table extraction.

What kind of paper is this?

Dominant: $\Psi_{\text{Evaluation}}$ The paper’s primary contribution is a new evaluation methodology for table extraction. It proposes LLM-as-a-judge as a replacement for rule-based metrics, validates that methodology against over 1,500 human ratings, and applies it to benchmark 21 parsers. The headline results concern measurement reliability and parser rankings, not a new model architecture.

Secondary: $\Psi_{\text{Systematic}}$ The work also surveys the landscape of existing table extraction metrics and document-level benchmarks, systematically characterizing the failure modes of TEDS, GriTS, and SCORE with concrete examples from real parser outputs.

What is the motivation?

Accurate table extraction from PDFs is increasingly important for scientific data mining, language model pretraining, and retrieval-augmented generation pipelines. Yet the dominant evaluation metrics, TEDS and GriTS, operate purely at the syntactic level. Both metrics compare structure and cell content as strings, which means they penalize semantically harmless representational differences (format conversions, symbol encoding variants, value equivalences like “85.0%” vs. “85%”) while being nearly insensitive to small but meaning-altering errors (a lost decimal point, a sign flip).

This creates a measurement gap: a parser that produces structurally different but semantically equivalent tables is unfairly penalized, while one that preserves structure but corrupts a handful of cell values receives an inflated score. Existing document-level benchmarks such as OmniDocBench and READoc adopt the same metrics without addressing this flaw. Separately, most benchmarks rely on manually annotated ground truth, which is expensive and limits scale and reproducibility.

What is the novelty?

The paper makes three interconnected contributions.

LLM-based semantic evaluation. The authors propose using an LLM as a judge to evaluate table extraction quality. Given a ground truth table and its parsed counterpart, the model assigns scores on a 0 to 10 scale for content accuracy and structural preservation, defined as whether every cell value can be unambiguously mapped to its row and column headers. This framing captures semantic equivalence that string-level comparison cannot.

Synthetic benchmark with exact ground truth. Rather than relying on manual annotation, the authors construct 100 benchmark pages by embedding real LaTeX tables sourced from arXiv papers (published December 2025, to avoid overlap with parser training data) into synthetically generated PDFs. Each page samples a random layout configuration (document class, font, margins, column layout) and fills available space with filler text and tables, compiled with pdflatex. The LaTeX source serves as exact ground truth at zero annotation cost. Tables are classified into three structural complexity tiers by an LLM classifier: simple (regular grid), moderate (limited cell merging), and complex (multi-dimensional merging, nested structures).

LLM-based table matching pipeline. Because parsers produce tables in diverse formats (HTML, Markdown, LaTeX, plain text) and may split, merge, or reorder content, aligning parser output to ground truth tables is non-trivial. The authors address this with a Gemini-3-Flash-Preview matching step that identifies and extracts the parsed counterpart of each ground truth table from the full parser output, followed by rule-based post-validation to correct minor LLM artifacts.

What experiments were performed?

Meta-evaluation of metrics. To compare automated metrics against human judgment, the authors collected 1,554 human ratings across 518 ground truth/parser output table pairs, sampled from diverse parsers, output formats, and table complexities. Three independent evaluators each rated all 518 pairs on a 0 to 10 scale. Inter-annotator agreement is reported as Krippendorff’s $\alpha = 0.77$ (interval), with average pairwise Pearson correlation $r = 0.85$ and a human ceiling of $r = 0.89$ (leave-one-out). To help surface subtle discrepancies, evaluators were shown LLM-generated hints identifying potential differences, though final scores remained entirely human decisions.

Four LLM judges were evaluated: DeepSeek-v3.2, GPT-5-mini, Gemini-3-Flash-Preview, and Claude Opus 4.6. All rule-based metrics (TEDS, GriTS variants, SCORE variants) were rescaled to a 0 to 10 range for comparability.

Correlations with mean human scores across 518 pairs:

$^\dagger$ Also used to generate error hints shown to evaluators; correlation may be inflated.

The authors note that Claude Opus 4.6 also generated the error hints shown to human evaluators, which may inflate its correlation. Gemini-3-Flash-Preview ($r = 0.927$) had no role in the annotation process, making it the recommended judge for the subsequent parser benchmark.

Parser benchmark. Using the validated Gemini-3-Flash-Preview judge, the authors evaluated 21 parsers on 451 tables from the 100 synthetic pages, reporting mean LLM score (0 to 10), per-complexity breakdowns, TEDS for comparison, and inference cost or wall-clock time on an NVIDIA RTX 4090. The parser set spans a broad spectrum: specialized OCR VLMs (LightOnOCR-2-1B, dots.ocr, MonkeyOCR-3B, GOT-OCR2.0, Chandra, olmOCR-2-7B, Nanonets-OCR-s, DeepSeek-OCR, MinerU2.5), commercial API services (Mathpix, Mistral OCR 3), general-purpose multimodal models prompted for Markdown output (Gemini 3 Pro, Gemini 3 Flash, Gemini 2.5 Flash, Qwen3-VL-235B, GLM-4.5V, GPT-5 mini, GPT-5 nano, Claude Sonnet 4.6), and rule-based tools (PyMuPDF4LLM, GROBID).

What are the outcomes/conclusions?

LLM evaluation vs. rule-based metrics. All rule-based metrics cluster in a narrow band of $r = 0.56$ to $0.70$ with human judgment. Even the weakest LLM judge (DeepSeek-v3.2, $r = 0.80$) surpasses the best rule-based metric. The authors argue this confirms that semantic assessment captures dimensions of table quality that string-level comparison systematically misses. TEDS scores for the 21 parsers cluster within 22% of its scale (0.66 to 0.88), while LLM scores span 38% (5.75 to 9.55), suggesting the former paints a misleadingly uniform picture of parser quality.

Parser rankings. Overall LLM scores range from 2.10 (GROBID) to 9.55 (Gemini 3 Pro). Selected results:

The top two systems are general-purpose multimodal models (Gemini 3), not dedicated OCR tools, suggesting broad visual-linguistic capabilities transfer well to table extraction. However, LightOnOCR-2-1B achieves 9.08 with only 1B parameters, running on a single consumer GPU within 30 minutes, narrowing the gap between proprietary APIs and self-hosted pipelines for privacy-constrained deployments.

Complexity sensitivity. Table complexity affects parsers unevenly. Gemini 3 Flash actually scores slightly higher on complex tables than on simple ones, while GLM-4.5V drops 2.19 points from simple to complex, Qwen3-VL-235B drops 1.56 points, and Mathpix drops 1.55 points. The authors identify handling multi-dimensional cell merging as a key differentiator.

Score distributions. Per-parser histograms reveal bimodal failure patterns that mean scores obscure. Top parsers (Gemini 3, LightOnOCR) concentrate more than 70% of tables at score 10. Claude Sonnet 4.6 and olmOCR-2-7B show strongly bimodal distributions: they frequently omit tables entirely (score 0) but extract near-perfectly when they succeed. Mid-tier parsers such as GPT-5 mini and Gemini 2.5 Flash produce broad distributions centered around scores 5 to 8, indicating pervasive partial errors. The authors note that, depending on the application, a missed table may be preferable to a corrupted one, making bimodal parsers more useful than uniformly mediocre ones in certain contexts.

Limitations. The authors acknowledge several scope restrictions. Synthetic PDFs do not capture the full diversity of real-world documents (scanned pages, non-standard layouts). The table source is exclusively arXiv, which may bias toward scientific formats and leave financial or medical table styles unrepresented. LLM-as-a-judge is not infallible and depends on proprietary models, though evaluation costs are described as modest (approximately $0.20 to score all 451 tables, roughly $1 per parser for a full benchmark run). The authors do not report statistical significance for parser score differences, which may matter for closely ranked systems.

Reproducibility

Models

The paper does not introduce a new model. The judge model used for the parser benchmark is Gemini-3-Flash-Preview (Google DeepMind), accessed via API. The matching pipeline also uses Gemini-3-Flash-Preview. For the human evaluation study, Claude Opus 4.6 was used to generate discrepancy hints shown to raters. No model weights are released as part of this work.

Algorithms

The benchmark pipeline proceeds as follows:

1. LaTeX tables are scraped from arXiv papers (December 2025), cleaned, and classified into simple/moderate/complex tiers by an LLM classifier.
2. Synthetic PDF pages are generated by sampling random layout configurations and filling pages with filler text and tables, compiled with pdflatex.
3. Each of the 21 parsers processes the 100 PDF pages.
4. A Gemini-3-Flash-Preview matching step identifies the parsed counterpart for each ground truth table in the parser output, followed by rule-based post-validation.
5. Matched pairs are scored by the LLM judge on a 0 to 10 scale for content accuracy and structural preservation.

The evaluation pipeline, parser configurations, exact prompts, and software versions used to produce the leaderboard are available in the pdf-parse-bench repository. The metric meta-evaluation code and human study data are in the table-metric-study repository.

Data

The benchmark consists of 100 synthetically generated PDF pages containing 451 tables derived from arXiv papers published in December 2025. Ground truth is the original LaTeX source, requiring no manual annotation. The dataset is released as part of the pdf-parse-bench repository and on Hugging Face (piushorn/pdf-parse-bench) under the MIT license.

For the human evaluation study, 518 ground truth/parser output table pairs were rated by three independent evaluators, yielding 1,554 total ratings. The human study data is available in the table-metric-study repository.

Evaluation

The primary metric is the LLM judge score (0 to 10) using Gemini-3-Flash-Preview, selected over Claude Opus 4.6 for cost efficiency and lack of confounding with the human annotation process. TEDS (0 to 1 scale) is reported alongside for comparison purposes.

Rule-based baselines evaluated in the meta-study: TEDS, GriTS$_{\text{Top}}$, GriTS$_{\text{Con}}$, GriTS-Avg, SCORE Index, SCORE Content, SCORE-Avg.

LLM judges evaluated in the meta-study: DeepSeek-v3.2, GPT-5-mini, Gemini-3-Flash-Preview, Claude Opus 4.6.

Human reference scores were collected from three evaluators rating 518 pairs, with Krippendorff’s $\alpha = 0.77$ and average pairwise Pearson $r = 0.85$.

Hardware

Parser benchmarking was run on a single NVIDIA RTX 4090 for GPU-based models, or via API for cloud-hosted models. API pricing in USD is reported at the time of writing. No training hardware is required as the benchmark evaluates existing parsers.

BibTeX

@article{horn2026benchmarking,
  title={Benchmarking PDF Parsers on Table Extraction with LLM-based Semantic Evaluation},
  author={Horn, Pius and Keuper, Janis},
  journal={arXiv preprint arXiv:2603.18652},
  year={2026}
}

TL;DR

CC-OCR is a four-track OCR evaluation benchmark from Alibaba Group and South China University of Technology covering 39 subsets, 7,058 fully annotated images, and 10 languages. It systematically evaluates large multimodal models (LMMs) on text reading, multilingual OCR, document parsing (including tables and formulas), and key information extraction, revealing specific weaknesses in text grounding, multi-orientation robustness, and structured output generation.

What kind of paper is this?

Dominant: $\Psi_{\text{Evaluation}}$: the primary contribution is a new benchmark and evaluation protocol. The paper introduces CC-OCR as a measurement framework for assessing LMM literacy capabilities across multiple tasks, scenarios, and languages. The headline novelty is “how we measure” these models, and the paper’s center of gravity lies in the benchmark design, annotation methodology, and systematic model evaluation rather than in proposing a new model or training recipe.

Secondary: $\Psi_{\text{Resource}}$: the benchmark itself represents a reusable evaluation asset: 7,058 annotated images spanning newly collected and re-annotated data, including the SOLD dataset (re-annotated open-category KIE data), all released to the research community.

What is the motivation?

Large multimodal models have demonstrated strong general visual understanding, but their performance on OCR-centric tasks that require fine-grained text perception, structured output generation, and multilingual reading is less well understood. Existing benchmarks for evaluating LMMs on OCR tasks suffer from several limitations: narrow scope (focusing on only one subtask such as line-level text recognition or formula detection), restricted language coverage (many are English-only or at best English and Chinese), and inconsistent annotation standards that make cross-model comparison difficult.

Benchmarks such as OCRBench emphasize line-level recognition, while FOX and DocLocal4K focus on document images but omit natural scenes and are restricted to one or two languages. KOSMOS2.5-Eval covers parsing but not key information extraction. The gap is a comprehensive benchmark that can simultaneously assess all major OCR-relevant capabilities of general-purpose LMMs. CC-OCR is constructed to fill that gap.

What is the novelty?

The main contribution is the CC-OCR benchmark itself, characterized by three design principles: diversity of scenarios, practicality through use of real-world captured images, and deliberate challenge construction targeting known weak points of LMMs.

Four tracks: The benchmark organizes evaluation into (1) multi-scene text reading (natural scenes, documents, web/UGC images in English and Chinese); (2) multilingual text reading across ten languages from four language families; (3) document parsing covering full-page document structuring, table recognition, and formula recognition; and (4) key information extraction in both constrained-category and open-category settings.

Challenge dimensions are orthogonal to the track structure and include orientation-sensitivity (multi-directional and curved text), grounding (predicting word-level bounding boxes), natural noise, artistic/stylized text, multilingual diversity, mathematical and chemical formula expressions, and structured output formats (LaTeX, HTML, JSON, SMILES).

SOLD dataset: A notable sub-contribution is SOLD (Structurally-rich Open Layout Dataset), created by re-annotating two existing open-category KIE datasets (SIBR and HUST-CELL) with a bottom-up annotation scheme that produces end-to-end JSON representations, including hierarchical structures and table-format key-value groups. The original link-based annotation format of those datasets was unsuitable for directly evaluating LMMs.

Evaluation metrics are tailored per track. For OCR text sequences, the authors use a full-text multi-set matching metric (Eval-Trans) that accounts for the fact that LMMs may predict text in different orders:

$$ \text{Recall} = \frac{\sum_{i}^{N} \min(c_i, c’_i)}{\sum_{i}^{N} c_i}, \quad \text{Precision} = \frac{\sum_{i}^{N} \min(c_i, c’_i)}{\sum_{i}^{N} c’_i} $$

where $u_i$ is a basic unit (character for CJK scripts, word for Latin/Cyrillic/Arabic), $c_i$ is its count in the ground-truth sequence, and $c’_i$ is its count in the predicted sequence. F1 is computed from recall and precision.

For document content structuring and formula recognition, normalized edit distance (NED) is used:

$$ \text{NED} = \frac{1}{N} \sum_{i=1}^{N} \left(1 - \frac{\text{EditDist}(P_i, G_i)}{\max(\text{len}(P_i), \text{len}(G_i))}\right) $$

For table parsing, the benchmark adopts Tree Edit Distance-based Similarity (TEDS), which accounts for both structural similarity and cell content accuracy:

$$ \text{TEDS}(T_{pred}, T_{gt}) = 1 - \frac{\text{EditDist}(T_{pred}, T_{gt})}{\max(|T_{pred}|, |T_{gt}|)} $$

For key information extraction, field-level F1 score is used, where a key-value pair is treated as a single field and any character mismatch counts as a failed extraction.

The paper also introduces a repetition ratio $R_{rep}$ to quantify hallucination in the form of repeated output, defined as the fraction of images for which the length of the longest continuous repetitive string exceeds 25% of the total output length.

What experiments were performed?

Nine LMMs are evaluated: five generalist models (GPT-4o-2024-08-06, Gemini-1.5-Pro-002, Claude-3.5-Sonnet-20241022, Qwen2-VL-72B, InternVL2-76B) and four specialist models (KOSMOS2.5, TextMonkey, Florence-2, GOT). Specialist models are those trained specifically for OCR or document understanding tasks.

The evaluation covers all four tracks. For multi-scene OCR, each model is prompted to output the full text content of the image. For multilingual OCR, prompts are in the target language. For document parsing, models are prompted to produce structured output (LaTeX for documents and formulas, HTML for tables, SMILES for molecular formulas). For KIE, models must fill in a provided JSON schema.

Three additional analyses probe specific challenge dimensions: text grounding (word-level bounding box prediction on English multi-scene subsets), multi-orientation robustness (rotating multi-scene images by 0, 90, 180, and 270 degrees), and repetition hallucination.

What are the outcomes/conclusions?

Overall ranking: Gemini-1.5-Pro achieves the highest average score (73.0), ranking first in three of four tracks. Qwen2-VL-72B ranks second overall (68.7) and first in the KIE track. Generalist LMMs consistently outperform specialist models, which the authors attribute to their larger parameter counts and broader training data.

Multi-scene OCR: Gemini-1.5-Pro scores 83.25, followed by Qwen2-VL-72B at 77.95. Performance in natural scenes is approximately 15 percentage points lower than in document scenes for most models, suggesting that text in natural environments remains a more difficult regime.

Multilingual OCR: The performance gap across languages is substantial. Asian languages (Japanese, Korean, Vietnamese) and Arabic show the weakest results across most models. Japanese scores are particularly low, which the authors attribute to the prevalence of vertical text in their collected images. Latin-family languages are generally easier, though German and Italian show lower scores, possibly due to special characters.

Document parsing: Gemini-1.5-Pro leads with 62.37. Even the top-performing model scores below 70 on individual document parsing sub-tasks (67.17 on document content structuring and 67.93 on table recognition), which the authors note as evidence of the benchmark’s difficulty. All models perform substantially better on English documents than Chinese documents; Gemini-1.5-Pro, for instance, scores about 5 percentage points higher on English tables than Chinese tables. Handwritten formula recognition is a clear weak point: nearly all models score below 50, with molecular formula recognition being a partial exception for Gemini-1.5-Pro.

Key information extraction: Qwen2-VL-72B (71.76 F1) and Gemini-1.5-Pro (67.28 F1) lead. GPT-4o and Claude-3.5-Sonnet perform notably weaker on the EPHOIE dataset. The highest-performing model still falls short of practical application thresholds, which the authors attribute to real-world noise, complex document structures, and the strict JSON output requirement.

Text grounding: Only four of the nine evaluated models support word-level grounding output. Even for these, performance drops sharply relative to recognition-only results. Gemini-1.5-Pro achieves the highest grounding score at 60.98%; all others remain below 50%. This finding indicates that fine-grained spatial understanding remains a major gap.

Multi-orientation robustness: Most models degrade significantly when images are rotated. InternVL2-76B shows the largest drop (38.92 percentage points), followed by GOT (34.86). Gemini-1.5-Pro is a notable outlier, declining by only 3.80 points, suggesting substantially better rotation invariance.

Repetition hallucination: TextMonkey produces repetitive output in 33.93% of images, and KOSMOS2.5 in 10.64%. Among generalist models, InternVL2-76B shows the highest repetition rate at 5.94%, while Claude-3.5-Sonnet shows the lowest at 0.09%. The authors identify this as a practical failure mode worth monitoring.

Failure mode observations: GPT-4o exhibits both hallucination (inventing plausible but incorrect text for long fields such as addresses) and instruction noncompliance (reformatting dates despite explicit prompts). These qualitative failures are documented alongside quantitative results.

Reproducibility

Models

No new model is introduced. The benchmark evaluates nine existing LMMs. The commercial APIs evaluated are GPT-4o-2024-08-06, Gemini-1.5-Pro-002, and Claude-3.5-Sonnet-20241022. The open-source models are KOSMOS2.5, TextMonkey, Florence-2, GOT, InternVL2-76B (76 billion parameters), and Qwen2-VL-72B (72 billion parameters). Model-specific details (parameter counts, architecture) are described in the original model papers and not repeated here.

Algorithms

Evaluation uses the metrics described above (Eval-Trans F1, NED, TEDS, field-level F1, $R_{rep}$). Normalization code for the KIE evaluation and evaluation scripts for all tracks are released at AlibabaResearch/AdvancedLiterateMachinery under MIT license, with VLMEvalKit integration supported via the HuggingFace dataset. The annotation pipeline for the SOLD dataset uses a bottom-up re-annotation scheme with rule-based multi-round quality rectification, LLM-assisted error correction, key normalization, and secondary manual review.

Data

CC-OCR comprises 7,058 fully annotated images across four tracks. Approximately 41% of the data are sourced from real-world applications and released for the first time. Data sources fall into three categories: existing benchmarks with qualified annotations (open-source, re-annotated as needed), newly collected data, and re-annotated data. All document parsing data and all Chinese multi-scene data are newly collected. KIE constrained-category data is re-annotated from SROIE, CORD, EPHOIE, and POIE. The open-category SOLD dataset is re-annotated from SIBR and HUST-CELL.

The benchmark data is publicly available on HuggingFace at wulipc/CC-OCR under an MIT license; this hosts a TSV-format version of the data integrated with VLMEvalKit. The dataset combines images from multiple source licenses: subsets derived from open-source benchmarks retain their original licenses (e.g., TotalText, IC15, CORD, SROIE are individually licensed), while newly collected and re-annotated images (including all document parsing data, Chinese multi-scene data, and SOLD) are released by the authors under the MIT license. The overall dataset package on HuggingFace carries MIT, though downstream users should verify the original source licenses for any open-source subsets they use independently.

Evaluation

Benchmark statistics: multi-scene OCR has 2,750 images; multilingual OCR has 1,500 images (150 per language); document parsing has 800 images (300 documents, 300 tables, 200 formulas); KIE has 2,008 images (1,008 constrained, 1,000 open-category).

For text evaluation, basic units are defined per script: characters for CJK languages (Chinese, Japanese, Korean), words for Latin/Cyrillic/Arabic. The repetition threshold $T_{rep}$ is set to 5 (a continuous unit must appear more than 5 times consecutively to qualify as repetitive).

No statistical uncertainty measures (confidence intervals, significance tests, multiple-seed runs) are reported. The evaluation is a single-pass inference with no reported variance.

A limitation noted by the authors is that most specialist models cannot produce LaTeX or HTML output, so the document parsing track primarily reflects generalist model performance. Results for specialist models on that track are incomplete or require post-hoc conversion (indicated by asterisks in Table 4).

Hardware

The paper does not report hardware specifications, inference latency, GPU counts, or cost estimates for the model evaluations. No deployment or inference requirements are discussed.

BibTeX

@InProceedings{Yang_2025_ICCV,
  author    = {Yang, Zhibo and Tang, Jun and Li, Zhaohai and Wang, Pengfei and Wan, Jianqiang and Zhong, Humen and Liu, Xuejing and Yang, Mingkun and Wang, Peng and Bai, Shuai and Jin, Lianwen and Lin, Junyang},
  title     = {CC-OCR: A Comprehensive and Challenging OCR Benchmark for Evaluating Large Multimodal Models in Literacy},
  booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
  month     = {October},
  year      = {2025},
  pages     = {21744-21754}
}

TL;DR

DocPTBench is a benchmark of over 1,300 photographed document images designed to measure how well current models handle parsing and translation under real-world capture conditions. Experiments show that shifting from digital-born to photographed documents causes average parsing edit distance to increase by roughly 18% for general MLLMs and 25% for specialized OCR models, with translation BLEU scores dropping by about 12%.

What kind of paper is this?

Dominant: $\Psi_{\text{Evaluation}}$. The central contribution is a new benchmark and evaluation protocol for photographed document parsing and translation. The paper does not introduce a new model or training procedure; it systematically measures the robustness gap in existing systems using a carefully constructed dataset.

Secondary: $\Psi_{\text{Resource}}$. DocPTBench itself is a released community resource: a dataset of over 1,300 labeled photographed document images with human-verified annotations across eight language-pair translation directions, along with evaluation code.

What is the motivation?

Document parsing and translation benchmarks have historically focused on digital-born or high-quality scanned documents with clean geometric structure. Benchmarks such as OmniDocBench, FoxPage, and olmOCR-Bench represent this well-structured distribution, as do translation benchmarks like DoTA and DITrans. The result is that reported model numbers do not reflect what happens when a user points a phone at a document in practice.

Real-world document photographs introduce a range of degradations absent from these benchmarks: perspective distortion from off-axis capture, page curvature and physical deformation, motion blur, uneven or harsh lighting, shadows, and overexposure. While the WildDoc benchmark incorporated in-the-wild images, it focused on visual question answering rather than structured parsing, and provided no translation annotations. The community therefore lacked a unified benchmark to measure end-to-end parsing and translation robustness under photographed conditions.

What is the novelty?

DocPTBench introduces three features that distinguish it from prior benchmarks:

1. Unified dual-task evaluation. A single dataset supports both document parsing and document translation evaluation. Prior benchmarks either covered parsing or translation in isolation; no earlier resource combined both tasks with photographed images.

2. Three-tier image collection. DocPTBench organizes its images into an Original set (981 born-digital documents from OmniDocBench), a Photographed set (981 simulated photographs plus 400 physically photographed images of 100 documents under four challenging conditions), and an Unwarping set (the Photographed images after a commercial geometric rectification API). This structure allows controlled ablation of geometric versus photometric degradation.

3. Broad multilingual translation coverage. Eight translation directions are supported: En-Zh, En-De, En-Fr, En-Ru, and their Zh-originated counterparts. Translation ground truth was produced by Qwen-Max generation followed by rigorous manual verification.

The evaluation metrics for parsing follow OmniDocBench conventions, using Levenshtein-based edit distances for text ($\text{Text Edit}$), formulas ($\text{Formula Edit}$), tables ($\text{Table Edit}$), and reading order ($\text{Read Order Edit}$), plus a Tree-Edit-Distance-based Similarity score (TEDS) for table structure. Translation is assessed with BLEU, chrF, METEOR, and STEDS. Lower edit distance indicates better parsing; higher BLEU indicates better translation.

What experiments were performed?

The paper benchmarks two model categories across all three image conditions:

Expert OCR models (evaluated on parsing only): PaddleOCR-VL, MinerU2.5, dots.ocr, MonkeyOCR, DeepSeek-OCR, olmOCR2, Dolphin, olmOCR, OCRFlux, SmolDocling, Nanonets-OCR, and Nanonets-OCR2. These systems are purpose-built for document content extraction.

General MLLMs (evaluated on both parsing and translation): Gemini 2.5-Pro, Qwen-VL-Max, GLM-4.5v, Kimi-VL, and Doubao-Seed-1.6-vision. Qwen2.5-VL-72B is also evaluated on parsing in the supplementary full results table. For translation, four smaller open-source models were also included: Qwen3-VL-4B, Qwen2.5-VL-3B, InternVL3-2B, and InternVL3.5-2B.

Parsing is evaluated across the Original, Photographed, and Unwarping conditions. Translation is evaluated with:

• A text-only baseline (the source Markdown text is provided directly, bypassing vision) to establish an upper bound for translation quality independent of OCR.
• Original and Photographed image inputs.
• Two prompting strategies: a Simple prompt (direct image-to-target-language output) and a Chain-of-Thought (CoT) prompt (explicit instruction to first parse the document into source-language Markdown, then translate).

The CoT formulation decouples perception from translation and helps diagnose where errors originate.

What are the outcomes/conclusions?

Parsing degradation is severe and widespread. Across the tested expert models, transitioning from Original to Photographed documents produces average overall edit distance increases far exceeding typical benchmark variance. DeepSeek-OCR’s Overall Edit Score (En) deteriorates from 13.4 to 54.4; SmolDocling’s rises from 49.3 to 90.1. General MLLMs fare comparatively better, with an average increase of roughly 18%, while expert models average roughly 25%. The authors attribute the relative resilience of MLLMs to training on more diverse and noisy data.

Geometric rectification is necessary but not sufficient. Processing photographed images through a commercial unwarping API before evaluation consistently recovers substantial performance. For example, DeepSeek-OCR’s Overall Edit Score (En) improves from 54.4 back to 22.1 after unwarping, and PaddleOCR-VL’s Table TEDS (En) rebounds from 54.3 to 82.5. However, a consistent gap remains between Unwarping and Original scores. The authors interpret this as evidence that photometric degradations (blur, uneven lighting) are not addressed by geometric correction and continue to impair parsing quality.

Translation exhibits a large modality gap. Comparing text-only inputs to image inputs on the same documents reveals a large gap. Qwen-VL-Max’s BLEU score on En-Zh drops from 69.41 (text) to 41.04 (image, original). For some models on low-resource scripts, the collapse is near-total: InternVL3-2B’s BLEU on En-Ru falls from 37.09 (text) to 8.15 (image, original), suggesting that complex scripts compound the difficulty.

Photographed images compound translation errors. When the image source is photographed rather than digital-born, BLEU scores drop a further average of 12% across models and directions. This degradation occurs consistently whether a simple or CoT prompt is used, indicating that visual image quality is the primary bottleneck rather than the prompting strategy.

CoT prompting mitigates some modality gap. For most models and language pairs, the CoT strategy (parse first, then translate) outperforms the simple direct approach. For example, Qwen-VL-Max’s En-Zh BLEU on original images rises from 41.04 with a simple prompt to 47.60 with CoT. The benefit, while consistent, does not close the gap to text-only baselines.

Dual bottleneck for translation. The paper identifies two independent constraints on end-to-end translation quality: the model’s visual parsing capability and its underlying translation competence. Doubao-Seed-1.6-vision achieves strong parsing scores but delivers only modest translation quality because its text-only translation performance is weaker than competing models. This finding suggests that improving either component in isolation will not fully solve end-to-end translation.

Gemini 2.5-Pro shows strongest overall robustness. Across parsing, translation, and photographed conditions, Gemini 2.5-Pro exhibits the smallest performance degradation among the general MLLMs tested, dropping only 3.4 Overall Edit Score (En) points on photographed images compared to original.

Reproducibility

Models

No new model weights are introduced. DocPTBench evaluates existing, externally released systems. All evaluated models are referenced by their original papers and repositories. Gemini 2.5-Pro was accessed via API; the open-source models (Qwen series, InternVL series) are available on HuggingFace under their respective licenses.

Algorithms

No training is performed. The evaluation pipeline applies each model’s default inference settings. Two prompting strategies are systematically varied: the Simple prompt and the Chain-of-Thought prompt. The exact prompts are illustrated in Fig. 5 of the supplementary material (page 14 of the paper). The commercial unwarping API used for the Unwarping tier is provided by textin.com (https://www.textin.com/market/detail/crop_enhance_image).

Inference scripts for each prompting mode (simple, CoT, and text-only) are provided in code/translation/model_infer/ in the repository, organized by source language. The translation evaluation script is at code/translation/evaluate_metric/evaluate.py. Parsing evaluation uses the OmniDocBench evaluation harness (v1_0 branch of https://github.com/OmniDocBench/OmniDocBench); the repository documentation (docs/parsing.md) includes a step-by-step walkthrough for reproducing parsing results with any model.

Data

The DocPTBench dataset is built on 981 documents from OmniDocBench. The Photographed tier combines:

• 981 photographed-simulation images (exact simulation pipeline details are in the paper and code repository).
• 400 physically photographed images of 100 source documents, each photographed under four conditions (strong illumination, shadows, perspective distortion, physical wear such as folds or wrinkles).

The 1,381 Unwarping images are derived from the Photographed set using a commercial API. Translation ground truth for all eight language directions was generated by Qwen-Max and then manually verified and corrected.

The dataset and code are released under Apache-2.0 at https://github.com/Topdu/DocPTBench. The dataset is also hosted on HuggingFace (topdu/DocPTBench) and ModelScope (topdktu/DocPTBench) and can be downloaded with:

huggingface-cli download topdu/DocPTBench --local-dir ./DocPTBench --repo-type dataset

The downloaded dataset contains a ground-truth file (DocPTBench_combined.json) and four image directories: images_synreal (981 simulated photographed images), images_synreal_dewarp (their unwarped counterparts), images_pic_synreal (400 physically photographed images), and images_pic_dewarp (their unwarped counterparts). Translation ground truth is stored under translation_gt/src_en/ and translation_gt/src_zh/ for the eight language pairs.

Evaluation

Parsing metrics follow OmniDocBench conventions: Levenshtein-based Text Edit, Formula Edit, Table Edit, and Read Order Edit (lower is better), plus TEDS for table structure (higher is better). An Overall Edit score aggregates these. Translation metrics are BLEU, chrF, METEOR, and STEDS.

Baselines cover 18 evaluated systems in total across the supplementary full results table (Tab. 5). The three-tier design (Original/Photographed/Unwarping) controls for degradation type. No statistical significance tests or error bars over multiple runs are reported; results reflect single-run evaluations. The authors acknowledge that the benchmark builds on OmniDocBench’s digital-born subset, so domain coverage reflects OmniDocBench’s composition (academic papers, invoices, forms, magazines, and others).

Hardware

No training hardware is relevant. Inference hardware specifics are not reported for the evaluated models. The paper does not report GPU-hours, inference latency, or throughput.

BibTeX

@article{du2025docptbench,
  title={DocPTBench: Benchmarking End-to-End Photographed Document Parsing and Translation},
  author={Du, Yongkun and Chen, Pinxuan and Ying, Xuye and Chen, Zhineng},
  journal={arXiv preprint arXiv:2511.18434},
  year={2025}
}

TL;DR

ENTRANT is a large-scale financial table dataset compiled from SEC EDGAR filings. It contains approximately 6.7 million tables extracted from roughly 330,000 XBRL-format reports filed between 2013 and 2021, covering ten report types including 10-K, 10-Q, and 8-K filings. Each table is stored as a JSON object with per-cell attributes (formatting, data type, formula references), spatial coordinates, and two explicit hierarchy trees encoding the top-down and left-right header structure. The authors demonstrate the dataset’s usability by pre-training a TUTA model on ENTRANT and applying it to cell type classification, where it either matches or slightly improves on the original model depending on the training mix.

What kind of paper is this?

Dominant: $\Psi_{\text{Resource}}$: the paper’s central contribution is the dataset itself, including the automated crawling and transformation pipeline, the JSON annotation format with bi-tree hierarchies, and the public release on Zenodo.

Secondary: $\Psi_{\text{Evaluation}}$: the pre-training and downstream cell classification experiments serve as a technical validation of the dataset’s usability rather than as a standalone evaluation contribution.

What is the motivation?

Table understanding covers a wide range of tasks: cell role prediction, table type classification, question answering over tables, schema mapping, and knowledge extraction. Most recent progress in this area relies on large pre-trained transformers (TaBERT, TaPas, TUTA, FORTAP) that consume millions of tables during pre-training.

The datasets most commonly used for this purpose come from general-purpose web sources: Wikipedia tables, Common Crawl, and the WDC Web Table corpus. These sources require substantial preprocessing to extract tables and attach per-cell features in a format that models can consume directly. Financial tables from spreadsheets, which interleave textual and numerical content and commonly exhibit multi-level hierarchical header structures, are largely absent from existing public pre-training corpora. The one prior financial table resource cited by the authors (TAT-QA) contains only 20K tables, which is several orders of magnitude smaller than what large pre-training runs require.

EDGAR, the SEC’s public filing database, provides a continuously updated source of structured financial spreadsheets in XBRL format. Starting from 2009, SEC rules required publicly traded companies to tag every number and table in their filings with XBRL identifiers, making automated extraction reliable and consistent. The authors argue that this source, once processed, can fill the gap for large-scale, machine-readable financial tabular data.

What is the novelty?

Dataset scale and coverage

ENTRANT covers approximately 331,448 reports filed between 2013 and 2021 across ten SEC form types. The 2013 start date reflects XBRL adoption maturity; the 2021 cutoff was chosen to align with prior research on financial misstatement detection, where average discovery lags of two to three years introduce noise in more recent filings. Processing these reports produced 6,735,407 tables distributed across roughly 330,000 JSON files.

The dataset is not limited to annual financial statements. It includes quarterly reports (10-Q), current event reports (8-K), registration statements (S-1, S-4), foreign issuer reports (20-F, 18-K), and investment company filings (485BPOS, 497). This breadth means the vocabulary spans standard XBRL financial concepts (e.g., “Net Income”, “Current Liabilities”) as well as short textual disclosures and notes that accompany significant corporate events.

JSON annotation format

Each JSON file corresponds to one filing workbook and contains a list of table dictionaries. Each table stores:

• Metadata: file description, language, and spreadsheet name
• Table dimensions via a “RangeAddress” field
• A “Cells” field structured as a list of row lists, where each cell is a dictionary

Per-cell attributes include:

In addition to cell content, each table records merged regions and the counts of top header rows and left header columns.

Bi-tree hierarchy representation

A distinctive feature of ENTRANT is its explicit encoding of the two hierarchical structures present in financial tables. Financial spreadsheets routinely use merged cells and indentation to imply both a top-down hierarchy (column group headers spanning multiple time periods or categories) and a left-to-right hierarchy (row label groups organizing related financial line items). ENTRANT encodes both hierarchies as nested node dictionaries:

node = {
  'RI': <RowIndex>,
  'CI': <ColumnIndex>,
  'Cd': [ <child node>, ... ]
}

The root node uses coordinates (-1, -1) to indicate it lies conceptually outside the table grid. This representation, adopted from the TUTA paper’s formalism, supports up to four levels of hierarchy depth, which the authors report covers all table types present in EDGAR filings. The two hierarchy trees are stored under the keys TopTreeRoot and LeftTreeRoot in each table dictionary.

Automated, tested curation pipeline

The crawling phase uses each company’s Central Index Key (CIK) and EDGAR accession number to construct API endpoints pointing to Financial_Report.xlsx files. Reports without valid XBRL tags are filtered out before downloading. The transformation phase uses OpenPyXL to extract tables from each workbook, annotate cells with the attributes above, and construct the bi-tree structures. The paper states that a test suite of 45 unit tests covering all transformation steps passed successfully at the time of publication. A post-processing pass verifies that no empty rows or columns remain in any extracted table, and all JSON files are validated by loading them into Python dictionaries.

What experiments were performed?

The authors pre-trained TUTA, a transformer-based table understanding model, on two data configurations to validate that ENTRANT is correctly formatted and consumable:

1. TUTA (WikiTables + WDC + ENTRANT): pre-trained on the three public datasets combined.
2. TUTA (ENTRANT only): pre-trained exclusively on ENTRANT.

Both models were then evaluated on cell type classification using two datasets from the DECO collection (Koci et al., ICDAR 2019), DeEx and SAUS, which annotate cells into six types: “metadata”, “notes”, “data”, “top attribute”, “left attribute”, and “derived”. These datasets draw from financial, educational, and public health domains.

Results are reported as Macro F1:

The model pre-trained only on ENTRANT performs approximately 2 Macro F1 points below the original TUTA baseline (2.4 on DeEx, 1.8 on SAUS). When ENTRANT is combined with the other two public datasets, the resulting model slightly outperforms the original baseline on both tasks.

What are the outcomes/conclusions?

The results suggest that ENTRANT can serve as a drop-in addition to existing public pre-training corpora for table understanding models. The ENTRANT-only model performs reasonably well despite being restricted to a single financial domain, which the authors attribute to the dataset’s vocabulary breadth (financial statements contain diverse numeric formats, date references, and conceptual labels) and its explicit structural annotations.

The paper notes several limitations worth considering:

• The dataset covers only filings from 2013 to 2021. More recent EDGAR filings are not included and would need to be crawled using the released code.
• The 2021 cutoff introduces a specific bias: filings close to the cutoff are less likely to have had misstatements discovered, which was the original research context but has no direct bearing on table understanding.
• Tables below a minimum size threshold and tables with sparse or broken content are filtered out during preprocessing. The exact filter criteria are described qualitatively but not as precise thresholds.
• The ENTRANT-only pre-training experiment uses downstream datasets (DeEx, SAUS) that include financial tables, which may partially favor a financial-domain pre-training corpus. Generalization to other domains is not directly measured.
• The paper does not report statistical confidence intervals or multiple pre-training runs, so the observed improvements over the TUTA baseline should be interpreted cautiously.

The code release allows users to extend the dataset by crawling more recent EDGAR filings or to apply the transformation pipeline to other XBRL-format corporate spreadsheets.

Reproducibility

Models

• TUTA: tree-based transformer for general table pre-training; originally described by Wang et al. (KDD 2021). The pre-training uses all three objectives described in the original TUTA paper. Model weights from the published TUTA codebase (https://github.com/microsoft/TUTA_table_understanding) were used as the starting point. Parameter count and layer configuration are not restated in the ENTRANT paper; readers should consult the original TUTA paper.

Algorithms

• Pre-training followed the TUTA procedure using the released Microsoft code. No modifications to the TUTA training recipe are described.
• Downstream cell type classification uses the fine-tuning setup from the original TUTA paper.
• Specific optimizer, learning rate, batch size, and compute budget for the ENTRANT pre-training runs are not reported.

Data

• Source: SEC EDGAR public filings, accessed via the EDGAR API. The underlying financial data is public domain. Spreadsheets are in XBRL-annotated XLSX format.
• Coverage: 2013-2021, ten report types, approximately 331,448 reports, 6,735,407 tables.
• File organization: One JSON file per company filing, organized in subfolders by report type. Each subfolder is provided as a zip archive on Zenodo.
• File naming convention: <CIK>_<YEAR>_<TYPE>_<ACCESSION_NUMBER>.json
• Dataset availability: Released on Zenodo at https://doi.org/10.5281/zenodo.10667088 under CC-BY-4.0 (open access, 17.3 GB total across 10 zip archives, one per report type). Note that the dataset license (CC-BY-4.0) is more permissive than the published article license (CC-BY-NC-ND-4.0).
• Code availability: Python scripts for crawling and transformation released at https://github.com/iit-Demokritos/entrant under CC-BY-4.0.
• Processing libraries: OpenPyXL for spreadsheet parsing.

Evaluation

• Metric: Macro F1 for multi-class cell type classification (six classes: metadata, notes, data, top attribute, left attribute, derived).
• Downstream datasets: DeEx and SAUS from the DECO collection (Koci et al., ICDAR 2019), covering financial, educational, and public health tables.
• Baselines: Original TUTA model pre-trained on WikiTables, WDC, and non-public web-crawled spreadsheets. This baseline uses pre-training data that is not fully reproducible by external parties.
• No error bars, significance tests, or multiple pre-training seeds are reported.

Hardware

• Computing and storage resources were provided by the SKEL AI Lab at NCSR “Demokritos”. Specific GPU types, GPU counts, pre-training duration, and memory requirements are not reported in the paper.

BibTeX

@article{zavitsanos2024entrant,
  title={ENTRANT: A Large Financial Dataset for Table Understanding},
  author={Zavitsanos, Elias and Mavroeidis, Dimitris and Spyropoulou, Eirini and Fergadiotis, Manos and Paliouras, Georgios},
  journal={Scientific Data},
  volume={11},
  pages={876},
  year={2024},
  publisher={Nature Publishing Group},
  doi={10.1038/s41597-024-03605-5}
}

TL;DR

MMSci is a dataset suite built from the SciGen corpus of scientific tables, comprising MMSci-Pre (52K image-to-HTML pairs for table structure learning), MMSci-Ins (12K chain-of-thought instruction samples), and MMSci-Eval (3,114 benchmark samples requiring numerical reasoning). The authors find that 52K domain-specific scientific tables outperform 150K general-domain tables on both in-distribution and held-out evaluation, suggesting that data source matters more than raw volume for this task.

What kind of paper is this?

Dominant: $\Psi_{\text{Resource}}$: The primary contribution is the construction and release of the three MMSci datasets (pre-training, instruction tuning, and evaluation), along with the construction methodology, quality control procedures, and reasoning-type taxonomy.

Secondary: $\Psi_{\text{Evaluation}}$: MMSci-Eval serves as a purpose-built benchmark for numerical reasoning over scientific tables, and a substantial portion of the paper is devoted to comparing models on this benchmark and analyzing cross-modal consistency.

What is the motivation?

Large language models have shown growing capability on table understanding tasks such as table question answering (TQA), table fact verification (TFV), and table-to-text generation (T2T). Most table-oriented LLMs, however, require converting tables into sequential text formats such as HTML strings, which can discard structural and positional information. Multimodal large language models (MLLMs) sidestep this by processing table images directly, but existing MLLM-based approaches suffer from at least three practical problems:

1. Fixed input image resolutions constrain applicability to tables of varying sizes.
2. Training corpora largely consist of general-domain tables (Wikipedia, financial reports, government documents) and do not cover the dense numerical content typical of scientific publications.
3. Models trained on these general corpora show substantial performance degradation on scientific tables that require arithmetic and statistical reasoning.

Prior datasets such as TAT-QA and FinQA address numerical reasoning in the financial domain, and SciGen introduces scientific table-to-text generation, but no existing resource combines multimodal (image-based) input with comprehensive numerical reasoning evaluation across multiple task types for scientific tables.

What is the novelty?

The primary novelty is the MMSci dataset suite itself. The authors build all three components by sourcing raw tabular data from the SciGen dataset (Moosavi et al., 2021), which pairs scientific tables from computer science papers with textual descriptions that naturally invoke arithmetic reasoning.

MMSci-Pre contains 52K image-to-HTML pairs. The authors convert SciGen’s LaTeX-formatted tables to HTML using a rendering pipeline (imgkit) and store the resulting table images alongside their HTML ground truth. The pre-training objective is standard sequence generation: given a table image, produce the corresponding HTML string. This stage develops table structure perception within the MLLM.

MMSci-Ins contains 12K instruction-tuning samples spanning TQA, TFV, and T2T. For each table, GPT-4o is prompted with both the table image and its SciGen description to generate a question-answer pair, a claim-and-verdict pair, and an extended table-to-text description, each accompanied by explicit chain-of-thought reasoning steps. To improve quality, self-consistency voting (Wang et al., 2023) is applied: multiple reasoning paths are sampled and a majority-vote mechanism selects the final output. The construction prompt is made available in the paper (Table 7). The dataset is balanced: each table contributes exactly one TQA, one TFV, and one T2T sample.

MMSci-Eval contains 3,114 test samples derived from SciGen’s test set using the same GPT-4o generation pipeline, with an additional round of human verification covering all samples. The benchmark covers eight reasoning types:

On the modeling side, the authors implement dynamic input resolution on two base architectures (Qwen2-VL-7B-Instruct and LLaVA-NeXT-7B), addressing the fixed-resolution limitation of prior table MLLMs. Qwen2-VL achieves this through 2D-RoPE positional encodings; LLaVA-NeXT splits images into grids and encodes them independently.

What experiments were performed?

Training procedure. Both models are fine-tuned in two stages on 4 × A100 80 GB GPUs using LoRA (rank 64, sequence length 4096). In the first stage (table structure learning), models are trained for one epoch on image-to-HTML data with varying combinations of MMSci-Pre (52K) and MMTab-Pre (150K from Table-LLaVA). In the second stage (visual instruction tuning), models are fine-tuned for 4 epochs on MMSci-Ins (12K), freezing the visual encoder and updating only the MLP connector and LLM weights.

Evaluation. The held-in evaluation uses MMSci-Eval’s TQA, TFV, and T2T splits. Held-out evaluation uses the MMTab-Eval benchmark (from Table-LLaVA), covering TABMWP, WTQ, HiTab, TAT-QA, FeTaQA (TQA), TabFact, InfoTabs (TFV), and HiTab_T2T, Rotowire, WikiBIO (T2T). Accuracy is used for TQA and TFV; BLEU is used for T2T.

Baselines. The authors compare against a range of MLLMs: GPT-4V, InternVL-2-76B, Qwen-2-VL-72B-Instruct, LLaVA-NeXT (72B/34B/13B/7B), Table-LLaVA (13B/7B), Pixtral-12B, Llama-3.2-11B-Vision-Instruct, MiniCPM-V-2.6-8B, and InternVL-2-8B.

Ablations. Three ablation axes are evaluated: (1) pre-training data source (MMSci-Pre 52K vs. MMTab-Pre 150K vs. combined MM-Pre 202K vs. no pre-training), (2) inclusion of explicit reasoning steps in instruction tuning, and (3) instruction data scaling from 3K to 12K samples.

Representational alignment analysis. The authors also measure language-vision alignment using kernel similarity metrics (CKA, CKNNA, SVCCA, and several KNN-based measures) on ImageNet, Wikipedia captions, and the MMSci T2T task. Qwen2-VL-7B-Instruct scores highest across most measures, which the authors correlate with its stronger task performance.

What are the outcomes/conclusions?

Data quality over quantity. Across both model architectures, training on MMSci-Pre (52K scientific domain) produces comparable or better results than training on MMTab-Pre (150K general domain), particularly on MMSci-Eval. For Qwen2-VL-7B-Instruct, MMSci-Pre (52K) + MMSci-Ins achieves 41.13% TQA accuracy and 72.92% TFV accuracy, versus 40.75% TQA and 72.73% TFV for MMTab-Pre (150K) + MMSci-Ins. Combining both (MM-Pre, 202K) pushes results to 42.10% TQA and 73.98% TFV.

Reasoning steps help consistently. Ablations show that including chain-of-thought reasoning steps during instruction tuning improves TQA by roughly 6-7 percentage points and TFV by 8-10 percentage points across all pre-training configurations. The benefit is consistent regardless of whether 3K or 12K instruction samples are used, and it generalizes to held-out datasets like TABMWP and TAT-QA.

Generalization to held-out data. Despite not training on any MMTab-Ins data, the MMSci-tuned Qwen2-VL models achieve 21.15% average TQA accuracy and 41.63% average TFV accuracy on the MMTab held-out benchmark, which is competitive with models explicitly trained on general-domain table data.

Limitations. The authors note three main limitations. First, the framework focuses on numerical tables; qualitative or methodology tables are not well covered. Second, complex statistical analyses and domain-specific mathematical notation (e.g., subscripted variables, special symbols) may still challenge the models. Third, very large tables with dense information remain difficult to process due to computational constraints and potential information loss during visual encoding. GPT-4V continues to lead most subtasks, suggesting that the fine-tuned 7B models leave substantial room for improvement.

Reproducibility

Models

Both fine-tuned model variants build on publicly available checkpoints:

• Qwen2-VL-7B-Instruct (ViT + MLP connector + Qwen2-7B-Instruct backbone)
• LLaVA-NeXT-7B (CLIP encoder + MLP connector + Vicuna-7B backbone)

The paper does not report releasing fine-tuned model weights. The base checkpoints are available via their respective repositories under permissive licenses.

Algorithms

• LoRA fine-tuning: rank 64, applied to both stages
• Sequence length: 4096 tokens
• Stage 1 (table structure learning): 1 epoch; LLaVA-NeXT updates only the MLP connector; Qwen2-VL updates the full model
• Stage 2 (instruction tuning): 4 epochs; visual encoder frozen; MLP and LLM weights updated
• Training time (4 × A100 80 GB): LLaVA-NeXT table structure learning takes approximately 15 hours for MMTab-Pre (150K), 3 hours for MMSci-Pre (52K), and 20 hours combined (one epoch); Qwen2-VL-7B takes approximately 15 hours, 8 hours, and 19 hours for the same three configurations respectively; instruction tuning takes approximately 1 hour for 4 epochs on 12K samples for both models
• Optimizer, learning rate, and warmup schedule are not reported in the paper
• Self-consistency voting used during dataset generation; exact number of sampled reasoning paths is not stated

Data

• MMSci-Pre: 52K image-to-HTML pairs derived from SciGen training and development splits; HTML rendered via the imgkit Python package
• MMSci-Ins: 12K samples from SciGen training split; generated by GPT-4o with self-consistency voting; 40% manually verified (94.36% accuracy after regeneration of failures)
• MMSci-Eval: 3,114 samples from SciGen test split; all samples manually verified (95.25% accuracy)
• The dataset JSON files and a zip of MMSci-Pre are publicly available at the GitHub repository; image data is referenced to HuggingFace but exact access path is not detailed in the paper
• The underlying SciGen dataset (Moosavi et al., 2021) is publicly available; its license should be consulted before use

Evaluation

• TQA and TFV: accuracy (exact match for TQA answers formatted as JSON)
• T2T: BLEU score
• No error bars or significance tests are reported; single-run results
• Held-out datasets (TABMWP, TAT-QA, TabFact, WTQ, etc.) are evaluated zero-shot relative to MMSci training distribution

Hardware

• Training: 4 × NVIDIA A100 80 GB GPUs
• Inference hardware not explicitly specified
• No cloud cost estimates provided

BibTeX

@article{yang2025mmsci,
  title={Does Table Source Matter? Benchmarking and Improving Multimodal Scientific Table Understanding and Reasoning},
  author={Bohao Yang and Yingji Zhang and Dong Liu and André Freitas and Chenghua Lin},
  journal={arXiv preprint arXiv:2501.13042},
  year={2025}
}

TL;DR

SPRINT is an image-to-sequence model for table structure recognition (TSR) that achieves script-agnostic generalization by downsampling table images to 128x128 pixels before encoding, thereby blurring language- and font-specific features. It uses a Global Context Attention (GCA) encoder paired with a transformer decoder trained on the compact Optimized Table Structure Language (OTSL) vocabulary. Alongside the model, the authors release MUSTARD, a multilingual TSR benchmark of 1,428 tables spanning 13 languages, including 11 Indic languages, English, and Chinese. On MUSTARD, SPRINT outperforms the best available comparison model (MTL-TabNet) by an absolute average of 11.12% TEDS-S while also achieving lower inference latency on English benchmarks.

What kind of paper is this?

Dominant: $\Psi_{\text{Method}}$: The paper’s primary contribution is the SPRINT architecture and training recipe. It introduces a deliberate design choice (aggressive downsampling to produce a script-agnostic feature representation) and an end-to-end methodology combining SPRINT with a loosely coupled table grid estimator. Ablations over decoder depth, image resolution, and training data mix support this framing.

Secondary: $\Psi_{\text{Resource}}$: The MUSTARD dataset is a meaningful standalone contribution: 1,428 annotated multilingual table images across 13 languages, the first such OTSL-labeled multilingual TSR benchmark. Code and dataset are both released under the MIT license.

What is the motivation?

Table structure recognition is a prerequisite for downstream tasks such as information retrieval, table reconstruction, and document understanding. While large English-language TSR benchmarks (PubTabNet, FinTabNet, PubTables-1M) and the models trained on them exist, no labeled multilingual TSR datasets with logical structure annotations were publicly available at the time of this work. Models trained on upsampled English table images implicitly learn language- and font-specific cues that do not transfer to documents in other scripts. The cost of collecting and annotating large labeled corpora for each target language is prohibitive, so the authors seek a model that generalizes across scripts without per-language retraining.

A secondary motivation is inference efficiency. Popular image-to-sequence models (MTL-TabNet, TableFormer + HTML) decode large HTML vocabularies, which slows inference relative to what is needed for practical deployment.

What is the novelty?

SPRINT reframes TSR as a script-agnostic cell arrangement prediction problem. The core insight is that two properties together should produce a model that ignores script-specific appearance: first, shrinking the input image to 128x128 pixels converts cell contents into indistinct blobs, removing text-level cues; second, using the minimal OTSL vocabulary (six tokens: F, E, L, X, U, N) rather than a full HTML tag set both speeds decoding and avoids learning vocabulary-linked biases.

OTSL representation. A table with $R$ rows and $C$ columns is represented as a flat sequence of length $R \times (C+1)$, where N acts as a row delimiter appearing at every $(C+1)^{\text{th}}$ position. The remaining tokens encode cell type:

• F (filled cell with content), E (empty cell)
• L (left-looking: column span), U (upward-looking: row span), X (cross: both spans)

This six-token vocabulary is substantially smaller than an HTML vocabulary (30+ tokens), which reduces decoder logit complexity and shortens sequences.

Architecture. SPRINT’s encoder is a ResNet-31 backbone with Multi-Aspect Global Attention (GCA) interleaved between convolutional layers. The encoder maps a 128x128 image to 512 channels of 16x32 feature maps, which are positionally embedded and fed to a six-layer transformer decoder. The decoder is trained with categorical cross-entropy against ground-truth OTSL sequences, with a maximum sequence length of 224 tokens.

Grid-based alignment. SPRINT predicts an unconstrained OTSL string, which is then aligned with row and column counts $R$ and $C$ obtained from a pre-trained Table Aware Transformer (TATR) grid estimator. The alignment step pads or trims the predicted string to length $R \times (C+1)$, enforces the periodicity of N tokens, and replaces syntactically invalid placements of L, U, X, and N with F. This produces a well-formed OTSL matrix that can be losslessly converted to HTML using a deterministic algorithm (Algorithm 1 in the paper).

MUSTARD. The dataset covers 1,214 scanned or printed document tables in 12 languages (11 Indic: Assamese, Bengali, Gujarati, Hindi, Kannada, Malayalam, Oriya, Punjabi, Tamil, Telugu, Urdu; plus Chinese from CTDAR) and 214 scene-text tables in English and Chinese sourced from TabRecSet. Each table is annotated with its OTSL logical structure. The split contains 662 simple tables (no merged cells) and 766 complex tables (at least one merged cell).

What experiments were performed?

The authors train two SPRINT variants:

• $\text{SPRINT}_{\text{FTN}}$: trained on the OTSL-labeled canonical subset of FinTabNet only.
• $\text{SPRINT}_{\text{ALL}}$: trained on a merged dataset combining FinTabNet, PubTabNet, and PubTables-1M canonical subsets.

Training runs for more than 80 epochs with a learning rate of 0.0001 on a single NVIDIA RTX A6000 GPU.

Evaluation metric. The primary metric is TEDS-S (Tree Edit Distance-based Similarity, structure-only), reported for simple tables, complex tables, and overall. Higher is better; scores are in the range [0, 100].

English benchmarks (canonical OTSL subsets). $\text{SPRINT}_{\text{FTN}}$ is evaluated on the FinTabNet canonical test set; $\text{SPRINT}_{\text{ALL}}$ on PubTabNet and PubTables-1M. Results are compared against the TableFormer + OTSL baseline:

English benchmarks (HTML baselines). On the original PubTabNet and FinTabNet test sets, SPRINT is compared against EDD, GTE, TableFormer, VAST, and MTL-TabNet. SPRINT reaches 95.71% overall TEDS-S on PubTabNet and 98.03% on FinTabNet, approximately 1.5 percentage points below the best-reported scores from MTL-TabNet (97.88% and 98.79% respectively). The authors attribute this gap to MTL-TabNet’s use of upscaled images and dual cascaded decoders trained on HTML.

Inference speed. On a random 400-image FinTabNet subset (50 iterations), SPRINT averages 1.52 seconds per image compared to 2.35 seconds for MTL-TabNet (both measured on the same A6000 GPU) and 3.26 seconds for TableFormer + HTML (a pre-reported figure measured on an AMD EPYC 7763 CPU, not directly comparable).

MUSTARD evaluation. $\text{SPRINT}_{\text{FTN}}$ and the FinTabNet-trained MTL-TabNet checkpoint are compared on all 1,428 MUSTARD tables across 14 language-modality groups. The table grid estimator achieves 54.62% exact match for both rows and columns on MUSTARD (versus over 80% on English datasets), reflecting that TATR was pretrained on English data, but the mean L1 error remains below 0.55, which the authors argue limits TEDS-S penalty in practice.

Ablations. Table 9 in the supplementary material explores combinations of decoder layer count (3, 4, 6, 8) and input image shape (32x32, 32x128, 128x128) across training data conditions. The 6-layer decoder with 128x128 input consistently produces the best or near-best results across all three English benchmarks.

What are the outcomes/conclusions?

On English TSR benchmarks, SPRINT is competitive with methods that use much larger image inputs and vocabularies, approaching but not matching the top HTML-based models. On MUSTARD, SPRINT outperforms MTL-TabNet by an absolute average of 11.12% TEDS-S overall, with gains visible across all 14 language-modality groups (ranging from roughly 2 to 29 percentage points). The largest gains occur in languages where MTL-TabNet performs most poorly (Punjabi: +29.39%, Bengali: +16.94%), which the authors attribute to MTL-TabNet’s implicit reliance on English-like visual features.

The authors acknowledge several limitations. The table grid estimator (TATR) was pretrained exclusively on English documents, so its row/column count estimates are less accurate on MUSTARD, which could suppress TEDS-S scores. Source code and checkpoints for VAST and the OTSL baseline are not publicly available, so direct replication of those comparisons is not possible. MUSTARD contains roughly 100 tables per language, which is a small evaluation set, and the dataset lacks a train split for non-English languages.

The conclusion suggests future work on integrating detected cell bounding boxes with the predicted logical structure to support end-to-end table reconstruction across scripts.

Reproducibility

Models

SPRINT consists of a ResNet-31 encoder with GCA-based attention (Multi-Aspect Global Attention) and a 6-layer transformer decoder. The encoder produces a 512-channel 16x32 feature tensor; decoder intermediate layers have width 2048. Total parameter count is not reported. Model weights are released via the GitHub repository under the MIT license. The architecture adapts the MASTER scene text recognition framework.

Algorithms

• Optimizer: Not explicitly stated; learning rate is 0.0001.
• Training duration: More than 80 epochs.
• Loss: Categorical cross-entropy over the OTSL vocabulary (6 characters plus start/stop tokens; vocabulary size of 8).
• Maximum sequence length: 224 tokens.
• Preprocessing: Images are resized to 128x128 pixels using standard resize operations; no augmentation details are reported.
• Grid estimator: TATR v1.1 (pre-trained on FinTabNet, PubTabNet, PubTables-1M); detection threshold 0.25; NMS IOU threshold 0.25 for table-row class. Two checkpoint variants are used: v1.1-all for PubTabNet, FinTabNet, and MUSTARD evaluations; v1.1-PubTables-1m for PubTables-1M evaluation.
• OTSL-to-HTML conversion: Deterministic algorithm released with the code.

Data

• Training data: OTSL-labeled canonical subsets of PubTabNet (320,000 train images), FinTabNet (88,441 train images), and PubTables-1M (522,874 train images), released by Lysak et al. (ICDAR 2023).
• Validation: Internal split of PubTabNet train set; non-overlapping PubTabNet validation images used for final reporting.
• MUSTARD: 1,428 tables used as a held-out evaluation set only; no train split is provided. Tables are sourced from Yojana magazine archives (11 Indic languages), CTDAR documents (102 Chinese tables), and a TabRecSet subset (214 English/Chinese scene tables). The HuggingFace release is at badrivishalk/MUSTARD under the MIT license.

Evaluation

• Primary metric: TEDS-S, computed from tree edit distance between predicted and ground-truth HTML structure sequences (cell content filtered out).
• Benchmarks: PubTabNet (original and canonical), FinTabNet (original and canonical), PubTables-1M (canonical), MUSTARD.
• Baselines on English: EDD, GTE, TableFormer (HTML and OTSL), VAST, MTL-TabNet. VAST and the OTSL baseline did not release code/checkpoints, so comparisons rely on pre-reported numbers.
• Baselines on MUSTARD: Only MTL-TabNet is compared, as VAST and the OTSL baseline have no public checkpoints.
• No error bars or significance tests are reported. Results are from single runs.

Hardware

• Training: Single NVIDIA RTX A6000 GPU.
• Inference timing: Single NVIDIA RTX A6000 GPU; 50 iterations over 400 FinTabNet images. SPRINT averages 1.52 seconds per image (1.36s for the SPRINT model itself, 0.16s for post-processing: grid estimation + alignment + HTML conversion). MTL-TabNet was also measured on the A6000 at 2.35 seconds. TableFormer timings (3.26s for HTML, 1.85s for OTSL) are pre-reported figures from the OTSL baseline paper, measured on an AMD EPYC 7763 CPU, not comparable to the A6000 measurements.
• No cost estimates or memory requirements are reported.
• SPRINT’s downsampled 128x128 inputs and compact OTSL vocabulary suggest modest inference memory requirements relative to models using full-resolution images.

BibTeX

@inproceedings{kudale2024sprint,
  title={SPRINT: Script-agnostic Structure Recognition in Tables},
  author={Kudale, Dhruv and Kasuba, Badri Vishal and Subramanian, Venkatapathy and Chaudhuri, Parag and Ramakrishnan, Ganesh},
  booktitle={Document Analysis and Recognition -- ICDAR 2024 -- 18th International Conference, Athens, Greece, August 30--September 4, 2024, Proceedings, Part V},
  pages={350--367},
  year={2024},
  publisher={Springer},
  doi={10.1007/978-3-031-70549-6_21}
}

TL;DR

Table2LaTeX-RL constructs a large-scale corpus of approximately 1.2 million table image-to-LaTeX pairs extracted from arXiv source files, then fine-tunes multimodal large language models in two stages: supervised fine-tuning on the full corpus followed by reinforcement fine-tuning via VSGRPO, a dual-reward extension of GRPO that combines a TEDS-Structure reward with a CW-SSIM reward computed on rendered output images. The resulting Qwen2.5-VL-3B model reaches TEDS-Structure of 0.9218 on complex tables, the first reported result above 0.9 on that subset, outperforming models with more than 20 times as many parameters.

What kind of paper is this?

Dominant: $\Psi_{\text{Method}}$ The primary contribution is the VSGRPO training framework: a visual-in-the-loop reinforcement learning strategy that renders generated LaTeX to PNG and computes CW-SSIM directly as an optimization signal alongside a structure-level TEDS-Structure reward. The paper devotes most of its space to method design, ablations on reward components, and comparative benchmarks.

Secondary: $\Psi_{\text{Resource}}$ The 1.2M table image-to-LaTeX dataset scraped from arXiv, together with the complexity-stratified test split covering simple, medium, and complex tables, represents a meaningful data contribution to a domain that previously lacked large-scale public training corpora.

What is the motivation?

Scientific documents depend on LaTeX tables to communicate experimental results, and digitizing or reusing those tables requires converting scanned or rendered table images back into source code. Most prior table structure recognition work targets HTML representations, which do not support the full expressiveness of LaTeX: nested multirow/multicolumn headers, mathematical content in cells, and typographic control. The handful of systems that do target LaTeX (LATTE, LaTeXNet, Nougat) either rely on from-scratch training with limited data or do not address the combined difficulty of large, deeply nested structures.

Two intertwined gaps motivate the work. First, no large-scale, publicly available dataset of table image-to-LaTeX pairs existed prior to this work. Second, standard evaluation metrics are poorly suited to the task. TEDS and BLEU compare LaTeX code at the token level, which penalizes semantically equivalent but syntactically different code (e.g., {} wrapper groups that do not affect rendering, or \textbf vs {\bf}). Visual similarity metrics like CW-SSIM capture rendered appearance but ignore structural correctness. Neither alone gives a reliable picture of generation quality.

What is the novelty?

The paper makes three interrelated contributions.

Large-scale table corpus. The authors scrape arXiv LaTeX source files for papers published between October 2017 and April 2023, extract table environments using regular expressions, and clean the code by removing references, color commands, and other non-structural elements. The result is 1,209,986 table-LaTeX pairs. Tables are classified into three complexity tiers based on cell count and the presence of \multirow/\multicolumn commands: complex (over 160 cells), medium (100-160 cells with 2 or more \multirow/\multicolumn commands), and simple (all others). In the training set, simple tables account for approximately 94%, with medium and complex each at roughly 3%.

Hybrid evaluation protocol. The authors propose combining TEDS-Structure (measuring structural tree edit distance, ignoring cell content) with a modified CW-SSIM tuned for binary, high-contrast table images. The CW-SSIM variant applies a one-level Haar wavelet decomposition and computes SSIM per sub-band:

$$ \text{CW-SSIM}(X, Y) = \frac{1}{4} \sum_{i \in \{A,H,V,D\}} \text{SSIM}(c^i_X, c^i_Y) $$

where $A$, $H$, $V$, $D$ denote the approximation, horizontal, vertical, and diagonal sub-bands respectively. TEDS-Structure is defined as:

$$ \text{TEDS-Structure} = 1 - \frac{\text{TED}_{\text{structure}}}{\max(|\mathcal{T}_{\text{pred}}|, |\mathcal{T}_{\text{gt}}|)} $$

VSGRPO dual-reward reinforcement fine-tuning. Standard GRPO optimizes text generation quality using correctness-based rewards without a separate value network. VSGRPO extends this to include a visual fidelity reward. For each training example, the model generates $N$ candidate LaTeX outputs. Each candidate is compiled to an image; if compilation fails the reward is 0. Successful compilations are scored by CW-SSIM against the ground-truth rendered image, yielding a binary reward at threshold 0.6. In parallel, each candidate is converted to HTML and scored by TEDS-Structure against the ground truth, yielding a second binary reward at threshold 0.9. The combined reward drives the GRPO objective:

$$ J_{\text{RFT}}(\theta) = \mathbb{E}\left[ \frac{1}{N} \sum_{i=1}^{N} \min!\left( \frac{\pi_\theta(o_i|q)}{\pi_{\theta_{\text{old}}}(o_i|q)} A_i,\ \text{clip}!\left(\frac{\pi_\theta(o_i|q)}{\pi_{\theta_{\text{old}}}(o_i|q)}, 1-\varepsilon, 1+\varepsilon\right) A_i \right) - \beta D_{\text{KL}}(\pi_\theta | \pi_{\text{ref}}) \right] $$

where $\varepsilon = 0.2$ and $\beta = 0.02$, and the group advantage $A_i$ is the z-score of reward $r_i$ within the group of $N$ samples:

$$ A_i = \frac{r_i - \text{mean}(\{r_1, \ldots, r_N\})}{\text{std}(\{r_1, \ldots, r_N\})} $$

Because LaTeX rendering is non-differentiable, this visual reward signal can only be incorporated via the reward mechanism in RL, not during gradient-based supervised training. That is the core motivation for using GRPO rather than extending SFT.

What experiments were performed?

Training data. SFT uses all 1,209,986 table-LaTeX pairs. VSGRPO fine-tunes on a curated subset of 5,936 complex-tier tables whose ground-truth LaTeX is under 3,000 characters, chosen to balance structural difficulty and rendering feasibility.

Test data. A separate crawl of arXiv papers from January to November 2024 yields 101,469 entries; from these, 496 simple, 354 medium, and 361 complex tables are sampled as the test set. An external dataset from the LATTE paper serves as an additional generalization benchmark.

Baselines. Commercial: Mathpix. General-purpose MLLMs: GPT-4o, Qwen2.5-VL-72B, InternVL2.5-78B. Task-specific: Nougat, LATTE.

Metrics. CW-SSIM and compile ratio measure rendered image quality. TEDS-Structure and TEDS measure structural and content alignment at the source level. Human preference evaluation covers 200 tables (50/50/100 by complexity), with majority-vote preference across four systems.

Ablations. The paper ablates (1) RL training data selection (simple-only vs. mixed vs. complex-only), (2) reward components (TEDS-Structure alone, CW-SSIM alone, both combined), and (3) the necessity of SFT initialization before VSGRPO.

Hardware. SFT for Nougat runs on 4 nodes of 8xA100 GPUs. InternVL2-1B SFT uses 4 nodes; VSGRPO uses 2 nodes. Qwen2.5-VL-3B SFT and VSGRPO each use 2 nodes.

What are the outcomes/conclusions?

Main benchmark results. On the authors’ own test set, Qwen2.5-VL-3B-VSGRPO achieves CW-SSIM scores of 0.8186 (simple), 0.7236 (medium), and 0.6145 (complex). On complex tables, the CW-SSIM gain over the SFT baseline is +0.034, and the compile ratio reaches 0.9917, matching or exceeding all baselines including Mathpix. On TEDS and TEDS-Structure, Qwen2.5-VL-3B-VSGRPO reaches 0.8673 TEDS and 0.9218 TEDS-Structure on complex tables. The reported TEDS gap over the next-best model is 0.1225 on complex tables. Intern2.5-VL-78B, a model more than 25 times larger, shows a TEDS collapse on complex tables (0.3379) while Qwen2.5-VL-3B-VSGRPO remains robust.

External benchmark. On the LATTE external dataset (predominantly simple tables), Qwen2.5-VL-3B-VSGRPO achieves CW-SSIM of 0.8225 and TEDS-Structure of 0.9461, outperforming LATTE itself (0.7615 / 0.9445) despite LATTE being a specialist system fine-tuned for this format.

Human evaluation. In the 200-table preference study, Qwen2.5-VL-3B-VSGRPO received the highest vote counts across all complexity levels: 42 vs. 29 for the SFT-only baseline on simple tables, 37 vs. 28 on medium, and 70 vs. 56 on complex.

Ablation findings. Training VSGRPO on complex tables only outperforms both simple-only and mixed-data variants across all metrics. Both reward components contribute independently; their combination achieves the highest performance on every metric. Skipping SFT and applying VSGRPO directly to the pre-trained base model results in substantially lower scores (CW-SSIM 0.4695 vs. 0.6145 on complex), confirming that a strong initialization is required.

Limitations. The authors acknowledge that the visual rendering step in VSGRPO training is computationally expensive. Each candidate output must be compiled to PDF and converted to PNG for CW-SSIM scoring, creating a training bottleneck even with multi-threading. This overhead restricted the VSGRPO training set to 5,936 tables. Additionally, the TEDS metric remains sensitive to syntactically irrelevant LaTeX differences (e.g., empty groups, different bold commands), which the paper illustrates with qualitative examples but does not fully resolve.

Reproducibility

Models

Two model families are reported. InternVL2-1B is fine-tuned via the VLM-R1 framework. Qwen2.5-VL-3B is fine-tuned via the ms-swift infrastructure. Both use full-parameter fine-tuning for SFT. Architecture details (layer count, attention heads) are not described in the paper; they are inherited from the pre-trained base models (Qwen2.5-VL-3B-Instruct from HuggingFace). During VSGRPO, the ViT visual encoder is frozen (--freeze_vit true, --freeze_parameters visual), so only the language decoder weights are updated in the RL phase. Trained weights for Qwen2.5-VL-3B-VSGRPO are publicly released at https://huggingface.co/LLLHHH/Table2Latex-RL (Apache-2.0, BF16 safetensors format, approximately 4B parameters).

Algorithms

SFT trains for one epoch with a maximum output length of 4,096 tokens. InternVL2-1B SFT uses batch size 4 with gradient accumulation steps of 2; Qwen2.5-VL-3B SFT uses batch size 4 with gradient accumulation steps of 2 on 2 nodes. For VSGRPO, InternVL2-1B uses 8 sampled generations per input, batch size 8, gradient accumulation 2, and 2 nodes. Qwen2.5-VL-3B VSGRPO uses 4 generations per input, batch size 4, gradient accumulation 2, and 2 nodes, for 1 epoch. Reward thresholds are 0.6 for CW-SSIM and 0.9 for TEDS-Structure. KL penalty $\beta = 0.02$; PPO clipping $\varepsilon = 0.2$.

The VSGRPO training script (published in the repository at examples/train/grpo/plugin/run_external_rm.sh) reveals additional hyperparameters not stated in the paper: learning rate 1e-5, warmup ratio 0.05, bfloat16 mixed precision, training-time sampling temperature 0.9, maximum completion length 3,000 tokens, maximum image resolution 524,288 pixels (--max_pixels), and 8 GPUs per node via torchrun. The optimizer is not named explicitly in the script but ms-swift defaults to AdamW. Testing uses maximum output length 8,192, batch size 1, temperature 0. Training requires a Docker environment; the README specifies a ModelScope image with CUDA 12.6, PyTorch 2.7.1, vLLM 0.10.1, and ms-swift 3.8.1.

Data

Training data covers arXiv papers from October 2017 to April 2023 (1,209,986 table entries). Test data covers arXiv papers from January to November 2024 (496 simple, 354 medium, 361 complex tables). VSGRPO uses 5,936 complex tables with ground-truth LaTeX under 3,000 characters. The full 1.2M SFT training corpus is not directly downloadable from the repository; instead, the authors provide arxiv_papers_get.py and a table.ipynb Kaggle notebook to re-crawl arXiv source files and reconstruct the dataset. The evaluation split (1,211 images, Apache-2.0) is released at https://huggingface.co/datasets/LLLHHH/Table2LaTeX-RL. The underlying arXiv source files are available under arXiv’s open-access policy, though the license status of individual paper LaTeX sources varies by author. The code repository is Apache-2.0.

Evaluation

CW-SSIM is computed using a Python implementation with Haar wavelet decomposition (provided as cw_ssim.ipynb in the repository). TEDS-Structure and TEDS use tree edit distance over structural HTML representations parsed from the LaTeX. All testing is conducted inside a texlive-full Docker environment to ensure consistent LaTeX rendering. No error bars, significance tests, or seed sensitivity analysis are reported; the NeurIPS checklist explicitly states “No” for statistical significance on the grounds that “running a complete experiment takes one week” and compute resources were limited. Human evaluation uses majority voting by 5 graduate students in computer science on 200 tables (50 simple, 50 medium, 100 complex), with anonymized and randomly shuffled model outputs.

Hardware

SFT for Nougat: 4 nodes of 8xA100 GPUs (32 A100s total). SFT for InternVL2-1B: 4 nodes of 8 GPUs; for Qwen2.5-VL-3B: 2 nodes of 8 GPUs (16 GPUs total). VSGRPO for both models: 2 nodes of 8 GPUs (16 GPUs total); the training script confirms 8 processes per node via --nproc_per_node=8. The GPU type for all non-Nougat experiments is not stated in the paper (only A100 is explicitly named for Nougat). Total GPU-hours, cost estimates, and inference latency figures are not reported. The NeurIPS checklist self-reports “Yes” on compute resources but the paper body only enumerates node counts. The rendering bottleneck in VSGRPO (PDF compilation per candidate per training step) is noted qualitatively but not measured; the authors state it was the reason for limiting VSGRPO training to 5,936 tables.

BibTeX

@inproceedings{ling2025table2latexrl,
  title={Table2LaTeX-RL: High-Fidelity LaTeX Code Generation from Table Images via Reinforced Multimodal Language Models},
  author={Ling, Jun and Qi, Yao and Huang, Tao and Zhou, Shibo and Huang, Yanqin and Yang, Jiang and Song, Ziqi and Zhou, Ying and Yang, Yang and Shen, Heng Tao and Wang, Peng},
  booktitle={Advances in Neural Information Processing Systems},
  year={2025}
}

TL;DR

SepFormer is a real-time table structure recognition model that directly regresses row and column separator lines using a DETR-style architecture. It combines the RT-DETR backbone (ResNet-34 + hybrid encoder) with dual decoder branches, each organized as two sequential transformer decoder stacks: a coarse decoder that predicts straight single-line separators, and a fine decoder that refines P evenly sampled points from each coarse line into a line-strip. This coarse-to-fine decomposition avoids segmentation masks entirely and eliminates the ROIAlign-based merge stage found in split-and-merge systems. On four benchmarks (SciTSR-COMP, PubTabNet, WTW, iFLYTAB), SepFormer runs at 25.6 FPS on average while staying within 0.6-1.2% of the top non-real-time methods on each dataset.

What kind of paper is this?

Dominant: $\Psi_{\text{Method}}$ The main contribution is the architecture: a coarse-to-fine dual decoder design for direct separator regression. The paper’s structure centers on model design, ablation studies, and comparisons against SOTA on four benchmarks.

Secondary: None. No new dataset or benchmark is released.

What is the motivation?

Table structure recognition (TSR) aims to reconstruct the logical layout of tables in document images as machine-readable representations. The two dominant paradigms for the split-and-merge family are segmentation-based methods (which predict pixel masks and apply post-processing to extract separator lines) and regression-based methods (which directly output line coordinates).

Segmentation-based methods such as SEMv2 and RTSR require high-resolution feature maps and additional post-processing steps to convert masks to lines. Regression-based approaches such as TSRFormer and its DQ-DETR variant already eliminate the segmentation stage, but retain a merge module relying on ROIAlign for the second stage, adding both computational cost and a dependency on the quality of the first stage.

SepFormer’s goal is to collapse the split-and-merge pipeline into a single forward pass. By framing TSR purely as separator line regression using a DETR architecture, the method removes both segmentation masks and the ROIAlign merge module, targeting practical real-time deployment (the authors note their system is deployed in a commercial product).

What is the novelty?

Architecture overview

SepFormer builds on the RT-DETR backbone: a ResNet-34 encoder extracts multi-scale feature maps $C_2, C_3, C_4$ at strides 8, 16, and 32; a hybrid encoder (one transformer encoder layer on $C_4$ plus a cross-scale fusion module) produces enhanced feature maps $\{M_1, M_2, M_3\}$. These are flattened and concatenated into a single sequence $M \in \mathbb{R}^{S \times C}$ with $C = 256$ channels, which is then fed into two parallel decoder branches for rows and columns respectively.

Coarse-to-fine decoder

Each branch contains two sequential stages (Fig. 2 in the paper).

Query selection. A detection head operating on the full encoder sequence scores $S$ candidate line proposals and selects the top $K = 300$ positions (by classification confidence) as initial reference points for the coarse decoder. Unlike the box proposals in Deformable DETR, SepFormer uses line proposals: row proposals take the form $\{x_i, y_i, x_i + 2^{l-1} \times s, y_i\}$ and column proposals $\{x_i, y_i, x_i, y_i + 2^{l-1} \times s\}$.

Coarse decoder. Three deformable transformer decoder layers refine the selected queries iteratively. The output for the $k$-th query is a single-line separator $l^k = \{(x_1^k, y_1^k), (x_2^k, y_2^k)\}$ and a classification score $c^k$.

Sampling. For each accepted coarse separator, $P = 16$ reference points are evenly interpolated between its two endpoints:

$$ \text{Sampling} = \left\{ \left(1 - \frac{t}{P}\right)(x_1, y_1) + \frac{t}{P}(x_2, y_2) ;\middle|; t = 1, \ldots, P \right\} $$

Fine decoder. Three more deformable transformer decoder layers refine the sampled reference points, producing a line-strip $ls^k = \{(x_i^k, y_i^k) \mid i = 1, \ldots, P\}$. The fine stage reuses the same content queries and encoder memory; only the reference points change.

Loss function

The training objective combines four terms per branch. Hungarian matching uses only single-line ($L_1$) distance for pairing predictions with ground-truth lines:

$$ \mathcal{L}_{\text{match}} = \sum_{i=1}^{N_{\text{row}}} \left( \lambda_{\text{coord}} |l_{gt}^i - l^{\sigma(i)}| + \lambda_{\text{cls}} , c^{\sigma(i)} \right) $$

After matching, the full loss for the row branch is:

$$ \mathcal{L}^{\text{row}} = \lambda_1 \mathcal{L}_{\text{cls}}^{\text{row}} + \lambda_2 \mathcal{L}_{\text{angle}}^{\text{row}} + \lambda_3 \mathcal{L}_{\text{line}}^{\text{row}} + \lambda_4 \mathcal{L}_{\text{linestrip}}^{\text{row}} $$

The angle loss uses cosine similarity between the predicted and ground-truth line direction vectors, with an additional penalty term $|c_{gt}^n| \times 4$ in the denominator to increase sensitivity for short separators:

$$ \mathcal{L}_{\text{angle}}^{\text{row}} = \frac{1}{N_{\text{row}}} \sum_{n=1}^{N_{\text{row}}} \left( 1 - \frac{\cos(c_{gt}^n, c_n)}{|c_{gt}^n| \times 4} \right) $$

The coordinate losses $\mathcal{L}_{\text{line}}$ and $\mathcal{L}_{\text{linestrip}}$ are $L_1$ distances on single-line and line-strip coordinates respectively. The total loss sums row and column branches:

$$ \mathcal{L}^{\text{sep}} = \mathcal{L}^{\text{row}} + \mathcal{L}^{\text{col}} $$

Loss coefficients are set to $\lambda_1 = 1, \lambda_2 = 1, \lambda_3 = 3, \lambda_4 = 1$; matching coefficients to $\lambda_{\text{cls}} = 2, \lambda_{\text{coord}} = 3$.

What experiments were performed?

Datasets

• SciTSR-COMP: 2,885 train / 716 test complex PDF tables. Metric: cell adjacency relationship F1.
• PubTabNet: 500,777 train / 9,115 validation / 9,138 test (validation set used, as the test labels are not released). Metric: TEDS-Struct.
• WTW: 10,970 train / 3,611 test; real-world wired tables including inclined, curved, and occluded examples. Metric: cell adjacency F1.
• iFLYTAB: 12,104 train / 5,187 test; wired and wireless tables from digital documents and camera capture. Metric: cell adjacency F1. Ablations also run on this dataset.

Baselines

Real-time comparison: RTSR (segmentation-based, 34.4 FPS average). Non-real-time comparisons on each benchmark include TSRFormer w/ DQ-DETR, SEMv2, SEMv3, GrabTab, LORE++, GridFormer, and TRUST among others.

Ablation studies

Table 5 in the paper evaluates two design choices on iFLYTAB: (1) two-stage vs. one-stage decoders and (2) single-line vs. line-strip matching. A one-stage 3-layer decoder achieves 90.8% F1; one-stage 6-layer achieves 89.7% (worse, suggesting overfitting at depth). The two-stage 3-layer decoder with single-line matching reaches 93.8%, a gain of 3.0% over the one-stage 3-layer baseline. Line-strip matching degrades performance in all configurations, indicating that the coarser $L_1$ single-line distance provides better optimization signal for the Hungarian algorithm.

Table 6 in the paper shows the angle loss contributes a marginal +0.2% F1 on iFLYTAB but improves consistency for short separators.

What are the outcomes/conclusions?

Speed: SepFormer averages 25.6 FPS across all four test datasets (RTX 3060), versus RTSR’s 34.4 FPS. It is the second real-time-capable method reported and the only regression-based real-time TSR system in the comparison.

Accuracy on SciTSR-COMP: 98.6% F1 (99.0 P / 98.2 R), within 0.7% of the best non-real-time method LORE++ (99.3%) and above RTSR (98.3%).

Accuracy on PubTabNet: 96.8% TEDS-Struct, 1.1 points above RTSR (95.7%) and within 0.7% of TSRFormer w/ DQ-DETR and SEMv3 (both at 97.5%).

Accuracy on WTW: 93.9% F1 (93.7 P / 94.2 R), above RTSR (92.9%) and within 1.2% of SEMv3 / GrabTab / LORE++ (all at 95.1%).

Accuracy on iFLYTAB: 93.8% F1 (94.6 P / 93.2 R), above RTSR (91.1%) and within 0.6% of SEMv3 (94.4%).

Limitations: The authors identify three failure modes: (1) heavily warped images where text lines exhibit varying degrees of warping within the same image; (2) two adjacent separators that are collapsed into a single predicted separator (attributed to insufficient discrimination in the $L_1$ single-line matching criterion); (3) tables with very high row or column density where over- or under-detection occurs. No code or model weights are publicly released.

Reproducibility

Models

• Backbone: ResNet-34 pretrained on ImageNet. Three feature levels $C_2$, $C_3$, $C_4$ at strides 8, 16, 32.
• Hybrid encoder: one transformer encoder layer (on $C_4$ only) plus cross-scale fusion; $C = 256$ channels.
• Dual decoder branches: query selection + 3 coarse deformable decoder layers + 3 fine deformable decoder layers per branch. 8 attention heads, 4 sampling points per deformable attention.
• $K_{\text{row}} = K_{\text{col}} = 300$ top queries selected.
• Model weights are not publicly released.

Algorithms

• Optimizer: not specified (implicitly AdamW or Adam for DETR-style models, but not stated).
• Learning rate: $3 \times 10^{-5}$; cosine annealing over 100 epochs (20 epochs for PubTabNet).
• Input resolution: longer side scaled to one of $\{864, 896, 928, 960\}$ during training (aspect ratio preserved); 896 at inference.
• Confidence thresholds: $\tau_{\text{row}} = \tau_{\text{col}} = 0.95$.
• $P = 16$ sampling points per separator.

Data

• SciTSR: MIT license. 12,000 train / 3,000 test; SciTSR-COMP subset used (2,885 train / 716 test).
• PubTabNet: CDLA-Permissive-1.0. ~500k train; validation set (9,115) used for evaluation.
• WTW: License unknown. 10,970 train / 3,611 test.
• iFLYTAB: Introduced with SEMv2. License unknown. 12,104 train / 5,187 test.

Evaluation

• Cell adjacency relationship F1 for SciTSR-COMP, WTW, and iFLYTAB.
• TEDS-Struct for PubTabNet (test labels unreleased; validation set used).
• No error bars, confidence intervals, or multi-seed results reported.
• FPS measured on RTX 3060; training on a single NVIDIA V100 32 GB.
• RTSR FPS figures taken from the prior paper [31]; direct head-to-head hardware comparison is approximate.

Hardware

• Training: single NVIDIA V100 32 GB.
• Inference: NVIDIA RTX 3060.
• GPU-hours, total compute cost, and memory at inference are not reported.

BibTeX

@inproceedings{nguyen2025sepformer,
  title={SepFormer: Coarse-to-fine Separator Regression Network for Table Structure Recognition},
  author={Nguyen, Nam Quan and Pham, Xuan Phong and Tran, Tuan-Anh},
  booktitle={Proceedings of the International Conference on Document Analysis and Recognition (ICDAR)},
  year={2025}
}

TL;DR

TableCenterNet is a one-stage, end-to-end table structure recognition network built on the CenterNet framework. It unifies spatial bounding-box regression and logical row/column index prediction into a single parallel multi-task pass, avoiding the serial two-stage pipelines of prior work. On the TableGraph-24k benchmark, it improves logical location accuracy by 7.2 percentage points over the prior best while running 15.7 times faster than LORE at comparable overall performance.

What kind of paper is this?

Dominant: $\Psi_{\text{Method}}$: The headline contribution is a new neural network architecture and training recipe. The paper proposes a regression-based approach to jointly predict spatial and logical cell structure from a single model, with ablation studies, SOTA comparisons across three benchmarks, and efficiency analysis all centered on validating the architectural design choices.

What is the motivation?

Table structure recognition (TSR) requires a system to recover both the physical location of each cell (bounding quadrilateral) and its logical position in the table grid (starting/ending row and column indices). Prior approaches fell into one of two camps, each with significant drawbacks.

The classic two-stage pipeline first detects row and column regions or individual cells, then runs a separate second-stage model to infer logical indices. Methods like TGRNet and LORE are representative: TGRNet uses a Graph Convolutional Network on detected cells, and LORE cascades a Transformer over spatial detections. These require independent training of multiple sub-modules and serial inference, resulting in high latency and compounding prediction errors.

Existing one-stage methods (TableNet, Cycle-CenterNet, SCAN, TRACE) simplified training but each suffered notable limitations in scenario coverage: they either assumed clean, bordered tables or were sensitive to geometric deformations, and none predicted logical indices directly in a single pass without complex post-processing.

The central gap was the absence of a method that could handle diverse real-world table scenarios (borderless, distorted, multi-column spanning, curved) while predicting both spatial and logical structure from a single model with simple training and fast inference.

What is the novelty?

TableCenterNet extends the CenterNet object detection framework into a multi-task regression architecture. The key idea is to represent the logical structure of the table as a pair of continuous interpolation maps (one for rows, one for columns) that can be regressed directly from features, then use those maps to read off integer row/column indices for any detected cell.

Overall architecture. A CNN backbone (DLA-34 or StarNet-s3) produces a shared feature map. Six parallel regression heads decode from that shared representation:

1. A keypoint heatmap for cell corner detection.
2. Keypoint offsets.
3. Two sets of corner-to-center offset vectors $\hat{u}$ (centers-to-corner) and $\hat{v}$ (corners-to-center).
4. A row interpolation map $\hat{I}_r \in \mathbb{R}^{H \times W}$.
5. A column interpolation map $\hat{I}_c \in \mathbb{R}^{H \times W}$.
6. Per-cell row/column span regression $\hat{s}_i = \{\hat{s}_i^r, \hat{s}_i^c\}$.

Spatial location regression. The spatial head follows a modified CenterNet/Cycle-CenterNet approach. Each cell is represented by its four corner points, and the network regresses offset vectors between corners and cell centers. The spatial loss is a weighted combination of L1 losses on these offset vectors, plus a regularization term on invalid (unmatched) corner-to-center vectors:

$$\mathcal{L}_{\text{spatial}} = \frac{1}{8n} \sum_{i=1}^{n} \omega(i) \left(\lambda_u \mathcal{L}_1(u_i, \hat{u}_i) + \lambda_v \mathcal{L}_1(v_i, \hat{v}_i)\right) + \lambda_e \mathcal{L}_e$$

where $\omega(i)$ is a sine-based quality weighting function and $\lambda_u = 1.0$, $\lambda_v = 0.5$, $\lambda_e = 0.2$.

Interpolation maps and logical location. The key novelty is how logical indices are obtained without requiring a graph or sequence model. Before training, a Delaunay triangulation-based polygon interpolation algorithm (Algorithm 1) renders a row interpolation map $I_r$ and column interpolation map $I_c$ for each training image. Each pixel in $I_r$ is assigned a value interpolated from the logical row boundaries of the cells overlapping that pixel. Once trained, the network regresses $\hat{I}_r$ and $\hat{I}_c$ directly. At inference, the logical indices of cell $i$ are extracted by sampling these maps at the upper-left corner of the predicted cell:

$$\begin{aligned} \hat{r}_i^{st} &= \text{round}(\hat{I}_r[\hat{y}_{i,1}, \hat{x}_{i,1}]) \\ \hat{r}_i^{ed} &= \hat{r}_i^{st} + \lfloor \hat{s}_r^i \rfloor - 1 \\ \hat{c}_i^{st} &= \text{round}(\hat{I}_c[\hat{y}_{i,1}, \hat{x}_{i,1}]) \\ \hat{c}_i^{ed} &= \hat{c}_i^{st} + \lfloor \hat{s}_c^i \rfloor - 1 \end{aligned}$$

Logical location supervision. Two auxiliary losses guide the interpolation map regression. The boundary loss $\mathcal{L}_{\text{boundary}}$ applies higher weight near logical boundaries (integer values in the map) via a squared sine weight function. The span loss $\mathcal{L}_{\text{span}}$ enforces consistency between the regressed per-cell span $\hat{s}_i$ and the spans implied by reading the interpolation map at the cell’s corners:

$$\mathcal{L}_{\text{logical}} = \mathcal{L}_{\text{boundary}} + \mathcal{L}_{\text{span}}$$

Selective gridding. For well-structured datasets without significant deformation, the authors propose decomposing merged (spanning) cells into unit grid cells before generating interpolation maps. This unifies the interpolation style and improves regression accuracy on such datasets.

Overall loss:

$$\mathcal{L}_{\text{overall}} = \mathcal{L}_{\text{keypoint}} + \mathcal{L}_{\text{spatial}} + \mathcal{L}_{\text{logical}}$$

What experiments were performed?

Datasets. Three public benchmarks were used:

• ICDAR-2013 (156 tables, PDF documents from government websites; modified ICDAR-2013.c version with cell bounding boxes; 80/20 train/test split following prior work).
• WTW (Wired Table in the Wild; 10,970 training and 3,611 test images from diverse real-world scenes including curved, occluded, and blurred tables).
• TableGraph-24k (20,000 training, 2,000 validation, 2,000 test images of document tables from arXiv papers; includes wired, borderless, and partial-border tables).

Metrics. Physical coordinate quality was assessed by precision, recall, and F1 at IoU threshold 0.5 (and 0.9 for WTW). Logical location was assessed by: (1) accuracy of logical location indices, (2) F1 of cell adjacency relations, and (3) TEDS (Tree-Edit-Distance-based Similarity).

Baselines. Comparisons were made against TGRNet, LORE, Cycle-CenterNet, SCAN, TRACE, FLAG-Net, NCGM, LGPMA, TabStrNet, TOD, TSRFormer, and CascadeTabNet, covering the main prior paradigms.

Ablation. Seven ablation experiments on WTW investigated: (a) whether cell-span regression helps, (b) whether the span-supervised loss $\mathcal{L}_{\text{span}}$ helps beyond direct span regression, (c) whether the boundary loss $\mathcal{L}_{\text{boundary}}$ adds value, and (d) whether interpolation map regression is better than regressing logical indices directly from upper-left or all four corners.

Two model variants. TableCenterNet-D uses DLA-34 as the backbone; TableCenterNet-S uses the lighter StarNet-s3.

What are the outcomes/conclusions?

ICDAR-2013. TableCenterNet-D improves physical F1 by 0.6 percentage points and logical accuracy by 4.3 percentage points over LORE. TableCenterNet-D and TableCenterNet-S both exceed Cycle-CenterNet and SCAN on adjacency metrics when trained only on WTW (no ICDAR-2013 training data).

WTW. At IoU 0.5, both variants match or exceed all prior methods on physical F1 (97.3%). Logical accuracy reaches 83.0% (TableCenterNet-D), versus 95.1% for LORE on adjacency F1 but 82.9% TEDS. TEDS improves 1% over SCAN’s 90.7% to reach 91.7%. At the stricter IoU threshold of 0.9, physical F1 (91.3%) improves 13 percentage points over Cycle-CenterNet.

TableGraph-24k. TableCenterNet-D achieves 95.1% overall logical accuracy versus 87.9% for LORE (+7.2 pp), with near-parity on physical F1 (96.0% vs. LORE’s unreported / TGRNet’s 90.6% baseline).

Efficiency. Using the same DLA-34 backbone, TableCenterNet-D has 27.3% fewer parameters (16.82 M vs. 27.86 M) and runs 15.7 times faster than LORE (227 ms vs. 3788 ms per image at 1024x1024). TableCenterNet-S (6.27 M parameters, 50.3 GFLOPs, 215 ms) offers further reduction suitable for edge deployment.

Ablation findings. Interpolation map regression outperforms direct logical index regression from either one corner (–3.2% Acc) or four corners (–1.2% Acc). The span loss $\mathcal{L}_{\text{span}}$ contributes the largest single gain (+6.5% Acc on WTW). The boundary loss adds +0.6% Acc. Combined, they account for the majority of the logical accuracy improvement.

Limitations acknowledged. The authors note that the WTW dataset focuses on bordered tables, so the method’s performance on diverse borderless scenes in the wild is not fully validated. Future work mentions collecting more challenging wireless table data. The paper does not report results on PubTables-1M or other large-scale benchmarks, and comparisons on TableGraph-24k are limited to two prior methods.

Reproducibility

Models

• Two backbone variants are evaluated: DLA-34 (18.8 M parameters in LORE; 16.82 M in TableCenterNet-D) and StarNet-s3 (TableCenterNet-S, 6.27 M parameters). Exact layer configurations are not detailed beyond the reference to the original DLA and StarNet papers.
• Six parallel regression heads, each with 256 hidden channels, appended to the backbone’s output feature map within the CenterNet regression framework.
• Backbones initialized from ImageNet-pretrained weights.
• Code released under Apache-2.0 at https://github.com/dreamy-xay/TableCenterNet. No information on whether pretrained TSR model weights are released separately.

Algorithms

• Optimizer: Adam.
• Batch size: 22 for ICDAR-2013 and WTW; 38 for TableGraph-24k.
• Training: 200 epochs. Learning rate starts at $1.25 \times 10^{-4}$, decayed to $1.25 \times 10^{-5}$ at epoch 140 and $1.25 \times 10^{-6}$ at epoch 180.
• Input images resized to 1024x1024 (768x768 for TableGraph-24k), aspect ratio maintained with padding.
• Data augmentation and normalization applied (specific augmentation types not enumerated).
• ICDAR-2013 training used fine-tuning from a WTW-trained model for 100 epochs, with ICDAR-2013 training images border-padded by 100 black pixels on all sides.
• Interpolation map GT generated once before training using Algorithm 1 (Delaunay triangulation-based polygon interpolation).
• Loss hyperparameters: $\lambda_u = 1.0$, $\lambda_v = 0.5$, $\lambda_e = 0.2$.

Data

• ICDAR-2013.c: modified version with cell bounding boxes rather than word boxes; 258 table images after preprocessing; 80/20 train/test split.
• WTW: 10,970 training / 3,611 test; publicly available; physical and logical coordinates with quadrilateral physical boxes.
• TableGraph-24k: 20,000 train / 2,000 val / 2,000 test; sourced from arXiv scholarly articles.
• All three datasets are publicly available. Licenses not stated in the paper.

Evaluation

• Physical coordinate metrics: precision, recall, F1 at IoU 0.5 (and additionally IoU 0.9 on WTW).
• Logical location accuracy: per-index accuracy of start/end row/col.
• Adjacency relation F1: follows ICDAR-2013 competition evaluation protocol.
• TEDS: Tree-Edit-Distance-based Similarity between predicted and ground-truth HTML table representations.
• No error bars or confidence intervals are reported. Single-run results throughout.
• Comparisons are not always on identical settings (some baselines use different training data marked with $\ddagger$), which limits direct head-to-head conclusions.

Hardware

• Training hardware: two NVIDIA GeForce RTX 3090 GPUs (24 GB VRAM each).
• Inference time measured at 1024x1024 input on the WTW test set; TableCenterNet-D: 227.4 ms/image; TableCenterNet-S: 214.8 ms/image. Hardware specification for inference timing is not separately reported (likely the same workstation).
• No cloud compute cost or energy consumption figures are provided.
• The compact TableCenterNet-S (6.27 M parameters, 50.3 GFLOPs) is described by the authors as feasible for edge device deployment.

BibTeX

@article{xiao2025tablecenternet,
  title={TableCenterNet: A one-stage network for table structure recognition},
  author={Xiao, Anyi and Yang, Cihui},
  journal={arXiv preprint arXiv:2504.17522},
  year={2025}
}

TL;DR

TRACE is a single-model, end-to-end table recognition system that detects corners and edges at the pixel level and reconstructs cell grids from those low-level features without needing a separate table detector. It also introduces SubTableBank, an in-house dataset with per-cell border visibility annotations. At publication, the authors report top results on ICDAR 2013 and WTW.

What kind of paper is this?

Dominant: $\Psi_{\text{Method}}$ (Methodological): The core contribution is a new architecture and post-processing pipeline for bottom-up table reconstruction. The paper provides ablations, baselines, and comparisons across two benchmarks to validate that the corner-and-edge prediction approach works better than prior two-stage pipelines.

Secondary: $\Psi_{\text{Resource}}$ (Infrastructure): The authors describe an in-house dataset, SubTableBank, with annotated cell bounding boxes and per-edge visibility flags. A portion is promised for public release, though no public link is provided in this preprint.

What is the motivation?

Table recognition in document images is typically split into two sequential steps: table detection (TD) to localize the table region, and table structure recognition (TSR) to identify rows, columns, and spanning cells. Running two separate models is expensive at training and inference time, and errors from the detection stage propagate into structure recognition with no feedback between the two. The authors argue that the field has studied end-to-end approaches only sparsely, and that the few examples that exist still require parallel branches for bordered versus borderless tables, or rely on OCR input to handle empty cells.

The deeper motivation is conceptual: a table is a hierarchical object built from cells, and each cell is a rectangle defined by its four corners and four edges. If a model can detect those fundamental visual elements directly, cells and tables can be assembled bottom-up through simple post-processing, without ever needing to detect the high-level bounding box first.

What is the novelty?

The central contribution is the framing of table reconstruction as low-level feature prediction rather than object detection.

A single U-Net style CNN with a ResNet-50 backbone outputs five segmentation maps:

• one corner heatmap (Gaussian blobs at every cell corner)
• two explicit edge maps (visible horizontal and vertical borders)
• two implicit edge maps (invisible horizontal and vertical separators)

The distinction between explicit and implicit edges is important for borderless tables. Previous split-merge methods either ignore invisible lines altogether or require separate model branches. TRACE instead labels every cell border by visibility at annotation time and predicts both types jointly in a shared network.

The training objective is a pixel-wise MSE loss across all five channels:

$$ L = \sum_{p}\sum_{i} | S_{i}(p) - S_{i}^{\ast}(p) |^{2}_{2} $$

where $S_{i}(p)$ is the ground truth for the $i$-th map at pixel $p$, and $S_{i}^{\ast}(p)$ is the model prediction.

The post-processing pipeline reconstructs tables through a split-merge strategy. Explicit and implicit edge maps are binarized and projected onto the horizontal and vertical axes. The midpoint of each projected edge cluster becomes a separator. For implicit horizontal lines where rows contain empty cells, ambiguity in the projection is resolved using corner map peak positions. Spanning cells are recovered by checking whether an edge is present at the midpoint between adjacent cell candidates; if not, they are merged. Table location is finally derived from the bounding coordinates of all reconstructed cells.

An optional image rectification step applies the Hough line transform followed by a homography to correct perspective distortion before reconstruction. This step is mostly relevant for scene-image tables in the WTW dataset.

What experiments were performed?

The authors evaluate on two public benchmarks.

ICDAR 2013 Table Competition contains 156 PDF-rendered document tables from EU/US government websites. The official evaluation uses character-level recall, precision, and F1 for table detection, along with Purity and Completeness scores. Table structure recognition is evaluated with adjacency-relation recall and precision. TRACE is tested without any fine-tuning on ICDAR 2013 data.

Wired Table in the Wild (WTW) includes 10,970 training and 3,611 test images of tables captured in natural scenes with perspective distortion, rotation, and varied lighting. Detection is measured at cell level with IoU = 0.9. Structure recognition is measured by cell adjacency relation recall and precision using IoU = 0.6 for cell matching.

Baselines: For ICDAR 2013, comparisons include Nurminen (PDF-based), TableNet, DeepDeSRT, GTE, SPLERGE, LGPMA, and GTE with ground-truth table crops. For WTW, comparisons include Cycle-CenterNet (TD+TSR), TSRFormer (TSR-only with GT table), and NCGM (TSR-only with GT table).

Training setup: ResNet-50 pretrained on ImageNet, input resized to 1280 on the long side, Adam optimizer, initial learning rate $1 \times 10^{-4}$ decayed every 10k steps, 100k total iterations, batch size 12, Online Hard Negative Mining, and standard augmentations. The in-house dataset of 9,717 images (7,783 train / 971 val / 963 test) is used for training.

What are the outcomes/conclusions?

On ICDAR 2013 table detection, TRACE reaches F1 = 97.53%, with the highest Completeness (150) and Purity (147) scores, outperforming all image-based methods on completeness and purity even though GTE leads on raw F1.

On ICDAR 2013 structure recognition, TRACE reaches F1 = 97.46% end-to-end, the highest reported result and the only method evaluated without pre-cropped GT table regions. Methods with GT crops score lower (GTE with GT at 96.24%).

On WTW structure recognition, TRACE achieves F1 = 94.5%, which is the best among compared methods, again without requiring GT table regions. Cell-level detection at IoU = 0.9 is 64.8% F1, which the authors note is inflated downward by annotation imprecision on small cells; at IoU = 0.7 the same model scores 94.9% F1.

Bordered vs. borderless breakdown (ICDAR 2013): TSR F1 on bordered tables is 99.4%, while borderless tables score 93.85%. The primary failure modes involve ambiguous implicit edge placement, mixed explicit-implicit separators along the same line, and partial table detection when cell content spacing is large.

Limitations acknowledged:

• Implicit edge prediction is the main source of errors, especially when content gaps are large or mixed-type separators appear.
• The model uses a CNN backbone (ResNet-50) without global attention; the authors suggest Swin Transformer or similar architectures as future work.
• Evaluation on tag-generation benchmarks (SciTSR, PubTabNet, TableBank) is out of scope because TRACE does not produce text content and those metrics require OCR.
• The internal dataset used for training is not yet publicly available at the time of publication.

Reproducibility

Models

• Architecture: U-Net style encoder-decoder, ResNet-50 backbone, five-channel segmentation head (one corner map, four edge maps).
• No pretrained TRACE weights are released.
• Backbone initialized from ImageNet-pretrained ResNet-50.

Algorithms

• Optimizer: Adam.
• Learning rate: $1 \times 10^{-4}$, decayed every 10k iterations.
• Total training: 100k iterations, batch size 12.
• Online Hard Negative Mining applied during training.
• Augmentations: color jitter, random rotations, random cropping.
• Binarization thresholds: 0.5 for explicit edge maps, 0.2 for implicit edge maps.
• Edge length filter: edges shorter than 25% of the corresponding table dimension are discarded before the split step.
• No fine-tuning performed on ICDAR 2013; the model trained on the in-house dataset is applied directly.

Data

• In-house training dataset: 9,717 annotated document images (financial and scientific documents plus some from TableBank). Split: 7,783 train / 971 val / 963 test.
• Annotations include cell bounding boxes with per-edge visibility flags (explicit vs. implicit).
• The dataset is described as “SubTableBank.” The authors state they “will soon make a portion” available publicly; no URL is provided in this preprint.
• WTW dataset (public): 10,970 train / 3,611 test; wired tables only from natural scenes. Source: Long et al., ICCV 2021.
• ICDAR 2013 benchmark: 156 tables from EU/US government PDFs; used for testing only.

Evaluation

• ICDAR 2013 TD: character-level recall, precision, F1, Purity, Completeness (official protocol).
• ICDAR 2013 TSR: adjacency-relation recall and precision.
• WTW: cell-level recall/precision/F1 at IoU = 0.9 for detection; adjacency-relation recall/precision at IoU = 0.6 for structure.
• The paper notes that IoU = 0.9 for cell detection is strict relative to annotation quality, and IoU = 0.7 may be more appropriate for small cells. Additional results at IoU 0.8, 0.7, and 0.6 are provided.
• No error bars or multiple-run statistics are reported.

Hardware

• Training hardware not specified.
• No inference latency or throughput numbers are reported.
• No cost or energy estimates provided.

BibTeX

@inproceedings{baek2023trace,
  title={TRACE: Table Reconstruction Aligned to Corner and Edges},
  author={Baek, Youngmin and Nam, Daehyun and Surh, Jaeheung and Shin, Seung and Kim, Seonghyeon},
  booktitle={International Conference on Document Analysis and Recognition},
  year={2023}
}

TL;DR

CascadeTabNet applies a three-stage Cascade Mask R-CNN with an HRNet backbone to jointly detect tables and recognize their cell structure in document images. The authors use two-stage iterative transfer learning from a merged general dataset down to a smaller annotated set, combined with dilation and smudge image augmentations, to achieve competitive results on ICDAR 2013, ICDAR 2019, and TableBank while training on relatively small amounts of labeled data.

What kind of paper is this?

Dominant: $\Psi_{\text{Method}}$: The central contribution is a complete training pipeline for an instance segmentation model adapted to table understanding. The paper devotes most of its space to architecture choices, augmentation strategies, ablation tables comparing model variants, and detailed transfer learning procedures.

Secondary: $\Psi_{\text{Resource}}$: The authors manually annotate 342 images from ICDAR 2019 with three-class labels (bordered table, borderless table, borderless cell masks) and release these annotations publicly. This is a supporting contribution rather than the headline.

What is the motivation?

Table recognition from document images involves two tightly coupled subproblems: locating table regions and identifying the internal cell structure. Earlier systems addressed these independently, often relying on heuristics (line detection, junction analysis) or sequential two-model pipelines where errors in detection propagate into structure recognition. More recent deep-learning approaches, such as DeepDeSRT and TableNet, attempted joint or sequential CNN-based solutions but still required large training sets or introduced separate post-processing steps.

The authors aim to handle both tasks with a single model that learns efficiently from limited labeled data, and that outputs cell-level segmentation masks directly without rule-based post-processing for the structure recognition component.

What is the novelty?

The paper combines three ideas that had not been assembled in this way for document table recognition:

1. Cascade Mask R-CNN with HRNet backbone. The model (CMRcnnHr, referred to as CascadeTabNet in the paper) is a three-stage Cascade Mask R-CNN using HRNetV2p-W32 as the backbone. HRNet maintains high-resolution feature maps throughout the network rather than progressively downsampling, which helps preserve fine-grained spatial detail needed for cell-boundary segmentation. The cascade stages allow the model to refine bounding box proposals at increasingly strict IoU thresholds, improving localization quality. The architecture is implemented via MMdetection using the default cascade_mask_rcnn_hrnetv2p_w32_20e configuration.

2. Two-stage iterative transfer learning. The training strategy proceeds as:

$$\theta_{\text{general}} \leftarrow \text{fine-tune}(\theta_{\text{ImageNet+COCO}},\ \mathcal{D}_{\text{general}})$$

$$\theta_{\text{specific}} \leftarrow \text{fine-tune}(\theta_{\text{general}},\ \mathcal{D}_{\text{specific}})$$

where $\mathcal{D}_{\text{general}}$ is a merged dataset of 1,934 images (ICDAR 2019 Modern, Marmot, and a GitHub borderless table set) with all tables labeled as one class, and $\mathcal{D}_{\text{specific}}$ is a smaller 342-image set annotated for three classes (bordered tables, borderless tables, borderless cell masks). No layers are frozen during any stage of fine-tuning.

3. Document-specific image augmentations. Two augmentation transforms are proposed:

• Dilation transform: binarizes the image and applies morphological dilation with a $2 \times 2$ kernel to thicken text strokes, making text regions more salient to an object detector trained on natural images.
• Smudge transform: applies Euclidean, linear, and max distance transforms to spread black pixel regions, producing a blurred, smeary representation. This is adapted from Gilani et al. (2017).

Adding both transforms to the training set (tripling its size) raises the baseline Faster R-CNN model’s weighted-average F1 on ICDAR 2019 from 0.758 to 0.835.

What experiments were performed?

Augmentation ablation (Table 1). A Faster R-CNN ResNeXt-101 baseline is trained on four variants of the general dataset: original only, original + dilation, original + smudge, and all three. Results are reported as F1 at IoU thresholds 0.6, 0.7, 0.8, and 0.9, plus a weighted average (WAvg.) that up-weights higher IoU thresholds. The combined augmentation set achieves WAvg. 0.835 vs. 0.758 for the original-only baseline.

Model comparison (Table 2). Seven model configurations are evaluated on the ICDAR 2019 Track A Modern test set, all trained on the general dataset with both augmentations: RetinaNet (ResNeXt-101), Faster R-CNN HRNet, Cascade R-CNN (ResNeXt-101 and HRNet), Cascade Mask R-CNN (ResNet-50 with deformable convolutions, ResNeXt-101, and HRNet). CascadeTabNet (CMRcnnHr) achieves WAvg. 0.918, outperforming the next best (CMRcnnX at 0.905).

Table detection benchmarks (Tables 3 and 4). After fine-tuning on ICDAR 2019 Track A training data, the model achieves 3rd place on the post-competition leaderboard (WAvg. 0.901), with the best reported F1 at IoU 0.9 (0.901). On TableBank, training on only 1,500 images per document type (Word, Latex, or both), the model achieves F1 of 94.33 (combined), 96.60 (Latex), and 94.92 (Word), exceeding the TableBank baseline ResNeXt-101 and ResNeXt-152 models trained on the full dataset.

ICDAR 2013 table detection (Table 5). Fine-tuning the general model on 40 ICDAR 2013 images (using 198 for testing, a harder evaluation than prior work that reserved most images for training) yields perfect precision, recall, and F1. The authors note their test set is larger and harder than those used by DeepDeSRT and TableNet.

ICDAR 2019 Track B2 structure recognition (Table 6). Evaluated on 100 test images using cell adjacency relations, CascadeTabNet achieves the highest post-competition WAvg. F1 of 0.232 (vs. 0.206 for NLPR-PAL). The absolute scores are low, reflecting the difficulty of precise cell localization at high IoU thresholds.

All experiments were run on Google Colaboratory with a P100 PCIE GPU (16 GB VRAM), Intel Xeon CPU at 2.30 GHz, and 12.72 GB RAM.

What are the outcomes/conclusions?

The results suggest that existing instance segmentation architectures originally designed for natural images can be adapted to document table understanding with relatively modest amounts of domain-specific training data, provided the training strategy (iterative transfer learning) and augmentations are chosen to close the gap between natural and document image statistics.

For bordered tables, the pipeline falls back to conventional line detection for cell extraction rather than relying on model predictions, which the authors describe as more efficient. The model is applied only to borderless cell segmentation, where line cues are insufficient.

Several limitations are worth noting:

• The ICDAR 2013 perfect-score result is difficult to interpret fairly because the test/train split differs from prior comparisons (the authors test on more images while fine-tuning on fewer).
• Structure recognition F1 scores remain low in absolute terms (WAvg. 0.232), and the authors explicitly note that “high-end post-processing can improve the results significantly,” suggesting the model alone is not sufficient for production-quality structure recognition.
• The annotated dataset for structure recognition contains only 342 images, which limits diversity and may explain the model’s failures on some document types (Figure 5d).
• No statistical significance testing or confidence intervals are reported.
• The comparison is framed as “no post-processing” vs. competitors that use post-processing, but the bordered-table branch of the pipeline does use conventional line detection and contour-based algorithms, so the system is not fully end-to-end in the strictest sense.

Reproducibility

Models

• Architecture: Cascade Mask R-CNN with HRNetV2p-W32 backbone, three Bbox stages plus one Mask Head at the final stage. Backbone width W32 indicates 32 channels in the high-resolution convolution stream. All models in the comparison use an FPN neck, as stated in the paper’s model comparison section.
• Parameter count: Not reported in the paper.
• Implementation: MMdetection toolbox, default configuration cascade_mask_rcnn_hrnetv2p_w32_20e (20 epochs by default, labeled “20e” in the config name).
• Pretrained initialization: ImageNet-pretrained HRNet weights (via COCO pretrained model from MMdetection). The paper refers to “imagenet coco model weights” for the first transfer learning stage.
• Weights release: The authors release code and annotations on GitHub (https://github.com/DevashishPrasad/CascadeTabNet) under the MIT license. The paper does not state whether trained model weight checkpoints are included or specify their license separately.

Algorithms

• Transfer learning stage 1: Fine-tune the COCO-pretrained model on the general dataset (1,934 images, single table class). All layers unfrozen.
• Transfer learning stage 2: Fine-tune the stage-1 model on the specific dataset (342 images, three classes: bordered table, borderless table, borderless cell). All layers unfrozen.
• Optimizer and schedule: Not explicitly stated in the paper. The paper uses MMdetection default configurations; the 20e schedule in MMdetection convention uses SGD with momentum and step-based learning rate decay, but these details are not confirmed in the text.
• Batch size: Not reported in the paper.
• Augmentation: Dilation transform (binary + $2 \times 2$ morphological dilation) and smudge transform (distance transform-based blurring) applied offline; augmented images added to the training set alongside originals, tripling its size.
• No test-time augmentation or ensembling is mentioned.

Data

• General dataset: 1,934 images, 2,835 table annotations. Merged from ICDAR 2019 cTDaR Modern (cTDaR), Marmot (Chinese and English subsets), and a GitHub borderless table dataset. Ground-truth errors in the Marmot dataset were corrected.
• Specific dataset: 342 manually annotated images selected from ICDAR 2019 Train set. Three-class annotations: 114 bordered tables, 429 borderless tables, 24,920 borderless cell masks. Released publicly via the GitHub repository.
• TableBank fine-tuning: 1,500 images randomly sampled per document type from the full TableBank dataset. Annotation quality issues in the Word subset led to exclusion of some images from the test set.
• ICDAR 2013 fine-tuning: 40 randomly selected images used for fine-tuning; 198 used for testing.
• Availability: ICDAR 2019, ICDAR 2013, Marmot, and TableBank are publicly available datasets; individual licenses are not stated in the paper and should be verified at their respective sources. The custom 342-image annotation set is available through the GitHub repository under the repository’s MIT license.

Evaluation

• ICDAR 2019 (Track A): Precision, Recall, F1 at IoU thresholds 0.6, 0.7, 0.8, 0.9. The WAvg. metric is defined by the cTDaR competition (Gao et al. 2019) and weights the four F1 scores more heavily at higher IoU thresholds; the exact weights (0.1, 0.2, 0.3, 0.4) are specified in the competition description, not directly in this paper.
• TableBank: Standard precision, recall, and F1 computed by summing overlap areas across all documents, following the method of Gilani et al. (2017) as described in the TableBank paper.
• ICDAR 2013: Cell-level completeness/purity metrics; precision and recall per table, then averaged. The paper follows the same protocol as TableNet (Paliwal et al. 2019).
• ICDAR 2019 (Track B2): Cell adjacency relation-based evaluation (Gobel et al. 2012); F1 at IoU 0.6, 0.7, 0.8, 0.9 and WAvg.
• No error bars, seeds, or significance tests are reported.
• Comparison fairness concern: The ICDAR 2013 evaluation uses a larger test split than prior published results, making direct comparison with DeepDeSRT and TableNet potentially misleading in either direction.

Hardware

• Training hardware: Google Colaboratory, P100 PCIE GPU (16 GB VRAM), Intel Xeon CPU at 2.30 GHz, 12.72 GB RAM.
• Training time: Not reported.
• Inference latency and throughput: Not reported.
• Inference hardware: Not separately characterized; all reported experiments used the same Colab environment.
• Deployment: The pipeline relies on MMdetection and standard image processing libraries; local deployment should be feasible on a single GPU with at least 16 GB VRAM based on the reported training environment.

BibTeX

@inproceedings{prasad2020cascadetabnet,
  title={CascadeTabNet: An approach for end to end table detection and structure recognition from image-based documents},
  author={Prasad, Devashish and Gadpal, Ayan and Kapadni, Kshitij and Visave, Manish and Sultanpure, Kavita},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops},
  pages={572--573},
  year={2020}
}

TL;DR

LGPMA proposes a Mask-RCNN-based framework that simultaneously learns local (RoI-level) and global (full-image) pyramid soft-mask predictions to produce accurately aligned cell bounding boxes for table structure recognition. A mask re-scoring module fuses the two prediction levels, and a three-step post-processing pipeline uses global segmentation cues to locate and merge empty cells. The method achieves 96.7 TEDS-Struct on the PubTabNet validation set and 98.8 F1 on SciTSR.

What kind of paper is this?

Dominant: $\Psi_{\text{Method}}$: The paper introduces LGPMA, an architecture with a specific training recipe combining dual pyramid mask supervision across local and global feature maps, a multi-task loss, and a structured inference pipeline. The center of gravity is the model design, demonstrated through ablation studies that attribute gains to each module.

Secondary: none significant.

What is the motivation?

Table structure recognition requires recovering the full cell grid from a cropped table image, including logical row/column indices and spanning relations. Two broad families of methods face complementary weaknesses:

• Global-object-based methods predict row/column separators or segmentation masks, but struggle with cells that span multiple rows or columns and with multi-line text that confuses line-cutting decisions.
• Local-object-based methods detect text regions as nodes and recover cell relations via heuristic rules or GNNs. Because these methods anchor predictions to visible text, they cannot reliably identify empty cells, which have no text and are visually ambiguous with neighboring cross-span cells. Empty cell handling directly affects downstream editable document conversion.

The authors observe that if one could obtain aligned bounding boxes (boxes expanded from the text region to the full cell extent, including empty padding) for every cell, the table grid becomes recoverable by simple coordinate overlap. The challenge is that aligned bounding boxes are difficult to regress because cell boundaries typically have no visible texture to guide the detector.

What is the novelty?

The core idea is soft pyramid mask supervision applied at two spatial granularities simultaneously, combined with a re-scoring procedure that fuses local and global predictions.

Pyramid mask label. For a cell proposal of height $H$ and width $W$, the text region corners are $(x_1, y_1)$ (top-left) and $(x_2, y_2)$ (bottom-right). Each pixel $(h, w)$ in the proposal is assigned a soft label in $[0, 1]$ that is maximal at the text midpoint and decreases linearly toward the cell boundary:

$$ t_h^{(w,h)} = \begin{cases} w/x_{mid} & w \le x_{mid} \\ \frac{W-w}{W-x_{mid}} & w > x_{mid} \end{cases}, \quad t_v^{(w,h)} = \begin{cases} h/y_{mid} & h \le y_{mid} \\ \frac{H-h}{H-y_{mid}} & h > y_{mid} \end{cases} $$

where $x_{mid} = (x_1 + x_2)/2$ and $y_{mid} = (y_1 + y_2)/2$. Because every pixel in the proposal participates in predicting the boundary, the network can push the predicted boundary beyond the initial proposal rectangle.

Local Pyramid Mask Alignment (LPMA). Within each RoI-aligned feature map, the mask head learns two tasks jointly: a binary text region segmentation and horizontal/vertical pyramid label regression. Local features provide reliable texture cues near the text content but have a limited receptive field.

Global Pyramid Mask Alignment (GPMA). A parallel branch operates on the full feature map and learns two tasks: (1) a global binary segmentation covering all cells including empty ones (empty cell ground-truth boundaries are derived from the maximum height/width of non-empty cells in the same row/column), and (2) global pyramid label regression for non-empty cells. The global feature captures long-range spatial context but lacks fine-grained texture precision.

Pyramid mask re-scoring. At inference, for each proposed aligned bounding box with text region midpoint $(x_{mid}, y_{mid})$ and overlap with the global segmentation map $P_o$, the local predictions $F^{(L)}$ and global predictions $F^{(G)}$ are combined with distance-weighted coefficients:

$$ F(x) = \begin{cases} \frac{x-x_1}{x_{mid}-x_1} F_{hor}^{(L)} + \frac{x_{mid}-x}{x_{mid}-x_1} F_{hor}^{(G)} & x_1 \le x \le x_{mid} \\ \frac{x-x_2}{x_{mid}-x_2} F_{hor}^{(L)} + \frac{x_{mid}-x}{x_{mid}-x_2} F_{hor}^{(G)} & x_{mid} < x \le x_2 \end{cases} $$

The fused pyramid map for each side of the box is used to fit a plane by least squares:

$$ \min \sum_{y_i=y_1}^{y_2} \sum_{x_i=x_{mid}}^{x_2} (ax_i + by_i + c - F(x_i, y_i))^2 $$

The zero-crossing of the fitted plane gives the refined boundary coordinate:

$$ x_{refine} = - \frac{1}{y_2 - y_1 + 1} \sum_{y_i=y_1}^{y_2} \frac{b y_i + c}{a} $$

All four boundaries (left, right, top, bottom) are refined analogously.

Table structure recovery pipeline. Three sequential steps follow bounding box refinement: (1) cell matching by coordinate midpoint overlap in horizontal and vertical directions, (2) empty cell searching using the Bron-Kerbosch maximum clique algorithm on the matching graph to identify row/column cliques and find vacant positions, and (3) empty cell merging guided by the fraction of GPMA segmentation pixels predicted as foreground in the region between adjacent empty cells; neighbor cells are merged when the ratio exceeds a threshold.

Multi-task loss. Training optimizes:

$$ \mathcal{L} = \mathcal{L}_{rpn} + \lambda_1(\mathcal{L}_{cls} + \mathcal{L}_{box}) + \lambda_2(\mathcal{L}_{mask} + \mathcal{L}_{LPMA}) + \lambda_3(\mathcal{L}_{seg} + \mathcal{L}_{GPMA}) $$

where $\mathcal{L}_{seg}$ uses the Dice coefficient loss and $\mathcal{L}_{LPMA}$, $\mathcal{L}_{GPMA}$ use pixel-wise L1 loss. The weights $\lambda_1 = \lambda_2 = \lambda_3 = 1$ were set empirically.

What experiments were performed?

Datasets:

• ICDAR 2013: 98 train / 156 test samples from PDF-extracted government reports.
• SciTSR: 12,000 train / 3,000 test scientific literature tables; SciTSR-COMP is a harder subset of 2,885 train / 716 test.
• PubTabNet: 500,777 train / 9,115 val / 9,138 test scientific tables.

Baselines: DeepDeSRT, Split, DeepTabStR, Siddiqui et al., ReS2TIM, GTE, GraphTSR, and TabStruct-Net on ICDAR 2013 and SciTSR; EDD, TabStruct-Net, and GTE on PubTabNet.

Metrics: Adjacency F1 (micro-averaged precision/recall of neighboring cell-pair relations) for ICDAR 2013 and SciTSR; TEDS and TEDS-Struct for PubTabNet. OCR for PubTabNet TEDS uses a separate attention-based recognizer (Lee and Osindero, CVPR 2016).

Ablations (Table 3, 60k train / 1k val from PubTabNet): Incremental addition of LPMA, GPMA, and Alignment Loss (AL from TabStruct-Net), measured on text region detection, aligned bounding box detection (IoU threshold 0.7), and TEDS-Struct. The combination LPMA+GPMA without AL achieves the best aligned bounding box Hmean (84.95) and TEDS-Struct (95.53). Adding AL on top of LPMA+GPMA degrades aligned bounding box Hmean by 0.27 pp, suggesting the two supervision signals conflict.

Ablations (Table 4): Three empty cell merging strategies compared on empty-cell detection F1, all-cell detection F1, and TEDS-Struct. The LGPMA-guided strategy achieves 70.21% empty-cell Hmean vs. 59.40% for minimum-cells and 14.70% for maximum-cells. When non-empty bounding box ground truth is provided (ideal upper bound), LGPMA recovery achieves 97.47% empty-cell Hmean and TEDS-Struct 99.77.

What are the outcomes/conclusions?

Main results on held-out test/validation sets:

The paper received the Best Industry Paper Award at ICDAR 2021.

Key findings:

• Pyramid soft supervision allows aligned bounding boxes to exceed initial proposal bounds, helping particularly for cross-span cells where the proposal often underestimates the true cell extent.
• GPMA’s global receptive field and LPMA’s local texture detail are complementary; the re-scoring fusion outperforms either branch alone.
• The empty cell merging strategy using GPMA visual guidance substantially outperforms naive minimum or maximum strategies on empty-cell detection metrics.
• When non-empty bounding boxes are provided as ground truth, the recovery pipeline achieves near-perfect TEDS-Struct (99.77), indicating the structural recovery logic itself is nearly lossless and that detection accuracy is the primary bottleneck.
• Alignment Loss (AL, from TabStruct-Net) degrades aligned bounding box detection by 3.1 pp Hmean when combined with LGPMA’s pyramid supervision, suggesting the approaches encode redundant or conflicting gradients.

Limitations:

• The method is designed for axis-aligned, print-format tables without rotation or perspective distortion. Wild or photographed tables are out of scope.
• Empty cell merging uses a fixed pixel-ratio threshold on GPMA segmentation output, which may be sensitive to heavily bordered or stylized tables.
• No inference latency or throughput figures are reported.
• OCR is handled by a separately trained recognizer; the full pipeline is not end-to-end.

Reproducibility

Models

• Backbone: ResNet-50 with Feature Pyramid Network (FPN), initialized from MS-COCO pretrained weights.
• LPMA: six anchor ratios [1/20, 1/10, 1/5, 1/2, 1, 2] to handle tall and wide cell shapes; RCNN NMS IoU threshold 0.1 at test time.
• GPMA: full-image segmentation branch with pyramid label regression; aligned bounding box ground-truth shrunk by 5% before generating targets to prevent overlap.
• Code is released in the DAVAR-Lab-OCR repository under Apache-2.0 (the repo-level license applies to the lgpma subdirectory; no subdirectory-specific license file exists).
• No pretrained model weights are publicly released by the authors. Reproduction requires training from scratch using the released code and configuration files.

Algorithms

• Optimizer: SGD, momentum 0.9, weight decay $1 \times 10^{-4}$.
• Batch size: 4.
• Learning rate: $1 \times 10^{-2}$, divided by 10 every 5 epochs.
• Training duration: 12 epochs for SciTSR and PubTabNet; 25 epochs for ICDAR 2013 fine-tuning.
• Input scale augmentation: random scale of longer side to [480, 1080] during training; fixed 768 at test time.
• Loss weights: $\lambda_1 = \lambda_2 = \lambda_3 = 1$ (empirical; no ablation over these values reported).
• Plane fitting for boundary refinement uses least squares; iterative refinement is optional following the Pyramid Mask Text Detector procedure.
• Empty cell merging threshold (the pixel-ratio cutoff for deciding whether to merge two adjacent empty cells via GPMA segmentation output) is not reported in the paper and is not exposed in the released configuration files based on available documentation.

Data

• Primary training for SciTSR and ICDAR 2013 experiments: SciTSR training split (12,000 images).
• ICDAR 2013 fine-tuning: 98-image training split, after SciTSR pretraining.
• PubTabNet experiments: full 500,777-image training split.
• Ablation experiments: 60,000 randomly sampled PubTabNet training images.
• ICDAR 2013: publicly available; hosted via ICDAR competition infrastructure; no explicit open license but widely used for academic research.
• SciTSR: publicly available on GitHub (https://github.com/Academic-Integrity-ML/SciTSR); no explicit open-source license stated by the authors.
• PubTabNet: publicly available on IBM’s GitHub (https://github.com/ibm-aur-nlp/PubTabNet) under CDLA-Permissive-1.0.

Evaluation

• Adjacency F1: micro-averaged correctness of neighboring cell-pair relations; standard for ICDAR 2013 and SciTSR benchmarks.
• TEDS / TEDS-Struct: tree edit distance similarity on full HTML (with and without cell content); introduced with PubTabNet.
• No error bars, confidence intervals, or multi-run seeds reported.
• Training data conditions vary across baselines (some use private pretraining data); LGPMA uses only public datasets.
• Full TEDS reproduction requires an OCR component: the authors use an attention-based recognizer (Lee and Osindero, CVPR 2016) trained separately. No checkpoint or training recipe for this recognizer is provided by the LGPMA authors; reproducing the TEDS (full) score requires sourcing or training a compatible OCR model independently. TEDS-Struct (structure only) is reproducible without OCR.

Hardware

• Training: 8 $\times$ Tesla V100 (32 GB each), PyTorch.
• No GPU-hours, per-sample latency, or throughput figures reported in the paper.
• The ResNet-50 + FPN + Mask-RCNN architecture is standard and deployable on a single modern GPU for inference.

BibTeX

@inproceedings{qiao2021lgpma,
  title     = {LGPMA: Complicated Table Structure Recognition with Local and Global Pyramid Mask Alignment},
  author    = {Qiao, Liang and Li, Zaisheng and Cheng, Zhanzhan and Zhang, Peng and Pu, Shiliang and Niu, Yi and Ren, Wenqi and Tan, Wenming and Wu, Fei},
  booktitle = {Document Analysis and Recognition -- ICDAR 2021},
  pages     = {99--114},
  year      = {2021},
  publisher = {Springer International Publishing},
  series    = {Lecture Notes in Computer Science},
  volume    = {12821}
}

TL;DR

NCGM introduces a graph-based architecture for table structure recognition that treats three modalities (geometry, appearance, and content) as separate graphs, then alternates between per-modality context extraction and cross-modality information synthesis in stacked collaborative blocks. The approach handles distorted tables considerably better than earlier graph methods that rely on early or late fusion.

What kind of paper is this?

Dominant: $\Psi_{\text{Method}}$ (new architecture). The headline contribution is a stack of collaborative blocks composed of Ego Context Extractors (ECE) and Cross Context Synthesizers (CCS). The paper is organized around ablations, comparisons on multiple benchmarks, and an analysis of learned attention diversity.

What is the motivation?

Table structure recognition (TSR) aims to recover the logical or physical grid structure of a table from an image, producing row/column/cell adjacency relationships. Two families of prior approaches fall short on tables with complex spanning cells or geometric distortion:

1. Early fusion concatenates multiple modality features before feeding them to a single graph model. This forces the model to learn a single inductive bias and ignores the fact that different modalities may be more or less informative depending on table type.
2. Late fusion models each modality in a separate graph and combines the outputs. This disentanglement sacrifices inter-modality interactions entirely.

The authors frame the unaddressed challenge as the “Heterogeneous TSR (Hetero-TSR)” problem: how should different modalities collaborate with each other in a way that adapts to the table at hand?

What is the novelty?

The paper introduces Neural Collaborative Graph Machines (NCGM), built from a stack of $L$ collaborative blocks. Each block has two successive modules.

Ego Context Extractor (ECE): For each modality $\sim \in \{G, A, C\}$, a fully-connected directed graph is built over the $N$ text segment bounding boxes. The edge feature for a pair $(i, j)$ is the asymmetric function:

$$h_e(x_i, x_j) = x_i ,|, (x_i - x_j)$$

ECE aggregates global context via a Compressed Multi-head Attention (CMHA) module. Standard MHA over all $N(N-1)/2$ edge features would be $O(NM)$ in complexity. CMHA reduces this by compressing keys and values through a reshape-and-project operation:

$$MC(H) = \text{Norm}(\text{Reshape}(H, \varepsilon), W^h)$$

with compression ratio $\varepsilon = N/M$, cutting complexity to $O(N^2)$. ECE outputs updated per-modality context embeddings $C^{(l)} \in \{C^G, C^A, C^C\}$.

Cross Context Synthesizer (CCS): Three parallel CMHA modules each take one modality as queries and the other two as keys/values, producing inter-modality embeddings $M^{(l)}$. For example, for the content branch:

$$M_C^{(l)} = \text{CMHA}(M_C^{(l-1)},; C_A^{(l)} \cup C_G^{(l)})$$

After $L=3$ blocks the three inter-modality streams are concatenated into collaborative graph embeddings $E \in \mathbb{R}^{N \times d_e}$. All $N^2$ pair vectors $U = \{u_{i,j}\}$ are formed by channel-wise concatenation of $e_i$ and $e_j$ and fed to three separate FC classifiers predicting binary same-row, same-column, and same-cell relationships.

Training loss: End-to-end with a combination of classification and contrastive terms for each relationship type:

$$L = L_{cell} + L_{col} + L_{row}$$

$$\tilde{L} = \lambda_1 L_{\text{class}} + \lambda_2 L_{\text{con}}$$

$$L_{con} = |e_{(a)} - e_{(b)}^+|_2^2 + \max\{0,, \alpha - |e_{(a)} - e_{(b)}^-|_2^2\}$$

During training, Monte Carlo sampling (sample size $S=10$) is used to keep memory costs tractable. At inference, all pairs are evaluated.

Augmented evaluation set: The authors also release a synthesizing procedure (SciTSR-COMP-A) that applies perspective transformation and quadratic Bézier curve distortion to SciTSR-COMP images, generating a harder variant for benchmarking under realistic capture conditions.

What experiments were performed?

Datasets:

• Physical structure (precision/recall/F1 on row/column adjacency): ICDAR-2013-P, ICDAR-2019, UNLV-P, WTW, SciTSR, SciTSR-COMP, and the new SciTSR-COMP-A.
• Logical structure (BLEU on TableBank; TEDS on PubTabNet): TableBank (145k train) and PubTabNet (339k train).

Evaluation setups follow TabStruct-Net:

• Setup-A: table image only (no bounding boxes or text content as input; FLAG-Net detection boxes and Tesseract OCR results are used for fair comparison).
• Setup-B: table image plus ground-truth (or OCR-derived) text segment bounding boxes and content.

Baselines: GraphTSR, DGCNN, TabStruct-Net, FLAG-Net, GTE, LGPMA, Cycle-CenterNet.

Ablations isolate: (a) early vs. late vs. collaborative fusion, (b) DGCNN vs. Transformer vs. ECE as the intra-modality aggregator, (c) concatenation vs. CCS as the inter-modality fusion strategy, (d) individual modality contributions (G, A, C alone and in pairs), (e) effect of block depth (1 to 9 blocks).

What are the outcomes/conclusions?

On SciTSR-COMP (complex structures), NCGM under Setup-B achieves F1 99.0% (P 98.8%, R 99.3%), outperforming FLAG-Net. The strongest result is on the distorted set SciTSR-COMP-A: without distorted training data, NCGM exceeds the second-best FLAG-Net by approximately 11 percentage points (F1) under Setup-A and roughly 12 points under Setup-B. When trained with distorted data, the margin remains around 7 to 9 points.

Ablation insights:

• Geometry (G) is the dominant modality for physical structure; appearance (A) contributes meaningfully; content (C) alone is weak (F1 50.2% on SciTSR-COMP).
• ECE with individual modality inputs (99.0 F1) beats all early-fusion and late-fusion alternatives on SciTSR-COMP.
• CCS raises F1 from 98.3% (ECE + late concatenation) to 99.0%.
• Three collaborative blocks is the practical optimum; deeper networks converge slower and become prone to training collapse beyond 7 blocks.

Limitations the authors acknowledge:

1. Computational cost grows with the number of modalities and decoupled processing (3.1M parameters, 12.7G FLOPs for 42 bounding boxes, which is heavier than FLAG-Net at 1.9M / 3.3G FLOPs).
2. Deeper collaborative blocks risk training collapse (observed for more than 7 blocks beyond 50 epochs).
3. Nested tables, where row/column boundaries are ambiguous, remain a failure mode.

The paper does not discuss inference latency beyond FLOPs counts, nor does it report results on FinTabNet or the PubTables-1M benchmark family that has since become standard.

Reproducibility

Models

• Backbone: ResNet-18 (conv1 to conv2_2) plus three 3$\times$3$\times$64 conv layers for appearance features; RoI Align for per-box pooling.
• Geometry: 4-dim normalized bounding box coordinates projected by a $d$-dim FC layer.
• Content: word2vec embeddings followed by a 7$\times$1$\times$d conv for sequential modeling.
• Hidden size $d=64$; 8 attention heads; $d_k = d_v = 8$; $d_m = 64$.
• 3 collaborative blocks (L=3); FC classifiers have 3 layers at 256 dimensions plus a 2-dim softmax head.
• Total: 3.1M parameters. No pretrained weights or code released.

Algorithms

• Framework: PyTorch.
• Input table images resized to $512 \times 512$.
• Pre-trained on SciTSR for 10 epochs; fine-tuned on each target benchmark for 50 epochs.
• Optimizer: not explicitly named; learning rate initialized at $1 \times 10^{-4}$, divided by 10 on loss plateau.
• Loss weights: $\lambda_1 = \lambda_2 = 1$; margin $\alpha = 1$ in contrastive loss.
• Monte Carlo sampling size $S=10$ during training; full pairing at inference.
• No data augmentation beyond the optional SciTSR-COMP-A distortion procedure (perspective transform + Bézier curve warp).

Data

• Training: SciTSR (12k), WTW (10.97k), ICDAR-2019 (600), PubTabNet (339k), TableBank (145k), and partial splits for ICDAR-2013 and UNLV (80/20 random 5-fold).
• Bounding box alignment: cell-level datasets (ICDAR-2019, UNLV, WTW) are converted to text-segment level via Tesseract OCR; text-segment-level datasets (ICDAR-2013, SciTSR) use parsed GT boxes directly.
• SciTSR-COMP-A synthesis procedure is described in the appendix but the augmented images are not separately distributed.

Evaluation

• Physical structure: precision, recall, and F1 on pairwise adjacency (same-row, same-column, same-cell).
• Logical structure: BLEU (TableBank, following Li et al.) and TEDS (PubTabNet, following Zheng et al.).
• ICDAR-2013 and UNLV: averaged over 10 random 5-fold splits.
• No error bars or significance tests reported for main results. No seed sensitivity study.
• Baselines use different input formats in some cases (flag-net detection vs. GT boxes), creating potential inconsistencies at the dataset level.

Hardware

• Training: one Nvidia Tesla V100 GPU with 32 GB memory.
• No wall-clock training time reported.
• FLOPs: 12.7G for a table with 42 text segment boxes; parameters: 3.1M.
• No inference latency or throughput numbers reported.

BibTeX

@inproceedings{liu2022neural,
  title={Neural Collaborative Graph Machines for Table Structure Recognition},
  author={Hao Liu and Xin Li and Bing Liu and Deqiang Jiang and Yinsong Liu and Bo Ren},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year={2022}
}

TL;DR

RobusTabNet is a two-stage table extraction system that replaces the standard region proposal network (RPN) in Faster R-CNN with CornerNet for higher-localization-accuracy table detection, and pairs a spatial CNN separator prediction module with a Grid CNN cell merger for table structure recognition. The approach is designed to handle geometrically distorted and even curved tables that confound axis-aligned methods.

What kind of paper is this?

Dominant: $\Psi_{\text{Method}}$: The paper’s center of gravity is the design and evaluation of two architectural components: a CornerNet-as-RPN table detector and a spatial CNN + Grid CNN structure recognizer. Extensive ablations and comparisons against published baselines on six public benchmarks support every claim.

Secondary: none significant; the authors release manually annotated structure labels for cTDaR TrackA modern, but this is a minor byproduct rather than the paper’s purpose.

What is the motivation?

Two gaps motivate this work.

For table detection, existing CNN detectors (CDeC-Net, Cascade Mask R-CNN variants) achieve good recall but poor localization at high IoU thresholds. The underlying cause is that standard RPN generates proposals whose IoU with ground-truth boxes falls mostly in the 0.7–0.9 range; proposals with IoU > 0.9 represent only 48.1% of positive samples. Because these lower-quality positives survive NMS with high scores, the final detector is penalized whenever evaluation uses tight IoU thresholds.

For table structure recognition (TSR), the dominant cell-detection-then-clustering paradigm (TabStruct-Net, LGPMA) assumes tables are axis-aligned. Camera-captured documents in settings such as the “Insert data from picture” feature in Excel routinely contain skewed or curved tables that violate this assumption. Separately, row/column segmentation models built on plain ResNet-FPN cannot propagate enough context to disambiguate large blank regions in borderless tables.

What is the novelty?

CornerNet-FRCN table detector. The authors replace RPN with CornerNet to generate table proposals. CornerNet predicts paired top-left and bottom-right corner heatmaps, with offsets to compensate for quantization error from the stride-16 backbone:

$$ p_i^x = \left\lfloor \frac{q_j^x}{s} \right\rfloor, \quad p_i^y = \left\lfloor \frac{q_j^y}{s} \right\rfloor $$

$$ \Delta_i = \left( \left\lfloor \frac{q_i^x}{s} \right\rfloor - \frac{q_i^x}{s},\ \left\lfloor \frac{q_i^y}{s} \right\rfloor - \frac{q_i^y}{s} \right) $$

All valid top-left/bottom-right corner pairs are enumerated as proposals (x-left $<$ x-right and y-top $<$ y-bottom) and then passed through a standard Fast R-CNN head for classification and box refinement. The authors report that this design raises the fraction of well-localized proposals (IoU $>$ 0.9 within IoU $>$ 0.7 positives) from 48.1% (RPN) to 96.3% (CornerNet). The combined training loss is:

$$ L_{\text{detector}} = \lambda_{\text{corner}} \cdot L_{\text{corner}} + L_{\text{frcn}}, \quad \lambda_{\text{corner}} = 0.2 $$

where $L_{\text{corner}}$ uses focal loss for heatmap classification and Smooth-$L_1$ for offset regression, and $L_{\text{frcn}}$ uses cross-entropy plus $L_1$ box regression.

Spatial CNN separation line prediction. A plain ResNet-18 + FPN feature map struggles with borderless tables because each pixel’s receptive field is local. The spatial CNN module (adapted from lane detection) propagates information sequentially across the feature map. For the row separator branch, slices $\{s_i^w\}$ along the width dimension are updated left-to-right and then right-to-left by convolving with a $9 \times 1$ kernel and adding to the adjacent slice. The column branch performs analogous top-to-bottom and bottom-to-top passes. The resulting context-enriched feature map drives pixel-level binary segmentation:

$$ L_{\text{split}} = \frac{1}{N_{\text{row}}} \sum_i L(R_i, R_i^\ast) + \frac{1}{N_{\text{col}}} \sum_j L(C_j, C_j^\ast) $$

Grid CNN cell merging. After cell generation via connected-component analysis and polynomial curve fitting, the detected cells are arranged in an $M \times N$ grid. RoI-aligned features per cell form a grid feature map $F_{\text{grid}} \in \mathbb{R}^{M \times N \times 512}$. Three stacked $3 \times 3$ convolutions aggregate spatial context across this grid. A relation network then predicts whether each pair of 4-adjacent cells should be merged, using an 18-dimensional spatial compatibility vector $l_{ij}$ encoding the box deltas:

$$ \begin{aligned} t_x^{ij} &= (x^i - x^j) / w^j, \quad & t_y^{ij} &= (y^i - y^j) / h^j, \\ t_w^{ij} &= \log(w^i / w^j), \quad & t_h^{ij} &= \log(h^i / h^j), \\ t_x^{ji} &= (x^j - x^i) / w^i, \quad & t_y^{ji} &= (y^j - y^i) / h^i. \end{aligned} $$

$$ L_{\text{merge}} = \frac{1}{N_p} \sum_i L(r_i, r_i^\ast) $$

The total structure recognizer loss is $L_{\text{recognizer}} = L_{\text{split}} + L_{\text{merge}}$.

What experiments were performed?

Datasets (TD): cTDaR 2019 TrackA (600/240 modern, plus 600/199 historical), PubLayNet (335k/11k/11k), IIIT-AR-13K (9.3k/2k/2.1k). Datasets (TSR): SciTSR (12k/3k), PubTabNet (500k/9k/9k), cTDaR TrackB2-Modern (100 test images). The authors also evaluate TSR on a private in-house dataset of 9,000/700 camera-captured, skewed, or curved table images.

Metrics: Weighted Average F1 (IoU $\in \{0.6, 0.7, 0.8, 0.9\}$) for cTDaR; COCO AP$^{0.5:0.95}$ for PubLayNet; PASCAL VOC AP for IIIT-AR-13K; adjacency relation F1 for SciTSR and cTDaR TrackB2; TEDS-Struct for PubTabNet.

Baselines: CDeC-Net (strongest published TD baseline), LGPMA (ICDAR 2021 TSR winner), TabStruct-Net, GTE, EDD, SPLERGE (for distorted table comparison).

Ablations: The paper systematically compares five message-passing methods for separation line prediction (no message passing, projection networks, Bi-GRU, CC Attention, Spatial CNN) and four cell merging methods (no merging, Relation Network, GCN, Grid CNN) on cTDaR TrackB2-Modern.

What are the outcomes/conclusions?

Table detection: RobusTabNet (ResNet-18 backbone) achieves WAvg. F1 of 94.9% on cTDaR TrackA (vs. 94.3% for CDeC-Net with a much heavier dual-ResNeXt-101 backbone), AP$^{0.5:0.95}$ of 97.0% on PubLayNet (vs. 96.7% for CDeC-Net), and test AP of 97.7% on IIIT-AR-13K (vs. 96.5% for Mask R-CNN with ResNet-101). The authors attribute these gains to proposal quality: CornerNet raises top-50 recall at IoU $\geq$ 0.9 from 89.3% (RPN) to 97.8%.

Table structure recognition: RobusTabNet achieves F1 of 99.3% / 98.7% on SciTSR / SciTSR-COMP (vs. 98.8% / 98.0% for LGPMA), TEDS-Struct of 97.0% on PubTabNet (vs. 96.7% for LGPMA), and the highest reported WAvg. F1 among evaluated methods on cTDaR TrackB2-Modern. On the private distorted-table dataset, RobusTabNet scores 94.6% WAvg. F1 against SPLERGE’s 63.8%.

Ablation findings: Spatial CNN outperforms CC Attention (93.9%), Bi-GRU (93.1%), and projection networks (93.0%) for separation line prediction (94.6% vs. all alternatives). Grid CNN outperforms GCN (94.0%) and Relation Network (93.2%) for cell merging (94.6%).

Limitations: The detector struggles with closely adjacent tables (difficulty disambiguating boundaries). The structure recognizer degrades on cells with multi-line content and on extremely dense tables where separator masks from adjacent rows or columns overlap. All public benchmarks consist of black-line/white-background tables; generalization to colored or stylized documents is untested.

Reproducibility

Models

• TD: ResNet-18 with dilations in Conv5 (stride 16), reduced to 64 channels with a $1 \times 1$ conv. CornerNet corner pooling + 3$\times$3 conv appended to produce Dilated-C5’. Fast R-CNN head with two 1,024-d fc layers. Total parameter count not reported.
• TSR: ResNet-18 + FPN (64 output channels) shared backbone. Spatial CNN branches with $\frac{H}{4} \times \frac{W}{32}$ intermediate resolution ($3 \times$ downsampling), propagating $9 \times 1$ kernels. Grid CNN: RoI Align $7 \times 7 \times 64$, two 512-d fc layers per cell, $3 \times$ stacked $3 \times 3$ convolutions, 2-hidden-layer MLP relation head.
• No pretrained weights or code released. Weights initialized from ImageNet-pretrained ResNet-18; new layers initialized from $\mathcal{N}(0, 0.01)$.

Algorithms

• Optimizer: SGD with momentum 0.9, weight decay $5 \times 10^{-4}$.
• Schedule: 15K iterations; base LR 0.032, decayed by $\times 0.1$ at 10K and 13K iterations. PubLayNet uses $3\times$ schedule. SciTSR and PubTabNet TSR models trained for 12 epochs.
• Batch size: 4 images per GPU across 8 GPUs; synchronized batch normalization.
• Multi-scale training (TD): shorter side randomly chosen from $\{320, 416, 512, 608, 704, 800\}$; cTDaR also uses random rotations in $\{0^\circ, 90^\circ, 180^\circ, 270^\circ\}$ with $\pm 5^\circ$ jitter. Multi-scale training (TSR): shorter side randomly chosen from $\{416, 512, 608, 704, 800\}$ using cropped table images.
• OHEM: Equal hard positive/negative sampling (32/32 proposals for FRCN; 1,024/1,024 separator/background pixels per branch for TSR split; 64/64 cell pairs for merge).
• Loss weights: $\lambda_{\text{corner}} = 0.2$ (tuned on in-house dataset; applied without per-dataset tuning).

Data

• Public training sets: cTDaR TrackA (600 modern images annotated by the authors for TSR structure), PubLayNet (86k table-containing images used for TD), IIIT-AR-13K, SciTSR, PubTabNet.
• Private in-house dataset: 9,000 training / 700 test camera-captured images with distorted/curved tables; not released.
• The authors annotated row/column separation lines and cell bounding boxes for the cTDaR TrackA 600 modern images for TSR training; they state these annotations “will be released publicly,” though no repository link was provided in either the preprint or the published journal version. As of this writing, no public release has been identified.

Evaluation

• Metrics: Standard per-dataset protocols. TEDS-Struct ignores OCR and evaluates structural HTML trees only. Adjacency F1 uses the official cTDaR measurement tool (https://github.com/cndplab-founder/ctdar_measurement_tool).
• Test-time scaling: TD images rescaled so the shorter side is 512 px (longer side capped at 1,024 px). TSR: cropped table images rescaled so the longer side is 1,024 px; SciTSR images are not resized.
• Comparisons: Single-model, single-scale inference throughout.
• Statistical rigor: No error bars, significance tests, or multi-run variance reported.
• Known limitations: LGPMA leverages the axis-aligned table constraint to boost PubTabNet performance; RobusTabNet does not use this constraint, making the comparison favorable to RobusTabNet on distorted images.
• Hyperparameter transfer: All thresholds ($C_{th}=0.3$, $S_{th}=0.8$, merge score threshold $0.8$, top-$K$=100) were tuned on the private in-house dataset and applied without modification to public benchmarks.

Hardware

• 8 Nvidia V100 GPUs.
• PyTorch 1.6.0.
• Training GPU-hours not reported.
• Inference: no latency figures reported; the paper does not quantify FPS or memory footprint at test time.

BibTeX

@article{ma2023robustabnet,
  title={Robust Table Detection and Structure Recognition from Heterogeneous Document Images},
  author={Ma, Chixiang and Lin, Weihong and Sun, Lei and Huo, Qiang},
  journal={Pattern Recognition},
  volume={133},
  pages={109006},
  year={2023},
  publisher={Elsevier},
  doi={10.1016/j.patcog.2022.109006}
}

TL;DR

GridFormer frames table structure recognition as predicting the vertices and edges of an $M \times N$ grid. A Deformable DETR backbone with parallel row and column decoders predicts this grid in a single forward pass, covering wired, wireless, oriented, and distorted tables. The method matches or improves on prior results across five benchmarks without requiring OCR annotations.

What kind of paper is this?

Dominant: $\Psi_{\text{Method}}$. The paper’s headline contribution is a novel architecture for TSR: a new table representation (the $M \times N$ vertex-edge grid) paired with a DETR-style two-stream decoder. The bulk of the paper describes the model design, loss functions, and ablation studies validating each module choice.

Secondary: none. The paper uses existing public benchmarks and does not release a dataset, metric, or code repository.

What is the motivation?

Table structure recognition (TSR) must handle tables that vary widely in structure, line style, and image quality. Prior methods each handle this diversity poorly in at least one dimension:

• Split-and-merge methods (region-based) use a two-stage pipeline: a segmentation model divides the table into an over-split grid of regions, then a merge model combines spanning cells. The two-stage design complicates end-to-end training and degrades on wireless or geometrically distorted tables.
• Graph-based methods model cells or text lines as graph nodes and predict pairwise relationships. They depend on OCR annotations at inference time, creating a hard coupling to text detection quality.
• Cell-based methods detect individual cells via object detection. They work well on wired tables but struggle with wireless tables where cell boundaries are implicit.
• Markup language-based methods generate HTML sequences autoregressively. They suffer from attention drift on long sequences and scale poorly to large tables.

None of these approaches cleanly handles all four challenging scenarios (wireless, oriented, distorted, multi-spanning-cell) within a single-stage, OCR-free pipeline. GridFormer is designed to close that gap.

What is the novelty?

Grid representation. The central observation is that any table can be expressed as an $M \times N$ grid of vertices and edges. Given a table with $r$ rows and $c$ columns, the corresponding grid has $(r+1) \times (c+1)$ vertices and two sets of edges (rightward and downward). A vertex is “positive” if it coincides with a cell corner; an edge is “positive” if it lies on a cell boundary. Both sets are binary-classified, and each positive vertex stores its physical $(x, y)$ coordinate. This representation is:

• Unified: wired and wireless tables differ only in which edges are positive; the representation itself does not change.
• Compact: no autoregressive decoding; the grid is a fixed-size tensor for a given dataset.
• Geometry-aware: physical coordinates are stored directly, enabling localization for distorted or rotated tables.

Two-stream decoder. GridFormer adapts Deformable DETR with two parallel transformer decoders. One decoder processes row queries $Q_{\text{row}} \in \mathbb{R}^{M \times d}$ and predicts $y$-axis coordinates; the other processes column queries $Q_{\text{col}} \in \mathbb{R}^{N \times d}$ and predicts $x$-axis coordinates. Decoupling the axes simplifies vertex localization because within a single row, the $y$ range is narrow while $x$ varies widely, and vice versa.

Query selection module. Rather than using learnable reference point embeddings (as in standard DETR), the query selection module generates image-conditioned reference points. Row proposals are placed at fixed $x = W/4$ positions evenly distributed along $y$; column proposals at fixed $y = H/4$ evenly along $x$. The top-80 scoring proposals (no NMS) initialize the decoder reference points, and L1 supervision is applied to these points during training.

Three prediction heads. Given row embeddings $Z_{\text{row}}$ and column embeddings $Z_{\text{col}}$:

1. Classification head: predicts positive/negative probability for each row and column via Focal loss.

$$L_{\text{cls}} = \text{Focal}(\hat{p}_{\text{row}}, p_{\text{row}}) + \text{Focal}(\hat{p}_{\text{col}}, p_{\text{col}})$$

2. Position head: predicts normalized $y$-coordinates from row embeddings and $x$-coordinates from column embeddings via L1 loss, with an auxiliary cell-level GIoU loss on reconstructed cell bounding boxes.

$$L_{\text{coord}} = L1(\hat{r}_{\text{row}}, r_{\text{row}}) + L1(\hat{r}_{\text{col}}, r_{\text{col}}) + \gamma_1 L_{\text{iou}}(\hat{g}, g)$$

3. Edge head: at each grid vertex, concatenates the corresponding row/column query embedding (global context) with a locally sampled CNN feature (local context), then applies binary Focal classification for rightward and downward edges separately.

$$L_{\text{edge}} = \text{Focal}(\hat{e}_{\text{row}}, e_{\text{row}}) + \text{Focal}(\hat{e}_{\text{col}}, e_{\text{col}})$$

The total loss is:

$$L = \lambda_1 L_{\text{cls}} + \lambda_2 L_{\text{coord}} + \lambda_3 L_{\text{ref}} + \lambda_4 L_{\text{edge}}$$

with $\lambda_1 = 1$, $\lambda_2 = \lambda_3 = \lambda_4 = 5$, $\gamma_1 = 0.1$. Auxiliary losses from all six decoder layers are also used.

Table reconstruction. After decoding, rows and columns with score $> \tau_1 = 0.5$ are kept and sorted by reference point coordinates. Edges with score $> \tau_2 = 0.4$ are kept. A breadth-first search on the resulting grid groups adjacent positive vertices into cells.

What experiments were performed?

GridFormer is evaluated on five benchmarks spanning both regular (PDF-derived) and scene (wild-image) tables.

Datasets:

• SciTSR: 12k train / 3k test; scientific PDFs; evaluated with cell adjacency F1.
• PubTabNet: 500k train / 9k val; scientific PDFs; evaluated with TEDS and TEDS-Struct on the validation set (test annotations are unreleased).
• FinTabNet: 92k train / 10k val; financial tables; evaluated with TEDS-Struct on the validation set.
• WTW: 11k train / 3.6k test; wild photos with deformation; evaluated with cell adjacency precision/recall/F1.
• TAL (TAL_OCR_TABLE): 12.3k train / 3k test (custom split from the competition set); wild scene tables; also evaluated on TAL_rotated ($\pm 30^\circ$) and TAL_curved (geometric distortion variants). Metrics: TEDS-Struct and cell F1 at IoU 0.6.

Baselines compared: EDD, TabStruct-Net, GTE, SEM, LGPMA, FLAG-Net, NCGM, TableFormer, TSRFormer, TRUST, VAST (PubTabNet/FinTabNet); GraphTSR, RobustTabNet (SciTSR); Cycle-CenterNet, TSRFormer, NCGM (WTW); SPLERGE, TableMaster (TAL).

Implementation: ResNet-50 backbone; 6 deformable encoder layers; 6 decoder layers per stream; AdamW, LR 2e-4; 8 V100 GPUs, batch size 24; multi-scale training with short side in {384, 416, 448, 480, 512}; long side capped at 640.

Ablations (on WTW):

• Removing the query selection module (random reference points): F1 drops from 94.1% to 86.3% (-7.8%).
• Replacing the two-stream decoder with a single decoder: F1 drops from 94.1% to 70.2% (-23.9%), suggesting that axis decoupling matters considerably.
• Using only query embeddings for edge classification (no local visual features): F1 drops to 62.2%.
• Using only local visual features for edge classification (no query embeddings): F1 drops to 88.7%.
• Both combined: 94.1% (best), suggesting the value of global-plus-local feature fusion.

What are the outcomes/conclusions?

Regular tables (PDF-derived):

On PubTabNet and SciTSR, GridFormer is competitive but not the best-reported number. On FinTabNet, it ties VAST at 98.63% (an improvement of 1.8 points over TableFormer).

Scene tables (wild images):

The gap between methods is largest on rotated and curved tables. TableMaster’s autoregressive decoder drops substantially under geometric distortion, from 27.8% (TAL_rotated) and 64.5% (TAL_curved) versus GridFormer at 92.9% and 96.8% respectively.

Limitations:

• No code or weights are released, limiting reproducibility.
• The TAL evaluation uses a custom train/test split from the competition set rather than an established evaluation protocol; comparison with future methods may be inconsistent.
• TEDS on PubTabNet requires external OCR (the authors use PSENet + MASTER); the quality of these OCR results directly affects reported TEDS scores, and the paper does not ablate OCR quality.
• The grid size $M \times N$ is matched to the largest table in each dataset. For datasets with very large tables, this increases query count and memory usage proportionally.
• The method does not output cell text content; it is a structure-only recognizer. Downstream content extraction requires a separate OCR stage.

Reproducibility

Models

• Backbone: ResNet-50 (ImageNet pre-trained; standard initialization).
• Encoder: 6 deformable transformer encoder layers from Deformable DETR; 256-channel features; multi-scale features from ResNet stages 3-5.
• Decoder: two parallel deformable transformer decoders, each with 6 layers; hidden dimension 256.
• Query counts: $M$ and $N$ are set to the maximum row and column counts in each dataset (e.g., 50 rows and 50 columns for TAL). Ablations show modest sensitivity to this choice.
• No pre-trained TSR checkpoints, model weights, or code repository are released.

Algorithms

• Optimizer: AdamW, initial LR 2e-4. No learning rate schedule (warmup, decay policy, or total training steps/epochs) is reported in the paper.
• Training hardware: 8 NVIDIA V100 GPUs; total batch size 24.
• Multi-scale training: short side randomly sampled from {384, 416, 448, 480, 512}; long side capped at 640.
• Inference resizing: long side fixed at 640 while preserving aspect ratio.
• Score thresholds: $\tau_1 = 0.5$ (row/column classification), $\tau_2 = 0.4$ (edge classification).
• Loss weights: $\lambda_1 = 1$, $\lambda_2 = \lambda_3 = \lambda_4 = 5$, $\gamma_1 = 0.1$.
• Bipartite matching: Hungarian algorithm used to assign ground-truth rows/columns to predicted queries, following DETR.

Data

• All training data come from existing public datasets (SciTSR, PubTabNet, FinTabNet, WTW, TAL). No new data is collected.
• TAL uses a custom 12,285/3,000 train/test split from the original competition training set; split file names are stated to be released but no repository is given.
• TAL_rotated and TAL_curved are derived from TAL via data augmentation (random rotation $\pm 30^\circ$; geometric distortion); these variants are not available as standalone downloads.
• Label generation for datasets with text-line level bounding box annotations (e.g., SciTSR, PubTabNet) requires a multi-step coordinate extension procedure described in the appendix; this preprocessing is non-trivial but documented in the paper.

Evaluation

• TEDS / TEDS-Struct: standard metrics from PubTabNet; TEDS-Struct ignores cell content and compares HTML structure trees only.
• Cell adjacency F1: used for SciTSR and WTW; measures whether pairs of horizontally or vertically adjacent cells are correctly identified.
• Cell F1 at IoU 0.6: localization metric used for TAL; measures how well predicted cell bounding boxes overlap with ground truth.
• PubTabNet TEDS results depend on external OCR (PSENet + MASTER); the paper does not provide TEDS-Struct-only comparisons that would isolate the structure predictor from OCR errors.
• No error bars, significance tests, or repeated runs are reported.

Hardware

• Training: 8 V100 GPUs; total batch size 24. Total GPU-hours and wall-clock time are not reported.
• Inference: long side 640; no inference latency or throughput numbers are reported.
• Deployment: no discussion of CPU-only or low-memory inference feasibility.

BibTeX

@inproceedings{lyu2023gridformer,
  title={GridFormer: Towards Accurate Table Structure Recognition via Grid Prediction},
  author={Lyu, Pengyuan and Ma, Weihong and Wang, Hongyi and Yu, Yuechen and Zhang, Chengquan and Yao, Kun and Xue, Yang and Wang, Jingdong},
  booktitle={Proceedings of the 31st ACM International Conference on Multimedia},
  year={2023},
  doi={10.1145/3581783.3611961}
}

TL;DR

MTL-TabNet proposes a single encoder-decoder model that jointly learns table structure recognition, cell bounding box detection, and cell content recognition as three coupled tasks. On PubTabNet and FinTabNet, it reported improvements over TableFormer on all three sub-tasks, achieving competitive results with the top entries in the ICDAR 2021 competition without ensembling or additional annotations.

What kind of paper is this?

Dominant: $\Psi_{\text{Method}}$: The paper’s core contribution is the architecture: a multi-task learning network that jointly handles three aspects of table recognition through shared and task-specific decoder components. The validation is comparison tables against strong baselines with ablation logic implicit in the sub-task decomposition.

Secondary: None significant. No new dataset, metric, or theoretical derivation is introduced.

What is the motivation?

Most table recognition pipelines at the time of publication were non-end-to-end: a separate system would handle table structure recognition, and a separate OCR system would handle cell content recognition. These two stages were trained independently, so errors from one stage could not be corrected by the other, and the systems could not share visual representations.

Prior end-to-end attempts (EDD, IM2TEX) existed but lagged substantially behind non-end-to-end methods. The authors argue that multi-task learning is a natural fit for table recognition because structure, cell layout, and cell content are interdependent.

What is the novelty?

The core architectural contribution is a five-component design:

1. Shared Encoder: ResNet-31 with Multi-Aspect Global Context Attention (GCAttention) after each residual block, producing a spatial feature grid that is unrolled column-by-column and passed through positional encoding.

2. Shared Decoder: Two standard Transformer decoder layers that consume the encoder output and a right-shifted sequence of structural HTML tokens, producing a context vector for each predicted cell token.

3. Structure Decoder: One Transformer decoder layer predicting an HTML tag sequence for the table structure. Cells are tokenized at the HTML tag level; spanning cells are decomposed into <td, span attribute, value, and > tokens.

4. Cell-BBox Decoder: Triggered whenever the structure decoder emits a cell-start token. Takes the corresponding shared-decoder output and predicts four bounding box coordinates via a linear layer and sigmoid activation.

5. Cell-Content Decoder: Similarly triggered per cell. Autoregressively generates the text content of that cell at character level, attending to both the shared encoder output and the per-cell shared decoder representation.

The overall loss is:

$$ \mathcal{L} = \lambda_{1}\mathcal{L}_{\text{struc.}} + \lambda_{2}\mathcal{L}_{\text{cont.}} + \lambda_{3}\mathcal{L}_{\text{bbox}} $$

where $\mathcal{L}_{\text{struc.}}$ and $\mathcal{L}_{\text{cont.}}$ are cross-entropy losses and $\mathcal{L}_{\text{bbox}}$ is an L1 regression loss. All three $\lambda$ values are set to 1.

The key design choice is that the shared decoder processes the structural token sequence, giving all three task-specific decoders a contextualized cell-level representation without requiring three separate encoder passes.

What experiments were performed?

Datasets:

• PubTabNet: 568k table images from PMCOA scientific papers, annotated with HTML structure, cell bounding boxes, and cell text. Split used: 500,777 train / 9,115 validation (dev phase) / 9,064 final evaluation.
• FinTabNet: 112k complex financial tables from S&P 500 annual reports. 81 / 9.5 / 9.5 train/val/test split.

Metrics:

• TEDS-Struct: Tree Edit Distance Similarity considering only structural HTML tokens (not cell content).
• TEDS: Full Tree Edit Distance Similarity including cell content.
• mAP (PASCAL VOC) for cell bounding box detection on PubTabNet.

Baselines compared:

• EDD (PubTabNet baseline)
• GTE and GTE (fine-tuned) for FinTabNet
• LGPMA (required additional text-line bounding box annotations)
• TableFormer (with and without PDF-based post-processing)
• SEM (3rd place, ICDAR 2021)
• VCGroup / TableMASTER (2nd place, ICDAR 2021, uses model ensembles)
• Davar-Lab-OCR (1st place, ICDAR 2021, uses additional annotation + ensembles)

What are the outcomes/conclusions?

Table Structure Recognition (TEDS-Struct):

• FinTabNet: 98.79% All (vs. 96.80% TableFormer, +2.0%)
• PubTabNet val: 97.88% All (vs. 96.75% TableFormer, +1.1%)

Cell Detection (mAP on PubTabNet val):

• 88.93% (vs. TableFormer 82.10%, +6.8%; vs. TableFormer+PP 86.80%, +2.1%)

Full Table Recognition (TEDS on PubTabNet val):

• 96.67% All (vs. TableFormer 93.60%, vs. SEM 93.70%, vs. VCGroup without ensemble 96.26%)

ICDAR 2021 Final Evaluation (PubTabNet final set):

• 96.17% TEDS All, approximately 4th place, without ensembles or additional training data.
• Davar-Lab-OCR (1st): 96.36%; VCGroup (2nd): 96.32%; XM (3rd): 96.27%.

The multi-task design appears to benefit cell detection in particular, likely because the structure decoder provides explicit cell token triggers, focusing the bbox decoder on exactly one cell at a time rather than running box detection over the whole table image. The content decoder benefits from this same focus mechanism.

Limitations the authors do not discuss: The architecture is autoregressive in the structure decoder, which means inference time scales with the number of cells in a table. Large tables with many cells could be slow. The paper does not report inference speed or latency. The cell-content decoder also autoregressively generates characters per cell, which may compound slow inference. There is no evaluation on wild or camera-captured table images (e.g., WTW dataset), so robustness outside clean document scans is unknown.

The paper also lacks a formal ablation study. No experiment isolates the contribution of individual decoders (for example, training without the cell-content loss, or replacing the shared decoder with per-task decoders). The gain over TableFormer is attributed to the MTL formulation, but the relative contributions of the stronger backbone (GCAttention), the shared decoder design, and multi-task gradient flow are not disentangled. Additionally, the shared decoder uses teacher forcing during training (conditioned on gold HTML tags) but must consume its own predicted output at inference. This train/inference mismatch is a known source of error accumulation in autoregressive sequence models and is not acknowledged.

Reproducibility

Models

• Backbone: ResNet-31 with GCAttention (Multi-Aspect Global Context Attention from MASTER, Lu et al. 2021) after each residual block.
• Input resolution: 480 $\times$ 480 pixels; CNN output feature map: 60 $\times$ 60.
• Shared decoder: $N = 2$ Transformer decoder layers; hidden size 512, FFN size 2048, 8 attention heads.
• Structure decoder: 1 Transformer decoder layer; same hidden/FFN/head config.
• Cell-BBox decoder: 1 Transformer decoder layer + linear + sigmoid.
• Cell-Content decoder: 1 Transformer decoder layer + linear + softmax.
• Max sequence lengths: 500 structural tokens; 150 character tokens per cell.
• Weights: Pretrained checkpoints for both PubTabNet and FinTabNet are released on Google Drive (see frontmatter artifacts). The repository and weights are licensed Apache-2.0 (inherited from the MMOCR codebase). Note: the repository README instructs using master_decoder_old20220923.py instead of the default master_decoder.py when loading the released checkpoints.

Algorithms

• Optimizer: SGD (implied; the paper says “stochastic gradient descent algorithms” without specifying Adam vs. SGD).
• Learning rate: 0.001 for the first 12 epochs; then 0.0001 for 8 more epochs or until convergence.
• Batch size: 4 per GPU (2 GPUs; effective batch size 8).
• Loss: Weighted sum of cross-entropy (structure), cross-entropy (content), and L1 (bounding box), with $\lambda_{1} = \lambda_{2} = \lambda_{3} = 1$.
• Framework: PyTorch + MMCV; specifically MMOCR-0.2.0, MMDetection-2.11.0, mmcv-full-1.3.4.
• Data augmentation: Not described.

Data

• PubTabNet: Publicly available (CDLA-Permissive-1.0 for annotations; underlying PMCOA images have per-article mixed licenses).
• FinTabNet: Published by IBM; CDLA-Permissive-1.0 for annotations; underlying S&P 500 report images are from copyrighted filings. The original IBM distribution is no longer easily accessible; the canonicalized FinTabNet.c version is available on HuggingFace.
• Both datasets provide structure annotations in HTML, cell bounding boxes, and cell text per non-empty cell.

Evaluation

• TEDS and TEDS-Struct as defined in Zhong et al. 2020.
• Simple vs. Complex splits: Simple = no spanning cells; Complex = tables with multi-row or multi-column cells.
• Cell detection evaluated with PASCAL VOC mAP protocol on PubTabNet.
• No error bars, significance tests, or multi-run statistics reported.
• Comparison with ICDAR 2021 competition entries is fair only in the sense that the same final evaluation set was used; competition entries may have used different training data augmentation or proprietary preprocessing.

Hardware

• Training: 2 $\times$ NVIDIA A100 80 GB GPUs.
• Training time: Not reported.
• Inference speed: Not reported.
• Deployment: The autoregressive structure and content decoders suggest non-trivial inference latency on large tables; no benchmarks provided.

BibTeX

@inproceedings{ly2023mtltabnet,
  title={An End-to-End Multi-Task Learning Model for Image-based Table Recognition},
  author={Nam Tuan Ly and Atsuhiro Takasu},
  booktitle={Proceedings of the 18th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 5: VISAPP},
  pages={626--634},
  year={2023},
  organization={SCITEPRESS},
  doi={10.5220/0011685000003417}
}

TL;DR

GraphTSR reformulates table structure recognition (TSR) in PDFs as an edge-classification problem on a cell adjacency graph, using alternating edge-to-vertex and vertex-to-edge graph attention blocks to predict horizontal and vertical cell relations. The paper also introduces SciTSR, a 15,000-table dataset of scientific PDF tables with structure labels derived from LaTeX source, split 12,000/3,000 train/test.

What kind of paper is this?

Dominant: $\Psi_{\text{Method}}$. The headline contribution is a new graph neural network architecture (GraphTSR) for recognizing table structure from PDF cell inputs. The paper’s center of gravity is the architectural design, comparison against three baselines, and the per-dataset, per-metric performance tables.

Secondary: $\Psi_{\text{Resource}}$. The paper simultaneously releases SciTSR, a large-scale dataset that did not exist before this work and which has since become a standard benchmark in the TSR community. Without SciTSR, the experimental section would have been limited to the tiny ICDAR-2013 set (156 tables, no training split).

What is the motivation?

PDF table structure recognition requires understanding which cells are adjacent to which, including cells that span multiple rows or columns (“spanning cells”). Prior work fell into two camps: rule-based methods that relied on explicit layout heuristics, and a small set of deep learning approaches that treated the problem as image segmentation into row/column regions. Both camps handled simple grid-like tables reasonably well but struggled with tables containing spanning cells, which carry disproportionately important semantic content (they are often headers).

The existing benchmark (ICDAR-2013) had only 156 tables and no training split, which severely constrained data-driven research. The authors identify both the methodological gap (no principled way to handle spanning cells) and the resource gap (no adequate training corpus) as jointly limiting progress.

What is the novelty?

GraphTSR treats each cell as a node and the task as predicting whether any two cells share a horizontal or vertical adjacency edge. The pipeline has four stages:

1. Pre-processing: extract cell text content and bounding boxes from the PDF using an existing tool (same procedure as Shigarov et al., 2016).
2. Graph construction: connect each cell to its $K = 20$ nearest neighbors by spatial distance, yielding an approximately sparse graph with $O(K|V|)$ edges rather than $O(|V|^2)$.
3. Relation prediction: run GraphTSR to classify each edge as vertical, horizontal, or no relation.
4. Post-processing: convert the labeled graph back into a structured table representation.

The core model maintains two representation streams, one over vertices (cells) and one over edges (candidate adjacency pairs), updated by alternating attention blocks. The edge-to-vertex block updates each cell’s representation by attending over its neighboring edge nodes; the vertex-to-edge block updates each edge’s representation by attending over its neighboring cell nodes. Graph attention is local (restricted to the $K$-NN neighborhood) rather than global:

$$ a_{iu} = \frac{e^{\mathbf{k}_i^\top \mathbf{q}_u / \sqrt{d_k}}}{\sum_{j \in N(u)} e^{\mathbf{k}_j^\top \mathbf{q}_u / \sqrt{d_k}}} $$

$$ \text{GraphAtt}(\mathbf{q}_u, \mathbf{K}, \mathbf{V}) = \sum_{j \in N(u)} a_{ju} \mathbf{v}_j $$

Each attention block wraps the graph attention with residual connections and layer normalization, following the Transformer block structure:

$$ \tilde{\mathbf{H}}_v^{(n)} = \text{Norm}!\left(\mathbf{H}_v + \text{GraphAtt}(\mathbf{Q}, \mathbf{K}, \mathbf{V})\right) $$

$$ \mathbf{H}_v^{(n)} = \text{Norm}!\left(\tilde{\mathbf{H}}_v^{(n)} + \text{FFN}(\tilde{\mathbf{H}}_v^{(n)})\right) $$

Vertex features include cell size, absolute bounding-box coordinates, and relative positions. Edge features include Euclidean, x-axis, and y-axis distances in both absolute and relative form, plus x-axis and y-axis overlap between cell pairs. These hand-crafted features are fed as initial representations to the two streams.

The key design decision is representing the original graph as a bipartite graph (cells and candidate edges as separate node sets), so that both vertex and edge states can be updated with the same graph attention mechanism.

What experiments were performed?

Datasets: SciTSR (12,000 train / 3,000 test), ICDAR-2013 (test-only, 156 tables), and SciTSR-COMP (the 716 complicated tables from the SciTSR test set, used as a harder held-out subset).

Metric: The ICDAR 2013 evaluation protocol, which converts a table to a list of horizontally and vertically adjacent cell pairs and computes precision, recall, and F1. Both macro- and micro-averaged scores are reported.

Baselines:

• Tabby (Shigarov et al., 2016): rule-based PDF table extractor
• DeepDeSRT (Schreiber et al., 2017): CNN-based semantic segmentation of row and column regions
• Adobe Acrobat DC SDK: commercial table-to-HTML extraction

Note that both DeepDeSRT and GraphTSR are trained on SciTSR training data and then evaluated zero-shot on ICDAR-2013, testing cross-domain generalization.

Training details: Adam optimizer, initial learning rate 0.0005, batch size 1 (one graph per step), 15 epochs, L2 weight decay $\lambda = 0.0001$, dropout $p = 0.4$ on each sub-layer output. Class imbalance is addressed with a manual rescaling weight: 0.2 for no relation edges and 1.0 for vertical and horizontal edges in the cross-entropy loss.

What are the outcomes/conclusions?

On ICDAR-2013, GraphTSR achieves macro-F1 of 0.837 and micro-F1 of 0.872, surpassing Tabby (0.816 / 0.854) and substantially outperforming DeepDeSRT (0.568 / 0.615). The cross-domain gap for DeepDeSRT suggests that image-based models are sensitive to table rendering style, while GraphTSR’s text-and-geometry features generalize better.

On SciTSR, GraphTSR achieves macro-F1 of 0.934 and micro-F1 of 0.953, ahead of Tabby (0.912 / 0.921) and DeepDeSRT (0.897 / 0.890).

The more informative comparison is SciTSR-COMP. All baselines drop at least 4 points relative to full SciTSR; GraphTSR retains macro-F1 0.934 and micro-F1 0.955, while Tabby falls to 0.855 / 0.882. On the most demanding sub-evaluation (spanning cells only, ignoring non-spanning adjacencies), the advantage is sharper: GraphTSR reaches macro-F1 0.703 vs. Adobe’s 0.485 and Tabby’s 0.379. DeepDeSRT cannot recover any spanning-cell structure at all (its outputs are always grid-like).

The authors conclude that framing TSR as graph edge prediction rather than image segmentation or sequence generation naturally handles spanning cells, because each cell-to-cell relation is predicted independently and the model is not forced into a grid assumption.

Limitations (stated and unstated):

• The pre-processing step depends on a rule-based PDF text extractor to supply cell bounding boxes; the model itself does not handle raw PDF pages or tables from scanned images.
• The SciTSR tables come entirely from scientific papers on arXiv compiled from LaTeX source; tables from other domains (business reports, government documents, HTML pages) are not represented, and the LaTeX-derived labels may not generalize to noisier real-world PDFs.
• Training requires a ground-truth cell-bounding-box-to-LaTeX-cell matching step (“cell matching”), which relies on accurate pre-processing and clean LaTeX source; this step is not evaluated separately.
• No ablation is provided over the number of attention blocks, the KNN $K$ value, or the feature engineering choices.
• GPU vs. CPU training is not discussed in the context of scalability or inference speed.
• DeepDeSRT was reimplemented by the authors rather than using original code (none was available). A reimplemented baseline may not match the original’s tuning, which puts the comparison’s fairness in question.
• SciTSR was designed and constructed to match GraphTSR’s input format: pre-extracted cell bounding boxes from clean LaTeX source. Image-based methods like DeepDeSRT are disadvantaged on this benchmark by design, since the benchmark natively favors text-and-geometry approaches.

Reproducibility

Models

GraphTSR is a 4-block model with hidden dimension $d = 64$, implemented in PyTorch 0.4.1. The bipartite-graph formulation has edge-to-vertex and vertex-to-edge attention blocks, each following the Transformer residual-norm-FFN pattern. Total parameter count is not reported. No pretrained checkpoints are released. The GitHub repository contains the SciTSR dataset (via Google Drive download) and evaluation scripts for computing precision/recall/F1 against ground-truth relation labels; it does not include GraphTSR training code.

Algorithms

• Optimizer: Adam, learning rate 0.0005
• Batch size: 1 graph per step
• Epochs: 15
• Dropout: $p = 0.4$ on each sub-layer output
• Weight decay: L2 with $\lambda = 0.0001$
• Loss: cross-entropy with class weights (0.2 for no relation, 1.0 for adjacency classes)
• KNN graph construction: $K = 20$
• No data augmentation or curriculum learning described

Data

SciTSR is collected by crawling LaTeX source files from arXiv, extracting table environments, compiling them to individual PDF files, and parsing \multirow/\multicolumn commands to produce JSON structure labels. The full dataset has 15,000 tables (12,000 train, 3,000 test). The 716 complicated-table test subset (SciTSR-COMP) is the subset of the test split that contains at least one spanning cell. The dataset is publicly available under the MIT license.

A notable caveat: since labels come from LaTeX source, the dataset cannot easily include tables from scanned documents or proprietary PDF producers. It is also restricted to the scientific paper style of arXiv submissions, which tend to be typeset consistently.

Evaluation

The ICDAR 2013 evaluation protocol converts each table to a set of adjacent-cell relation tuples and computes precision, recall, and F1 by set overlap. Both macro (per-table average) and micro (per-relation aggregate) variants are reported. No error bars, significance tests, or multi-run averages are provided.

ICDAR-2013 has no training split; results on it test zero-shot generalization from SciTSR training data.

Hardware

Training runs on Intel Xeon CPUs. Each epoch over the 12,000 SciTSR training graphs takes approximately 20 minutes, giving roughly 5 hours of total training time. No GPU is used or reported. Inference hardware and latency are not discussed.

BibTeX

@misc{chi2019complicated,
  title={Complicated Table Structure Recognition},
  author={Chi, Zewen and Huang, Heyan and Xu, Heng-Da and Yu, Houjin and Wanxuan Yin and Xian-Ling Mao},
  year={2019},
  eprint={1908.04729},
  archivePrefix={arXiv},
  primaryClass={cs.IR}
}

TL;DR

PubTables-v2 is a new large-scale dataset from Kensho Technologies for table extraction from scientific documents. It introduces three collections organized by context: 136k large cropped tables, 548k single-page tables with caption and footer annotations, and 9,172 full documents with 9,492 multi-page tables spanning up to 13 pages. It is the first large benchmark for multi-page table structure recognition, and the authors also demonstrate a cross-page table continuation classifier that significantly improves document-level extraction performance.

What kind of paper is this?

Dominant: $\Psi_{\text{Resource}}$ (the primary contribution is the dataset itself: a large-scale, quality-controlled collection of annotated tables at multiple levels of document context, including the first large multi-page table benchmark).

Secondary: $\Psi_{\text{Evaluation}}$ (the paper introduces generalized GriTS and TEDS metrics for multi-table inputs and establishes baseline performance across four distinct tasks); $\Psi_{\text{Method}}$ (the paper introduces POTATR, a page-object adaptation of TATR, and a cross-page table continuation classifier).

What is the motivation?

Table extraction (TE) has traditionally been decomposed into two sequential stages: detecting individual tables, then recognizing their internal structure from cropped images. This decomposition has known drawbacks. Multi-stage pipelines are complex to maintain, and processing tables in isolation discards potentially useful surrounding context. Interest has grown in methods that extract tables directly from full page or document images, including smaller specialized vision-language models (VLMs), but demonstrating progress on this richer problem has been difficult because no adequate benchmarks existed.

Specific gaps the authors identify:

• No large-scale dataset annotates table structure within full-page context with strict quality control and complete coverage (every table on a page must be annotated).
• No large dataset connects tables to their captions and footers with hierarchical relationships.
• No published dataset exists for multi-page table extraction, despite its clear practical importance.
• Existing benchmarks like PubTables-1M have been in use long enough that current model pre-training practices create data leakage concerns.
• Existing datasets skew toward short, narrow tables; challenging long and wide tables are underrepresented.

What is the novelty?

Three context-level collections: Rather than splitting TD and TSR into separate subtasks, PubTables-v2 organizes its data by the amount of surrounding context available:

1. Cropped Tables (135,578 samples): exclusively long ($\geq 30$ rows) or wide ($\geq 12$ columns) tables, representing the challenging tail that prior datasets undersample. This collection includes 32% more long or wide tables than PubTables-1M, despite having a fraction of the total count.
2. Single Pages (467,541 pages; 548,414 tables): full-page annotations with bounding boxes for tables, rows, columns, spanning cells, column headers, projected row headers, captions, and footers, plus a hierarchical relation set connecting each table to its associated captions and footers.
3. Full Documents (9,172 documents; 137,095 pages; 24,862 tables): document-level annotations where every document contains at least one multi-page or split table. This collection includes 9,492 multi-page tables and 630 single-page split tables (split across two columns of a two-column page).

Stricter quality control: PubTables-v2 tightens the annotation quality thresholds from PubTables-1M. The mean normalized edit distance threshold is $\alpha = 0.02$ (down from $0.05$) and a new maximum per-cell edit distance threshold $\beta = 0.2$ is added. For the page- and document-level collections, if any table in a document fails quality control, the entire document is discarded, ensuring no page has missing table annotations.

Generalized evaluation metrics: Standard GriTS and TEDS metrics are defined for a single predicted table against a single ground truth table. The authors generalize both to handle a predicted set of tables against a ground truth set. Using the Hungarian algorithm to find the optimal one-to-one matching, they compute a pseudo-F1 score aggregated over all samples:

$$ \text{GriTS}_f(\mathbf{A}, \mathbf{B}) = \frac{2 \sum_{i,j} f(\tilde{\mathbf{A}}_{i,j}, \tilde{\mathbf{B}}_{i,j})}{|\mathbf{A}| + |\mathbf{B}|} $$

where $\mathbf{A}$ and $\mathbf{B}$ are the ground truth and predicted table matrices and $f$ is a cell-level similarity function. For multi-table inputs, the true positive scores are summed across all samples before computing the final pseudo-F1.

POTATR (Page-Object Table Transformer): An adaptation of TATR for full-page extraction. POTATR extends TATR by adding two page-level object classes (caption, footer) and their rotated counterparts (16 classes total), doubling the object queries to 250, and adding a relation head (a three-layer MLP) to predict parent-child relationships between a table and its associated structures.

Cross-page table continuation classifier: A binary image classifier trained to predict whether the last table on one page continues onto the next. The input is two contiguous page images concatenated side by side. A ViT-B-16 classifier trained on PubTables-v2 achieves recall of 0.995 and precision of 0.987, suggesting strong visual cues for this task in the dataset.

What experiments were performed?

The authors evaluate several recent smaller, specialized VLMs (SmolDocling-256M, GraniteDocling-258M, Qwen2.5-VL-3B, Granite-Vision-3.2-2B, DeepSeek-OCR, DeepSeek-OCR 2, dots.ocr) plus TATR-based models trained on PubTables-v2 data.

Experiment 1: Cropped table structure recognition. Models are evaluated on the Cropped Tables collection (long and wide tables). TATR-v1.2-Pub (fine-tuned on PubTables-v2) is compared against seven VLMs and prior TATR versions. TATR is also compared with four text sources: EasyOCR, PaddleOCR, docTR, and text extracted directly from the PDF (DT).

Experiment 2: Page-level table extraction. Models are evaluated on the Single Pages collection, requiring prediction of all tables on a page. POTATR-v1.0-Pub is introduced here for the first time. Metrics are the generalized GriTS and TEDS variants.

Experiment 3: Document-level table extraction. Models are evaluated on the Full Documents collection. Two scenarios are compared: (1) providing all pages of a document at once, and (2) processing pages individually while evaluating at the document level. Only Qwen2.5-VL-3B was able to process full documents at once due to context window constraints.

Experiment 4: Cross-page table continuation prediction. ResNet-50 and ViT-B-16 are trained on 15,830 pairs (9,866 positive, 5,964 negative) from the Full Documents collection and evaluated on classification metrics (recall, precision, F1, AUC).

Experiment 5: Document-level TE with cross-page merging. The best-performing document-level model (dots.ocr, processed page-by-page) is augmented with the ViT-B-16 continuation classifier. When the classifier predicts a table continuation and the table column counts match, tables are concatenated vertically.

Additional experiments (appendix): Single-page split table extraction, analysis of how often models predict the correct number of tables per page, comparison of image-to-graph architectures (Relationformer, EGTR, POTATR) in a small-scale experiment, and an ablation on training set size for the continuation classifier.

What are the outcomes/conclusions?

Key findings:

• Among VLMs on cropped long/wide tables, dots.ocr performs best with $\text{GriTS}_{\text{Con}} = 0.801$. TATR-v1.2-Pub with direct text extraction achieves $\text{GriTS}_{\text{Con}} = 0.980$, roughly halving the error of the prior TATR-v1.1-Pub model, showing the value of additional training data for this challenging subset.
• At the page level, dots.ocr again leads among VLMs ($\text{GriTS}_{\text{Con}} = 0.899$), still below POTATR with direct text extraction ($\text{GriTS}_{\text{Con}} = 0.957$).
• At the document level (pages processed individually), the best VLM, dots.ocr, achieves only $\text{GriTS}_{\text{Con}} = 0.577$ and perfect content-and-structure exact match on only 11.8% of tables ($\text{Acc}_{\text{Con}} = 0.118$). The only model tested on full documents at once, Qwen2.5-VL-3B, scores $\text{GriTS}_{\text{Con}} = 0.047$.
• The cross-page continuation classifier (ViT-B-16) achieves F1 = 0.991, indicating strong visual cues in the dataset. Even with only 250 training samples, models exceed F1 = 0.86.
• Augmenting dots.ocr with the continuation classifier improves document-level $\text{GriTS}_{\text{Con}}$ from 0.577 to 0.684, a substantial gain from a lightweight auxiliary model.

Failure modes observed:

• Qwen2.5-VL-3B enters infinite repetition loops on full-document inputs, even for short two-page documents.
• Granite-Vision-3.2-2B always predicts exactly one table per page, regardless of actual table count, which artificially inflates its split-table metrics.
• No model achieves $\text{Acc}_{\text{Con}} > 0$ on single-page split tables.

Limitations acknowledged by the authors:

• PubTables-v2 is sourced exclusively from English-language scientific articles (PubMed); it does not cover other domains or languages.
• The dataset is not intended to replace benchmarks covering broader variation in table and page appearance.
• Row and column bounding boxes are not annotated for multi-page tables in the Full Documents collection.

Reproducibility

Models

• TATR-v1.2-Pub: 28M parameters, identical architecture to TATR-v1.1-Pub, fine-tuned on the PubTables-v2 Cropped Tables collection. Weights initialized from TATR-v1.1-Pub (pre-trained on PubTables-1M cropped tables).
• POTATR-v1.0-Pub: 29M parameters, extends TATR with 125 additional object queries (250 total), two additional object classes (caption, footer) plus rotated counterparts (16 classes total), and a three-layer MLP relation head. Initialized from TATR-v1.1-Pub; relation head and extra queries are randomly initialized.
• Cross-page classifiers: Standard ResNet-50 and ViT-B-16 image classifiers. Input is two page images concatenated horizontally.
• All models will be released at the HuggingFace dataset page under MIT license.

Algorithms

Both TATR-v1.2-Pub and POTATR-v1.0-Pub follow the default TATR training setup (AdamW optimizer, standard DETR hyperparameters) unless noted otherwise.

• TATR-v1.2-Pub training: 8 Nvidia T4 GPUs, batch size 2 per GPU (effective batch size 16). Epoch defined as 100,000 samples; trained for 160 epochs. Initial learning rate $5 \times 10^{-5}$, gamma 0.9 every 4 epochs. No-object class weight eos_coef = 0.3.
• POTATR-v1.0-Pub training: Same hardware setup. 100 epochs on Single Pages collection. Initial learning rate $5 \times 10^{-5}$, gamma 0.9 every 2 epochs. eos_coef = 0.1. Relation loss weight = 0.05.
• Cross-page classifiers: Trained for 200 epochs, best checkpoint selected by validation F1.

Data

• Source: PubMed Central articles published 2023-2025 (not included in PubTables-1M).
• Annotation leverages author-supplied HTML/XML table markup aligned to PDF via sequence matching.
• Quality thresholds: mean normalized edit distance $\alpha < 0.02$, maximum per-cell edit distance $\beta < 0.2$.
• Table structure canonicalization scheme from PubTables-1M is applied, plus the improved column header annotation correction from Smock et al. (ICDAR 2023).
• Splits: train, validation, public test, and hidden test sets. The hidden test set (4.3% of cropped tables) is withheld to allow future leakage detection.
• Annotation formats: PASCAL VOC for object detection training, grid matrix format for GriTS, HTML for TEDS, JSON for cells and relations.
• Dataset license: CDLA-Permissive-2.0.

Evaluation

• GriTS variants: $\text{GriTS}_{\text{Top}}$ (cell topology), $\text{GriTS}_{\text{Con}}$ (cell content via normalized edit distance). Aggregated as pseudo-F1 weighted by cell count across all samples.
• TEDS and TEDS-S: tree-edit distance similarity on HTML table representation, generalized to sets via Hungarian matching.
• Exact match accuracy: $\text{Acc}_{\text{Top}}$ (topology) and $\text{Acc}_{\text{Con}}$ (content + topology).
• Graph metrics (POTATR): Edge F1 at IoU threshold 0.8 for relation prediction.
• No statistical significance testing or multi-run reporting. Single training run for each model configuration.
• TEDS computation for full documents has a 10-minute timeout; affected samples are bounded $[0, 1]$.

Hardware

• TATR and POTATR training: 8 Nvidia T4 GPUs. GPU-hours not reported.
• VLM inference: vLLM used for SmolDocling, GraniteDocling, and Qwen2.5-VL-3B. Inference hardware and latency not reported.
• Larger models (Qwen2.5-VL-7B) evaluated only qualitatively on a small number of documents due to computational cost.

BibTeX

@article{smock2025pubtablesv2,
  title={PubTables-v2: A new large-scale dataset for full-page and multi-page table extraction},
  author={Smock, Brandon and Faucon-Morin, Valerie and Sokolov, Max and Liang, Libin and Khanam, Tayyibah and Ramesh, Amrit and Courtland, Maury},
  journal={arXiv preprint arXiv:2512.10888},
  year={2025}
}

TL;DR

TRivia is a self-supervised fine-tuning framework that enables vision-language models (VLMs) to learn table recognition from unlabeled table images. It uses Group Relative Policy Optimization (GRPO) with table question-answering as a proxy reward, sidestepping the need for costly human annotations or proprietary model distillation. The resulting TRivia-3B, fine-tuned from Qwen2.5-VL-3B, outperforms Gemini 2.5 Pro and MinerU2.5 on three standard table recognition benchmarks.

What kind of paper is this?

Dominant: $\Psi_{\text{Method}}$: The core contribution is a self-supervised fine-tuning framework (TRivia) built around a training algorithm combining GRPO, response-consistency sampling, and attention-guided QA generation. The paper includes extensive ablations establishing which components drive performance.

Secondary: $\Psi_{\text{Evaluation}}$: The paper evaluates across four benchmarks (PubTabNet, OmniDocBench, CC-OCR, OCRBench v2), compares against 12 baselines spanning three model categories, and includes a robustness analysis to QA noise and complex table types.

What is the motivation?

Table recognition (TR), the task of converting table images into structured representations such as HTML or Markdown, is a core component of document parsing pipelines. Recent VLMs have pushed TR quality forward significantly, but the gap between open-source and proprietary systems remains large.

The root cause is data. Acquiring labeled TR data follows three established paradigms, each with a ceiling:

1. Synthetic data (rendered HTML tables) offers scale but misses real-world visual diversity.
2. Human-annotated real-world data captures complexity but is expensive and slow to produce.
3. Distillation from proprietary models (e.g., Gemini 2.5 Pro) is still costly, may violate service agreements, and hard-caps downstream performance at the teacher’s level.

Even large-scale open-source efforts like MinerU2.5, trained on millions of samples with manual annotations and Gemini 2.5 Pro distillation, cannot exceed Gemini’s own performance. This motivates a different question: can unlabeled table images from the wild, which are cheap to collect at scale, be used to train TR models that go beyond the limits of labeled data?

What is the novelty?

TRivia introduces a closed-loop self-supervised fine-tuning pipeline built around three components:

1. Table QA-driven GRPO

Rather than predicting full HTML markup as a supervised target, TRivia treats TR as a reinforcement learning (RL) problem. The TR model acts as the policy and generates multiple recognition responses $o_1, \ldots, o_R$ for each table image $I$. A separate LLM $M_{\text{LLM}}$ then answers table-specific questions based on each response, and the reward is defined as the average F1-score across the QA set:

$$\text{Reward}(o_j) = \frac{1}{|QA|} \sum_{(q,a) \in QA} \text{F1}(M_{\text{LLM}}(q; o_j),, a)$$

GRPO optimizes relative reward differences across the response group rather than chasing absolute scores, which means imperfect QA pairs tend to cancel out during within-group normalization. To prevent destabilization from invalid (illegal) recognition outputs, which receive zero reward by default, TRivia filters those responses out before computing advantages.

2. Response-consistency sampling

Not all unlabeled images contribute equally to learning. TRivia selects samples that cause the model to produce diverse recognition outputs, since GRPO benefits most from samples where relative advantages are informative. For each image $I$, the TR model generates $K$ recognition outputs $\{o_i\}_{i=1}^K$, and the consistency score is defined as the average pairwise TEDS similarity:

$$\text{Consistency}(I) = \frac{2}{K^2 - K} \sum_{1 \le i < j \le K} \text{TEDS}(o_i, o_j)$$

Images with lower consistency scores (more disagreement across responses) are prioritized for training. To keep the process practical, this sampling is done offline once rather than re-evaluated online. Images with consistency scores below 0.4 tend to be noisy or non-tabular and are discarded; the remaining images are uniformly sampled across score intervals from 0.4 to 1.0.

3. Attention-guided diverse QA generation

QA pairs serve as the supervisory signal, so they must cover different table regions and be verifiable. Naive single-pass prompting often covers only part of a table, while multi-pass prompting generates paraphrased duplicates from the same region. TRivia instead uses the VLM’s own attention weights during answer generation to identify the visual source of each QA pair:

$$VS((q,a);, I, M_{\text{QA}}) = \{v \mid A_{M_{\text{QA}}}(v \mid a) > \tau_A,; v \in \mathcal{V}\}$$

Candidate QA pairs are then selected greedily so that no two selected pairs share significant visual overlap (IOU below a threshold $\tau_{\text{IOU}}$). Before selection, each candidate is cross-checked by a separate VLM $M_{\text{Val}}$ to verify that it can be answered correctly with the image but not without it.

What experiments were performed?

Benchmarks. TRivia-3B is evaluated on four benchmarks:

• PubTabNet: academic papers with digital tables.
• OmniDocBench v1.5: 512 samples from digital PDFs with diverse table types.
• CC-OCR: 300 scanned/photographed images, including handwritten, long, and complex tables.
• OCRBench v2 (table parsing subset): 700 images with varied real-world layouts.

The primary metric is TEDS (Tree Edit Distance-based Similarity), which captures both content accuracy and structure. S-TEDS is a structure-only variant.

Baselines. The paper compares against 12 models in three categories:

• Expert TR models: SLANNet-plus, UniTable.
• General-purpose VLMs: InternVL3.5-241B, Qwen2.5-VL-72B, Qwen3-VL-235B, Gemini 2.5 Pro, GPT-4o, GPT-5.
• Document-parsing VLMs: dots.ocr, DeepSeek-OCR, PaddleOCR-VL, MinerU2.5.

Ablations. The paper ablates each of the three main TRivia components:

• Removing attention-guided QA generation significantly drops performance on structurally complex tables.
• Replacing response-consistency sampling with random sampling slows convergence and reduces final TEDS from 63.5 to 52.0 on the analysis set.
• Removing illegal-sample filtering destabilizes GRPO rewards, increasing convergence steps by approximately 25% and reducing final TEDS by 3 points.

The paper also compares the QA-based approach directly against distillation alternatives: using Qwen2.5-VL-72B pseudo-labels for SFT decreases TEDS by 8.37 on average, and GRPO with those same pseudo-labels still lags by 4.92 TEDS compared to TRivia.

What are the outcomes/conclusions?

TRivia-3B achieves 89.88 average TEDS across OmniDocBench, CC-OCR, and OCRBench v2, compared to 88.93 for Gemini 2.5 Pro and 86.82 for MinerU2.5. It is the top-performing model overall, with the only gap being on CC-OCR where Gemini 2.5 Pro scores 85.56 versus 84.90 for TRivia-3B.

Several secondary findings are notable:

• TRivia-3B annotations can be used to distill into a smaller Stage-2 model via standard SFT with nearly no performance loss (89.99 vs 89.88 overall TEDS), which contrasts sharply with the degraded results from distilling Qwen2.5-VL-72B pseudo-labels.
• The improvements are most pronounced on challenging table types: rotated tables gain +21.76 TEDS over the base Stage-2 model, with large gains on large tables and borderless merged-cell tables as well.
• TRivia is robust to moderate QA noise: injecting 20% incorrect QA pairs causes only limited degradation, explained by the relative reward normalization in GRPO canceling out uniformly wrong signals.
• A preliminary multi-task study shows TRivia is compatible with key information extraction (KIE), suggesting the approach may generalize beyond TR.

Limitations. The pipeline relies on large auxiliary models (Qwen2.5-VL-72B as $M_{\text{QA}}$ and InternVL3-78B as $M_{\text{Val}}$), which imposes significant compute cost during data preparation even though TRivia-3B itself is compact. The method is validated only on table recognition and one KIE task; broader applicability to other document understanding tasks remains to be demonstrated.

Reproducibility

Models

• TRivia-3B: fine-tuned from Qwen2.5-VL-3B-Instruct. 3 billion parameters. Model weights are released on HuggingFace at opendatalab/TRivia-3B (Apache-2.0). Training code is at opendatalab/TRivia (Apache-2.0). A Huggingface demo is available at opendatalab/TRivia-3B (Space).
• Visual encoder: supports image resolutions from $256 \times 28 \times 28$ to $1280 \times 28 \times 28$, yielding 256 to 1280 visual tokens per image.

Algorithms

Training proceeds in three stages:

Stage 3 hyperparameters: learning rate $2 \times 10^{-7}$ (ViT), $1 \times 10^{-6}$ (MLP + LM); GRPO group size $G = 16$; sampling temperature 1.2; sequence length 8192; constant LR scheduler.

• QA generation: Qwen2.5-VL-72B-Instruct as $M_{\text{QA}}$, attention layer 72, $\tau_A = 0.01$, $\tau_{\text{IOU}} = 0.3$.
• Validity cross-checking: InternVL3-78B as $M_{\text{Val}}$.
• Answer LLM: Qwen3-8B as $M_{\text{LLM}}$ during GRPO.

Data

• Stage 1: PubTabNet (200K), SynthTabNet (4 subsets, 100K each), MMTab (100K), converted to OTSL format.
• Stage 2: approximately 50K real-world samples from open-source datasets and web sources.
• Stage 3: 100K unlabeled PDF table images collected from web; after response-consistency filtering, 50K selected; QA generation yields a final dataset of 48,470 images with an average of 28.3 QA pairs each.
• Dataset construction details and prompt templates are provided in the supplementary appendix of the paper.

Evaluation

• Metrics: TEDS (full) and S-TEDS (structure only), both computed via tree edit distance against reference HTML.
• Benchmarks: OmniDocBench v1.5 (512 samples), CC-OCR (300 samples), OCRBench v2 table subset (700 samples), PubTabNet, FinTabNet.
• Inference: vLLM used for compatible models; temperature 0.2 for general-purpose VLMs to reduce repetitions; Gemini 2.5 Pro evaluated with thinking mode enabled.
• The authors acknowledge that PubTabNet and FinTabNet in-domain performance may drop after Stage 2 real-world adaptation, as those benchmarks are only present in Stage 1 training.

Hardware

• All experiments run on 8x A100 80GB GPUs.
• Stage 1: approximately 1 day. Stage 2: approximately 2 hours. Stage 3 (GRPO): approximately 2 days.
• TRivia-3B is designed for offline deployment and can run on hardware that fits a 3B parameter VLM.

BibTeX

@article{zhang2025trivia,
  title={TRivia: Self-supervised Fine-tuning of Vision-Language Models for Table Recognition},
  author={Junyuan Zhang and Bin Wang and Qintong Zhang and Fan Wu and Zichen Wen and Jialin Lu and Junjie Shan and Ziqi Zhao and Shuya Yang and Ziling Wang and Ziyang Miao and Huaping Zhong and Yuhang Zang and Xiaoyi Dong and Ka-Ho Chow and Conghui He},
  journal={arXiv preprint arXiv:2512.01248},
  year={2025}
}

TL;DR

This paper proposes an end-to-end benchmark for table extraction (TE) from scientific PDF documents. It introduces two new heterogeneous datasets (Table-arXiv: 36k samples from arXiv LaTeX sources; Table-BRGM: 124 tables from French geological reports), alongside a formally justified set of metrics that jointly evaluate table detection and structure recognition. Experiments across nine methods show that models trained on homogeneous collections like PubTables-1M suffer substantial performance drops on heterogeneous data, and that table extraction remains an unsolved problem.

What kind of paper is this?

Dominant: $\Psi_{\text{Evaluation}}$. The central contribution is a new benchmark framework: two new datasets, a formal metric design covering TD, TSR, and end-to-end TE, and a rigorous evaluation protocol that captures model uncertainty and cascading errors. The paper does not introduce a new model or training algorithm.

Secondary: $\Psi_{\text{Resource}}$. Table-arXiv and Table-BRGM are released as community resources. Their heterogeneity (diverse LaTeX typesetting, Word-processed geological reports, multilingual content) makes them useful for training and evaluation beyond the paper’s own experiments.

What is the motivation?

Table extraction from PDFs is a prerequisite for many downstream tasks: question answering over tables, data lake integration, and automated document analysis pipelines. Despite a large number of competing tools, choosing among them is difficult because existing evaluations have two systematic limitations.

First, prior benchmarks assess table detection (TD) and table structure recognition (TSR) as independent problems, each given a clean pre-cropped input. In practice, a TE pipeline chains these steps: the TSR model receives whatever the TD model detected. Errors in detection degrade structure recognition, but this cascade is invisible when the two tasks are scored separately.

Second, the dominant large-scale dataset, PubTables-1M, draws exclusively from PubMed Central biomedical articles produced with professional publishing systems. Models trained on this distribution learn typesetting conventions that do not transfer to LaTeX-compiled preprints from other domains, Word-processed reports, or documents in languages other than English. Benchmarks constructed from only this source inflate reported scores and mask generalization failures.

The authors address both limitations: they design an end-to-end evaluation protocol and construct heterogeneous test data that stress-tests out-of-distribution behavior.

What is the novelty?

Formal end-to-end metrics

The paper introduces new metrics that propagate table detection quality into the TSR score. For a model whose predictions include bounding boxes and optionally a confidence score $\tau$, the TE precision and recall are defined as:

$$ P_{\text{TSR}} = \frac{1}{|\mathcal{P}|} \sum_{\delta \in \mathcal{P}} \mathbb{1}_{\delta > \lambda} , B^{\text{TSR}}(\delta) \qquad R_{\text{TSR}} = \frac{1}{|\mathcal{G}|} \sum_{\delta \in \mathcal{G}} \mathbb{1}_{\delta > \lambda} , B^{\text{TSR}}(\delta) $$

where $\delta$ is the Jaccard index between a detected table and its ground-truth match, $\lambda$ is the matching threshold, and $B^{\text{TSR}}(\delta)$ is the TSR score (GriTS or TEDS) for that detected table. Missed detections receive $B^{\text{TSR}} = 0$, so a TD failure directly penalizes the end-to-end score.

For probabilistic models that output a confidence score $\tau_p$ per prediction, the paper additionally derives expected precision and recall by treating the Jaccard threshold as a random variable with a probability density function. Under the uniform density $S_0$, the expected value of the indicator $\mathbb{1}_{\lambda \geq \tau}$ for a prediction with Jaccard index $\delta$ is $2\delta$, providing a smooth, principled summary that avoids dependence on a single threshold choice.

Calibration of probabilistic models is measured with the object detection version of Expected Calibration Error (D-ECE):

$$ \text{D-ECE} = \sum_{i=1}^{M} \frac{|\mathcal{P}_i|}{|\mathcal{P}|} \left| \text{prec}_i - \text{conf}_i \right| $$

where predictions are binned by confidence and the gap between average precision and average confidence within each bin is accumulated.

Two new heterogeneous datasets

Table-arXiv is built from 2,443 LaTeX source files downloaded from arXiv, spanning the last 20 years and all arXiv subject areas. TD annotations are generated by instrumenting the LaTeX tabular environment to inject anchors that survive PDF compilation. TSR annotations (with cell content) are produced by converting the LaTeX source to HTML using LaTeXML. The result is 36,869 pages with 6,308 annotated tables, all from documents compiled with LaTeX and covering mathematics, physics, astrophysics, and other domains where PubTables-1M has no representation.

Table-BRGM is a small, manually annotated domain-specific set from geological survey reports published by BRGM, France’s national geological survey institution. It contains 6 PDFs, 499 pages, and 124 tables in English and French. Most documents were produced with Word processing software rather than LaTeX. Tables include large multi-column structures, borderless layouts, empty cells, and merged cells, representing challenges absent from PubTables-1M.

Both datasets include negative pages (pages with no tables). Prior benchmarks consisting only of positive instances inflate precision scores because models are never penalized for predicting tables on table-free pages. The new datasets expose this gap by including pages where false positives can occur.

Content-based table detection evaluation

For models like the LVLM (GPT-4o mini) that produce HTML tables without bounding boxes, the paper introduces a content-Jaccard metric based on character-level 2-grams. Given multisets $\mu(P)$ and $\mu(G)$ of character bigrams from predicted and ground-truth table cells:

$$ \text{Jaccard}_c(P, G) = \frac{|\mu(P) \cap \mu(G)|}{|\mu(P) \cup \mu(G)|} $$

This allows all methods to be evaluated for TD even when spatial coordinates are unavailable, using a content-based notion of whether a table was found.

What experiments were performed?

The benchmark evaluates nine end-to-end TE methods across three datasets.

Baseline methods (no confidence scores): PDFPlumber, PyMuPDF, Camelot (rule-based Python libraries); Grobid (ML-based CRF extraction pipeline); and GPT-4o mini used as a zero-shot LVLM given a page image and a prompt to return HTML tables.

Probabilistic methods (output confidence scores): Docling (RT-DETR for TD + TableFormer for TSR + EasyOCR for content); TATR-extract (Table Transformer detector + Table Transformer structure model, with a modified inference loop that retains low-confidence table predictions before NMS); XY+TATR-extract (XY-cut page segmentation prepended to TATR-extract to handle pages with multiple tables); and VGT+TATR-structure (Vision Grid Transformer for TD, which also uses text token grid information, combined with the same TATR-structure model).

Metrics reported include AP and F1 for TD, GriTS-Topology, GriTS-Content, and TEDS for TSR and end-to-end TE, and D-ECE for calibration of probabilistic models. Inference speed is measured as CPU seconds per image on 4 CPUs with 2 concurrent tasks.

What are the outcomes/conclusions?

No single method dominates across datasets. Docling achieves the best overall TD F1 scores across all three datasets (F1 of 0.99 on PubTables, 0.89 on Table-arXiv, 0.90 on Table-BRGM), benefiting from its RT-DETR model trained on the diverse DocLayNet dataset. Among probabilistic models, TATR achieves near-perfect AP on PubTables (1.00) but drops to 0.84 AP on Table-BRGM, and VGT produces better-calibrated confidence scores on heterogeneous data while achieving lower raw AP than TATR on PubTables.

Rule-based tools collapse on heterogeneous data. Camelot achieves F1 of 0.88 on Table-BRGM (where tables often have explicit grid lines) but only 0.25 on PubTables and 0.33 on Table-arXiv. PyMuPDF follows a similar pattern. These tools depend on visual lines and alignment cues that are absent from the LaTeX-typeset documents in PubTables and Table-arXiv.

PubTables-1M-specialized models do not generalize. TATR is trained on PubTables-1M and achieves near-perfect scores on the PubTables test set. On Table-arXiv and Table-BRGM, its scores drop substantially. Its confidence scores also become poorly calibrated: on Table-arXiv, predictions with 80-90% confidence scores correspond to fewer than 2% true positives on average, making the scores essentially uninformative.

End-to-end metrics are more pessimistic than subtask metrics. Once TD errors are propagated into TSR evaluation, all scores decrease. GriTS-Topology scores remain relatively high (tables that are detected tend to have their row/column structure recognized correctly), but GriTS-Content and TEDS drop more, because token extraction (via PDFAlto) misses stylized text like mathematical formulas, limiting content accuracy for TATR-structure-based models.

The LVLM hallucinates cell content. GPT-4o mini in zero-shot mode replaces capital-I characters with the digit 1, reformats numbers, fills in empty cells with fabricated values, and changes numeric values. Its content TD is accordingly low on Table-arXiv. It also achieves only modest F1 scores on TD across all datasets, consistent with prior reports that GPT-4o mini struggles with precise spatial localization.

Inference speed differs by an order of magnitude. Rule-based tools and Grobid require roughly 0.4-1.2 seconds per image. Docling, TATR, and XY require 9-13 seconds. VGT is substantially slower at 128 seconds per image due to its grid transformer pipeline.

The authors conclude that table extraction is not a solved problem. Heterogeneous data exposes model fragility that homogeneous benchmarks conceal, and the cascade from TD to TSR means that strong subtask numbers do not guarantee strong end-to-end quality.

Reproducibility

Models

No new model weights are introduced. The benchmark uses existing off-the-shelf models:

• TATR-detect and TATR-structure: Table Transformer models from Microsoft, trained on PubTables-1M. Available via the table-transformer repository and HuggingFace.
• VGT: Vision Grid Transformer, fine-tuned on PubLayNet with DiT-base ViT weights. Available via the VGT repository. Uses BERT tokenizer with LayoutLM word embeddings.
• Docling: Available as a Python library. Uses RT-DETR (trained on DocLayNet) for TD and TableFormer for TSR.
• Grobid: Available as an open-source library.
• GPT-4o mini: Accessed via OpenAI API in zero-shot mode with the prompt shown in Appendix A.2 of the paper.

Algorithms

The benchmark evaluation code is available at the GitLab repository listed in the artifacts. The authors state that their pipelines were built starting from the microsoft/table-transformer repository. AP is computed with Scikit-learn’s average_precision_score (area under the full precision-recall curve, following the sklearn convention rather than the Pascal VOC interpolation).

No training is performed. Key pipeline parameters:

• TATR-extract retains all predictions with confidence above 0.05 for “table” or “table rotated” (modifying the original inference which kept only the top-1 label), then applies NMS.
• XY+TATR-extract runs XY-cut recursive page segmentation (counting black pixels along both axes) before TATR-detect; each chunk retains only the top 2 predictions.
• Table padding for TATR-structure crops: $\delta_{\text{pxs}} = 100$ pixels added around detected tables; $\delta_{\text{pxd}} = 10$ pixels padding for XY-cut sub-images.
• Content-Jaccard matching threshold for LVLM TD evaluation: $\iota = 0.5$.
• Camelot uses the “lattice” mode (demarcated cell lines), which outperformed other modes in this evaluation.

Data

Table-arXiv: 2,443 LaTeX source files from arXiv, sampled from all subject domains over 20 years. TD annotations are generated automatically by instrumenting LaTeX compilation; TSR annotations are produced by LaTeXML. Contains 36,869 pages, 5,214 positive pages, 6,308 annotated tables. All documents are LaTeX-compiled.

Table-BRGM: 6 PDF documents from BRGM geological reports (French and English). 499 pages, 91 positive pages, 124 tables. Manually annotated; an end-to-end model was used as a starting point and outputs were manually corrected. Mix of Word-processed (67%) and LaTeX (33%) documents.

PubTables-Test: Subset of the PubTables-1M test set for which PDF files could be retrieved from PubMed Central (Open Access subset). 23,175 PDFs, 46,942 pages (all positive), 55,990 annotated tables. Ground-truth HTML markup was generated from the original dataset annotations. Note that not all PubMed PDFs were available; the subset used here covers only papers with valid retrievable PDFs. The PubTables-1M original annotations are available at the table-transformer repository; the reconstructed HTML ground truth and the PDF subset used are provided via the benchmark GitLab repository.

Datasets are available at https://gitlab.inria.fr/msoric/table-extraction-benchmark under MIT license.

Evaluation

Metrics used:

• TD: Precision, Recall, F1, and AP at IoU threshold $\lambda = 0.5$; expected Precision/Recall (integral over $\lambda$ weighted by density $S_{0.5}$); D-ECE for calibration.
• TSR: GriTS-Topology, GriTS-Content, TEDS computed on HTML markup normalized to only <table>, <tr>, <td> tags (header/body distinctions stripped for comparability).
• End-to-end TE: $P_{\text{TSR}}$ and $R_{\text{TSR}}$ with each of the three TSR scores; AP versions for probabilistic models and F1 versions for baseline models.

TSR evaluation is conditioned on TD: only true-positive detections (IoU $\geq 0.5$) are passed to the structure model. Missed tables receive a score of 0 in the end-to-end aggregation. This means TSR scores cannot be compared directly across models because each model evaluates on a different subset of detected tables.

The benchmark does not report statistical significance tests or error bars. Each model is run once. LVLM results carry a contamination caveat: GPT-4o mini may have been trained on benchmark datasets.

Hardware

Inference is measured on CPU (4 CPUs per task, 2 tasks concurrently). No GPU-hours or training hardware are relevant. Approximate inference times per image: PDFPlumber 0.35s, PyMuPDF 0.41s, Grobid 0.78s, Camelot 1.18s, LVLM 8.88s (API), XY+TATR 9.32s, Docling 11.36s, TATR 13.33s, VGT 127.98s.

BibTeX

@article{soric2025benchmarking,
  title={Benchmarking Table Extraction from Heterogeneous Scientific Documents},
  author={Soric, Marijan and Manolescu, Ioana and Gracianne, C\'{e}cile and Senellart, Pierre},
  journal={arXiv preprint arXiv:2511.16134},
  year={2025}
}

TL;DR

TABLET is a split-merge TSR model that treats row and column splitting as 1D sequence labeling and cell merging as OTSL-based grid cell classification, using only Transformer encoders throughout. Trained on FinTabNet and PubTabNet, it reaches 98.54 TEDS on FinTabNet while processing 18 FPS end-to-end, roughly 2.5 times faster than the prior fastest reported split-merge method.

What kind of paper is this?

Dominant: $\Psi_{\text{Method}}$ The headline contribution is the TABLET architecture: dual Transformer encoders for split sequence labeling and a third Transformer encoder for merge classification. The paper devotes most pages to the model design, ablations on encoder depth, and benchmark comparisons on FinTabNet and PubTabNet.

Secondary: None. No dataset, benchmark, or tool is released alongside the method.

What is the motivation?

Table structure recognition in production environments faces two competing pressures: accuracy and throughput. End-to-end autoregressive approaches (e.g., TableMaster, TableFormer, VAST, TFLOP) generate HTML or OTSL sequences token by token, which gives them a natural advantage on TEDS because TEDS itself evaluates cell-level accuracy in a way that mirrors their output format. However, autoregressive decoding is inherently sequential, making it slow on large, densely populated tables. For financial documents (annual reports, earnings announcements), tables often have dozens of rows and columns with small fonts, creating long output sequences that slow autoregressive methods further.

The authors also observe that for end-to-end methods, predicted cell bounding boxes frequently misalign with OCR-extracted text positions, causing errors in the final TEDS evaluation even when structure is correct. This TEDS-Struct vs. TEDS gap is visible in the paper’s comparison tables (e.g., Zhu et al. show a gap exceeding 2 points; Ly et al. exceeding 3 points).

Top-down split-merge methods avoid bounding box prediction and matching entirely, but existing approaches (TSRFormer, SEMv2, SEMv3) rely on Spatial CNN feature extractors or RNN-based sequence models that either introduce complexity or lag behind Transformer-based alternatives. The paper’s goal is a split-merge model that uses only Transformer encoders while maintaining a concise, high-resolution feature pipeline suited for dense financial tables.

What is the novelty?

Overall Architecture

TABLET is a two-stage pipeline. A split model segments the table into an $R \times C$ grid; a merge model then classifies each grid cell to reconstruct spanning cells. The two models are trained separately.

Before inference, all table images are rescaled so that the longer side is 960 pixels (preserving aspect ratio) and padded to $960 \times 960$.

Split Model

A modified ResNet-18 (Max Pooling removed, all channel counts halved) combined with an FPN at 128 channels produces a feature map $F_{1/2}$ of size $(H/2) \times (W/2) \times 128$, half the input resolution. Removing Max Pooling preserves fine-grained spatial detail that matters for small-font, dense tables.

Global and local features are extracted along each axis independently:

• Horizontal (row splitting): Global projection along the horizontal axis yields $F_{RG}$ of shape $(H/2) \times 128$. Local features apply Average Pooling ($1 \times 2$) then $1 \times 1$ convolution to obtain $F_{RL}$ of shape $(H/2) \times (W/4)$. Concatenation gives $F_{RG+L}$ of shape $(H/2) \times (128 + W/4)$.
• Vertical (column splitting): The same procedure along the vertical axis produces $F_{CG+L}$ of shape $(W/2) \times (128 + H/4)$.

Each feature map is fed into its own Transformer encoder (3 layers, 8 heads, $d_{ff} = 2048$, dropout 0.1). The fixed input sequence lengths are $H/2 = 480$ and $W/2 = 480$; positional embeddings are 1D, randomly initialized, and learned end-to-end.

Each encoder output position is passed through a linear head for binary classification (split vs. non-split line). Training uses Focal Loss:

$$ FL_{\text{split}} = \frac{1}{n_h} \sum_{i=1}^{n_h} \alpha_i (1-p_i)^\gamma (-\log p_i) + \frac{1}{n_v} \sum_{j=1}^{n_v} \alpha_j (1-p_j)^\gamma (-\log p_j) $$

with all class weights $\alpha = 1$ and focusing parameter $\gamma = 2$. Predictions are upsampled 2$\times$ to match input resolution. A post-processing rule reclassifies any non-split region containing no OCR text projection as a split region, handling empty rows and columns.

Merge Model

A standard ResNet-18 + FPN (256 channels) extracts a feature map $F_{1/4}$ at $(H/4) \times (W/4) \times 256$. Grid cells from the split output are mapped to $F_{1/4}$ via RoIAlign ($7 \times 7$), yielding a feature tensor $F_{\text{grids}}$ of shape $R \times C \times (7 \times 7 \times 256) = R \times C \times 12544$.

A two-layer MLP (Linear + ReLU, both layers 512-dimensional) compresses each cell to 512 dimensions, forming a sequence $S_{\text{grids}}$ of length $R \times C$. A Transformer encoder (3 layers, 8 heads, $d_{ff} = 2048$, dropout 0.1, maximum sequence length 640) with learnable 2D positional embeddings models cross-cell interactions. A linear classification head then assigns each grid cell one of four OTSL labels:

• C: new cell (may contain content)
• L: left-looking span (merges with the cell to its left)
• U: up-looking span (merges with the cell above)
• X: cross span (merges with both left and upper neighbors)

The NL (newline) token from OTSL is unnecessary in a split-merge context and is not used. Merging again uses Focal Loss:

$$ FL_{\text{merge}} = \frac{1}{R \times C} \sum_{k=1}^{R \times C} \alpha_k (1-p_k)^\gamma (-\log p_k) $$

with $\alpha = 1$, $\gamma = 2$.

The OTSL output is converted to HTML, and OCR-extracted text blocks are assigned to cells by position.

What experiments were performed?

Datasets

• FinTabNet v1.0.0: 91,596 train / 10,656 val / 10,635 test tables from financial announcements. A manually refined subset of 9,681 test tables (Hou et al., same authors) with corrected annotations is also used.
• PubTabNet v2: 500,777 train / 9,115 val tables from scientific articles. Note: PubTabNet does not provide original PDF files, so OCR is required to extract cell text.

Metrics

TEDS (content-inclusive), TEDS-Struct (structure only, ignoring cell content), and Accuracy (proportion of tables with fully correct structure and content).

Ablations

Tables 2 and 3 in the paper sweep the Transformer encoder depth (0, 1, 3, 6 layers) independently for the split and merge models. The 3-layer configuration consistently achieves the best results; deeper encoders (6 layers) degrade performance, suggesting the task does not benefit from additional capacity at this scale.

Speed Comparison

Inference speed is measured end-to-end (image resizing, Split-Merge inference, OCR text matching, HTML post-processing) on a single NVIDIA A100 80GB GPU at FP32.

The authors also benchmark Qwen2.5-VL-7B on FinTabNet: zero-shot TEDS is more than 10 points below TABLET; after fine-tuning, the gap narrows to roughly 2 points, but FPS is over 100$\times$ slower.

What are the outcomes/conclusions?

FinTabNet (test set): TEDS Simple/Complex/All = 98.97 / 98.14 / 98.54; TEDS-Struct = 99.10 / 98.35 / 98.71; Accuracy = 88.18%. The authors report this as the highest TEDS among listed methods with image-only input. The near-zero gap between TEDS and TEDS-Struct (0.17 points) confirms that bounding box misalignment is absent for split-merge methods.

FinTabNet (refined test set): TEDS = 98.87; TEDS-Struct = 98.99; Accuracy = 90.00%.

PubTabNet (val set): TEDS = 96.79; TEDS-Struct = 97.67. Competitive but below DRCC (97.80 TEDS / 98.90 TEDS-Struct). The authors note that DRCC’s semi-autoregressive approach is less sensitive to resolution than pixel-classification split-merge methods, which partly explains the gap.

Error analysis (FinTabNet): The authors identify five failure modes: adjacent columns with overlapping text projections that prevent correct splitting; misaligned column headers; multi-line single-cell text mistakenly split into multiple rows; empty cell annotation inconsistencies; and cases where the model’s output is actually correct but the FinTabNet annotation is wrong.

Acknowledged limitations: The method assumes tables have already been extracted and dewarped; it does not handle raw document pages, distorted images, or tables in natural scenes. It depends on an OCR engine as preprocessing, so OCR quality directly affects final TEDS scores (especially visible on PubTabNet, where different methods use different OCR tools). The merge model sequence length is capped at 640 grid cells; very large tables that produce more grid cells than this are truncated.

Unacknowledged gaps: No code or model weights are released. No error bars or multi-run statistics are reported. The Accuracy metric (fully correct tables) is only reported on FinTabNet, not PubTabNet. The comparison table on FinTabNet is incomplete (several methods report only TEDS-Struct, omitting TEDS), making direct comparison across all metrics impossible.

Reproducibility

No code, pretrained weights, or evaluation scripts were released with this paper.

Models

• Split model: Modified ResNet-18 (no MaxPool, half channels) + FPN (128 channels); two Transformer encoders (3 layers, 8 heads, $d_{ff} = 2048$, dropout 0.1); 16.1M parameters total.
• Merge model: Standard ResNet-18 + FPN (256 channels); RoIAlign $7 \times 7$; 2-layer MLP (512-dim); Transformer encoder (3 layers, 8 heads, $d_{ff} = 2048$, dropout 0.1); 32.5M parameters total.
• Total: approximately 48.6M parameters.
• No pretrained weights or code repository released.

Algorithms

• Optimizer: AdamW ($\beta_1 = 0.9$, $\beta_2 = 0.999$, $\epsilon = 10^{-8}$, weight decay $5 \times 10^{-4}$).
• Learning rate: $3 \times 10^{-4}$ for the split model (constant); polynomial decay schedule (power 0.9) for the merge model.
• Gradient clipping: L2 norm with max_norm = 0.5.
• Batch size: 32.
• Training epochs: 16 (split model), 24 (merge model).
• Focal Loss: $\alpha = 1$, $\gamma = 2$ for both split and merge.
• Input resolution: 960$\times$960 (proportional resize + zero-padding).
• Split region annotation preprocessing: regions narrower than 5 pixels are expanded symmetrically to a minimum of 5 pixels.

Data

• FinTabNet v1.0.0: 91,596 train / 10,656 val / 10,635 test. Available under CDLA-Permissive-1.0 from IBM Research.
• PubTabNet v2: 500,777 train / 9,115 val. Available under CDLA-Permissive-1.0. PubTabNet does not provide original PDFs; cell text must be obtained with an OCR tool.
• Refined FinTabNet test set (9,681 tables): released by the same authors in prior work (Hou et al.); exact availability not specified in this paper.
• No new datasets are introduced or released.

Evaluation

• TEDS and TEDS-Struct computed over all tables, split into simple (no spans) and complex (at least one spanning cell) subsets.
• Accuracy = percentage of tables with fully correct structure and content.
• Baselines vary in what they report; direct comparison on all three metrics is not possible for most methods.
• No error bars, significance tests, or multiple training runs reported.

Hardware

• Training: 2$\times$ NVIDIA A100 80GB GPUs.
• Inference: 1$\times$ NVIDIA A100 80GB GPU; 18.01 FPS end-to-end (960$\times$960 input, full pipeline including OCR text matching and HTML post-processing).
• No training time, total GPU-hours, or memory breakdown reported.

BibTeX

@inproceedings{hou2025tablet,
  title={TABLET: Table Structure Recognition using Encoder-only Transformers},
  author={Hou, Qiyu and Wang, Jun},
  booktitle={Proceedings of the International Conference on Document Analysis and Recognition (ICDAR)},
  year={2025}
}

TL;DR

OG-HFYOLO extends YOLOv5-based instance segmentation with three new modules (a gradient orientation extractor, a heterogeneous kernel cross-fusion block, and a scale-aware loss) plus mask-driven NMS, targeting the problem of accurately localizing cells in physically deformed table images. The authors also construct DWTAL, a large-scale synthetically augmented dataset of deformed wired tables with pixel-level segmentation annotations, using a custom wave and cylindrical warping generator applied to existing datasets.

What kind of paper is this?

Dominant: $\Psi_{\text{Method}}$ The paper’s primary contribution is a new model architecture with several interacting modules. The bulk of the content covers architectural design choices, loss function derivations, and ablation experiments validating each component.

Secondary: $\Psi_{\text{Resource}}$ The paper also releases DWTAL, a dataset of 28,285 images across two difficulty tiers (DWTAL-s and DWTAL-l) with instance segmentation annotations for deformed table cells. This is a meaningful contribution, since no prior public dataset offered pixel-level annotations for this task.

What is the motivation?

Physical deformations in photographed or scanned tables (bending, perspective distortion, page curl from binding) disrupt the alignment between cell content and structure. Downstream tasks such as content extraction and OCR depend on accurate spatial coordinate localization of individual cells, but most existing table structure recognition methods assume near-planar inputs.

Prior approaches to cell coordinate localization fall into two categories. Contour-based detection methods lose critical information when deformations are severe. Text box segmentation methods handle non-deformed wired tables but struggle with geometric distortion. Keypoint detection partially addresses the problem but corner regression accuracy degrades under heavy curvature. Instance segmentation offers pixel-level boundary descriptions that can represent irregular, curved cell shapes, but two practical barriers remained: (1) existing datasets lacked the fine-grained segmentation annotations needed to train such a model, and (2) instance segmentation methods had not been adapted to the specific challenges of deformed table cells: dense object arrangements, extreme aspect ratio variation from merged cells, and overlapping bounding boxes from complex cell shapes.

What is the novelty?

The paper introduces four technical contributions that work in concert:

Gradient Orientation-aware Extractor (GOE). Placed after the first downsampling convolution in the backbone, GOE applies separate horizontal and vertical decoupling operators $G_x$ and $G_y$ (initialized as edge operators, then trained freely) to the input feature map. The gradient magnitude and gradient direction are computed as:

$$I_{GM} = \sqrt{G_x^2 + G_y^2}$$

$$I_{GD} = \text{Cat}(G_x, G_y)$$

An orientation attention mechanism $\Phi$, inspired by HOG’s orientation binning, initializes its $1 \times 1$ convolutional kernels from polar directional basis vectors:

$$\mathbf{e}_i = \begin{bmatrix} \cos \theta_i \ \sin \theta_i \end{bmatrix}, \quad \theta_i = \frac{i\pi}{n}$$

These are assembled into a weight matrix $W \in \mathbb{R}^{n \times 2 \times 1 \times 1}$ mapping gradient components to $n$ orientation channels. The final output combines both cues:

$$I_o = \text{IN}(\text{Softmax}(\Phi(I_{GD}))) \odot I_{GM}$$

where IN denotes instance normalization. This gives the network a geometric prior about edge directionality that standard convolutions lack.

Heterogeneous Kernel Cross Fusion (HKCF). Inserted after skip connections in the FPN-PAN neck, HKCF addresses the extreme width and height variation of merged table cells. It uses a bottleneck to reduce channel count, then runs parallel $1 \times k$ and $k \times 1$ asymmetric convolutions (horizontal and vertical cross-convolutions) whose outputs are summed:

$$HXConv^{(k)}(I) = HConv^{(k)}(I) + VConv^{(k)}(I)$$

A Channel Attention Bridge (CAB) before the cross-convolutions compensates for information loss from dimensionality reduction. Following the Heterogeneous Kernel Select Protocol from YOLO-MS, kernel sizes 3, 5, and 7 are applied at progressively deeper fusion stages.

Scale-aware loss function. The standard YOLO mask normalization divides by instance area $A$, but $1/A$ has a second derivative $2/A^3$ that causes near-vertical gradient growth for very small cells. The authors introduce a composite weighting term $W_s = 1 + \log(1/A)$, yielding:

$$L_{\text{scale-aware}} = \frac{1}{n} \sum_{i=1}^{n} \left[ \left(1 + \log \frac{1}{A_i}\right) \frac{1}{A_i} \sum_{(x,y) \in \text{crop}(\Omega)} \frac{1}{H’_i W’_i} L_{\text{base}}(x,y) \right]$$

where $L_{\text{base}} = L_{\text{BCE}} + L_{\text{Dice}}$. The $\log(1/A)$ term moderates the curvature of compensation, so gradient magnitudes scale more smoothly across the object size distribution.

Mask-driven NMS. Standard bounding-box IoU NMS incorrectly suppresses valid cells when deformed bounding boxes substantially overlap. The proposed replacement computes IoU directly on predicted binary masks:

$$\text{Mask_IoU} = \frac{|M_i \cap M_j|}{|M_i \cup M_j|}$$

suppressing a lower-confidence mask only when this pixel-level overlap exceeds the threshold.

DWTAL dataset. The authors construct a data generator applying wave warping (sine/cosine-based, amplitude $A \in [10, 50]$) and cylindrical warping (cosine-based along the y-axis) to mildly deformed source images from TAL-OCR and WTW, with an illumination adjustment stage. The result is two datasets: DWTAL-s (8,765 images, simpler backgrounds from TAL-OCR) and DWTAL-l (19,520 images, more complex backgrounds from WTW), both with pixel-level instance segmentation annotations.

What experiments were performed?

All experiments used a single NVIDIA RTX 3090 (24 GB). For ablation studies, no pretrained weights were used. For comparison experiments against non-YOLO models, ResNet-101 backbones were initialized from ImageNet-pretrained MSRA weights via MMDetection.

Training settings: SGD optimizer, momentum 0.9, learning rate 0.001, weight decay 0.0005, $640 \times 640$ input, batch size 2 (batch size 1 for DWTAL-l comparisons), 200 epochs (ablation) and 100 epochs (comparisons).

Baselines spanned two-stage models (Mask R-CNN, Cascade Mask R-CNN), single-stage models (SOLOv2, YOLACT), Transformer-based segmentation (Mask2Former), and YOLO variants (YOLOv5l-seg, YOLOv8l-seg, YOLOv11l-seg).

Evaluation metrics: mask mAP@50, mask mAP@50:95, bounding box mAP@50, bounding box mAP@50:95, parameter count, and GFLOPs.

Ablation studies isolated each of the four proposed components, examining individual and joint contributions on DWTAL-s. A separate backbone ablation compared the YOLOv5 C3 baseline against substituting YOLOv11’s C3k2 and C2PSA components. An anchor mechanism ablation confirmed that anchor-based detection substantially outperforms anchor-free variants on this task.

What are the outcomes/conclusions?

On DWTAL-s, the full OG-HFYOLO model achieves mask mAP@50:95 of 74.23%, compared to 71.96% for YOLOv5l-seg, 64.4% for Mask2Former, 62.5% for Mask R-CNN, 57.5% for YOLOv8l-seg, and 57.8% for YOLOv11l-seg. On DWTAL-l, OG-HFYOLO reaches mask mAP@50:95 of 62.38% versus 61.34% for YOLOv5l-seg (the next-best on this metric) and 44.7% for Mask R-CNN.

The ablation results indicate that each module delivers modest gains in isolation (GOE alone: +0.44% mask mAP@50:95; HKCF alone: +0.09%; scale-aware loss alone: +0.48%) but their joint use yields substantially larger improvements. Mask-driven NMS provides an additional +0.94% increase when added on top of the three structural modules, and is identified as the single component with the largest individual contribution in the combined setting. The authors note that the challenges in this dataset are interdependent, so the modules benefit from each other.

The backbone ablation found that the simpler YOLOv5 C3 architecture outperforms YOLOv11-style components within the same training setup, which the authors take as evidence that the proposed modules are more important than the backbone choice.

Anchor-based detection shows a greater than 10% absolute improvement over anchor-free variants on both datasets, suggesting that the dense, regular structure of table cell arrangements favors anchor-based priors.

Qualitative results on natural scene photographs and the CamCap dataset (using a model trained exclusively on DWTAL-l) suggest some cross-domain generalization, though the evaluation is limited to visual examples rather than held-out metrics.

Limitations. The datasets are entirely derived from two source datasets (TAL-OCR and WTW), so the diversity of deformation types and backgrounds is constrained by those sources. All images contain only a single table instance. Downstream logical coordinate recovery is left as future work. Statistical reporting (standard deviations, multiple runs) is absent. The model’s parameter count (125.39 M) is higher than several baselines, which may create deployment constraints.

Reproducibility

Models

OG-HFYOLO is built on a modified YOLOv5 backbone (CSPDarknet with C3 modules) augmented by GOE after the first downsampling layer. The neck uses FPN-PAN with HKCF blocks replacing standard skip-connection convolutions. The detection head retains YOLOv5’s anchor-based design. Total parameters: 125.39 M. GFLOPs: 170 (at $640 \times 640$).

Code is released under AGPL-3.0 at https://github.com/justliulong/OGHFYOLO. The model architecture is defined in cfg/models/segment/og-hfyolo.yaml. A demonstration checkpoint (best.pt) is available via GitHub releases (v1.0.2). No fully reproduced checkpoint from the paper’s complete training runs is separately documented; the released weight is a demonstration model. Environment setup is provided via requirements.txt or environment.yaml (Python >= 3.8, PyTorch >= 1.8).

Algorithms

• Optimizer: SGD, momentum 0.9
• Learning rate: 0.001 (schedule not specified)
• Weight decay: 0.0005 (ablation), 0.0001 (comparison experiments)
• Batch size: 2 for DWTAL-s; 1 for DWTAL-l
• Training epochs: 200 (ablation), 100 (comparison)
• Input resolution: $640 \times 640$
• Bounding box regression: EIoU loss replacing CIoU
• Mask loss: $L_{\text{base}} = L_{\text{BCE}} + L_{\text{Dice}}$, scaled by $(1 + \log(1/A)) \cdot (1/A)$
• Post-processing: Mask IoU NMS replacing bounding-box IoU NMS
• GOE orientation bins: $n$ (value not stated explicitly in the paper)
• HKCF kernel sizes: 3, 5, 7 at three FPN levels

Data

• DWTAL-s: 8,765 images total; 7,012 train / 1,753 test. Primarily derived from TAL-OCR with 150 collected images. Simpler backgrounds.
• DWTAL-l: 19,520 images total; 15,616 train / 3,904 test. Primarily derived from WTW. More complex backgrounds.
• 80/20 train/test split, stratified by deformation type.
• Labels are pixel-level instance segmentation masks per table cell.
• A logical coordinate annotation version has also been released (via GitHub releases, without images; images must be downloaded separately).
• DWTAL is distributed in two formats. YOLO format is hosted on Google Drive (separate links for DWTAL-s and DWTAL-l). COCO format is hosted on Hugging Face at justliulong/DWTAL and can be downloaded via the datasets library. The data generator code lives in the GitHub repo at dataprocess/data_gen.py. Labelme-format annotation archives are also available on Google Drive.
• The underlying source datasets (TAL-OCR and WTW) have their own licenses; the paper does not state an explicit license for DWTAL beyond referencing the AGPL-3.0 repo and describing it as open-source.

Evaluation

• Primary metrics: mask mAP@50 and mAP@50:95 (COCO-style AP over 101 recall points)
• Secondary metrics: bounding box mAP@50 and mAP@50:95
• Non-YOLO baselines: ResNet-101 backbone, pretrained on ImageNet, fine-tuned 100 epochs via MMDetection
• No error bars, significance tests, or repeated runs are reported
• Comparison against Mask2Former uses ResNet-101 backbone for consistency with other non-YOLO models; a ViT-based Mask2Former is not evaluated
• CamCap generalization evaluation is qualitative only

Hardware

• Training hardware: NVIDIA RTX 3090, 24 GB VRAM
• Framework: Python 3.8.19, PyTorch 1.13.0, CUDA 12.4
• Training time and energy consumption are not reported
• Inference speed is discussed qualitatively (OG-HFYOLO described as retaining single-stage speed) but no concrete latency numbers are given

BibTeX

@article{liu2025oghfyolo,
  title={OG-HFYOLO: Orientation Gradient Guidance and Heterogeneous Feature Fusion for Deformation Table Cell Instance Segmentation},
  author={Long Liu and Cihui Yang},
  journal={arXiv preprint arXiv:2504.20682},
  year={2025}
}

TL;DR

CISOL (Construction Industry Steel Ordering Lists) is a medium-sized, human-annotated dataset of 844 scanned civil engineering documents with over 120,000 annotated instances for table detection (TD) and table structure recognition (TSR). All documents are anonymized real-world steel ordering lists in German, sourced from 10 structural engineering firms. The dataset is released under CC-BY-4.0 on Zenodo and is accompanied by a public evaluation server. Benchmarking with YOLOv8 reaches 67.22 mAP@0.5:0.95:0.05, outperforming the TSR-specific Table Transformer (TATR) on this domain, while both results remain below the annotation-derived convergence threshold.

What kind of paper is this?

Dominant: $\Psi_{\text{Resource}}$. The primary contribution is the dataset itself: a new, openly licensed, human-annotated collection of real-world civil engineering documents that fills a gap in available table extraction data. The paper documents a thorough creation pipeline, an annotation guideline, a stratified splitting strategy, and a long-term public evaluation server. The dataset release is the headline novelty.

Secondary: $\Psi_{\text{Evaluation}}$. The authors measure annotation consistency using Krippendorff’s $\alpha$ and derive a label convergence threshold that defines a theoretical upper bound on model performance. They then benchmark YOLOv8 and TATR against this bound, providing a measurement framework as well as a dataset.

What is the motivation?

Table extraction from scanned documents is a long-standing problem in document analysis. Most publicly available table extraction datasets come from born-digital government documents, scientific articles, or financial reports. Civil engineering documents, such as steel ordering lists, have a distinctive structure: large tables with spanning cells, embedded tables, technical drawings, and highly domain-specific content. No dataset for this sub-domain existed at the time of writing.

The authors also identify a broader problem in dataset creation: inconsistent annotation conventions, opaque collection pipelines, inadequate licensing, and poor extensibility. Prior human-annotated datasets such as ICDAR 2013, WTW, and TabRecSet lack metadata that would allow others to replicate or extend the annotation process. CISOL is designed from the outset to address these documentation gaps by following the data development lifecycle described by Hutchinson et al., the datasheet framework of Gebru et al., and the FAIR principles (Findability, Accessibility, Interoperability, Reusability).

What is the novelty?

The novelty is the CISOL dataset itself and the process used to create it. Key properties that distinguish it from prior human-annotated TSR datasets include:

1. Domain specificity: all documents come from real construction industry projects, a domain not previously represented in TSR benchmarks.
2. Anonymized real-world data: raw documents from 10 structural engineering firms (24 projects, creation dates 2015-2023) were anonymized via an automated pass (replacing names, locations, and personal data with length-preserving substitutions) followed by manual review. This allowed real industry data to be licensed under CC-BY-4.0.
3. Rich metadata: each document image is tagged with table size (XS/S/M/L by cell count) and table type (open/closed/mixed by separator line usage), plus provenance metadata. This metadata enables stratified sampling and is available to downstream researchers.
4. Extensible annotation pipeline: the annotation guideline and the full pipeline (using the publicly available CVAT tool) are released, making it straightforward for others to add new documents or sub-domains without writing a new guideline from scratch.
5. Physical structure focus: the dataset annotates physical structure (column, row, spanning cell, and header bounding boxes) rather than logical structure. This design choice makes downstream heuristic inference of logical structure simpler, supports embedded table recognition extensions, and reduces the need for speculative inference during annotation.
6. Annotation quality measurement and convergence threshold: 20 images were re-annotated by all four annotators. Krippendorff’s $\alpha$ ($K\text{-}\alpha$) was computed at multiple IoU thresholds and converted to mAP to yield a label convergence threshold, i.e., a theoretical ceiling on model performance given the annotation variance. The authors interpret $K\text{-}\alpha > 0.8$ as reliable agreement and $K\text{-}\alpha > 0.667$ as moderate agreement; CISOL achieves the former at IoU 0.5 and the latter at the aggregate $0.5\text{:}0.95\text{:}0.05$ range. The conversion from $K\text{-}\alpha$ to equivalent mAP follows the formula from Tschirschwitz and Rodehorst (WACV 2025). The convergence threshold sits above the best benchmark result, indicating the problem is not yet solved.

What experiments were performed?

The paper benchmarks two models on CISOL: YOLOv8 (a general-purpose object detector) and TATR (Table Transformer, a TSR-specific model). Both models are trained on the CISOL training split and evaluated on the test split using the COCO mAP metric at IoU thresholds $0.5\text{:}0.95\text{:}0.05$, the same metric used in the PubTabNet ICDAR 2021 Challenge Track A.

The primary reported result is:

The paper reports that YOLOv8 outperforms TATR on this dataset and attributes this to TSR-specific models generalizing poorly to new domains, consistent with a prior reproducibility study by Ajayi et al. Exact per-model and per-class numbers are not included in the paper; the authors direct readers to the public evaluation server at EvalAI for the current best results.

Annotation consistency is evaluated as a separate experiment. The four annotators each labeled the same 20 randomly selected images, and $K\text{-}\alpha$ was computed at IoU thresholds $0.5$, $0.75$, and $0.5\text{:}0.95\text{:}0.05$. The $K\text{-}\alpha$ values are converted to equivalent mAP values and plotted alongside benchmark results.

Dataset statistics are also characterized and compared to seven existing human-annotated TSR datasets (ICDAR 2013, ICDAR 2019 Track-B2, UNLV, UW3, TabRecSet, TUCD, WTW) on instance count, image count, class distribution, and annotation density. CISOL falls in the medium-size range and is distinguished by its larger average table size (more rows and columns per table) and the provision of metadata.

What are the outcomes/conclusions?

The CISOL dataset provides a functional benchmark for TSR in a domain not previously covered. Key findings reported by the authors include:

• YOLOv8 achieves 67.22 mAP@0.5:0.95:0.05 on CISOL, outperforming TATR despite TATR being designed specifically for TSR. Results suggest that domain shift remains a significant challenge for TSR-specific models, and that general-purpose detectors can be competitive on out-of-distribution data.
• Annotation quality is moderate to high: $K\text{-}\alpha > 0.8$ at IoU 0.5 and $K\text{-}\alpha > 0.667$ at the aggregate IoU range. This indicates that the annotation guideline was sufficiently clear and that the labeling process was reproducible.
• The best benchmarked model remains below the convergence threshold, leaving room for further model improvement specific to this domain.
• Embedded tables are present in the raw data but were intentionally excluded from annotations in this release. They are identified as a prime extension target.

The authors identify several directions for future work: annotating embedded tables, extending to logical structure and cell content, and expanding to other sub-domains within civil engineering or other industries.

Limitations:

• The dataset covers a single narrow sub-domain (German-language steel ordering lists from 10 firms). Generalization beyond this sub-domain is unclear.
• Exact benchmarking numbers for TATR and per-class results are not in the paper; they require querying the external evaluation server.
• The dataset is openly licensed under CC-BY-4.0, which permits commercial use with attribution, but the underlying documents reflect a single geographic and industry context, limiting diversity.
• No model weights or training configurations are released for the benchmarked models.

Reproducibility

Models

• YOLOv8: general-purpose object detector (Ultralytics YOLO, January 2023 release). No architectural modifications reported. No weights released by the CISOL authors.
• TATR (Table Transformer): DETR-based TSR model from Smock et al. (CVPR 2022), pre-trained on PubTables-1M. Fine-tuned on CISOL. No fine-tuned weights released.

Algorithms

• Training details for the benchmarked models are not described in the paper beyond model selection. Hyperparameters, optimizer, learning rate, batch size, and epoch count are not reported.
• Annotation was performed using CVAT (Computer Vision Annotation Tool, CVAT.ai), a publicly available open-source tool.
• The annotation guideline is released alongside the dataset on Zenodo.
• Train/validation/test splitting used stratified sampling primarily on data origin (company), with size and type tags used to verify balance across subsets rather than as hard stratification criteria.

Data

• Total collected: 3,288 document images from 24 reinforced concrete projects at 10 German structural engineering firms (2015-2023). Documents are scanned and in German.
• Annotated subset: 844 images, selected by stratified sampling with a preference for companies with fewer than 100 documents to ensure provenance diversity.
• Annotation format: COCO-JSON.
• Classes: table (for TD), column, row, spanning cell, header (for TSR).
• Instance count: over 120,000 annotated instances across both tasks.
• Splits: training, validation, and test sets. Stratified sampling on data origin (company) ensures provenance diversity; size and type distributions are verified to be consistent across splits.
• License: CC-BY-4.0.
• Zenodo release contents (five files, 772.5 MB total): annotation guidelines, tagged metadata, TD+TSR annotated dataset, TSR-only annotated dataset, and unlabeled images (full 3,288-image collection).
• Anonymization: automated pattern matching for names, locations, and personal data (length-preserving substitution), followed by manual review. Eight images removed for containing only cover page content.
• Public availability (DOI): https://doi.org/10.5281/zenodo.10829550

Evaluation

• Primary metric: COCO mAP at IoU $0.5\text{:}0.95\text{:}0.05$, the same metric used in the PubTabNet ICDAR 2021 Challenge.
• Annotation quality metric: Krippendorff’s $\alpha$ computed at IoU thresholds $0.5$, $0.75$, and $0.5\text{:}0.95\text{:}0.05$ on 20 re-annotated images.
• Convergence threshold: $K\text{-}\alpha$ values are converted to mAP to yield a theoretical performance ceiling. The formula follows Tschirschwitz and Rodehorst (WACV 2025).
• Evaluation server: https://eval.ai/web/challenges/challenge-page/2257, with held-out test annotations not publicly released.
• No error bars or significance tests are reported for the benchmark results. Number of training runs and seeds are not stated.

Hardware

• Training and evaluation hardware are not reported in the paper.
• No GPU-hour, VRAM, or inference latency figures are provided.

BibTeX

@InProceedings{Tschirschwitz_2025_WACV,
  author = {Tschirschwitz, David and Rodehorst, Volker},
  title = {CISOL: An Open and Extensible Dataset for Table Structure Recognition in the Construction Industry},
  booktitle = {Proceedings of the Winter Conference on Applications of Computer Vision (WACV)},
  month = {February},
  year = {2025},
  pages = {7594-7602}
}

TL;DR

TFLOP replaces the conventional dual-decoder TSR pipeline (predict cell bounding boxes, then match them to OCR text regions) with a layout pointer mechanism: text region bounding boxes are encoded as context from the start, and the decoder directly associates each predicted table data tag with the corresponding text region. An additional span-aware contrastive supervision improves recognition of tables with row and column spans. On PubTabNet, FinTabNet, and SynthTabNet, TFLOP achieves the highest reported TEDS scores at the time of submission, and the authors demonstrate practical utility on watermarked and non-English industrial documents.

What kind of paper is this?

Dominant: $\Psi_{\text{Method}}$ The headline contribution is the TFLOP architecture: the layout pointer mechanism and span-aware contrastive supervision. The paper devotes most pages to the model design, ablations, and benchmark comparisons.

Secondary: None. No dataset is released (the Korean evaluation set is small and kept in the supplementary; the watermark dataset is synthesized from FinTabNet and not distributed).

What is the motivation?

Recent image-to-text TSR systems follow a dual-decoder pattern: one decoder generates HTML or OTSL structure tokens, and a second decoder predicts cell bounding boxes conditioned on those tokens. The bounding boxes are then matched to text regions obtained from an OCR engine or PDF parser to produce the complete table.

The matching step requires carefully calibrated post-processing heuristics to pair predicted bounding boxes with the correct text regions. When a predicted box does not precisely align with the true cell boundary, the matched text may be wrong or incomplete, degrading the final table structure. The gap between TEDS-Struct (structure only) and TEDS (structure + content) scores in prior work is a direct indicator of this misalignment problem.

TFLOP’s premise is that text region bounding boxes are already available (from OCR or PDF parsing) before inference. Rather than predicting those boxes and then matching them, the model can treat the available boxes as input context and directly point to them from the generated structure tokens, skipping the matching stage entirely.

What is the novelty?

Overall Architecture

TFLOP comprises four modules operating on an input table image and its associated text region bounding boxes:

1. Image Encoder (Swin Transformer): Produces visual patch features ${z_i}_{i=1}^{P} \in \mathbb{R}^{d}$.
2. Layout Encoder (MLP): Embeds each text region bounding box combined with $2 \times 2$ RoI Align features from ${z_i}$ into layout embeddings ${l_j}_{j=1}^{B} \in \mathbb{R}^{d}$.
3. Logical Structure Decoder (BART): Auto-regressively generates a sequence of $T$ table tags ${y_k}_{k=1}^{T} \in \mathbb{R}^{v}$, conditioned on both visual features (cross-attention) and layout embeddings (context prompt). Tags are in OTSL format, which has a 1-to-1 mapping with HTML.
4. Layout Pointer: Associates the decoder’s last hidden states with the text region bounding boxes to resolve which text belongs to which cell.

Feature dimension $d = 1{,}024$; input resolution $768 \times 768$; output sequence length $N = 1{,}376$.

Layout Pointer Mechanism

The decoder’s final hidden states ${h_i}_{i=1}^{N}$ are split into bounding box features ${b_j}_{j=1}^{B}$ and tag features ${t_k}_{k=1}^{T}$. Both are linearly projected:

$$ b_j = \text{proj}_b(b_j), \quad t_k = \text{proj}_t(t_k) $$

For bounding boxes that are not empty, the pointer loss is an InfoNCE-style objective over all data tag positions in set $D$:

$$ L_{\text{ptr}} = -\frac{1}{B} \sum_{j=1}^{B} \log \frac{\exp(b_j \cdot t_{k=j} / \tau)}{\sum_{k’ \in D} \exp(b_j \cdot t_{k’} / \tau)} $$

where $k=j$ is the index of the data tag corresponding to the $j$-th bounding box and $\tau = 0.1$ is the temperature.

Empty data cells (no corresponding bounding box) receive a separate BCE loss using a special learnable embedding $b_0$:

$$ L_{\text{ptr}}^{\text{empty}} = \frac{1}{|D|} \sum_{k’ \in D} \text{BCE}!\left(\sigma(b_0 \cdot t_{k’}),; I(k’)\right) $$

where $I(k’)$ is 1 if data tag $k’$ has no associated bounding box.

Span-aware Contrastive Supervision

To improve recognition of tables with row or column spans, TFLOP applies contrastive supervision across bounding box embeddings. For the $j$-th bounding box projected to $\hat{b}_j = \text{proj}_s(b_j)$:

$$ L_{\text{contr},j} = -\sum_{p \in P(j)} \frac{c_p(j)}{|P(j)|} \log \frac{\exp(\hat{b}_j \cdot \hat{b}_p / \tau)}{\sum_{a \in A(j)} \exp(\hat{b}_j \cdot \hat{b}_a / \tau)} $$

where $A(j)$ is all bounding boxes except $j$, and $P(j)$ is the set of positive samples (same row or column as $j$). The span coefficient $c_p(j)$ weighs each positive pair by the degree of span overlap:

$$ c_p(j) = \frac{(\text{overlap}(p, j))^2}{\text{span}(p) \cdot \text{span}(j)} $$

$\text{span}()$ is the row or column span count; $\text{overlap}(p,j)$ is the number of rows or columns shared between $p$ and $j$. This formulation assigns higher weight to cells that overlap more completely, and lower weight to partially overlapping cells in multi-span scenarios.

Overall Loss

The training objective combines five terms:

$$ L = \lambda_1 L_{\text{cls}} + \lambda_2 L_{\text{ptr}} + \lambda_3 L_{\text{ptr}}^{\text{empty}} + \lambda_4 L_{\text{contr}}^{\text{row}} + \lambda_5 L_{\text{contr}}^{\text{col}} $$

where $\lambda_1 = \lambda_2 = \lambda_3 = 1$ and $\lambda_4 = \lambda_5 = 0.5$. $L_{\text{cls}}$ is the cross-entropy loss for tag classification.

What experiments were performed?

Datasets

• PubTabNet: 500,777 train / 9,115 validation / 9,064 test. Tables from scientific articles; cell-level annotations available for train and val. For the test set, text regions were obtained using PSENet + MASTER (same OCR pipeline as prior work).
• FinTabNet: 112,887 tables from financial reports; cell-level annotations provided via PDF parsing (no OCR noise).
• SynthTabNet: 600,000 synthetic tables with diverse styles; cell-level annotations provided.

Baselines

PubTabNet: TableMaster, LGPMA, TableFormer, VAST, RobusTabNet, DRCC. FinTabNet and SynthTabNet: TableFormer, GridFormer, VAST, DRCC.

Ablation (Table 3)

Four configurations on PubTabNet test TEDS (%):

The gain from span-aware over uniform contrastive is concentrated in complex tables (+0.35 vs +0.12 for complex), supporting the claimed benefit for spanning cells.

Industrial Experiments

Watermark TSR: A watermarked FinTabNet set was synthesized by inpainting 20 candidate texts per image (20% probability per box). TFLOP extended with a 2-layer MLP filter (Layout Filter) achieves 99.54% TEDS-S / 99.41% TEDS vs. TableMaster’s 82.18% / 72.83%.

Cross-lingual TSR: 30 manually annotated Korean financial report tables (15 simple, 15 complex) and 175 QA pairs. TFLOP trained on English PubTabNet generalizes to Korean: 95.76% / 89.41% simple/complex TEDS vs. GPT-4V 79.43% / 68.39% and TableMaster 89.96% / 83.94%.

Implementation

• Image encoder: Swin Transformer.
• Structure decoder: BART (configured similarly to Donut).
• Input resolution: $768 \times 768$.
• Output sequence length: $N = 1{,}376$; bounding box context length $B = 640$ for PubTabNet/FinTabNet, $864$ for SynthTabNet.
• Optimizer: AdamW (assumed; not stated explicitly), learning rate $8 \times 10^{-5}$ with cosine scheduling over 250K steps.
• Hardware: 4 Nvidia A100 GPUs.

What are the outcomes/conclusions?

PubTabNet (val): TFLOP_FULL reaches 98.3% TEDS-S and 98.0% TEDS, ahead of DRCC (98.9% TEDS-S, 97.8% TEDS) on TEDS-S but better on TEDS. Note: DRCC reports no TEDS on the test set.

PubTabNet (test): 98.38% TEDS-S, 96.66% TEDS. The TEDS-Struct to TEDS gap for TFLOP_FULL (0.11 on FinTabNet) is substantially smaller than for prior dual-decoder methods (0.42 for comparison), directly reflecting the reduction in bounding box misalignment.

FinTabNet: 99.56% TEDS-S, 99.45% TEDS, ahead of VAST (98.63% TEDS-S, 98.21% TEDS).

SynthTabNet: 99.42% TEDS-S, 99.40% TEDS, ahead of DRCC (98.70% TEDS-S).

Limitations acknowledged: The Korean TSR evaluation uses only 30 tables (15 simple, 15 complex), which is too small for reliable statistical conclusions. The watermark dataset is synthesized; performance on natural watermarks may differ. TFLOP requires text region bounding boxes as input, which means it is not a fully end-to-end system and depends on an upstream OCR pipeline. For datasets without cell-level annotations (like PubTabNet test), OCR quality affects downstream TEDS scores.

Unacknowledged gaps: No comparison with non-English training baselines; the Korean experiment only tests zero-shot transfer. No latency or throughput measurements. The optimizer is not stated in the main paper. No error bars or multi-seed runs.

Reproducibility

Models

• Image encoder: Swin Transformer (configuration matches Donut; exact size not specified in paper).
• Structure decoder: BART architecture; feature dimension $d = 1{,}024$.
• Layout encoder: MLP with $2 \times 2$ RoI Align; output dimension $d = 1{,}024$.
• Layout pointer: two linear projection heads for $b_j$ and $t_k$; one additional projection $\text{proj}_s$ for contrastive.
• Watermark extension: 2-layer MLP (Layout Filter) with BCE and sigmoid.
• Model weights: the GitHub repository exists (license: CC-BY-NC-4.0); weight availability should be checked directly.

Algorithms

• Learning rate: $8 \times 10^{-5}$ with cosine scheduling.
• Training steps: 250K.
• Hardware: 4 Nvidia A100 GPUs.
• Input resolution: $768 \times 768$.
• $N = 1{,}376$ (max sequence length); $B = 640$ (PubTabNet/FinTabNet), $B = 864$ (SynthTabNet).
• Temperature $\tau = 0.1$ for both pointer and contrastive losses.
• Loss weights: $\lambda_1 = \lambda_2 = \lambda_3 = 1$; $\lambda_4 = \lambda_5 = 0.5$.
• OTSL tokenization used (from Lysak et al., 2023; arXiv:2305.03393).
• Optimizer not explicitly stated in the paper text.

Data

• PubTabNet: 500,777 train / 9,115 val / 9,064 test; license CDLA-Permissive-1.0. Test set has no cell-level annotations; OCR (PSENet + MASTER) used.
• FinTabNet: 112,887 tables; PDF-parsed cell annotations; license CDLA-Permissive-1.0.
• SynthTabNet: 600,000 synthetic tables; cell-level annotations; license CDLA-Permissive-1.0.
• Korean tables: 30 manually annotated tables + 175 QA pairs; not released.
• Watermark dataset: synthesized from FinTabNet (not released); synthesis procedure described in Appendix B.

Evaluation

• TEDS and TEDS-Struct as primary metrics (tree-edit-distance similarity on HTML trees).
• PubTabNet: both val and test splits reported.
• Ablation on PubTabNet test TEDS.
• Watermark evaluation: custom synthesized dataset.
• Korean evaluation: 30 tables; tiny set, no confidence intervals.
• No error bars, significance tests, or multi-seed runs.

Hardware

• Training: 4 Nvidia A100 GPUs.
• Training time: 250K steps; approximate GPU-hours not reported.
• Inference latency and memory requirements not reported.
• No deployment or cost estimates.

BibTeX

@inproceedings{khang2024tflop,
  title={TFLOP: Table Structure Recognition Framework with Layout Pointer Mechanism},
  author={Khang, Minsoo and Hong, Teakgyu},
  booktitle={Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence},
  pages={947--955},
  year={2024},
  doi={10.24963/ijcai.2024/105}
}

TL;DR

TabSniper is an end-to-end bank statement processing pipeline from American Express that chains table detection and categorization (TDC) with table structure recognition (TSR), both built on DETR fine-tuned from PubTables-1M. The accompanying BankTabNet dataset provides 11,607 page-level and 5,165 table-level annotations from real bank statements. The primary contribution is a deployed workflow for extracting transactions from diverse bank templates, with CIoU loss substitution and a long-table split-merge strategy as the key model-level changes.

What kind of paper is this?

Dominant: $\Psi_{\text{Impact}}$ (translational/operational). The headline contribution is a validated end-to-end pipeline for a specific industrial use case: extracting credit and debit transactions from bank statements for credit underwriting. The paper describes the full workflow from raw PDF pages to a structured JSON of transactions, includes a post-processing checksum (balance reconciliation), reports CPU inference time as a primary metric, and is explicitly motivated by real-time deployment constraints. The “win” is the validated operational outcome, not a new modeling paradigm.

Secondary: $\Psi_{\text{Method}}$ (the CIoU loss substitution and split-merge strategy for long tables are genuine, if modest, algorithmic contributions) and $\Psi_{\text{Resource}}$ (BankTabNet is a purpose-built annotated dataset for this domain).

What is the motivation?

Bank statement processing is a core task in credit underwriting: lenders need to assess cash flows, spending patterns, and transaction histories before making lending decisions. Unlike structured financial reports, bank statements vary substantially across issuers in layout, template, and table structure. Scanned and digitally generated PDFs coexist, and tables routinely span multiple pages with densely packed multi-line rows.

Standard TSR datasets (PubTabNet, FinTabNet, PubTables-1M) are derived from scientific articles and corporate financial reports. The authors argue these do not capture the specific characteristics of bank statement tables: long tables (often more than 20 rows) with variable intra-cell text spacing that misleads column boundary detection, multiple table categories on a single page (credit, debit, check, transaction balance, summary), and the need to categorize table types for downstream processing.

No existing public dataset addressed bank statement TSR specifically, and no end-to-end system had demonstrated reliable transaction extraction across multiple bank templates. The paper is motivated by this operational gap.

What is the novelty?

The novelty is concentrated in three areas:

1. End-to-end bank statement pipeline. TabSniper chains TDC and TSR into a single workflow designed for real-time CPU deployment. A one-time OCR call is shared across both stages, and a postprocessing layer sorts tables by page order, matches column headers to synonyms for standard transaction fields (Date, Amount, Debit, Credit, Balance), and computes a balance checksum:

$$ \text{Checksum} = \text{OpenBal} - \sum_{t \in \text{Debit}} t + \sum_{t \in \text{Credit}} t - \text{EndBal} $$

A zero checksum confirms that all transactions were extracted for a given statement.

2. CIoU loss for TSR. The baseline DETR uses GIoU loss. The authors identify that GIoU does not account for overlapping area, center distance, or aspect ratio, which causes false positive row bounding boxes in densely packed bank tables. They substitute CIoU, which incorporates a normalized center-distance penalty (DIoU) plus an aspect-ratio consistency term:

$$ L_{\text{CIoU}} = L_{\text{DIoU}} + \alpha \cdot \upsilon $$

$$ \upsilon = \frac{4}{\pi^2} \left(\arctan \frac{w^{gt}}{h^{gt}} - \arctan \frac{w}{h}\right)^2, \quad \alpha = \frac{\upsilon}{(1 - \text{IoU}) + \upsilon} $$

The full TSR loss becomes:

$$ L_{\text{TSR}} = \sum_{n=1}^{N} \lambda_{ce} \cdot L_{CE} + \lambda_{l1} \cdot L_{L1} + \lambda_{ciou} \cdot L_{\text{CIoU}} $$

3. Long-table split-merge. Tables with more than 20 rows are split horizontally into two sub-images before passing to the TSR model (which uses a fixed query count of $N = 125$). Predictions are merged at postprocessing time. This addresses the attention drift and missed detections that arise when the fixed-capacity transformer processes tables that are longer than those in its training distribution.

4. Multi-modal TDC. The TDC stage runs a DETR vision model first, then refines Credit/Debit categorization using a text-based Multinomial Naive Bayes classifier trained on table caption and header text. Because caption text is outside the table bounding box (and therefore invisible to the vision model), this two-stage multi-modal approach separates detection from categorization.

5. BankTabNet dataset. The authors introduce two new annotated datasets sourced from real, PII-masked bank statements: a TDC dataset (11,607 page images, 10 table categories) and a TSR dataset (5,165 table images, 5 object classes). Inter-annotator agreement is measured with Krippendorff’s Alpha ($\kappa \geq 0.955$ at $\text{IoU} > 0.5$ for TDC; $\kappa \geq 0.99$ for TSR).

What experiments were performed?

TDC evaluation. TabSniper-TDC is compared against Faster R-CNN, Mask R-CNN, Cascade R-CNN, DiT-B (Cascade), and Dynamic Head, all evaluated on the BankTabNet TDC test split. Metrics are $AP$, $AP_{50}$, and $AP_{75}$ (COCO-style). CPU inference time on Apple M1 Max (32 GB) is also reported.

TabSniper-TDC sits below DiT and Dynamic Head on AP but delivers the lowest CPU inference time among all models that support CPU inference.

TSR evaluation with ablations. Three ablations are stacked incrementally on BankTabNet TSR test:

Each ablation contributes meaningfully, with padding variations delivering the largest single jump in $AP_{75}$ (from 80.8 to 89.8).

TSR on external public datasets. TabSniper-TSR is trained from scratch on PubTables-1M and FinTabNet respectively and compared to Table Transformer:

TabSniper shows modest gains on AP and AR (localization over a range of IoU thresholds) while being slightly below baseline on $AP_{50}$ for FinTabNet.

Text classifier evaluation. Three Multinomial Naive Bayes classifiers (Header NB, Caption NB, Header_Caption NB) are evaluated on per-category F1. Header_Caption NB achieves the best results across all categories.

What are the outcomes/conclusions?

The system demonstrates that a DETR-based pipeline, adapted with domain-specific techniques (long-table split-merge, CIoU loss, padding augmentation, multi-modal TDC), can reliably extract transactions from diverse bank templates at CPU inference speeds suitable for real-time processing (1.91 minutes for a 20-page statement on Apple M1 Max).

The balance checksum mechanism provides a form of deterministic validation: if the extracted debits and credits reconcile to the opening and closing balances, the extraction is likely complete.

On the internal BankTabNet TSR test set, TabSniper achieves AP 83.1 and AR 90.6, substantially above all baselines. On public datasets, it is competitive with Table Transformer, particularly on AP and AR, suggesting the model modifications generalize modestly beyond the bank statement domain.

Limitations acknowledged by the authors:

• Tilted tables (from mobile phone captures) cause bounding box misalignment and partial OCR capture. The authors note this could be addressed with document angle correction, which they plan for future versions.
• The paper does not report end-to-end transaction extraction accuracy (e.g., percentage of statements with checksum zero, false transaction rates), limiting assessment of the pipeline’s actual production performance.

Limitations not acknowledged:

• BankTabNet is proprietary (AmEx internal, PII-masked). The dataset is described but not released, making the bank-statement-specific results unreproducible for external researchers.
• No code or model weights are released. All components (DETR fine-tune, Naive Bayes text classifier, postprocessing heuristics) are described but not available.
• Evaluation uses AP/AR rather than TEDS or GriTS, making direct comparison to most TSR literature difficult.
• The DOI in the paper (https://doi.org/XXXXXXX.XXXXXXX) is a placeholder; the actual ACM DOI was not available at time of arXiv submission.
• The paper relies on an in-house OCR service and an in-house statement summary extraction service. These dependencies are opaque and unavailable to external researchers.

Reproducibility

Models

• TDC: DETR with ResNet-50 backbone, implemented via Detectron2. Pre-trained weights: Table Transformer (PubTables-1M). Fine-tuned on BankTabNet TDC. No weights released.
• TSR: DETR with ResNet-50 backbone, pre-trained on PubTables-1M, fine-tuned on BankTabNet TSR with CIoU loss substitution. Query count $N = 125$. No weights released.
• Text classifier: Three Multinomial Naive Bayes models trained on caption/header text using count vectorization (vocabulary sizes: 840 caption words, 860 header words). Lightweight; reproducible from the paper’s description if data were available.

Algorithms

• TDC training: LR $1 \times 10^{-5}$; 100 epochs; batch size 16; $\lambda_{ce} = 1$, $\lambda_{l1} = 5$, $\lambda_{giou} = 2$. Scale augmentation: shortest side 400-800 px. Optimizer not stated for TDC (Adam stated only for TSR).
• TSR training: LR $5 \times 10^{-5}$ with linear decay every 4 epochs; 100 epochs; batch size 2; $\lambda_{ce} = 1$, $\lambda_{l1} = 5$, $\lambda_{ciou} = 2$. Adam optimizer. Scale augmentation: longest side 1100-1300 px. Inference: $1200 \times 1200$ px.
• Long-table split: tables with more than 20 rows are split into two sub-images horizontally. Merge happens in postprocessing.
• Postprocessing heuristics: NMS, gap filling, addition of missing rows/columns, rule-based row separation from OCR date column.
• Balance checksum: $\text{OpenBal} - \sum_{\text{Debit}} + \sum_{\text{Credit}} - \text{EndBal} = 0$.

Data

• BankTabNet TDC: 11,607 page images (7,544 train / 1,741 val / 2,322 test). 10 table categories. PII-masked via in-house service. Not publicly available.
• BankTabNet TSR: 5,165 table images from 310 bank statements (9,724 train / 2,000 val / 2,200 test after augmentation). 5 object classes. Not publicly available.
• Annotation quality: Krippendorff’s Alpha $\geq 0.955$ (TDC, $\text{IoU} > 0.5$), $\geq 0.994$ (TDC, $\text{IoU} > 0.9$); $\geq 0.99$ (TSR, $\text{IoU} > 0.5$), $\geq 0.98$ (TSR, $\text{IoU} > 0.9$).
• External datasets used for generalization tests: PubTables-1M (public, CDLA-Perm-2.0 annotations) and FinTabNet (public, CDLA-Permissive-1.0 annotations).

Evaluation

• Primary metrics: $AP$, $AP_{50}$, $AP_{75}$, $AR$ (COCO-style object detection metrics). Text classifier: per-category F1.
• No TEDS, GriTS, or Adjacency F1 reported. Direct comparison to Im2Seq TSR literature is not straightforward.
• Baselines for TSR: TabStructNet, LGPMA, Table Transformer (all trained on BankTabNet TSR). Comparisons on external datasets: Table Transformer trained on the respective dataset.
• No error bars, significance tests, or multi-seed averaging reported.
• The end-to-end transaction extraction accuracy (checksum pass rate) is mentioned conceptually but not quantified in the results tables.

Hardware

• Training: NVIDIA A100 Tensor Core GPU, 40 GB. Number of GPUs not stated.
• Inference: Apple M1 Max CPU (32 GB). End-to-end pipeline for a 20-page bank statement: approximately 1.91 minutes. Per-image TDC inference: 1.25 s (TabSniper-TDC) on this hardware.
• No GPU-hour estimates or energy consumption figures are reported.

BibTeX

@inproceedings{trivedi2024tabsniper,
  title={TabSniper: Towards Accurate Table Detection \& Structure Recognition for Bank Statements},
  author={Trivedi, Abhishek and Mukherjee, Sourajit and Singh, Rajat Kumar and Agarwal, Vani and Ramakrishnan, Sriranjani and Bhatt, Himanshu Sharad},
  booktitle={8th International Conference on Data Science and Management of Data (12th ACM IKDD CODS and 30th COMAD)},
  year={2024},
  address={Jodhpur, India},
  publisher={ACM}
}

TL;DR

OmniDocBench is a document parsing benchmark from Shanghai AI Laboratory that covers 981 pages across nine document types (academic papers, books, textbooks, magazines, slides, financial reports, newspapers, handwritten notes, and exam papers), with human-verified annotations for layout detection, text, formulas, and tables, including dual HTML+LaTeX table annotations for TEDS-style evaluation. The authors evaluate three classes of systems: pipeline tools, expert VLMs, and general VLMs. Pipeline-based tools generally lead on standard document types while general VLMs generalize better to uncommon formats and degrade less under visual noise. No single method dominates across all nine document types or both language settings (English and Chinese).

What kind of paper is this?

Dominant: $\Psi_{\text{Evaluation}}$ The primary contribution is a new benchmark and evaluation methodology for document parsing. The dataset design, annotation protocol, multi-level evaluation pipeline, and systematic comparison of methods are all in service of a measurement goal: determining how well current document parsing systems handle diverse real-world document types.

Secondary: $\Psi_{\text{Resource}}$ The benchmark itself, including 981 annotated pages, over 100,000 annotations, dual HTML+LaTeX table annotations, and a publicly released evaluation codebase, is a reusable community resource.

What is the motivation?

Document parsing, the task of extracting structured, machine-readable content from PDFs, underpins data pipelines for LLM pretraining and retrieval-augmented generation systems. Two main paradigms have emerged: pipeline-based approaches (layout detection followed by specialized OCR, formula, and table recognition modules) and end-to-end vision-language models. Despite progress on both fronts, fair and comprehensive comparison between systems has been difficult because existing benchmarks carry several well-documented shortcomings.

Document diversity is narrow: most benchmarks focus on academic papers from arXiv, overlooking document types common in real workflows such as textbooks, financial reports, handwritten notes, newspapers, and slides. Evaluation metrics are inconsistent or superficial: benchmarks relying solely on edit distance or BLEU do not account for the syntactic variety of valid LaTeX or HTML table representations, leading to systematic under-scoring of structurally correct but syntactically non-identical predictions. Evaluations are also typically flat, reporting one aggregate score without breakdown by document type, content category, language, or layout complexity.

The combination of narrow document coverage and weak metrics makes it difficult to identify where particular systems actually fail, or to determine whether a reported improvement generalizes beyond the training distribution.

What is the novelty?

OmniDocBench addresses these gaps with three main contributions.

Diverse evaluation corpus. The dataset spans nine document types: academic papers (129 pages), slides/PPT-to-PDF (133 pages), financial reports (81 pages), colorful textbooks (96 pages), exam papers (114 pages), magazines (97 pages), handwritten and typed notes (116 pages), newspapers (111 pages), and books (104 pages). The corpus covers three language settings: English (290 pages), Simplified Chinese (612 pages), and mixed (79 pages), with multiple layout types and three classes of visual degradations (fuzzy scans, watermarks, and colorful backgrounds).

Comprehensive annotations. Each page carries layout bounding boxes for 19 region categories with reading order and caption-footnote affiliation annotations, nine bounding-box-level attribute labels (three text attributes: language, background, rotation; six table attributes: language, frame type, merged cells, colorful background, formula content, rotation), and content recognition annotations in plain text for paragraphs and titles, LaTeX for formulas, and dual HTML plus LaTeX for tables.

The dual HTML+LaTeX annotation for tables is a direct response to the known problem that TEDS-style metrics penalize syntactically different but semantically equivalent table representations.

Multi-level evaluation pipeline. OmniDocBench supports three evaluation granularities:

1. End-to-end evaluation: full-page Markdown prediction versus per-element ground truth
2. Task-specific evaluation: isolated assessment of layout detection (mAP), OCR (normalized edit distance), table recognition (TEDS), and formula recognition (CDM + ExpRate)
3. Attribute-based evaluation: performance broken down by document type, language, background, rotation, table frame type, and special conditions

A key technical contribution is the Adjacency Search Match algorithm, which addresses the paragraph-splitting problem that arises when different parsers apply different line-break and merge conventions. The algorithm computes a matrix of normalized edit distances between ground truth and predicted paragraph sequences, applies fuzzy matching for unmatched pairs, then iteratively merges adjacent paragraphs in either direction until similarity no longer improves.

The CDM (Character-level Differential Metric) is applied for formula evaluation. For tables, TEDS is computed against both HTML and LaTeX annotations and the better of the two scores is retained, reducing the syntactic-equivalence penalty.

What experiments were performed?

Systems evaluated. The authors evaluate three categories of methods:

• Pipeline tools: MinerU v0.9.3, Marker v1.2.3, Mathpix (commercial)
• Expert VLMs: GOT-OCR 2.0, Nougat 0.1.0-base (350M parameters)
• General VLMs: GPT-4o (2024-08-06), Qwen2-VL-72B, InternVL2-Llama3-76B

For general VLMs, deterministic decoding is applied (do_sample=False) with per-model max_token settings (32,000 for Qwen2-VL-72B; 4,096 for InternVL2; default for GPT-4o).

End-to-end evaluation. Each model produces a full-page Markdown prediction. The pipeline extracts structured elements, matches them to ground-truth elements via the adjacency search algorithm, and computes per-category scores for text, formulas, tables, and reading order, reported separately for English and Chinese pages. Selected end-to-end table TEDS results:

Layout detection. DocLayout-YOLO achieves a mean average precision of 47.38 averaged across nine document types, substantially ahead of LayoutLMv3 (28.84) and DIT-L (26.90). Performance drops markedly on handwritten notes and slides across all models evaluated.

Table recognition (component-level). Six models are evaluated on an isolated table subset using TEDS broken down by language, frame type, and special conditions. OCR-based models (RapidTable: 82.5 overall, PaddleOCR: 73.6) outperform expert VLMs (StructEqTable: 75.8, GOT-OCR: 74.9) and general VLMs (Qwen2-VL-7B: 71.0, InternVL2-8B: 71.5). Table rotation degrades all models substantially, with StructEqTable showing the best relative robustness on rotated tables.

OCR (component-level). PaddleOCR achieves the lowest normalized edit distance on the OCR subset across standard and complex-background conditions. GPT-4o achieves the lowest English edit distance (0.020) but is substantially weaker on Chinese (0.224). Rotation to 270 degrees is challenging for most systems.

Formula recognition. GPT-4o leads on CDM (86.8) and ExpRate@CDM (65.5); UniMERNet achieves the best normalized edit distance (0.238). Mathpix reports high CDM (86.6) but low ExpRate (2.8), consistent with high character-level precision combined with occasional punctuation omissions.

What are the outcomes/conclusions?

Pipeline tools (MinerU, Mathpix) generally lead on structured content tasks for standard document types such as academic papers and financial reports. Their strength comes from specialized downstream models and layout segmentation preprocessing.

General VLMs (Qwen2-VL-72B, GPT-4o) generalize better to uncommon or visually unusual document types such as handwritten notes, slides, and exam papers, and degrade less under visual degradations. The authors attribute this to broader training data covering long-tail scenarios.

VLMs struggle with high-density documents like newspapers, where limitations in input resolution and context length cause content truncation. The failure modes differ qualitatively from pipeline failures: VLMs tend to miss content on dense pages or produce hallucinated content on hard-to-read ones, while pipeline tools tend to misclassify layout regions on uncommon formats.

All systems perform worse on Chinese than English pages. All systems degrade substantially on rotated text and rotated tables. No single method achieves strong performance across all nine document types, both language settings, and all content categories simultaneously. This finding suggests that OmniDocBench captures a meaningfully broader evaluation space than prior benchmarks focused narrowly on academic papers.

Limitations acknowledged by the authors include that the dataset is evaluation-only and not suitable for training, the Chinese-language subset represents a specific subset of Chinese document styles, and some annotation categories such as code blocks are represented by too few examples for reliable evaluation.

Reproducibility

Models

The evaluated systems range widely in openness. MinerU and Marker are open-source pipeline tools. Nougat (350M) and GOT-OCR 2.0 are open expert VLMs with released weights. Qwen2-VL-72B and InternVL2-Llama3-76B are open general VLMs with released weights. Mathpix and GPT-4o are commercial API services with no public weights.

Algorithms

For pipeline tools, default settings are used throughout. For general VLMs, do_sample=False is set for reproducibility. Nougat uses its 0.1.0-base checkpoint; GOT-OCR uses its “format OCR” mode to produce structured output. The Adjacency Search Match algorithm applies normalized edit distance with a similarity threshold, followed by iterative paragraph merging until similarity ceases to improve.

Data

The 981 PDF pages were collected from Common Crawl, Google and Baidu search, and internal data at Shanghai AI Laboratory. Pages were selected by clustering visual features from 6,000 candidate pages using ResNet-50 features and Faiss at 10 cluster centers, with manual balancing across nine document types and page attributes. Annotation used a three-stage pipeline: automatic pre-annotation with LayoutLMv3 (layout), PaddleOCR (text), UniMERNet (formulas), and GPT-4o (tables); manual annotator correction with rendering verification using Tables Generator and latexlive; and expert quality inspection using CDM rendering to flag unrenderable elements.

The evaluation code is available on GitHub under Apache-2.0. The dataset on Hugging Face carries no formal SPDX license; a copyright statement on the dataset page explicitly restricts use to research purposes and prohibits commercial use. The dataset files include an annotation JSON (OmniDocBench.json), a corresponding images directory, and a PDF version of the evaluation pages added in December 2024.

Evaluation

Primary metrics: normalized edit distance for text and reading order (lower is better); CDM and ExpRate@CDM for formulas (higher is better); TEDS for tables (higher is better, computed against both HTML and LaTeX annotations with the better score taken); mAP for layout detection. Inline formulas are converted to Unicode for cross-model fairness before text evaluation. Ignore handling is applied to reduce noise from inter-system differences in header and footer retention conventions and in caption placement conventions.

Statistical details (number of runs, seeds, confidence intervals) are not reported; single-run results are presented for all methods.

Hardware

Hardware requirements are not reported. The evaluation mixes API calls to commercial services (Mathpix, GPT-4o) with local inference for open-weight models. No GPU hours, memory requirements, or cost estimates are provided.

BibTeX

@inproceedings{ouyang2025omnidocbench,
  title={OmniDocBench: Benchmarking Diverse PDF Document Parsing with Comprehensive Annotations},
  author={Ouyang, Linke and Qu, Yuan and Zhou, Hongbin and Zhu, Jiawei and Zhang, Rui and Lin, Qunshu and Wang, Bin and Zhao, Zhiyuan and Jiang, Man and Zhao, Xiaomeng and Shi, Jin and Wu, Fan and Chu, Pei and Liu, Minghao and Li, Zhenxiang and Xu, Chao and Zhang, Bo and Shi, Botian and Tu, Zhongying and He, Conghui},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year={2025}
}

TL;DR

SynFinTabs is a dataset of 100,000 synthetic financial table images from Queen’s University Belfast, generated to resemble tables found in UK Companies House filings and related financial documents. Each image ships with ground-truth HTML, JSON, and CSV representations and bounding-box annotations at the table, row, cell, and word levels. The authors also fine-tune LayoutLM on an extractive table question-answering task to produce FinTabQA, demonstrating the dataset’s utility for training information-extraction models.

What kind of paper is this?

Dominant: $\Psi_{\text{Resource}}$: the primary contribution is the dataset itself. The paper’s center of gravity is the curation pipeline, the characteristics of the 100,000 synthetic tables, their annotation scheme, and the licensing and public release. The headline claim is a reusable asset for the financial document AI community.

Secondary: $\Psi_{\text{Method}}$: the FinTabQA model is a secondary contribution. It demonstrates one practical use of the dataset by fine-tuning LayoutLM for extractive question answering, but the model is not the headline novelty; the dataset is.

What is the motivation?

Table extraction from document images sits at the intersection of computer vision, NLP, and information retrieval, and it remains difficult in part because high-quality labelled data is scarce outside the scientific domain. Most existing table datasets (ICDAR 2013, SciTSR, TableBank, PubTabNet, PubTables-1M) draw heavily from scientific articles because large repositories of such articles are available with source markup that can be used to construct annotations automatically. Financial tables differ from scientific tables in layout, typography, and structural conventions: they tend to be larger, use less dense borders, include currency units, section headings, note columns, and parenthesized negative values, and span a wider range of date formats and font choices.

A second gap is the absence of word-level positional ground truth. Many datasets rely on OCR to recover the spatial positions of words after the fact. The authors demonstrate that OCR is unreliable on tabular data: using default EasyOCR parameters on SynFinTabs test images yields approximately 75% exact-match accuracy, compared to near-perfect accuracy when ground-truth positions are used. For training layout language models such as the LayoutLM family, which consume 2D bounding-box coordinates as part of their input, accurate positional annotations are important.

Privacy constraints compound the data-scarcity problem. Businesses are often reluctant to share financial documents for model training, particularly when those documents contain sensitive information and training would involve a third-party service provider. A synthetic dataset avoids this constraint by design.

What is the novelty?

SynFinTabs makes two interconnected contributions. First, it introduces a publicly available dataset of 100,000 synthetic financial table images with:

• Ground-truth structure in three formats: HTML, JSON, and CSV.
• Bounding-box annotations at four levels of granularity: full table, row, cell, and word. Cell bounding boxes represent the full spatially meaningful cell region, not just the minimum pixel region containing text, which is a noted limitation of FinTabNet.
• Semantic cell-type labels: “section title”, “currency unit”, “row header”, “column header”, and “data”.
• A predefined question-answer pair per table, with start and end span positions stored explicitly rather than derived from OCR.

Second, it releases the dataset generation code (SynFinTabGen), enabling others to extend the pipeline to new domains or larger scales. The generation process proceeds as: specification (structural blueprint) $\rightarrow$ table object (rows, cells, words) $\rightarrow$ HTML document $\rightarrow$ headless browser rendering $\rightarrow$ screenshot image plus bounding-box extraction via the browser DOM. This approach sidesteps OCR entirely for annotation and produces pixel-accurate bounding boxes.

Dataset composition. The 100,000 tables are spread across six visual themes. Theme 0 (40%) replicates the style of financial statements filed with UK Companies House, designed by inspecting a large sample of filings. The remaining five themes (12% each) represent financial spreadsheet styles and one company-report style (styled annual report). An 80/10/10 train/validation/test split is applied with each theme represented proportionally. Textual cell content uses random English words from a 10,000-word vocabulary; numerical cells contain random numbers. Row headers draw from a list of real financial account names.

What experiments were performed?

The experiments center on extractive table question answering. Each table in SynFinTabs has question-answer pairs for every non-empty cell; one pair per table is designated as the “competition pair” used for training or evaluation depending on the split. Questions take the form “What is the value of [row header] for the year [column header]?” and the target answer is the cell value at that intersection.

FinTabQA training. LayoutLM base (113 million parameters) is fine-tuned on SynFinTabs using a batch size of 2 and a learning rate found via PyTorch Lightning’s Tuner (initial max $3 \times 10^{-5}$). Two model variants are trained:

• FinTabQA: input is the table image cropped to the table boundary.
• FinTabQA-A4: input is an A4 page-size image with the table in the top-left corner. This variant is motivated by the observation that when coordinates are rescaled to the 0-1000 “virtual” coordinate space used by LayoutLM, very small or large cropped images suffer more distortion than a consistently-sized A4 canvas.

Training uses ground-truth word bounding boxes. Evaluation uses EasyOCR output, with parameters tuned beyond the defaults through a parameter search.

The exact-match accuracy metric checks that both the predicted start and end span positions are correct. The F1 score used in standard QA benchmarks is not reported because a partial match in financial cell extraction is considered uninformative: either the full cell value is extracted or it is not.

Real-world evaluation. A small test set of 50 real-world table images, sampled from UK Companies House filings from March 2023, is manually annotated with two question-answer pairs each, yielding 100 questions. This set tests how well models trained on synthetic data transfer to real documents.

Baseline comparison. FinTabQA models are compared against GPT-4V under several prompting conditions: question-only, and question plus an instruction specifying that the image contains tabular financial data and requesting parentheses and negation signs be preserved in the response. GPT-4V responses are manually evaluated.

OCR sensitivity analysis. To separate OCR errors from model errors, both FinTabQA and FinTabQA-A4 are also evaluated on the SynFinTabs test split using the ground-truth words and bounding boxes directly.

What are the outcomes/conclusions?

SynFinTabs test split. Using tuned EasyOCR, FinTabQA achieves 95.87% exact-match accuracy and FinTabQA-A4 achieves 94.97% (Table 1). The less-than-one-point difference between the two image-input strategies suggests the choice of image size has limited impact in this setting.

Real-world evaluation. Results on the 100-question real-world set are reported in Table 2:

FinTabQA trained on synthetic data achieves 89% on real-world tables, outperforming GPT-4V with a question-only prompt (76%). When GPT-4V is prompted with an explicit instruction to preserve parentheses and negation signs, it reaches 94%, a gap of 18 percentage points attributed almost entirely to parenthesis handling: 22 of 24 GPT-4V errors with the question-only prompt were caused by the model omitting parentheses around negative values.

OCR impact. When ground-truth words and bounding boxes are provided, FinTabQA and FinTabQA-A4 achieve 99.98% and 99.99% accuracy respectively, indicating the models themselves are near-perfect on the task. The gap to the OCR-based results (roughly 4-5 percentage points on the synthetic test split) is attributable to OCR errors in two categories: (1) headers not recognized correctly, meaning the question context is corrupted, and (2) answer text not recognized correctly, so the span cannot be located. Using default EasyOCR parameters instead of the tuned parameters reduces accuracy to roughly 75%, a drop of about 20 percentage points. Tesseract, which is commonly used in related work, is reported to perform substantially worse on tabular data.

Limitations. The paper identifies several explicit constraints. Table content is semantically random: words are drawn from a vocabulary at random and numbers are arbitrary, so models cannot learn to interpret meaning from cell values. All question-answer pairs follow the same grammatical template, which may limit generalization to natural-language paraphrases of the same question. GPT-4V experiments were limited by API cost, restricting the extent of prompt engineering explored. The real-world evaluation set is small (100 questions from 50 tables), making it difficult to draw strong statistical conclusions from the real-world results.

Reproducibility

Models

FinTabQA and FinTabQA-A4 are both fine-tuned from the base variant of LayoutLM (microsoft/layoutlm-base-uncased), which has 113 million parameters. The base LayoutLM is pre-trained for document understanding using text, layout (2D bounding-box coordinates), and image features. The fine-tuned checkpoints are released separately on HuggingFace under the MIT license: ethanbradley/fintabqa (table-boundary variant) and ethanbradley/fintabqa-a4 (A4-page variant).

Algorithms

The extractive QA objective follows the standard span prediction setup: the model predicts a start logit and end logit over the context (the flattened list of table words), and the loss is computed against the ground-truth span start and end positions. A post-processing fix is applied at inference: the end position is constrained to come after the predicted start position (argmax over end logits restricted to positions after the start), which yielded approximately 2-point accuracy improvements.

Fine-tuning hyperparameters: batch size 2; learning rate selected with PyTorch Lightning Tuner with initial max $3 \times 10^{-5}$; FinTabQA trained for 12 epochs (63 GPU hours); FinTabQA-A4 trained for 11 epochs (57 GPU hours). Best checkpoint selected by lowest validation loss at epoch end.

Data

SynFinTabs comprises 100,000 images split 80/10/10 (train/validation/test). The dataset includes HTML, JSON, and CSV representations of each table; word, cell, row, and table bounding-box annotations; semantic cell-type labels; and pre-computed question-answer pairs with stored span positions. For test-split tables, EasyOCR-extracted words and bounding boxes are also included.

The dataset is publicly available on HuggingFace at ethanbradley/synfintabs under the MIT license. The generation code is available at ethanbradley/synfintabgen under the MIT license.

Textual row headers are drawn from a curated list of real financial account names observed in Companies House filings. Other textual cell content uses random words from a 10,000-word English vocabulary. Numerical cell content is random. No personally identifiable information or actual financial figures from real companies appear in the dataset.

Evaluation

The primary metric is exact-match accuracy requiring both span start and end positions to be correct. Evaluation on SynFinTabs uses the competition pair (one randomly selected question-answer pair per table). For the real-world set, two question-answer pairs per table were manually defined. The real-world evaluation set is small (50 tables, 100 questions) and statistical uncertainty measures are not reported for any result in the paper.

Hardware

All training was performed on a single NVIDIA GRID M60-8Q GPU using the Northern Ireland High Performance Computing (NI-HPC) service. FinTabQA required 63 GPU hours; FinTabQA-A4 required 57 GPU hours. No inference latency or memory figures are reported.

BibTeX

@inproceedings{bradley2026synfintabs,
  title     = {{SynFinTabs}: A Dataset of Synthetic Financial Tables for Information and Table Extraction},
  author    = {Bradley, Ethan and Roman, Muhammad and Rafferty, Karen and Devereux, Barry},
  booktitle = {Document Analysis and Recognition -- ICDAR 2025 Workshops},
  publisher = {Springer Nature Switzerland},
  address   = {Cham},
  pages     = {85--100},
  year      = {2026},
  month     = jan,
  doi       = {10.1007/978-3-032-09371-4_6}
}

TL;DR

UniTabNet is an image-to-text model for table structure recognition (TSR) that pairs a Swin Transformer encoder with a BART decoder, then adds two auxiliary modules: a Vision Guider that supervises attention toward row/column regions, and a Language Guider that aligns structural decoding tokens with text-reading tokens to improve handling of descriptive table cells. Results on PubTables1M, PubTabNet, WTW, and iFLYTAB are competitive with or exceed prior split-and-merge methods.

What kind of paper is this?

Dominant: $\Psi_{\text{Method}}$. The headline contribution is a new model architecture (UniTabNet) with two auxiliary training modules (Vision Guider and Language Guider). The paper is structured around ablation tables, baseline comparison tables, and architecture figures, all of which suggest that the model design is the primary claim.

Secondary: none significant. Experiments use existing benchmarks; no new dataset or tool is released.

What is the motivation?

Most prior TSR methods treat table parsing as a purely visual problem: bottom-up methods detect cells from visual features, and split-and-merge methods segment rows and columns with pixel-level classifiers. Both paradigms ignore the textual content of cells, which matters when table structure is visually ambiguous.

Wireless tables with descriptive cells are a clear failure case. If two adjacent cells both span multiple rows but differ in content length, a purely visual model can confuse the cell boundaries. The authors illustrate this with a comparison against SEMv2 (Figure 1): the visual-only baseline merges cells that should remain separate because it cannot distinguish them by layout alone.

Image-to-text models (Donut, Pix2Struct) have shown strong text perception in general document understanding, but prior TSR work in this paradigm (TableFormer, VAST) did not fully exploit text understanding within cells. The authors position UniTabNet as the first image-to-text TSR model to explicitly condition structural decoding on textual semantics; this framing is consistent with the reviewed prior work but is the authors’ own claim rather than an externally verified fact.

What is the novelty?

Architecture overview

UniTabNet follows a “divide-and-conquer” strategy:

1. An image-to-text model (Swin encoder + BART decoder) decodes a sequence of cell tokens <C> and row-boundary tokens <NL>.
2. A Physical Decoder converts each cell’s hidden state into a polygon (bounding box with 8 coordinates).
3. A Logical Decoder predicts rowspan and colspan for each cell.
4. A Vision Guider and a Language Guider are injected at the final decoder layer to improve both prediction types.

Physical Decoder

Polygon coordinates are quantized into 1,000 bins. Rather than taking the argmax over the bin vocabulary (as in Pix2Seq), the physical decoder computes the expected location:

$$ E(p_j) = \sum_{i=0}^{999} i \cdot a_i^{p_j} $$

where $a^{p_j} = \text{softmax}(\mathbf{h}_i^{p_j} \mathbf{Loc}^\top)$ is the distribution over the location vocabulary. The regression loss is mean squared error over the eight coordinate points:

$$ \mathcal{L}_{\text{poly}} = \frac{1}{8} \sum_{j=1}^{8} \left(E(p_j) - p_j^\ast\right)^2 $$

Logical Decoder

Span prediction uses argmax over the same location vocabulary. Because span distributions are heavily imbalanced (most cells have span 1), the training loss is sigmoid focal loss:

$$ \mathcal{L}_{\text{span}} = L_f!\left(a^{\text{row}}, l_{\text{row}}^\ast\right) + L_f!\left(a^{\text{col}}, l_{\text{col}}^\ast\right) $$

Vision Guider

The Vision Guider supervises which visual positions a cell token attends to. For each decoded cell, the last-layer hidden state $\mathbf{h}_i$ is projected to query vectors $\mathbf{h}_i^{\text{row}}$ and $\mathbf{h}_i^{\text{col}}$, which compute attention scores over the visual feature map $\mathbf{Z}$. The loss penalizes deviations from ground-truth row and column mask maps using sigmoid focal loss:

$$ \mathcal{L}_{\text{vis}} = L_f!\left(a^{\text{row}}, g_{\text{row}}^\ast\right) + L_f!\left(a^{\text{col}}, g_{\text{col}}^\ast\right) $$

This is an auxiliary loss; the attention weight itself is not changed, only supervised.

Language Guider

An auxiliary “Table Read” (TR) task trains the model to output the text content of each cell in sequence. A cell token in the TSR task is mapped to a hidden vector $\mathbf{h}_i^{\text{lang}}$, which is aligned with the mean-pooled hidden states $\mathbf{h}_{\text{lang}}^\ast$ of the corresponding TR tokens via MSE loss:

$$ \mathcal{L}_{\text{lang}} = \text{MSE}!\left(\mathbf{h}_i^{\text{lang}},, \mathbf{h}_{\text{lang}}^\ast\right) $$

This alignment gives structural tokens access to textual representations without requiring OCR outputs at inference time.

Total loss

Five losses with large differences in scale are combined using learnable homoscedastic uncertainty weights (Kendall et al., 2018). The five components are: the language model cross-entropy loss $\mathcal{L}_{\text{lm}}$, the polygon regression loss $\mathcal{L}_{\text{poly}}$, the span focal loss $\mathcal{L}_{\text{span}}$, the Vision Guider focal loss $\mathcal{L}_{\text{vis}}$, and the Language Guider alignment loss $\mathcal{L}_{\text{lang}}$. These are combined as:

$$ \mathcal{L}_{\text{total}} = \sum_{k=1}^{5} \frac{1}{2\sigma_k^2}\mathcal{L}_k + \log(1 + \sigma_k^2) $$

The $\sigma_k$ parameters are learned end-to-end and remove the need for manual loss weighting.

What experiments were performed?

Datasets