Document Layout Analysis

TL;DR

VILA introduces two methods for incorporating visual layout group structure (text lines and blocks) into BERT-based models for scientific PDF parsing. I-VILA inserts boundary indicator tokens and improves Macro F1 by up to 1.9%; H-VILA uses hierarchical encoding to reduce inference time by 47% with minimal accuracy loss. Both require only fine-tuning, cutting training cost by up to 95% compared to layout-aware pretraining. The paper also introduces S2-VL, a human-annotated dataset of 1,337 pages across 19 scientific disciplines.

What kind of paper is this?

Dominant: $\Psi_{\text{Method}}$. The headline contribution is two novel methods (I-VILA and H-VILA) for incorporating visual layout group information into existing language models without expensive pretraining. Ablations, efficiency comparisons, and accuracy improvements are the paper’s center of gravity.

Secondary: $\Psi_{\text{Resource}}$. The paper introduces the S2-VL dataset (1,337 human-annotated pages across 19 disciplines) and the S2-VLUE benchmark suite unifying three evaluation datasets.

Secondary: $\Psi_{\text{Evaluation}}$. The paper introduces the group category inconsistency metric $H(G)$ and systematically evaluates how different VILA group detectors (ground-truth, vision model, PDF parsing) affect downstream performance.

What is the motivation?

Extracting structured content from scientific PDFs (titles, abstracts, references, body text, etc.) is a critical first step for NLP over scientific papers. Layout-aware language models like LayoutLM improve accuracy by encoding each token’s 2D position, but they do not explicitly model higher-level layout structures: the grouping of tokens into text lines and blocks. These models also require expensive pretraining (1,000+ GPU-hours for LayoutLM, several thousand for LayoutLMv2).

Existing evaluation datasets (GROTOAP2, DocBank) rely on automatic labeling from source files, have limited domain coverage (life sciences or math/physics/CS only), and contain systematic annotation errors. No manually annotated multi-discipline evaluation set existed.

What is the novelty?

The core insight is the group uniformity assumption: tokens within a visual layout group (text line or text block) generally share the same semantic category. This is exploited in two ways:

I-VILA: Layout Indicator Injection

A special [BLK] token is inserted at each group boundary in the linearized token sequence. The resulting input has the form:

$$[CLS], T_1^{(1)}, \ldots, T_{n_1}^{(1)}, [BLK], T_1^{(j+1)}, \ldots, T_{n_m}^{(m)}, [SEP]$$

These boundary tokens signal possible category changes between groups. When used with a layout-aware base model like LayoutLM, the [BLK] token inherits the 2D positional embedding of its group’s bounding box. No pretraining is needed; the model learns to use these indicators during fine-tuning.

H-VILA: Hierarchical Encoding

A two-level transformer architecture:

1. Group encoder ($l_g = 1$ layer): encodes tokens within each group into a single vector $\tilde{\mathbf{h}}_j$, combined with 2D positional embeddings:

$$\mathbf{h}_j = \tilde{\mathbf{h}}_j + p(b_j)$$

2. Page encoder ($l_p = 12$ layers): models inter-group relationships across the full page, producing group-level predictions via an MLP classifier.

Both encoders are initialized from pretrained BERT/LayoutLM weights (layer 1 for the group encoder, full model for the page encoder), so no additional pretraining is required.

S2-VLUE Benchmark Suite

Three datasets unified with VILA structure annotations:

• GROTOAP2: 119K pages, 22 categories, life sciences, automatic labels from CERMINE PDF parsing.
• DocBank: 498K pages, 12 categories, math/physics/CS, automatic labels from LaTeX source. VILA structures regenerated via Mask R-CNN.
• S2-VL (new): 1,337 pages from 87 papers across 19 disciplines, 15 categories, human-annotated using the PAWLS tool. Inter-annotator agreement: 0.95 token-level accuracy.

Group Category Inconsistency Metric

A new diagnostic metric $H(G)$ measuring the entropy of predicted token categories within each group:

$$H(G) = \frac{1}{m} \sum_i^m H(g_i), \quad H(g) = -\sum_c p_c \log p_c$$

Lower values indicate more consistent predictions within groups.

What experiments were performed?

Experiments are conducted on all three S2-VLUE datasets. For S2-VL, 5-fold cross-validation is used (split by paper, not page).

I-VILA results (Table 2):

I-VILA with text blocks also reduces group inconsistency $H(G)$ by 32.1% on GROTOAP2.

H-VILA results (Table 3):

Generalization across base models (Table 4, GROTOAP2):

I-VILA improves all tested base models: DistilBERT (+1.60), BERT (+1.53), RoBERTa (+0.88), LayoutLM (+1.04 Macro F1).

Training cost comparison (Table 5):

LayoutLM pretraining requires 1,000+ GPU-hours. VILA methods require only fine-tuning: 4.7 GPU-hours for I-VILA and 3.5 GPU-hours for H-VILA on GROTOAP2, representing a 95%+ cost reduction.

VILA group quality ablation (Table 6, S2-VL):

Using ground-truth text blocks yields the best I-VILA performance (86.50 F1) compared to vision model predictions (83.44 F1) or PDF parsing (83.95 F1). H-VILA is more sensitive to group detection errors since it cannot assign different categories within a group.

What are the outcomes/conclusions?

I-VILA provides consistent accuracy improvements (+1-2% Macro F1) and better prediction consistency across all three benchmarks, without any pretraining. H-VILA trades modest accuracy for large efficiency gains (up to 69% inference time reduction). Both methods generalize across different BERT variants.

The S2-VL dataset fills a gap as the first human-annotated, multi-discipline evaluation set for scientific document parsing, covering 19 disciplines with 15 token categories.

Key limitations:

• VILA methods are evaluated only on scientific documents; generalization to other document types is untested.
• H-VILA is sensitive to group detection quality: incorrect block boundaries propagate errors since all tokens in a group receive the same label.
• S2-VL is relatively small (1,337 pages, 87 papers), limiting its use as a training set.
• The group uniformity assumption does not always hold, particularly at block boundaries or when block detectors make errors.

Reproducibility

Models

• I-VILA: Fine-tuned LayoutLM-BASE with [BLK] indicator tokens. 110M params (same as LayoutLM-BASE).
• H-VILA: Group encoder ($l_g = 1$, initialized from LayoutLM layer 1) + Page encoder ($l_p = 12$, initialized from full LayoutLM). Positional embeddings from group bounding boxes.
• Eight fine-tuned model checkpoints released on HuggingFace (baseline, I-VILA, H-VILA variants).

Algorithms

• Optimizer: AdamW, lr $5 \times 10^{-5}$ ($2 \times 10^{-5}$ for S2-VL), $\beta = (0.9, 0.999)$.
• Linear warmup over 5% steps, then linear decay.
• Batch sizes: 40 (GROTOAP2), 40 (DocBank), 12 (S2-VL).
• Epochs: 24 (GROTOAP2), 6 (DocBank), 10/20 (S2-VL).
• Mixed precision training.
• S2-VL uses 5-fold cross-validation, split by paper.

Data

• GROTOAP2: 119K pages from PubMed Central. Automatic annotations. Available via the VILA repo.
• DocBank: 498K pages from arXiv. Automatic annotations. VILA structures regenerated via Mask R-CNN.
• S2-VL: 1,337 pages from 87 papers across 19 disciplines. Human-annotated with PAWLS. 15 categories. Inter-annotator agreement 0.95. Available via the VILA repo.

Evaluation

• Primary metric: Macro F1 (token-level).
• Diagnostic metric: Group category inconsistency $H(G)$ (entropy-based).
• Inference time: average over 3 runs on 1,000 GROTOAP2 test pages, single V100 GPU.
• S2-VL: 5-fold cross-validation with standard deviations reported.

Hardware

• Training: 4-GPU RTX 8000 or A100 machines.
• Training cost: 4.7 GPU-hours (I-VILA) to 3.5 GPU-hours (H-VILA) on GROTOAP2, compared to 1,000+ GPU-hours for LayoutLM pretraining.
• Inference: single V100 GPU. I-VILA: ~53 ms/page. H-VILA (text line): ~28 ms/page. H-VILA (text block): ~16 ms/page.

BibTeX

  Copy
  

@article{shen2022vila,
  title={{VILA}: Improving Structured Content Extraction from Scientific {PDFs} Using Visual Layout Groups},
  author={Shen, Zejiang and Lo, Kyle and Wang, Lucy Lu and Kuehl, Bailey and Weld, Daniel S. and Downey, Doug},
  journal={Transactions of the Association for Computational Linguistics},
  volume={10},
  pages={376--392},
  year={2022},
  publisher={MIT Press},
  doi={10.1162/tacl_a_00466}
}

TL;DR

IBM trains five real-time document layout detectors (RT-DETRv2 and D-FINE variants) on a mixed corpus of 150K documents spanning 17 layout classes. The best model, heron-101, achieves 0.780 mAP on a filtered DocLayNet test set with 28 ms/image inference on an A100, a 23.9 absolute mAP-point improvement over Docling’s previous baseline. All model weights are released under Apache-2.0.

What kind of paper is this?

Dominant: $\Psi_{\text{Method}}$. The headline contribution is training and evaluating a family of object detection architectures (RT-DETRv1, RT-DETRv2, D-FINE) for document layout analysis. The paper devotes most of its pages to architecture selection, training recipes, post-processing pipelines, and multi-benchmark evaluation.

Secondary: $\Psi_{\text{Resource}}$. Five publicly released model checkpoints under Apache-2.0, plus the introduction of DocLayNet-v2 (a proprietary 17-class extension of DocLayNet) and canonical-DocLayNet (a filtered subset of DocLayNet that removes annotation noise from the 6 “delta” classes).

Secondary: $\Psi_{\text{Evaluation}}$. The paper devotes significant analysis to comparing two evaluation methodologies (COCO-tools vs. docling-eval), documents a “post-processing paradox” where qualitatively better outputs score lower on mAP, and concludes that mAP may not be suitable for document layout evaluation.

What is the motivation?

Docling is IBM’s open-source document conversion pipeline. Its previous layout model (RT-DETRv1 with ResNet-50, referred to as “old-docling”) only supported DocLayNet’s 11 original classes and missed important structural elements: checkboxes, code blocks, forms, key-value regions, and document indices. The authors needed to:

1. Expand the label taxonomy from 11 to 17 classes to support richer document types (forms, code, etc.).
2. Improve detection accuracy while maintaining real-time inference speed.
3. Use only permissively licensed architectures (no YOLO due to AGPL licensing concerns).

What is the novelty?

Expanded taxonomy and data curation

The 17-class taxonomy adds six “delta” classes to DocLayNet’s original 11: Checkbox-Selected, Checkbox-Unselected, Code, Document Index, Form, and Key-Value Region. The training mix unifies three sources:

• DocLayNet (public, filtered): A “canonical-DocLayNet” variant created by training a filtering detector at confidence threshold 0.3 to flag and exclude pages likely containing mislabeled delta-class elements. This removed ~32% of DocLayNet, yielding 22,101 training pages.
• DocLayNet-v2 (proprietary): IBM’s extended dataset covering all 17 classes with a 7,613-page test split.
• WordScape (public, filtered): Documents from the 2013 Common Crawl snapshot, with all table-containing pages removed due to systematic “Table”/“Form” label confusion.

The combined training set contains ~150K pages and 2.3M bounding-box annotations.

Architecture selection

The authors chose transformer-based detectors with permissive licenses:

Post-processing pipeline

Raw detections undergo PDF-aware refinement:

1. Each predicted bounding box is matched to overlapping native PDF cells and snapped to their boundaries.
2. Pictures covering >90% of a page are discarded.
3. Regular elements overlapping “wrapper” elements (Form, Key-Value Region, Table, Document Index) become children; wrapper boxes expand to enclose children.
4. Overlapping groups are resolved by a rule-based selector that considers label priority, element size, and confidence.

What experiments were performed?

Evaluation datasets

• DocLayNet (original 11-class test split)
• DocLayNet-v2 (17-class, 7,613-page proprietary test split)
• canonical-DocLayNet (filtered 11-class, 1,574-page test split)

Evaluation methodologies

Both methods report mean Average Precision averaged over IoU thresholds from 0.50 to 0.95 in steps of 0.05:

$$\text{mAP} = \frac{1}{|\mathcal{T}|} \sum_{t \in \mathcal{T}} \text{AP}_t, \quad \mathcal{T} = \{0.50, 0.55, \ldots, 0.95\}$$

1. COCO-tools: Standard $\text{mAP}@[0.50{:}0.95]$ with per-size AP (small/medium/large). Evaluated under three conditions: all predictions (no threshold), confidence $\geq 0.50$, and Docling post-processing (confidence $\geq 0.50$ + PDF-cell snapping + overlap resolution).
2. docling-eval: Docling’s own evaluation package. Applies a 0.50 confidence threshold, sets all surviving scores to 1.0, restricts to the label intersection of predictions and ground truth, skips samples with mismatched box counts, then computes $\text{mAP}@[0.50{:}0.95]$.

Runtime benchmarks

Inference time measured across three hardware configurations:

• CPU: AMD EPYC 7763 (4 threads)
• GPU: NVIDIA A100-80GB (batch sizes 100, 200, 500)
• MPS: Apple M3 Max (batch sizes 50, 100)

What are the outcomes/conclusions?

Accuracy

On canonical-DocLayNet (COCO-tools, no post-processing, no threshold):

• heron-101: 0.780 mAP (best overall)
• heron: 0.776 mAP
• egret-m: 0.765 mAP

On DocLayNet-v2 (COCO-tools, no post-processing, no threshold):

• heron-101: 0.758 mAP
• heron: 0.751 mAP

All new models improve over old-docling by 20.6 to 23.9 absolute mAP points (the paper reports these as percentage-point gains, not relative improvements). On the original DocLayNet test set, heron leads with 0.699 mAP, a 38.4% relative improvement over old-docling’s 0.505.

Post-processing paradox

The Docling post-processing pipeline (PDF-cell snapping, overlap resolution) visually improves layout quality, producing cleaner, less fragmented bounding boxes. However, it consistently lowers mAP scores by 10+ points. The authors attribute this to mAP penalizing geometric adjustments that do not exactly match ground-truth boxes, even when the post-processed output is qualitatively better. They conclude that mAP may not be the most suitable metric for document layout evaluation.

Runtime

On A100 with batch size 200:

• egret-m: 24 ms/image (fastest)
• heron-101: 28 ms/image
• heron: 31 ms/image

On CPU (4 threads, batch 32):

• egret-m: 334 ms/image
• heron: 643 ms/image
• heron-101: 988 ms/image

egret-m is roughly $3\times$ faster than heron-101 on CPU, making it a better choice for CPU-only deployments.

Limitations

• DocLayNet-v2 is proprietary. The 17-class test set that most directly measures full-taxonomy performance is not publicly available, limiting independent reproducibility.
• No comparison to non-DETR baselines. YOLO-family models (DocLayout-YOLO, PP-DocLayout) are excluded due to licensing. This makes it difficult to position these results against the broader DLA landscape.
• mAP critique is observational, not formal. The authors note mAP’s shortcomings for document layout but do not propose an alternative metric.
• Single training run per model. No error bars, variance estimates, or seed sensitivity analysis.
• Annotation ambiguity acknowledged but unaddressed. The qualitative analysis (Figures 2-4) reveals that ground truth and model predictions can both be “valid” layouts, but the paper does not explore multi-annotation or soft-label approaches.

Reproducibility

Models

• Five new models released on HuggingFace in safetensors format under Apache-2.0, plus the pre-existing old-docling baseline.
• Parameter counts: 19.5M (egret-m) to 76.7M (heron-101).
• Backbones initialized from pre-trained weights (ResNet, HGNet-V2). Pre-training source not specified beyond “pre-trained” (likely ImageNet, but not stated).
• Models were trained with native PyTorch code from the original RT-DETR and D-FINE repositories, then converted to HuggingFace Transformers safetensors for inference. Training scripts, configs, and reproduction recipes are not released; only inference-ready checkpoints are available.

Algorithms

• RT-DETRv2 models (heron, heron-101): 72 epochs, lr $10^{-4}$, AdamW ($\beta_1$=0.9, $\beta_2$=0.999), weight decay $10^{-4}$.
• D-FINE models (egret-m/l/x): 132/80/80 epochs, lr $2 \times 10^{-4}$ / $2.5 \times 10^{-4}$ / $2.5 \times 10^{-4}$, AdamW ($\beta_1$=0.9, $\beta_2$=0.999), weight decay $10^{-4}$ / $1.25 \times 10^{-4}$ / $1.25 \times 10^{-4}$.
• Augmentation: randomized distortions, zoom out, horizontal flips, resize to $640 \times 640$.
• No mention of learning rate warmup, gradient clipping, mixed precision, or EMA.

Data

• Training set: ~150K pages, 2.3M annotations across 17 classes.
  • DocLayNet (public, filtered to ~22K pages via canonical filtering at threshold 0.3)
  • DocLayNet-v2 (proprietary, not publicly available)
  • WordScape (public, table-containing pages excluded)
• Test splits: DocLayNet original (public), DocLayNet-v2 (proprietary, 7,613 pages), canonical-DocLayNet (public, 1,574 pages).
• The proprietary DocLayNet-v2 component means the full training set cannot be reconstructed. DocLayNet-v2 is not available on the docling-project HuggingFace org or the deprecated ds4sd org.

Evaluation

• COCO-tools mAP@[0.50:0.95] with size-stratified AP.
• docling-eval (MIT license): custom evaluation that filters predictions at 0.50 confidence, normalizes scores, and restricts to shared labels.
• No cross-dataset evaluation (e.g., M6Doc, D4LA) beyond the DocLayNet family.
• Single run per model; no error bars or significance tests.

Hardware

• Training hardware not specified (GPU type, count, total hours not reported).
• Inference benchmarked on three platforms: A100-80GB, AMD EPYC 7763 CPU (4 threads), Apple M3 Max (MPS). Batch sizes varied (32 for CPU, 50-500 for GPU/MPS).
• Models are small enough to run on consumer hardware; egret-m at 19.5M params runs at 334 ms/page on CPU.

BibTeX

@article{livathinos2025advanced,
  title={Advanced Layout Analysis Models for Docling},
  author={Livathinos, Nikolaos and Auer, Christoph and Nassar, Ahmed and Teixeira de Lima, Rafael and Lysak, Maksym and Ebouky, Brown and Berrospi, Cesar and Dolfi, Michele and Vagenas, Panagiotis and Omenetti, Matteo and Dinkla, Kasper and Kim, Yusik and Weber, Valery and Morin, Lucas and Meijer, Ingmar and Kuropiatnyk, Viktor and Strohmeyer, Tim and Gurbuz, A. Said and Staar, Peter W. J.},
  journal={arXiv preprint arXiv:2509.11720},
  year={2025}
}

TL;DR

American Stories applies a modular, cost-efficient deep learning pipeline (YOLOv8 layout detection, MobileNetV3 legibility classification, EfficientOCR) to nearly 20 million Library of Congress newspaper scans, producing 1.14 billion content region bounding boxes and 65.6 billion tokens of structured article text. The dataset is public domain content released under CC-BY-4.0.

What kind of paper is this?

Dominant: $\Psi_{\text{Resource}}$. The headline contribution is the American Stories dataset itself: 1.14B layout annotations and structured OCR text extracted from the Chronicling America collection. The paper’s applications section, evaluation, and limitations all center on the dataset’s fitness for downstream use.

Secondary: $\Psi_{\text{Method}}$. The paper develops a full extraction pipeline (layout detection, legibility classification, custom OCR, content association) and makes specific architectural choices (mobile-phone-class models) to meet a $60K cloud compute budget constraint. The pipeline is modular and released for reuse.

What is the motivation?

Library of Congress’s Chronicling America project provides approximately 20 million historical newspaper scans with page-level OCR, but the digitized text does not respect page layout. Articles, headlines, captions, advertisements, and other regions are scrambled together at the page level. A non-trivial share of scans are illegible, introducing noise. These limitations prevent effective use of modern NLP methods, language model pre-training on historical English, and structured social science analyses that require article-level text.

Existing open-source alternatives (notably Newspaper Navigator) detect visual content classes (photographs, illustrations, maps, comics, cartoons, ads, headlines) but do not detect article bounding boxes, which is the prerequisite for legibility filtering, custom OCR, and article association.

What is the novelty?

The core contribution is the end-to-end pipeline for converting newspaper scans into structured article text at scale:

1. Layout detection: YOLOv8-Medium trained on 2,202 labeled pages (48,874 layout objects) with 9 content classes: Article, Headline, Byline, Caption, Image, Advertisement, Table, Masthead, and Header. Achieves 91.3 mAP@50:95 on articles and 88.6 on headlines.

2. Legibility classification: MobileNetV3-Small classifies each text region as legible, borderline, or illegible. Trained on 979 labeled crops with weighted cross-entropy. No legible texts are misclassified as illegible in evaluation.

3. OCR (EfficientOCR): A character/word-level image retrieval approach using MobileNetV3-Small encoders trained with supervised contrastive loss. Words are recognized by nearest-neighbor lookup against an offline embedding index of 22,230 dictionary terms rendered across 43 fonts. When cosine similarity falls below 0.82, the system falls back to character-level recognition. This design meets the $60K compute budget while achieving 4.3% CER.

4. Content association: Rule-based methods associate headlines, bylines, and article bounding boxes using spatial overlap heuristics. Achieves 97.0 F1 on headline-article association.

The pipeline prioritizes mobile-class architectures (YOLOv8, MobileNetV3) throughout, making it over an order of magnitude cheaper to deploy than alternatives like TrOCR.

What experiments were performed?

Evaluation uses four hand-constructed datasets:

• Full page scans (10 pages, 597 content regions, 196,655 characters): end-to-end pipeline evaluation including content association (214 headline-article pairs).
• Day-per-decade sample (50 lines per decade, 1850-1920): OCR evaluation across diverse historical periods.
• Random lines from Carlson et al. (64 textlines): comparison with other detection frameworks and OCR engines.
• Legibility sample (100 bounding boxes): legibility classifier evaluation.

Key metrics:

• End-to-end CER: 0.051 (0.044 with spellchecking)
• OCR-only CER: 0.043
• Layout/line detection CER contribution: 0.012
• Content association F1: 97.0
• Legibility: 0/81 legible texts misclassified as illegible; 1/16 illegible texts misclassified as legible

Downstream application comparisons against Chronicling America page-level OCR:

• Topic classification (politics): RoBERTa on American Stories articles achieves F1 83.6 (article-level) and 96.0 (page-level), compared to 83.3 for neural methods on Chronicling America pages.
• Reproduced content detection: Neural method on American Stories achieves ARI 86.2 (page-level), compared to 74.6 for Viral Texts on Chronicling America.

What are the outcomes/conclusions?

The American Stories dataset contains 1.14 billion content region bounding boxes across nearly 20 million scans, with 65.6 billion tokens of OCR text. Coverage spans all U.S. states, with content dating from the 17th century through the mid-20th century (bulk from early 1900s).

The structured article texts enable analyses that are impossible with page-level OCR: article-level topic classification, reproduced content detection, and news story clustering. The authors demonstrate that neural methods on structured articles substantially outperform both neural and sparse methods on unstructured page text.

Limitations the authors acknowledge:

• Foreign-language newspapers are excluded (off-the-shelf OCR performs poorly on diverse scripts)
• Only 3.8% of articles span multiple bounding boxes; these are not associated in the current release
• Historical language reflects cultural biases of the period; the authors recommend against using the dataset for training generative models
• The pipeline is optimized for cost over accuracy; swapping in larger models (ViT, two-stage detectors) could improve quality

Reproducibility

Models

• Layout detection: YOLOv8-Medium, initialized from official pretrained checkpoint. Trained 100 epochs on 2,202 pages. Settings: imgsz=1280, iou=0.2, max_det=500, conf=0.1.
• Legibility classification: MobileNetV3-Small (timm mobilenetv3_small_050). Trained 50 epochs on 979 crops. Resolution 256, lr 2e-3 with 0.1 decay every 20 epochs. Weighted CE loss [2.0, 1.0, 1.0].
• Line detection: YOLOv8-Small. 100 epochs on 4,000 synthetic lines, then 50 epochs on 373 annotated crops. Resolution 640.
• Word/character localization: YOLOv8-Small. 100 epochs on 8,000 synthetic textlines, then 100 epochs on 684 annotated lines. Resolution 640.
• Word recognition: MobileNetV3-Small (timm mobilenetv3_small_050). SupCon loss, $\tau=0.1$, AdamW, cosine annealing with warm restarts (max lr 2e-3). 50 epochs without hard negatives, 40 epochs with offline hard negative mining. Batch size 1024 ($m=4$ views $\times$ 256 words).
• Character recognition: EffOCR-C (Small) from Carlson et al.
• All model checkpoints available via Dropbox link in the GitHub repo.

Data

• Source: Library of Congress Chronicling America collection (~20M scans). Public domain (pre-1925 content).
• Layout annotations: 2,202 pages labeled by undergraduate research assistants. Active learning sampling. 13% from Chronicling America, remainder from other public domain/off-copyright newspapers.
• OCR training data: 43 digital fonts for synthetic rendering; silver-quality labels from EffOCR-C on random sample; small set of hand-labeled word crops. Dictionary of 22,230 terms.
• Output dataset: 1.14B bounding boxes, 65.6B tokens. JSON format on HuggingFace Hub. CC-BY-4.0.

Evaluation

• CER (character error rate) via Levenshtein distance. Non-word rate as proxy metric on larger samples.
• mAP@50:95 for layout and line detection.
• F1 for content association. ARI (adjusted rand index) for reproduced content clustering.
• Evaluation sets are small (10 full pages, 64-225 textlines). No error bars or multi-run analysis reported.

Hardware

• Training: Single NVIDIA A6000 GPU.
• Inference (pipeline deployment): Azure F-series CPU nodes. Total compute budget: $60,000 USD for processing ~20M scans.
• Cost comparison: EfficientOCR is reported as over an order of magnitude cheaper than TrOCR Base on cloud CPUs.

BibTeX

  Copy
  

@inproceedings{dell2023american,
  title={American Stories: A Large-Scale Structured Text Dataset of Historical {U.S.} Newspapers},
  author={Dell, Melissa and Carlson, Jacob and Bryan, Tom and Silcock, Emily and Arora, Abhishek and Shen, Zejiang and D'Amico-Wong, Luca and Le, Quan and Querubin, Pablo and Heldring, Leander},
  booktitle={Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track},
  year={2023}
}

TL;DR

CATMuS Medieval Segmentation is a 1,680-page layout dataset covering 159 medieval manuscripts (8th-16th century) in 10 languages, annotated with 19 classes following the SegmOnto vocabulary (13 zone types, 6 line types). It is released as a companion to the CATMuS Medieval HTR dataset, which was the primary focus of the ICDAR 2024 paper. Since the segmentation dataset has no standalone publication, this note draws on the ICDAR 2024 paper for context on dataset construction, sources, and methodology. The segmentation dataset provides hierarchical block/line annotations with polygons, bounding boxes, and baselines.

What kind of paper is this?

Dominant: $\Psi_{\text{Resource}}$ The primary contribution is a large-scale, multi-project, interoperable dataset for medieval manuscript analysis. The segmentation dataset is a companion release providing layout annotations derived from the HTR ground truth creation pipeline.

Secondary: $\Psi_{\text{Evaluation}}$ The paper devotes a full section to benchmark split design (general vs. feature-based) and HTR baseline experiments with two engines across three seeds, reporting CER with standard deviations.

What is the motivation?

Digitization initiatives have made thousands of medieval manuscripts available as images, but most lack machine-readable text. Handwritten Text Recognition (HTR) is the key technology for conversion, yet building consistent ground truth across projects is difficult due to conflicting philological traditions, varying transcription standards, and project-specific annotation practices.

Prior to CATMuS, no single dataset provided cross-century, cross-language, and cross-script evaluation capability for medieval Latin-script manuscripts. Each project worked in isolation with incompatible conventions.

For layout analysis specifically, medieval manuscripts present challenges that modern document datasets do not cover: marginal glosses, drop capitals, music notation, quire marks, seals, and other elements unique to historical codices. The SegmOnto vocabulary was designed to provide a shared ontology for these elements across projects.

What is the novelty?

CATMuS Medieval is a macro-dataset assembled from 14 contributing repositories across multiple institutions. The core contribution is the unification effort: harmonizing transcription guidelines, segmentation practices, and metadata across projects that previously used incompatible standards.

The dataset construction followed three phases:

1. Phase 1: CREMMA and HTRomance projects formulated primary transcription guidelines for literary manuscripts in book scripts (Old French, Latin, Castilian, Italian).
2. Phase 2: The DEEDS project joined, adding Latin cartulary material.
3. Phase 3: Adoption by the Carthusian Monastery of Herne team and Middle English researchers.

Segmentation was performed primarily in eScriptorium using the BLLA model, with corrections. Transkribus data was repolygonized and post-corrected for compatibility. The segmentation dataset applies SegmOnto vocabulary for both zone-level and line-level classification.

Transcription Guidelines

A significant part of the paper’s contribution is the graphemic transcription approach, which enables interoperability across projects:

• Allograph normalization: Many-to-one mapping (e.g., round s and long f merged; round r and short r merged).
• Abbreviations preserved: Treated as an NLP task, not an HTR task. Diacritics in Unicode decomposed form (NFD).
• u/v and i/j normalization: The “Ramist” distinction is disregarded; v is transcribed as u, j as i.
• Punctuation standardized: Reduced to four signs: . (full stop), ; (double signs), , (comma), ? (question mark).
• Word segmentation: Semantic (lexical spaces) rather than imitative, addressing inconsistencies across medieval scribal practices.

These choices allow manuscripts from different projects, languages, and centuries to coexist in a single training set without conflicting conventions.

SegmOnto Classes (19 total)

Zone types (13):

Line types (6):

Note: The class inventory above is sourced from the HuggingFace dataset card. The ICDAR 2024 paper mentions region and line classes as metadata but does not enumerate the full SegmOnto vocabulary.

What experiments were performed?

The ICDAR 2024 paper evaluates HTR baselines (not layout analysis) on the companion CATMuS Medieval HTR dataset. Two HTR engines were benchmarked across two split strategies:

• General split (90/5/5): Every manuscript appears in all partitions. This tests in-domain generalization.
• Feature-based split (~85/7/7): Entire manuscripts are assigned to a single partition. This tests out-of-domain generalization to unseen manuscripts, scripts, and centuries.

The large CER gap between general and feature-based splits (e.g., 4.7% vs. 13.1% for Kraken) highlights the difficulty of generalizing to unseen manuscripts. Space-related errors are a notable component, reflecting inconsistent word separation practices across medieval languages and centuries.

No layout analysis models were evaluated on the segmentation dataset. The segmentation annotations serve as ground truth for line extraction in the HTR pipeline but have not been benchmarked independently for object detection or instance segmentation tasks.

What are the outcomes/conclusions?

The CATMuS Medieval project demonstrates that cross-institutional collaboration on annotation standards is feasible for historical manuscripts, producing a dataset of 208 documents, 165,347 lines, and 5.77 million characters across 10 languages and 9 centuries.

Key limitations:

• Imbalance: The Carthusian Monastery of Herne project alone contributes 27% of lines, creating overrepresentation of 14th-century Dutch Textualis. Old French, Latin, and Castilian together exceed 50% of the data.
• Rare languages: Venetian (2 documents), Navarese (1), and Old English (1) have minimal coverage.
• Image quality variation: 30 of 203 documents are grayscale (from microfilm), with varying resolution across sources.
• Layout coverage: The segmentation dataset (1,680 pages, per the HuggingFace release) is smaller than the full HTR dataset (165K+ lines across 208 documents), suggesting not all pages have full region-level annotations.
• No layout baselines: The paper does not evaluate layout detection performance, so it is unknown how well modern detectors handle the SegmOnto class set on this data.
• No segmentation pipeline code: The XML-to-Parquet conversion, SegmOnto remapping, and polygon extraction steps used to produce the segmentation dataset are not released as code.

Reproducibility

Models

Not applicable. This is a dataset paper. No layout models are proposed or released. The HTR baselines use default Kraken and recommended Pylaia architectures without modification.

Algorithms

HTR training hyperparameters reported in the paper:

• Batch size: 64
• Seeds: 21, 42, 84 (three runs per configuration)
• Early stopping: 5-epoch patience
• Learning rate: $1 \times 10^{-4}$ (Kraken), $5 \times 10^{-4}$ (Pylaia)
• Deterministic mode: Enabled
• Configuration: Default specifications for Kraken; recommended specifications for Pylaia (per Maarand et al.)

No layout-specific training algorithms are reported.

Data

The segmentation dataset is available on HuggingFace in Parquet format (statistics from the HuggingFace dataset card):

• Train: 1,340 pages
• Validation: 191 pages
• Test: 158 pages
• Total: 1,680 pages from 159 manuscripts

Each record includes the page image, bounding boxes, polygon masks, baselines, SegmOnto category labels, block/line type, parent-child hierarchy, manuscript shelfmark, century, and source project.

The broader HTR dataset is available separately on HuggingFace (CATMuS/medieval) and Zenodo, containing 165,347 extracted line images with transcriptions and rich metadata (language, script type, genre, century).

Annotation:

• Tool: eScriptorium (primary), Transkribus (Middle Dutch subset)
• Annotators: 45+ contributors across 14 source projects
• Process: Automatic segmentation via BLLA model, followed by manual correction
• Format: Polygonal masks with baselines; hierarchical (lines linked to parent blocks)

Evaluation

No layout evaluation metrics or baselines are reported for the segmentation dataset. HTR evaluation uses Character Error Rate (CER):

$$ \text{CER} = \frac{S + D + I}{N} $$

where $S$, $D$, and $I$ are the number of substitutions, deletions, and insertions respectively, and $N$ is the number of characters in the reference. The paper also reports a “space-related” CER isolating spacing errors, which is relevant given medieval word separation variability.

Hardware

HTR training was performed on NVIDIA RTX 8000 GPUs with 12 CPU cores per run. Kraken training took substantially longer than Pylaia ($\sim$2100 min vs. $\sim$300 min on the general split). No layout-specific training was reported.

BibTeX

  Copy
  

@inproceedings{clerice2024catmus,
  author    = {Cl{\'e}rice, Thibault and Pinche, Ariane and Vlachou-Efstathiou, Malamatenia and Chagu{\'e}, Alix and Camps, Jean-Baptiste and Gille Levenson, Matthias and Brisville-Fertin, Olivier and Boschetti, Federico and Fischer, Franz and Gervers, Michael and Boutreux, Agn{\`e}s and Manton, Avery and Gabay, Simon and O'Connor, Patricia and Haverals, Wouter and Kestemont, Mike and Vandyck, Caroline and Kiessling, Benjamin},
  title     = {{CATMuS Medieval: A multilingual large-scale cross-century dataset in Latin script for handwritten text recognition and beyond}},
  booktitle = {Document Analysis and Recognition -- ICDAR 2024},
  pages     = {174--194},
  year      = {2024},
  publisher = {Springer},
  address   = {Athens, Greece},
  doi       = {10.1007/978-3-031-70543-4_11}
}

TL;DR

This paper introduces DOC (Detect-Order-Construct), a three-stage pipeline for hierarchical document structure analysis that casts each stage as a relation prediction problem solved by multi-modal transformers. Alongside the method, the authors release Comp-HRDoc, a comprehensive benchmark built on HRDoc-Hard (1,500 documents) that evaluates four tasks simultaneously: page object detection, reading order prediction, table of contents extraction, and hierarchical structure reconstruction. DOC improves over prior methods across all four tasks, with margins detailed below.

What kind of paper is this?

Dominant: $\Psi_{\text{Method}}$. The headline contribution is the DOC framework itself: a three-stage architecture that decomposes hierarchical document structure analysis into Detect (page object detection + text region grouping), Order (reading order prediction), and Construct (TOC and tree building). Each stage is formulated as a relation prediction problem with dedicated transformer-based models, ablations, and comparisons against published results across multiple benchmarks.

Secondary: $\Psi_{\text{Resource}}$. Comp-HRDoc is a new benchmark extending HRDoc-Hard with annotations for four tasks, publicly released with evaluation scripts under MIT license. Document images must be obtained separately from the original HRDoc dataset.

Secondary: $\Psi_{\text{Evaluation}}$. The paper introduces REDS (Reading Edit Distance Score), a new metric for reading order evaluation that uses Levenshtein distance with Hungarian matching across multiple reading order groups.

What is the motivation?

Prior work on document structure analysis treats sub-tasks (layout detection, reading order, TOC extraction, hierarchical reconstruction) independently, using separate models and datasets. No unified benchmark existed to evaluate all four tasks together on the same documents. The original HRDoc dataset only provided line-level semantic labels and hierarchical annotations, lacking standard page-object-level detection and reading order evaluation.

The authors also argue that existing reading order metrics (BLEU, nDCG, Spearman) are borrowed from other domains and do not adequately capture document-specific errors like paragraph segmentation mistakes.

What is the novelty?

DOC Framework

The core insight is that all three stages of hierarchical document structure analysis can be unified as relation prediction problems:

1. Detect: A hybrid approach combining top-down graphical object detection (DINO or Mask2Former) with bottom-up text region detection. Text-lines from a PDF parser are grouped into text regions via intra-region reading order prediction, formulated as dependency parsing:

$$P(j \mid i) = \frac{\exp(f_{ij})}{\sum_{k} \exp(f_{ik})}$$

where $f_{ij}$ is a pairwise compatibility score between text-lines $i$ and $j$, computed from multi-modal features (visual via RoIAlign, text via BERT-BASE, 2D positional embeddings). Union-Find groups text-lines into regions based on predicted links.

2. Order: Detected page objects are encoded with attention-fused text-line features and region type embeddings, processed through a 3-layer transformer encoder. Inter-region reading order is predicted using the same dependency parsing formulation, with a separate classification head distinguishing text region reading order from graphical region relationships (caption-to-figure/table links).

3. Construct: Section headings (in predicted reading order) are encoded with Rotary Positional Embeddings (RoPE) and processed by a transformer encoder. A tree-aware TOC relation prediction head predicts both parent-child and sibling relationships:

$$P_{\text{parent}}(j \mid i) = \frac{\exp(f^{p}_{ij})}{\sum_{k} \exp(f^{p}_{ik})}$$

$$P_{\text{sibling}}(j \mid i) = \frac{\exp(f^{s}_{ij})}{\sum_{k} \exp(f^{s}_{ik})}$$

A serial tree insertion algorithm (Algorithm 1) scores candidate positions by multiplying parent and sibling probabilities along the rightmost sub-tree and inserts each heading at the best-scoring position.

Comp-HRDoc Benchmark

Extends HRDoc-Hard (1,500 documents, 1,000 train / 500 test) with:

• Page object detection annotations (12 categories: Title, Author, Mail, Affiliate, Section heading, Paragraph, Table, Figure, Caption, Footnote, Header, Footer) evaluated via segmentation-based mAP (not box-based, since logical paragraphs may span multiple columns)
• Reading order prediction evaluated via REDS
• TOC extraction and hierarchical structure reconstruction evaluated via Semantic-TEDS (Micro and Macro)

REDS Metric

REDS (Reading Edit Distance Score) addresses shortcomings of borrowed metrics (BLEU, nDCG) by using Levenshtein edit distance with Hungarian matching across reading order groups:

$$\text{REDS} = 1 - \frac{D}{N}$$

where $D$ is the minimum total Levenshtein distance after Hungarian matching across predicted and ground-truth reading order groups, and $N$ is the total number of basic evaluation units. It includes a special paragraph-ending tag to penalize paragraph segmentation errors.

What experiments were performed?

The authors evaluate DOC on four benchmarks:

• PubLayNet (360K pages, 5 classes): Page object detection. COCO box-based mAP.
• DocLayNet (80K pages, 11 classes): Page object detection. COCO box-based mAP.
• HRDoc (Simple: 1,000 docs; Hard: 1,500 docs): Semantic unit classification (F1) and hierarchical structure reconstruction (Semantic-TEDS).
• Comp-HRDoc (1,500 docs): All four tasks with dedicated metrics.

Baselines include Mask R-CNN, Faster R-CNN, YOLOv5, DINO, Mask2Former, DiT-L, VSR, LayoutLMv3, and the DSPS Encoder from HRDoc.

Ablation studies cover: (1) hybrid detection strategy effectiveness, (2) modality contributions in the Construct module, (3) component contributions to TOC extraction (sibling head, tree insertion algorithm, loss function choice).

What are the outcomes/conclusions?

Page Object Detection

The large Comp-HRDoc gap (+14.52) comes from the hybrid strategy: top-down detection handles graphical objects well, while bottom-up text-line grouping produces more precise text region boundaries than pure detection approaches.

Reading Order, TOC, and Hierarchy (Comp-HRDoc)

Key Ablation Findings

• Section number information is critical for the Construct module: without section numbers in the text features, Micro-STEDS drops from 0.861 to 0.641.
• The tree insertion algorithm is essential: removing it drops Micro-STEDS from 0.861 to 0.711.
• Softmax cross-entropy loss significantly outperforms binary cross-entropy for relation prediction (0.861 vs. 0.700).

Limitations

• The Construct module depends on accurate section heading recognition; errors in detection propagate to tree construction.
• Section number information is critical for TOC extraction; the approach may not generalize to documents without numbered sections.
• Only tree-structured documents are supported; graph-based logical structures are not addressed.
• The system assumes text-line bounding boxes and content are provided by a PDF parser or OCR engine, limiting direct applicability to image-only documents without a prior OCR step.
• Inconsistent paragraph annotations in HRDoc may affect paragraph detection performance.

Reproducibility

Models

• Detect: ResNet-50 backbone (PubLayNet/DocLayNet) or ResNet-18 (HRDoc/Comp-HRDoc). Graphical object detector: DINO (PubLayNet/DocLayNet) or Mask2Former (HRDoc/Comp-HRDoc). Custom bottom-up text region detector with 1-layer transformer encoder (12 heads, 768-dim, 2048-dim FFN).
• Order: 3-layer transformer encoder for inter-region relation prediction
• Construct: Transformer encoder with RoPE; tree-aware relation prediction head with parent-child and sibling sub-heads
• Text encoder: BERT-BASE (pretrained), averaged token embeddings per text-line
• No model weights or training code released. The GitHub repo contains only benchmark annotations and evaluation scripts.

Algorithms

• Optimizer: AdamW (betas 0.9/0.999, epsilon 1e-8)
• Learning rates: 1e-5 (backbone) / 2e-5 (BERT) for PubLayNet/DocLayNet; 2e-4 (backbone) / 4e-5 (BERT) for HRDoc/Comp-HRDoc
• Weight decay: 1e-4 (backbone, PubLayNet/DocLayNet); 1e-2 (backbone, HRDoc/Comp-HRDoc); 1e-2 (BERT, all)
• Schedule: PubLayNet/DocLayNet: step decay (LR / 10 at epoch 11 / 20 respectively). HRDoc/Comp-HRDoc: linear warmup (2 epochs) then linear decay to 0.
• Epochs: 12 (PubLayNet), 24 (DocLayNet), 20 (HRDoc/Comp-HRDoc)
• Batch size: 16 (PubLayNet/DocLayNet), 1 (HRDoc/Comp-HRDoc, multi-page documents)
• Multi-scale training: Shorter side randomly sampled from [512, 640, 768] (max 800) or [320, 416, 512, 608, 704, 800] (max 1024)
• Test resolution: Shorter side 640 (PubLayNet/DocLayNet), 512 (HRDoc/Comp-HRDoc)
• Loss: Softmax cross-entropy for relation prediction; standard detection losses for DINO/Mask2Former
• Framework: PyTorch v1.10

Data

• Comp-HRDoc: 1,000 train / 500 test documents, built on HRDoc-Hard. Annotations publicly available at GitHub under MIT license. Document images must be obtained separately from the HRDoc dataset (license unknown).
• PubLayNet: 340K train / 12K val / 12K test. CDLA-Perm-1.0 (annotations); underlying PDFs non-commercial.
• DocLayNet: 69K train / 6.5K test / 5K val. CDLA-Perm-1.0 (annotations).
• HRDoc: Simple (1,000 docs) and Hard (1,500 docs). Mixed licenses (CC-BY, CC-BY-NC-SA).

Evaluation

• Page object detection: COCO-style mAP (box-based for PubLayNet/DocLayNet; segmentation-based for Comp-HRDoc)
• Reading order: REDS (new metric introduced in this paper)
• TOC extraction and hierarchy: Semantic-TEDS Micro and Macro (from HRDoc)
• Semantic unit classification (HRDoc): Per-category F1, Micro/Macro averages
• Comp-HRDoc evaluation scripts released at GitHub
• No error bars or multi-run statistics reported

Hardware

• Training: 8 NVIDIA Tesla V100 GPUs (32 GB each)
• No inference latency, throughput, or cost estimates reported
• Batch size of 1 for Comp-HRDoc due to multi-page document processing; memory constraints likely driven by processing dozens of pages per document

BibTeX

@article{wang2024detect,
  title={Detect-Order-Construct: A Tree Construction based Approach for Hierarchical Document Structure Analysis},
  author={Wang, Jiawei and Hu, Kai and Zhong, Zhuoyao and Sun, Lei and Huo, Qiang},
  journal={Pattern Recognition},
  volume={156},
  pages={110836},
  year={2024},
  publisher={Elsevier},
  doi={10.1016/j.patcog.2024.110836}
}

TL;DR

DocHieNet is a manually annotated, multi-page, bilingual (English + Chinese) dataset of 1,673 documents (15,610 pages) spanning legal, financial, educational, and scientific domains for document hierarchy parsing. It annotates 187K+ layout elements across 19 categories with hierarchical and sequential relationships, including 37.4% cross-page relations. The authors also propose DHFormer, a transformer-based baseline that uses sparse text-layout encoding and a global layout element decoder, reaching 77.82 F1 on DocHieNet while achieving 93-99 F1 on prior datasets.

What kind of paper is this?

Dominant: $\Psi_{\text{Resource}}$. The headline contribution is the DocHieNet dataset: a multi-domain, bilingual, multi-page dataset for document hierarchy parsing with manually verified annotations, filling a gap left by existing single-domain or single-page resources. The paper devotes its primary sections to data collection, annotation design, dataset statistics, and comparisons with existing resources.

Secondary: $\Psi_{\text{Method}}$. DHFormer is a non-trivial methodological contribution: a sparse text-layout encoder with chunked attention plus a global layout element decoder for multi-page hierarchy prediction, evaluated across five datasets with ablations.

What is the motivation?

Existing document hierarchy parsing datasets are limited in scope:

• arXivdocs is single-page and English-only; E-Periodica is single-page (though multilingual: EN, DE, FR, IT). Neither captures cross-page relationships.
• HRDoc covers only scientific papers with automatically generated (not manually verified) annotations at the text-line level
• No existing dataset spans multiple document domains (legal, financial, educational, technical)

The authors argue that layout-block-level annotation is more practical than text-line-level because (a) it leverages existing layout analysis models, (b) text-line evaluation inflates scores since trivial “connect” relations dominate, and (c) layout-block evaluation better reflects actual hierarchy prediction quality.

What is the novelty?

DocHieNet Dataset

• 1,673 documents (1,110 English, 563 Chinese) across 15,610 pages with 187K+ layout elements
• 19 layout element categories: title, sub-title, section-title, text, formula, TOC-title, TOC, figure, figure-title, figure-caption, table, table-title, table-caption, header, footer, page-number, footnote, endnote, sidebar
• Two relationship types: hierarchical (parent-child) and sequential (reading order)
• Multi-domain: Legal, financial, educational, technical, scientific, government publications
• Cross-page relations: 37.4% macro-average ratio with 5.4-page average span (vs. HRDoc’s 24.9% ratio and 2.4-page span)
• Manual annotation: 12 experienced annotators with 3 rounds of specialist quality checks
• Split: 1,512 train / 161 test documents. 835 training documents are truncated (partially annotated) to reduce pattern redundancy; test documents are fully annotated.

DHFormer

A transformer-based framework for document hierarchy parsing with three components:

1. Sparse Text-Layout Encoder: Uses a pretrained layout-aware language model (GeoLayoutLM). Documents are split into $K$ chunks where each chunk contains the maximum number of layout elements whose total tokens do not exceed a limit $l = 512$. Attention is dense within chunks but factorized across chunks, reducing complexity from $O(N^2)$ to $O(l \cdot N)$.

2. Page and Inner-Layout Embeddings: Sinusoidal page embeddings encode absolute page number; 1D inner-layout position embeddings encode relative token position within each layout element.

3. Global Layout Element Decoder: A 2-layer transformer decoder with Shifted Sparse Attention (window size 48) that pools each layout element’s representation via its first token. A learnable root embedding is prepended. Relation prediction uses a bilinear layer with sigmoid activation trained with cross-entropy loss.

What experiments were performed?

The authors evaluate DHFormer on five datasets, all mapped to DocHieNet’s annotation format for comparability:

• DocHieNet (1,673 docs, multi-domain, bilingual)
• arXivdocs (362 single-page scientific documents)
• HRDoc-Simple (1,000 docs) and HRDoc-Hard (1,500 docs)
• E-Periodica (542 single-page magazine pages)

Metrics: F1 and TEDS (Tree Edit Distance Score).

Baselines: DocParser, DSPS (HRDoc baseline), DOC (Comp-HRDoc baseline), DSG (graph-based), GPT-4 (in-context learning), Llama2-7b (LoRA fine-tuned).

Ablations: (1) encoder choice (GeoLayoutLM vs. LayoutLMv3 vs. BROS vs. XLM-RoBERTa), (2) chunking strategy and window size, (3) page and inner-layout embeddings, (4) text-line vs. layout-block annotation paradigm, (5) LLM comparison across document lengths.

What are the outcomes/conclusions?

Main Results (all datasets in DocHieNet format)

DocHieNet is considerably more challenging than prior datasets: even DHFormer achieves only 77.82 F1, compared to 93+ on the others.

Key Findings

• Encoder matters: GeoLayoutLM (layout-aware) outperforms XLM-RoBERTa (text-only) by 8.7 F1 points (77.82 vs. 69.13).
• Both embedding types help: Page embeddings add ~2 F1 and inner-layout position embeddings add ~1.5 F1; combined they add ~4 F1 (73.66 to 77.82).
• Chunking strategy: Layout-element-aligned chunking at 512 tokens (77.82 F1) outperforms page-level chunking (75.66) and stride-based chunking (73.98).
• LLM baselines lag behind: DHFormer (~80 F1) outperforms fine-tuned Llama2-7b (~55-65 F1) and GPT-4 ICL (~25-50 F1), with LLM performance degrading sharply as document length increases (approximate values read from Figure 5; no table provided).
• Annotation paradigm: Text-line-level evaluation (as in HRDoc) inflates scores because trivial “connect” relations dominate. Layout-block evaluation is more informative.

Limitations

• The dataset, while multi-domain, may not cover all document variations in the wild.
• 835 of 1,673 training documents are truncated (partially annotated), which may affect model training for long-document scenarios.
• The dataset requires layout element detection as a prerequisite; end-to-end evaluation with a layout detector (CenterNet) drops F1 from 99.34 to 94.32 on HRDoc-Simple.
• No arXiv preprint available; the paper is only accessible through ACL Anthology.

Reproducibility

Models

• DHFormer: Sparse text-layout encoder (GeoLayoutLM default) + 2-layer transformer decoder with Shifted Sparse Attention (window size 48)
• Text encoder variants evaluated: GeoLayoutLM, LayoutLMv3, BROS, XLM-RoBERTa
• The GitHub repo contains only a README and LICENSE file; no runnable training or inference code is available. The README states data and model “will be made publicly available in the near future.” The dataset is hosted on ModelScope.
• No pretrained model weights or evaluation scripts released.

Algorithms

• Optimizer: AdamW, base LR 4e-5 decaying to 1e-6
• Epochs: 100
• Max tokens per chunk: 512
• Max chunks: 32 (training), 128 (testing)
• Decoder: 2-layer transformer, window size 48 for Shifted Sparse Attention
• Relation prediction: Bilinear layer with sigmoid, cross-entropy loss
• Batch size: Not reported
• Framework: Not explicitly stated (likely PyTorch given GeoLayoutLM dependency)

Data

• DocHieNet: 1,512 train / 161 test documents. Available on ModelScope under Apache-2.0.
• Languages: English (1,110 docs) + Chinese (563 docs)
• Annotation: Manual by 12 annotators with 3 rounds of specialist QC
• 835 training documents are truncated (partially annotated); all 161 test documents are fully annotated
• Comparison datasets (arXivdocs, HRDoc, E-Periodica) available separately with their own licenses

Evaluation

• Metrics: F1 (relation-level) and TEDS (Tree Edit Distance Score: 1 minus the normalized tree edit distance between predicted and ground-truth document hierarchies; document-level)
• All datasets mapped to DocHieNet annotation format for cross-dataset comparison
• Cross-format evaluation also performed (train in one format, test in another)
• No error bars or multi-run statistics reported
• LLM comparison included (GPT-4 ICL, Llama2-7b LoRA)

Hardware

• DHFormer training: 2 NVIDIA Tesla V100 GPUs
• Llama2-7b fine-tuning: 2 NVIDIA A100 GPUs, LoRA (rank 8, alpha 32, dropout 0.05)
• No inference latency or throughput reported

BibTeX

@inproceedings{xing2024dochiennet,
  title={DocHieNet: A Large and Diverse Dataset for Document Hierarchy Parsing},
  author={Xing, Hangdi and Cheng, Changxu and Gao, Feiyu and Shao, Zirui and Yu, Zhi and Bu, Jiajun and Zheng, Qi and Yao, Cong},
  booktitle={Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing},
  pages={1129--1142},
  year={2024},
  doi={10.18653/v1/2024.emnlp-main.65}
}

TL;DR

LayoutLMv2 is the second generation of the LayoutLM family, introducing visual features directly into the pre-training stage (rather than only at fine-tuning time, as in LayoutLMv1). It combines a multi-modal Transformer encoder with a ResNeXt-101 FPN visual backbone, a spatial-aware self-attention mechanism for 2D relative position modeling, and two new pre-training objectives (Text-Image Alignment and Text-Image Matching). It achieved SOTA on six document understanding benchmarks at the time of publication.

What kind of paper is this?

Dominant: $\Psi_{\text{Method}}$

The headline contribution is the model architecture and pre-training strategy. The paper introduces three key innovations: (1) early visual integration via a trainable CNN backbone, (2) spatial-aware self-attention with 2D relative position biases, and (3) two new cross-modal pre-training objectives. The majority of the paper describes these components and their ablations.

Secondary: $\Psi_{\text{Evaluation}}$

The paper evaluates on six diverse benchmarks (FUNSD, CORD, SROIE, Kleister-NDA, RVL-CDIP, DocVQA) and provides a systematic ablation study isolating the contribution of each component.

What is the motivation?

LayoutLMv1 demonstrated that jointly pre-training text and layout (2D position) embeddings from large-scale unlabeled documents improves downstream document understanding tasks. However, v1 had a significant limitation: visual features from a CNN were only added during fine-tuning, not during pre-training. This meant the model could not learn cross-modal interactions between text, layout, and visual appearance from the 11M unlabeled IIT-CDIP pages.

The authors also identified that absolute 2D position embeddings (as used in v1) encode where a token is on the page but not how tokens relate to each other spatially. Two tokens in the same relative arrangement (e.g., a label above a value) at different absolute positions would produce different representations.

What is the novelty?

1. Early Visual Integration

Instead of adding visual features only at fine-tuning time, LayoutLMv2 passes the document page image through a ResNeXt-101 FPN visual encoder during pre-training. The output feature map is average-pooled to $W \times H = 7 \times 7 = 49$ visual tokens, each projected to the same dimensionality as text tokens. Visual and text tokens are concatenated into a single sequence:

$$x_i^{(0)} = X_i + l_i, \quad X = \{v_0, \ldots, v_{WH-1}, t_0, \ldots, t_{L-1}\}$$

where $l_i$ is the layout (2D position) embedding and $X_i$ is either a visual or text embedding. The CNN backbone is updated through backpropagation during pre-training, initialized from a Mask R-CNN trained on PubLayNet.

2. Spatial-Aware Self-Attention

The standard self-attention score $\alpha_{ij}$ is augmented with learnable relative position biases:

$$\alpha’_{ij} = \alpha_{ij} + b^{(1D)}_{j-i} + b^{(2D_x)}_{x_j - x_i} + b^{(2D_y)}_{y_j - y_i}$$

where $(x_i, y_i)$ are the top-left bounding box coordinates of the $i$-th token. The biases are shared across encoder layers but differ across attention heads, adding minimal parameters while encoding spatial relationships.

3. New Pre-training Objectives

In addition to Masked Visual-Language Modeling (MVLM, inherited from v1):

• Text-Image Alignment (TIA): Randomly cover line-level image regions and train the model to predict which text tokens have been covered. This enforces fine-grained spatial correspondence between text and image. Image regions corresponding to MVLM-masked tokens are also covered to prevent visual leakage.
• Text-Image Matching (TIM): Binary classification on whether the image and text come from the same page. 15% of images are replaced with a different document page; 5% are dropped entirely.

What experiments were performed?

Pre-training

Pre-trained on 11M pages from IIT-CDIP Test Collection. Text is extracted via OCR (Microsoft Read API). Two model sizes: BASE (200M params, 12 layers, $d=768$) and LARGE (426M params, 24 layers, $d=1024$). Both use the same ResNeXt-101 FPN visual backbone. Text encoder initialized from UniLMv2; visual backbone initialized from a Mask R-CNN trained on PubLayNet.

Downstream Tasks

Six benchmarks spanning four task types:

*SROIE F1 of 0.9781 excludes OCR mismatch errors; the raw F1 is 0.9661. The DocVQA score of 0.8672 uses train+dev fine-tuning with question-generation data augmentation.

Ablation Study (DocVQA val, BASE size)

The ablation isolates each component’s contribution:

TIA contributes more than TIM. Spatial-aware self-attention adds roughly 1 ANLS point. Better text-side initialization (UniLMv2 over BERT) provides a further 2-point gain.

What are the outcomes/conclusions?

LayoutLMv2 LARGE set new SOTA on all six benchmarks at the time of publication, with particularly large gains on DocVQA (+13.8 ANLS over v1) and FUNSD (+5.3 F1 over v1). The results demonstrate that integrating visual features during pre-training, rather than only at fine-tuning, yields substantial improvements across diverse document understanding tasks.

The ablation confirms that each component contributes: visual integration, TIA, TIM, and spatial-aware attention all provide additive gains. The largest single contributor is the visual module with TIA, suggesting that fine-grained text-image spatial alignment is the most impactful pre-training signal.

Limitations

• The model requires OCR as a preprocessing step, inheriting OCR errors.
• The visual backbone (ResNeXt-101 FPN) adds computational cost; the 49 visual tokens increase sequence length.
• Pre-training on 11M IIT-CDIP pages is computationally expensive (exact GPU-hours not reported).
• The CC-BY-NC-SA-4.0 license on released model weights restricts commercial use.
• No layout detection experiments are reported; the paper focuses on document understanding (entity extraction, classification, VQA) rather than bounding-box detection.

Reproducibility

Models

• LayoutLMv2 BASE: 200M parameters. 12-layer, 12-head Transformer encoder, $d = 768$. ResNeXt-101 FPN visual backbone.
• LayoutLMv2 LARGE: 426M parameters. 24-layer, 16-head Transformer encoder, $d = 1024$. Same visual backbone.
• Both model checkpoints released on Hugging Face under CC-BY-NC-SA-4.0.
• Text encoder initialized from UniLMv2; visual backbone initialized from Mask R-CNN trained on PubLayNet (“MaskRCNN ResNeXt101_32x8d FPN 3X” from detectron2).

Algorithms

• Pre-training: Maximum sequence length $L = 512$. Visual tokens: $7 \times 7 = 49$. MVLM masks 15% of tokens (80% [MASK], 10% random, 10% unchanged). TIA covers 15% of lines. TIM replaces 15% of images and drops 5%.
• Optimizer: Adam with learning rate $2 \times 10^{-5}$, weight decay $1 \times 10^{-2}$, $(\beta_1, \beta_2) = (0.9, 0.999)$. Linear warmup over the first 10% of steps, then linear decay.
• Pre-training schedule: BASE is trained with batch size 64 for 5 epochs; LARGE is trained with batch size 2048 for 20 epochs on IIT-CDIP. (The ablation study uses 1-epoch runs for efficiency.)
• Fine-tuning: Task-specific heads on top of encoder outputs. DocVQA additionally benefits from a question-generation data augmentation step. Fine-tuning hyperparameters (learning rates, batch sizes, epochs per task) are not reported in the paper.

Data

• Pre-training: IIT-CDIP Test Collection 1.0 (11M document images). Text extracted via Microsoft Read API (a commercial OCR service). The IIT-CDIP dataset requires institutional access and is not freely downloadable.
• Downstream: FUNSD (149/50 train/test), CORD (800/100/100), SROIE (626/347), Kleister-NDA (254/83/203), RVL-CDIP (320K/4K/4K), DocVQA (39K/5K/5K).

Evaluation

• Entity extraction: entity-level F1 score.
• Classification: accuracy.
• DocVQA: Average Normalized Levenshtein Similarity (ANLS).
• Ablation on DocVQA validation set with BASE models pre-trained for one epoch.
• No error bars or multi-seed runs reported.

Hardware

• Not specified in the main paper. Pre-training on 11M pages with a 200M-426M parameter model with a ResNeXt-101 FPN backbone requires substantial GPU resources. The original LayoutLM was pre-trained on 8x V100 GPUs; LayoutLMv2 likely requires similar or greater resources.

BibTeX

@inproceedings{xu2021layoutlmv2,
  title={LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding},
  author={Xu, Yang and Xu, Yiheng and Lv, Tengchao and Cui, Lei and Wei, Furu and Wang, Guoxin and Lu, Yijuan and Florencio, Dinei and Zhang, Cha and Che, Wanxiang and Zhang, Min and Zhou, Lidong},
  booktitle={Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)},
  pages={2579--2591},
  year={2021},
  publisher={Association for Computational Linguistics},
  doi={10.18653/v1/2021.acl-long.201}
}

TL;DR

LayoutXLM is the multilingual extension of LayoutLMv2. It uses the same architecture (multimodal Transformer with ResNeXt-FPN visual backbone and spatial-aware self-attention) but pre-trains on 30M documents across 53 languages. The paper also introduces XFUND, a manually labeled multilingual form understanding benchmark in 7 languages. LayoutXLM outperforms multilingual text-only baselines (XLM-R, InfoXLM) on cross-lingual document understanding tasks.

What kind of paper is this?

Dominant: $\Psi_{\text{Method}}$. The headline contribution is extending LayoutLMv2 to the multilingual setting with a large-scale cross-lingual pre-training corpus.

Secondary: $\Psi_{\text{Resource}}$. The XFUND dataset (1,393 forms across 7 languages with key-value pair annotations) is a meaningful data contribution.

Secondary: $\Psi_{\text{Evaluation}}$. Benchmarks on XFUND with zero-shot transfer and multitask fine-tuning settings.

What is the motivation?

Nearly 40% of digital documents on the web are in non-English languages. LayoutLMv2 was pre-trained only on English documents (IIT-CDIP), limiting its cross-lingual capabilities. Multilingual text-only models (mBERT, XLM-R) lack the layout and visual understanding needed for document tasks. Translating documents via machine translation is unsatisfactory because it loses layout structure.

What is the novelty?

1. Multilingual pre-training corpus. 22M multilingual PDFs scraped from Common Crawl across 53 languages, combined with 8M English documents from IIT-CDIP (30M total). The multilingual PDFs are filtered for quality and classified by language using fastText.

2. Architecture. Same as LayoutLMv2 (multimodal Transformer + ResNeXt-FPN + spatial-aware self-attention), but initialized from InfoXLM (a multilingual pre-trained model) instead of UniLMv2. Three pre-training objectives: MVLM, TIA, TIM.

3. XFUND benchmark. 1,393 manually annotated forms across 7 languages (Chinese, Japanese, Spanish, French, Italian, German, Portuguese). 199 forms per language (149 train / 50 test). Two tasks: Semantic Entity Recognition (SER) and Relation Extraction (RE) for key-value pairs.

What experiments were performed?

Three evaluation settings on XFUND:

• Language-specific fine-tuning: Train and test on the same language.
• Zero-shot transfer: Fine-tune on English FUNSD, evaluate on each target language.
• Multitask fine-tuning: Fine-tune on all 8 languages jointly.

LayoutXLM BASE and LARGE are compared against XLM-R, InfoXLM, and LayoutLMv2.

What are the outcomes/conclusions?

LayoutXLM outperforms text-only multilingual baselines on both SER and RE tasks. On XFUND SER (language-specific), LayoutXLM BASE achieves an average F1 of 0.8056 across 7 languages, compared to 0.7047 for XLM-R BASE. Zero-shot transfer from English to other languages is feasible but shows substantial performance drops, confirming that layout understanding transfers better than textual semantics across languages.

Limitations

• CC-BY-NC-SA-4.0 license on model weights restricts commercial use.
• Requires OCR as preprocessing.
• The 30M pre-training corpus is not publicly released.
• XFUND covers only form understanding; no layout detection or document classification benchmarks for multilingual evaluation.

Reproducibility

Models

• LayoutXLM BASE: 369M parameters. Initialized from InfoXLM BASE + ResNeXt-101 FPN.
• LayoutXLM LARGE: 625M parameters. Initialized from InfoXLM LARGE + ResNeXt-101 FPN.

Data

• Pre-training: 30M documents (22M multilingual PDFs + 8M IIT-CDIP English). OCR via Microsoft Read API.
• XFUND: 1,393 forms, 7 languages, manually annotated key-value pairs.

Algorithms

• Optimizer, learning rate, batch size, warmup schedule, and total training steps are not reported in the paper.
• Three pre-training objectives are identical to LayoutLMv2: Masked Visual-Language Modeling (MVLM), Text-Image Alignment (TIA), and Text-Image Matching (TIM).
• OCR preprocessing uses the commercial Microsoft Read API.

Evaluation

• Semantic Entity Recognition (SER) is evaluated with entity-level F1.
• Relation Extraction (RE) is evaluated with pair-level F1.
• No error bars or multi-run statistics are reported.

Hardware

• Not specified in the paper.

BibTeX

@inproceedings{xu2022layoutxlm,
  title={LayoutXLM: Multimodal Pre-training for Multilingual Visually-rich Document Understanding},
  author={Xu, Yiheng and Lv, Tengchao and Cui, Lei and Wang, Guoxin and Lu, Yijuan and Florencio, Dinei and Zhang, Cha and Wei, Furu},
  booktitle={Findings of the Association for Computational Linguistics: ACL 2022},
  pages={3184--3196},
  year={2022},
  doi={10.18653/v1/2022.findings-acl.253}
}

TL;DR

LiLT (Language-independent Layout Transformer) is a dual-stream Transformer that separates text and layout processing into parallel flows connected by a Bi-directional Attention Complementation Mechanism (BiACM). The key insight: pre-train the layout stream on English-only structured documents (IIT-CDIP), then swap in any off-the-shelf text encoder (monolingual or multilingual) at fine-tuning time. This avoids the costly multilingual pre-training corpus that LayoutXLM requires while achieving competitive or superior results on 8 languages.

What kind of paper is this?

Dominant: $\Psi_{\text{Method}}$. The core contribution is the disentangled dual-stream architecture and BiACM mechanism that enables language-independent layout pre-training.

Secondary: $\Psi_{\text{Evaluation}}$. Comprehensive benchmarking on FUNSD, CORD, EPHOIE, RVL-CDIP, and the multilingual XFUND benchmark across 8 languages.

What is the motivation?

LayoutXLM demonstrated that multilingual document understanding requires both layout and text signals. However, it requires scraping 22M multilingual PDFs, cleaning them, and pre-training a 369M+ parameter model from scratch. LiLT asks: can we learn layout structure from English-only documents and transfer that knowledge to other languages by simply swapping the text encoder?

The key observation is that document layout structure is largely language-independent. The spatial arrangement of fields on a form looks similar whether the text is in English, Chinese, or Arabic. If the layout knowledge can be separated from the language knowledge, each can be learned independently.

What is the novelty?

Dual-Stream Architecture

LiLT uses two parallel Transformer streams:

• Text flow: Processes token embeddings + 1D positional embeddings. Standard Transformer architecture.
• Layout flow: Processes 2D positional (bounding box) embeddings only. Reduced hidden size for efficiency (adds only 6.1M parameters to the text flow).

BiACM (Bi-directional Attention Complementation Mechanism)

At each layer, the attention weights from each stream are added to the other stream’s attention computation before the softmax:

$$A_T’ = \text{Softmax}(\frac{Q_T K_T^T}{\sqrt{d_T}} + \text{DETACH}(\frac{Q_L K_L^T}{\sqrt{d_L}}))$$

$$A_L’ = \text{Softmax}(\frac{Q_L K_L^T}{\sqrt{d_L}} + \text{DETACH}(\frac{Q_T K_T^T}{\sqrt{d_T}}))$$

The DETACH operation (gradient stopping during pre-training) prevents the layout flow from corrupting the text encoder’s representations. At fine-tuning time, DETACH is removed to allow full end-to-end optimization.

Pre-training Objectives

Three objectives, all operating on text tokens:

1. MVLM: Standard masked visual-language modeling.
2. KPL (Key Point Location): Predict the top-left and bottom-right corner coordinates of masked bounding boxes.
3. CAI (Cross-modal Alignment Identification): Binary classification of whether each text-layout pair is correctly aligned or randomly shifted.

Asynchronous Optimization

The text flow’s learning rate is slowed down by a factor of 1000x relative to the layout flow during pre-training. This prevents the text encoder from overfitting to English while allowing the layout encoder to fully learn spatial structure.

What experiments were performed?

Monolingual Benchmarks

LiLT with English RoBERTa reports the highest F1 among the compared baselines on FUNSD and competitive results on CORD and RVL-CDIP, outperforming both LayoutLMv2 and LayoutXLM at BASE size. Note that on CORD, DocFormer BASE (0.9633 F1) outperforms LiLT; the table above shows only the most relevant layout-aware baselines.

Multilingual (XFUND)

The paper evaluates three settings on XFUND: (1) language-specific fine-tuning (train and test on the same language), (2) zero-shot transfer (train on English, test on target language), and (3) multitask fine-tuning (train on all languages jointly). On zero-shot transfer, LiLT with InfoXLM achieves competitive performance with LayoutXLM despite using only English pre-training data. On language-specific fine-tuning, LiLT often matches or exceeds LayoutXLM.

What are the outcomes/conclusions?

LiLT demonstrates that layout structure is sufficiently language-independent that it can be learned from a single language and transferred to others. The disentangled architecture enables a practical workflow: pre-train layout once on English, then combine with any text encoder for the target language.

The BiACM mechanism is critical. Without it (plain concatenation), performance drops sharply. Replacing it with co-attention (a common dual-stream approach) also hurts performance, because deeper cross-modal interaction damages the text flow’s consistency during pre-training.

Limitations

• No visual features (no CNN backbone). LiLT uses only text + layout, unlike LayoutLMv2/LayoutXLM which also use document images.
• Only BASE size evaluated; no LARGE variant.
• Pre-trained on IIT-CDIP only (English); adding multilingual layout data could further help.

Reproducibility

Models

• LiLT BASE: 12-layer layout encoder with 192 hidden size, 768 FFN size, 12 attention heads (6.1M parameters). Combined with a text encoder at fine-tuning time.
• Released checkpoints: lilt-roberta-en-base (293 MB, English), lilt-infoxlm-base (846 MB, multilingual), and lilt-only-base (21 MB, layout flow only). Available on both GitHub (OneDrive links) and HuggingFace under MIT license.
• The text flow is initialized from an existing pre-trained model (RoBERTa BASE or InfoXLM BASE) and can be swapped at fine-tuning time.

Algorithms

• Pre-training: Adam optimizer ($\beta_1 = 0.9$, $\beta_2 = 0.999$), learning rate $2 \times 10^{-5}$ with linear warmup over the first 10% of steps followed by linear decay, weight decay $1 \times 10^{-2}$, batch size 96, 5 epochs on IIT-CDIP.
• Text flow slow-down ratio: 1000x (text learning rate = base rate / 1000).
• Max sequence length: 512.

Data

• Pre-training: IIT-CDIP Test Collection 1.0 (English only): 6M+ documents, 11M+ scanned images. OCR via TextIn API.
• Ablation pre-training: 2M randomly sampled documents from IIT-CDIP (5 epochs).
• Fine-tuning benchmarks: FUNSD (149/50 train/test forms), CORD (800/100/100), EPHOIE (1183/311 Chinese exam papers), RVL-CDIP (320K/40K/40K), XFUND (149/50 per language, 7 languages + English FUNSD).

Evaluation

• Metrics: Entity-level F1 for SER (FUNSD, CORD, EPHOIE, XFUND); overall accuracy for document classification (RVL-CDIP); F1 for relation extraction (XFUND RE).
• Baselines: LayoutLM, LayoutLMv2, LayoutXLM, BROS, DocFormer, and text-only models (BERT, RoBERTa, InfoXLM).
• Statistical rigor: No error bars, confidence intervals, or multi-run statistics reported.

Hardware

• 4x NVIDIA A40 (48 GB) GPUs for pre-training.
• Training time not reported.

BibTeX

@inproceedings{wang2022lilt,
  title={LiLT: A Simple yet Effective Language-Independent Layout Transformer for Structured Document Understanding},
  author={Wang, Jiapeng and Jin, Lianwen and Ding, Kai},
  booktitle={Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  pages={7747--7757},
  year={2022},
  doi={10.18653/v1/2022.acl-long.534}
}

TL;DR

The PAL Database is a large-scale document layout analysis dataset covering 37,910 Spanish government gazettes (441,300 pages) with over 8 million layout labels across 8 categories. Annotations were generated through a semi-automatic pipeline combining PDF mining, heuristic rules, and human-in-the-loop Random Forest classifiers. The dataset is gated behind a signed license agreement.

What kind of paper is this?

Dominant: $\Psi_{\text{Resource}}$. The headline contribution is the PAL Database itself: a new domain-specific dataset for document layout analysis in the public affairs domain. The paper devotes most of its space to describing the data collection pipeline, annotation process, and dataset statistics.

Secondary: $\Psi_{\text{Method}}$. The semi-automatic annotation pipeline (heuristic rules followed by iterative human-in-the-loop Random Forest classifiers) is a reusable methodological contribution beyond the dataset itself.

What is the motivation?

Existing DLA datasets (PubLayNet, DocBank, DocLayNet) focus on scientific articles, financial reports, and similar domains. Government and legal documents have distinct visual layouts (multi-column gazettes, structured metadata blocks, legal identifiers) that are underrepresented in current benchmarks. The authors argue that public affairs documents require their own annotated corpus because:

• The layout structure differs from academic and business documents
• Existing models may not generalize well to this domain
• Government document processing is a practical need for legal and administrative automation

Beyond layout analysis, the authors also frame the corpus as a large-scale NLP pre-training resource for the legal domain in Spanish and four co-official languages (Basque, Catalan/Valencian, Galician).

What is the novelty?

The core novelty is the dataset and its semi-automatic annotation pipeline:

1. Data Collection: Web scraping of 24 Spanish administration sources (3 national, 19 regional, 2 local) using Scrapy. Only born-digital PDFs from 2014 onward are included.

2. Feature Extraction: PyMuPDF extracts images, links, and text blocks with 18 features per block (bounding box, font size, bold/italic proportions, capitalization ratios, etc.). Camelot extracts tables.

3. Two-step Annotation:

   • Step 0 (Heuristic): Rule-based labeling using font features and keyword matching, followed by human validation through a custom annotation application
   • Step 1 (ML-assisted): Random Forest classifiers trained on validated documents (minimum 50 validated pages per source), with iterative human correction

4. Label Taxonomy: 8 categories split into 4 basic layout blocks (Image, Table, Link, Text) and 4 text semantic categories (Identifier, Title, Summary, Body).

The per-source Random Forest approach acknowledges that government gazette layouts vary significantly across administrative regions, each requiring its own classifier.

What experiments were performed?

The evaluation focuses on validating the text-block classification quality:

• Model: Random Forest classifiers (max depth 1000), one per data source (24 total)
• Features: 17 of 18 extracted features (raw text excluded). Positional features normalized by document dimensions. Font type encoded via dictionary mapping.
• Protocol: 10-fold cross-validation with 80/20 document-level splits on the validation set (1,444 documents, 19,276 pages)
• Metric: Accuracy on 4-class text block classification (Identifier, Title, Summary, Body)

What are the outcomes/conclusions?

Overall accuracy ranges from 93.37% (Source 16, Ceuta) to 99.93% (Source 20), with 20 of 24 sources exceeding 96%.

Per-class performance:

• Identifier: Highest accuracy; mean above 99% for 22 of 24 sources. These blocks have consistent position, token count, and font characteristics across documents.
• Body: All sources above 95%. The dominant class, easily distinguished by font regularity.
• Title: Hardest category; all sources above 84%, 19 above 90%. The authors note titles are the hardest to label consistently due to varying editorial conventions across sources.
• Summary: Generally high, but Source 16 (Ceuta) is an outlier at 61.08% ($\pm$12.35 std), indicating high instability for that source.

Limitations:

• Restricted to born-digital PDFs only; scanned/image-based documents are excluded
• Feature extraction depends on internal PDF structure, which varies by editing software
• Single human supervisor validated all documents (no inter-annotator agreement metrics)
• The training set (422K pages) uses ML-generated labels that were only spot-checked (“visual inspection of an arbitrary selection of documents”), not systematically validated
• No cross-source generalization experiments; each source gets its own classifier, so it is unknown whether the dataset supports training a single unified model
• Only Random Forest classifiers evaluated; no comparison with deep learning approaches
• The dataset is not freely available (requires signed license agreement)

Reproducibility

Models

• Random Forest classifiers with max depth 1000, one per source
• No deep learning models trained
• No pretrained weights released
• No source code released; the GitHub repository contains only a README and license agreement

Algorithms

• Feature extraction via PyMuPDF (text blocks, images, links, fonts) and Camelot (tables)
• 18 features per text block (Table 2 in the paper): page number ($f_1$), bounding box ($f_{2\text{-}5}$: $x_0, y_0, x_1, y_1$), center coordinates ($f_{6\text{-}7}$: $x_c, y_c$), distances to page edges ($f_{8\text{-}11}$), raw text/data ($f_{12}$), bold proportion ($f_{13}$), italic proportion ($f_{14}$), average font size ($f_{15}$), font types tuple ($f_{16}$), capital letter proportion ($f_{17}$), word count ($f_{18}$)
• For classification, 17 of 18 features used ($f_{12}$ excluded). Positional features ($f_2$-$f_{11}$) normalized by page dimensions.
• Font type ($f_{16}$) encoded via dictionary mapping (details not provided)
• Full heuristic rule set for Step 0 annotation not disclosed; the paper provides only one example rule (bold proportion $> 0.5$ for titles)
• 10-fold cross-validation with 80/20 document-level splits

Data

• Sources: 24 Spanish administration websites (official gazettes)
• Scale: 37,910 documents, 441,300 pages, 138.1M tokens, 8M+ labels
• Split: Train: 36,466 docs (422K pages); Validation: 1,444 docs (19,276 pages)
• Languages: Spanish (primary), Basque, Catalan/Valencian, Galician
• Availability: Gated. Requires downloading and signing a license agreement from the BiDAlab GitHub repo, then emailing credentials to atvs@uam.es
• Annotation: Semi-automatic (heuristic + ML-assisted) with single-annotator human validation

Evaluation

• Metric: Accuracy (4-class text block classification)
• Protocol: 10-fold cross-validation, document-level splits
• No cross-dataset evaluation against established DLA benchmarks (PubLayNet, DocLayNet)
• Standard deviations reported across 10-fold CV, but no formal significance tests (e.g., paired t-tests between sources or against baselines)
• No comparison with deep learning baselines (Faster R-CNN, YOLO, LayoutLM, etc.)

Hardware

• Not reported. Random Forest classifiers are lightweight and likely trained on CPU.
• No training time, memory, or compute cost information provided.

BibTeX

@inproceedings{pena2023document,
  title={Document Layout Annotation: Database and Benchmark in the Domain of Public Affairs},
  author={Pe{\~n}a, Alejandro and Morales, Aythami and Fierrez, Julian and Ortega-Garcia, Javier and Grande, Marcos and Puente, {\'I}{\~n}igo and C{\'o}rdova, Jorge and C{\'o}rdova, Gonzalo},
  booktitle={Document Analysis and Recognition -- ICDAR 2023 Workshops},
  series={Lecture Notes in Computer Science},
  volume={14194},
  pages={126--141},
  year={2023},
  publisher={Springer},
  doi={10.1007/978-3-031-41501-2_9}
}

TL;DR

TexBiG (Text-Bild-Gefuge, “Text-Image-Structure”) is a document layout analysis dataset for historical documents from the late 19th and early 20th centuries. It provides over 52,000 expert polygon annotations across 19 classes, with inter-annotator agreement measured using Krippendorff’s Alpha. Each document image was annotated by at least two independent annotators. Baselines are provided using Mask R-CNN and VSR.

What kind of paper is this?

Dominant: $\Psi_{\text{Resource}}$

The headline contribution is the dataset itself: its annotation guidelines, 19-class taxonomy, polygon annotations, and inter-annotator agreement evaluation. The paper is organized around data curation and quality assessment.

Secondary: $\Psi_{\text{Evaluation}}$

The paper benchmarks Mask R-CNN and VSR on the dataset and uses Krippendorff’s Alpha to formally quantify annotation reliability, which is unusual for layout datasets.

What is the motivation?

Digital humanities researchers work with historical documents that differ substantially from the born-digital, English-language STEM papers that dominate existing DLA datasets. Late 19th and early 20th century publications exhibit complex text-image arrangements, varied typography, decorative elements, and advertisements interleaved with body content. Existing datasets either lack the class granularity needed for these documents or do not provide polygon-level annotations that can capture the irregular region shapes common in historical layouts.

The authors also note that most DLA datasets lack formal inter-annotator agreement measurements, making it difficult to assess annotation reliability. TexBiG addresses both gaps: fine-grained polygon annotations for historical documents with formal agreement metrics.

What is the novelty?

1. 19-class polygon annotations. Instance segmentation annotations (bounding boxes + polygons/masks) in COCO format across 19 classes covering both textual and non-textual document elements.

2. Expert annotation with formal agreement. Each document image was annotated by at least two independent expert annotators. Inter-annotator agreement is measured using Krippendorff’s Alpha ($\alpha$), providing a principled estimate of annotation reliability. Krippendorff’s Alpha is defined as:

$$\alpha = 1 - \frac{D_o}{D_e}$$

where $D_o$ is the observed disagreement between annotators and $D_e$ is the disagreement expected by chance. Values of $\alpha \geq 0.8$ are generally considered reliable, while $0.667 \leq \alpha < 0.8$ allows tentative conclusions.

3. Historical document focus. The dataset draws from 7 publicly available German-language publications from the late 19th and early 20th centuries, covering a range of layout complexity from simple book pages to magazine-style layouts with embedded images, advertisements, and decorative elements.

4. OCR data included. Tesseract-generated OCR output is provided alongside the annotations to support multimodal models like VSR, though the authors note this OCR is imperfect.

What experiments were performed?

Two baseline models were evaluated:

• Mask R-CNN (via Detectron2): standard instance segmentation baseline
• VSR (Vision-Semantics-Relations): multimodal model using both visual features and OCR text, pre-trained on PubLayNet

Results are reported as mAP on the test split. The paper DOI (10.1007/978-3-031-16788-1_22) provides full benchmark numbers (the paper itself is not open access).

What are the outcomes/conclusions?

TexBiG fills a gap for historical document layout analysis with a well-annotated, multi-class dataset. The formal inter-annotator agreement evaluation via Krippendorff’s Alpha sets it apart from most DLA datasets, which either report no agreement metrics or use ad-hoc measures. The dataset has been adopted by subsequent work (e.g., AnnoPage re-annotates TexBiG pages under its own schema).

Limitations

• The paper is behind a paywall (Springer DAGM GCPR), limiting accessibility.
• The GitHub repository for the VSR benchmark code has no license file.
• The dataset is relatively small by deep learning standards (7 books), though the annotation density (52K+ instances) is high.
• OCR data is Tesseract-generated and imperfect.

Mapping to Matter vs. Meaning Framework

How does TexBiG’s 19-class taxonomy map to our Matter vs. Meaning framework?

Coverage strengths: TexBiG provides a well-balanced taxonomy for historical documents. It distinguishes Logo from generic Image, separates Advertisement from editorial content, and includes decoration, frame, and noise classes that are important for historical material but absent from most modern-document datasets. The editorial note class is unique among DLA datasets.

Coverage gaps: No explicit classes for ListItem, BibEntry, Abstract, Blockquote, Code, OpticalCode, or form primitives. No relation annotations (reading order, hierarchy).

Reproducibility

Models

• Mask R-CNN: Standard Detectron2 implementation. Pretrained weights released on Zenodo.
• VSR: Multimodal model from Hikvision’s DAVAR-Lab-OCR, pre-trained on PubLayNet. Code adapted for TexBiG and released on GitHub.

Algorithms

• Framework: Detectron2 for Mask R-CNN; mmdetection 2.11.0 for VSR.
• Pre-training: VSR initialized from PubLayNet-pretrained weights.
• Training details: Optimizer, learning rate, batch size, epochs, and augmentation details are not available outside the paywalled paper.

Data

• Source: 7 publicly available German-language historical publications (late 19th to early 20th century).
• Total annotations: 52,000+ instances across 19 classes.
• Format: COCO JSON with polygon masks and bounding boxes. Train/val/test splits provided.
• Annotation process: Manual expert annotation using a detailed guideline (provided as PDF on Zenodo). At least 2 annotators per document. Agreement measured with Krippendorff’s Alpha.
• OCR: Tesseract-generated text provided in DAVAR format for VSR compatibility.
• License: CC-BY-4.0 for both annotations and images. Attribution required.
• Access: Zenodo (doi:10.5281/zenodo.6885144).

Evaluation

• Metric: mAP (COCO-style) for detection and segmentation.
• Agreement: Krippendorff’s Alpha for inter-annotator reliability.
• Baselines: Mask R-CNN and VSR.
• Full benchmark results in the published paper (Springer, paywalled).

Hardware

• NVIDIA Tesla V100-PCIE (16 GB), Ubuntu 18.04.

BibTeX

@inproceedings{tschirschwitz2022texbig,
  title={A Dataset for Analysing Complex Document Layouts in the Digital Humanities and its Evaluation with Krippendorff's Alpha},
  author={Tschirschwitz, David and Klemstein, Franziska and Stein, Benno and Rodehorst, Volker},
  booktitle={DAGM German Conference on Pattern Recognition},
  pages={354--374},
  year={2022},
  publisher={Springer},
  doi={10.1007/978-3-031-16788-1_22}
}

TL;DR

UDOP (Universal Document Processing) is a generative foundation model for document AI that unifies vision, text, and layout in a single encoder-decoder architecture. It converts all document tasks (understanding, classification, QA, layout analysis) into a sequence-to-sequence generation format, including generating document images from text and layout. UDOP ranked first on the Document Understanding Benchmark (DUE) at the time of publication and reported leading results on 8 tasks across diverse document domains.

What kind of paper is this?

Dominant: $\Psi_{\text{Method}}$. The headline contribution is the Vision-Text-Layout (VTL) Transformer architecture and unified generative framework that handles diverse document tasks without task-specific heads.

Secondary: $\Psi_{\text{Evaluation}}$. Comprehensive evaluation on 8+ benchmarks (DUE-Benchmark, FUNSD, CORD, RVL-CDIP, DocVQA) with ablation studies.

What is the motivation?

Prior document AI models (LayoutLMv2, LayoutLMv3) treat layout as shallow positional embeddings added to text tokens, underutilizing the strong spatial correlation between text content and its visual appearance. They also require task-specific heads for each downstream task (classification head for RVL-CDIP, sequence labeling head for FUNSD, etc.), limiting flexibility.

UDOP addresses both issues: (1) a layout-induced representation that directly fuses text tokens with the image patches where they appear, and (2) a unified seq2seq generation framework where all tasks use the same decoder.

What is the novelty?

Layout-Induced Vision-Text Representation

Instead of encoding text and image separately, UDOP adds each text token’s embedding to the visual features of the image patch where that token is located. This creates a unified representation where each token carries both its textual semantics and its visual appearance.

Vision-Text-Layout Transformer

A T5-based encoder-decoder with:

• Modality-agnostic encoder: Processes the fused vision-text tokens.
• Text-Layout decoder: Generates text and bounding box tokens autoregressively.
• Vision decoder: Reconstructs masked image patches (MAE-style).

Bounding boxes are discretized into tokens and added to the text vocabulary, allowing the model to generate layout as part of its output sequence.

Unified Pre-training

Four self-supervised objectives:

1. Joint Text-Layout Reconstruction: Reconstruct masked text tokens and their bounding boxes.
2. Visual Text Recognition: Predict text from image patches (OCR-like objective).
3. Layout Modeling: Predict bounding boxes from text.
4. Masked Autoencoding: Reconstruct masked image patches.

Plus 11 supervised datasets (1.8M examples) incorporated into pre-training via task prompts, including FUNSD, CORD, DocVQA, and others.

What experiments were performed?

DUE-Benchmark (7 tasks)

UDOP achieves first place overall with an average score of 64.8, compared to 62.9 for LayoutLMv3 LARGE. Strong results on DocVQA (84.7), InfoVQA (47.4), KLC (82.8), and DeepForm (85.5).

Additional Benchmarks

Document Generation

UDOP can generate document images from text and layout, enabling document editing and content customization. The authors describe this as the first document AI model with this capability, though the generated images are at MAE resolution (not publication quality).

What are the outcomes/conclusions?

The authors argue that UDOP demonstrates a unified generative framework can match or exceed task-specific models across diverse document AI tasks. The layout-induced representation is effective at exploiting the spatial correlation unique to documents. Incorporating supervised datasets into pre-training alongside self-supervised objectives provides additional gains.

Limitations

• 794M parameters (LARGE size only); no BASE variant released.
• Requires OCR as preprocessing.
• Document generation quality is limited to MAE reconstruction resolution.
• Pre-trained on 11M IIT-CDIP pages + 1.8M supervised examples; substantial compute requirements.

Reproducibility

Models

• UDOP: ~794M parameters. Based on T5 LARGE architecture with added vision encoder (ViT) and vision decoder.
• Checkpoint released on Hugging Face under MIT license.
• Vision decoder weights are withheld. The authors chose not to release the vision decoder due to ethical concerns around potential misuse for document forgery. The released checkpoint includes the encoder and text-layout decoder only.

Algorithms

• Pre-training: 11M IIT-CDIP pages (self-supervised) + 11 supervised datasets (1.8M examples).
• Optimizer: Adam with $\beta_1 = 0.9$, $\beta_2 = 0.98$, weight decay $1 \times 10^{-2}$.
• Learning rate: $5 \times 10^{-5}$ with 1000 warmup steps.
• Batch size: 512.
• Curriculum: 3-stage resolution scaling ($224 \rightarrow 512 \rightarrow 1024$), 1 epoch per stage.
• Self-supervised tasks: Joint text-layout reconstruction, visual text recognition, layout modeling, masked autoencoding.

Data

• Pre-training (unlabeled): IIT-CDIP Test Collection (11M documents).
• Pre-training (labeled): FUNSD, CORD, RVL-CDIP, DocVQA, InfoVQA, KLC, PWC, DeepForm, WTQ, TabFact, SQA (11 datasets, 1.8M examples total).

Evaluation

• Metrics: F1 for entity extraction tasks (FUNSD, CORD), accuracy for document classification (RVL-CDIP), ANLS for visual question answering (DocVQA, InfoVQA).
• Benchmarks: DUE-Benchmark (7 tasks), plus standalone evaluation on FUNSD, CORD, RVL-CDIP, and DocVQA.
• Baselines: LayoutLMv3 LARGE, LiLT, BROS, StructuralLM, and other document AI models.
• Statistical reporting: The authors report results over 5 runs with different random seeds and include standard deviations (Tables 9-10 in the paper), which is notable for this domain.

Hardware

• Not specified in the main paper. A model of this scale (794M params with vision encoder) typically requires multi-GPU training on A100-class hardware.

BibTeX

@inproceedings{tang2023udop,
  title={Unifying Vision, Text, and Layout for Universal Document Processing},
  author={Tang, Zineng and Yang, Ziyi and Wang, Guoxin and Fang, Yuwei and Liu, Yang and Zhu, Chenguang and Zeng, Michael and Zhang, Cha and Bansal, Mohit},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={15254--15264},
  year={2023},
  doi={10.1109/CVPR52729.2023.01462}
}

TL;DR

This paper formalizes cross-domain document object detection (DOD) as a task, introduces a benchmark suite of three PDF document datasets (PubMed, Chn, Legal), and proposes an FPN-based method with three adversarial alignment modules (Feature Pyramid Alignment, Region Alignment, and Rendering Layer Alignment) that leverage PDF rendering layers as domain-consistent supervision signals.

What kind of paper is this?

Dominant: $\Psi_{\text{Method}}$ : The headline novelty is the three-module alignment framework for cross-domain DOD. The paper dedicates the most space to describing the FPA, RA, and RLA modules and includes ablation studies isolating each module’s contribution. SOTA-style comparisons across six cross-domain transfer directions demonstrate the method’s performance across transfer directions.

Secondary: $\Psi_{\text{Resource}}$ : The paper establishes a benchmark suite of three document datasets (PubMed, Chn, Legal) with page images, bounding box annotations, raw PDF files, and PDF rendering layers. Two of three datasets (PubMed and Chn) are publicly released.

What is the motivation?

Document object detection requires large labeled training sets, but documents vary widely in layout, language, and genre. Constructing a comprehensive labeled dataset for every document domain is infeasible. The authors identify two key gaps:

1. No cross-domain DOD benchmark exists. Existing DOD datasets annotate only certain object types (tables, formulas) and do not provide auxiliary data like PDF rendering layers that could help bridge domain gaps.
2. Cross-domain object detection methods from natural images are suboptimal for documents. Document objects have extreme variance in aspect ratio and scale (a page number vs. a full-page table), and documents contain modular elements where context provides limited inpainting cues, unlike natural scenes.

The paper argues that adapting cross-domain techniques from natural image detection requires both document-specific data assets and document-specific alignment strategies.

What is the novelty?

Benchmark Suite

Three datasets sharing the same 5-class label space (text, list, heading, table, figure):

Each dataset includes page images, bounding box annotations, raw PDF files, and PDF rendering layers (text, vector, raster). The authors re-processed PubLayNet’s “list” annotations, splitting whole-list bounding boxes into individual items.

Cross-Domain DOD Method

Built on FPN with ResNet-101 backbone, the method introduces three adversarial alignment modules:

Feature Pyramid Alignment (FPA): Four binary domain classifiers $\{D_1, D_2, D_3, D_4\}$ operating on feature pyramid layers $\{P_1, P_2, P_3, P_4\}$. Each classifier predicts whether a pixel belongs to the source or target domain, trained adversarially via gradient reversal:

$$L_p = -\frac{1}{4W^sH^s}\sum_{i=1}^{4}\sum_{w=1}^{W^s}\sum_{h=1}^{H^s}\log(D_i(P_{i,w,h}^s)) - \frac{1}{4W^tH^t}\sum_{i=1}^{4}\sum_{w=1}^{W^t}\sum_{h=1}^{H^t}\log(1 - D_i(P_{i,w,h}^t))$$

The authors note that since each pyramid layer mixes high- and low-level features, aligning pyramids jointly addresses both semantic levels simultaneously, unlike prior methods that align them separately.

Region Alignment (RA): A focal-loss-weighted binary domain classifier on region proposals. Unlike prior work that treats all proposals equally, the focal loss focuses on hard-to-align proposals:

$$L_r = -\frac{1}{R}\sum_{i=1}^{R}(1 - D_r(r_i^s))^\gamma \log(D_r(r_i^s)) - \frac{1}{R}\sum_{i=1}^{R}(D_r(r_i^t))^\gamma \log(1 - D_r(r_i^t))$$

with $\gamma = 5.0$.

Rendering Layer Alignment (RLA): A DeepLab-V2-style segmentation network that predicts pixel-level rendering layer masks (background, text, raster; $C = 3$) from the FPN feature map $C_4$. Since rendering layers are available for both source and target PDFs, this auxiliary segmentation task provides domain-consistent supervision. The vector layer is merged into background because vector drawings are too thin to carry reliable semantics.

The total training loss is:

$$\mathcal{L} = \mathcal{L}_{det}^s + \lambda_1 \mathcal{L}_p + \lambda_2 \mathcal{L}_r + \lambda_3 \mathcal{L}_s$$

At inference, all alignment modules are removed and only the standard FPN detector remains.

What experiments were performed?

Cross-Domain Transfer Directions

Six transfer experiments across three datasets (each used as both source and target):

• Legal $\leftrightarrow$ Chn (cross-lingual)
• Chn $\leftrightarrow$ PubMed (cross-lingual)
• Legal $\leftrightarrow$ PubMed (cross-category, both English)

Baselines

• FRCNN (source-only): Faster R-CNN trained only on source labels, no adaptation.
• FPN (source-only): Feature Pyramid Network trained only on source labels.
• SWDA: Strong-Weak Distribution Alignment (Saito et al., CVPR 2019), a cross-domain detector for natural images.
• Oracle: FPN trained with labeled target data (upper bound).

Ablation Study (Legal $\rightarrow$ PubMed)

RLA was also tested as a plug-in on SWDA, boosting its mAP from 65.3 to 68.7 (+3.4).

Natural Image Transfer

Without RLA (which requires PDF data), FPA + RA were tested on Kitti $\leftrightarrow$ Cityscape car detection, outperforming SWDA (73.3 vs. 70.6 on Cityscape $\rightarrow$ Kitti).

Metric

Mean Average Precision (mAP) at IoU threshold 0.5.

What are the outcomes/conclusions?

Key results

• The full method (FPA + RA + RLA) consistently outperforms all baselines across all six transfer directions.
• Best improvements appear on Legal $\rightarrow$ PubMed (+5.8 mAP over FPN baseline) and PubMed $\rightarrow$ Chn (+20.5 mAP over FPN baseline).
• RLA is a modular component that improves any detector when PDF rendering layers are available.
• PubMed-as-source transfers show the largest domain adaptation gains, likely because PubMed’s homogeneous formatting (scientific journals with similar templates) limits its diversity, making the source-only model especially brittle on out-of-domain targets.

Limitations

• Large oracle gap persists. The best adapted results (e.g., 70.7 on Legal $\rightarrow$ PubMed) remain well below oracle performance (86.1), indicating domain shift is far from fully addressed.
• RLA requires PDF access. The rendering layer module only works when raw PDFs are available for both domains, which excludes scanned or camera-captured documents. This limits the method’s applicability to born-digital PDFs.
• Legal dataset not released. Only PubMed and Chn are publicly available; the Legal dataset (the largest at 19K pages) is withheld due to company policy.
• Limited class diversity. All datasets share the same 5 coarse classes. The benchmark does not test fine-grained or overlapping label spaces.
• No comparison with self-training methods. The related work discusses self-training as an alternative paradigm for cross-domain detection, but no self-training baselines are included in experiments.
• Single evaluation metric. Only mAP@0.5 is reported; no mAP@[0.5:0.95] or per-class analysis of failure modes.

Reproducibility

Models

• Backbone: ResNet-101 (pre-trained on ImageNet), fine-tuned end-to-end. Total parameter count not reported.
• Detection framework: FPN with RPN (standard Faster R-CNN two-stage pipeline).
• FPA: Four binary domain classifiers, each with three conv layers (kernel size 1, padding 0). ReLU after first two, Sigmoid after last. Classifiers share structure but not weights.
• RA: Three FC layers with ReLU + Dropout after the first two.
• RLA: DeepLab-V2 architecture predicting rendering layer segmentation from $C_4$ features.
• Initialization strategy for alignment modules not reported.
• No pretrained weights released.

Algorithms

• Optimizer: SGD, initial learning rate 0.001, divided by 10 every 8 epochs, total 12 epochs. Momentum and weight decay not reported.
• Batch size not reported.
• Loss weights: $\lambda_1 = \lambda_2 = 0.1$, $\lambda_3 = 0.01$.
• Focal loss parameter: $\gamma = 5.0$.
• Image shorter side resized to 600 pixels.
• No mention of warmup, gradient clipping, mixed precision, or other training tricks.
• At inference, FPA/RA/RLA modules are removed; only the standard FPN remains.

Data

• PubMed: 12,871 pages (9,653 train / 3,218 test) from PubLayNet. “List” annotations re-processed to split whole-list boxes into individual items.
• Chn: 8,005 pages (5,000 train / 3,005 test) synthesized from Chinese Wikipedia via layout/style randomization.
• Legal: 19,355 pages (9,684 train / 9,671 test) of human-annotated legal reports. Not publicly released due to company policy. Inter-annotator agreement not reported.
• PubMed and Chn available via Google Drive (Pascal VOC format) from the GitHub repo. Legal is withheld. The repo is MIT-licensed, but no explicit dataset license is stated.
• All datasets provide page images, bounding box annotations, raw PDFs, and rendering layers.

Evaluation

• mAP at IoU 0.5, per-class AP for 5 classes.
• No error bars, significance tests, or multi-run statistics.
• Oracle (supervised target training with FPN) provides an upper bound for each transfer direction.
• SWDA baseline was re-implemented by the authors. Compute budget parity across baselines is not discussed.

Hardware

• No training hardware, GPU hours, or memory requirements reported.
• No inference latency or throughput measurements.
• Implementation in PyTorch. The ResNet-101 + FPN backbone is a standard architecture that can run on a single consumer GPU at inference.

BibTeX

@inproceedings{li2020crossdomain,
  title={Cross-Domain Document Object Detection: Benchmark Suite and Method},
  author={Li, Kai and Wigington, Curtis and Tensmeyer, Chris and Zhao, Handong and Barmpalios, Nikolaos and Morariu, Vlad I. and Manjunatha, Varun and Sun, Tong and Fu, Yun},
  booktitle={IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  pages={12912--12921},
  year={2020},
  doi={10.1109/CVPR42600.2020.01293}
}

TL;DR

DocSegTr is a hybrid CNN-transformer architecture for bottom-up instance-level document layout segmentation. It combines a ResNeXt-101-FPN backbone with a twin-attention transformer module and dynamic convolution for mask generation, avoiding dependence on bounding box detection. The authors report mAP scores of 89.4 (PubLayNet), 40.3 (PRImA), 83.4 (Historical Japanese), and 93.3 (TableBank), competitive with or slightly above top-down Mask R-CNN baselines on three of four benchmarks.

What kind of paper is this?

Dominant: $\Psi_{\text{Method}}$ : The headline contribution is a new transformer-based architecture for document instance segmentation. The paper introduces a twin-attention mechanism adapted from Guo et al. for document-specific use, combined with a bottom-up dynamic mask prediction strategy. Ablation studies and baseline comparisons dominate the experimental sections.

Secondary: None. While the paper benchmarks across four datasets with per-category AP breakdowns and ablation studies, this evaluation work is standard method-paper validation rather than a standalone evaluation contribution (no new metrics, protocols, or test suites are proposed).

What is the motivation?

Document layout analysis has traditionally relied on top-down approaches (Faster R-CNN, Mask R-CNN) that first detect bounding boxes, then predict segmentation masks within them. The authors identify three problems with this paradigm:

1. Overlapping objects: Bounding box detection struggles with overlapping layout elements (e.g., captions overlapping figures), which are common in complex document layouts.
2. Computational cost: The two-stage detect-then-segment pipeline is expensive at inference time.
3. Limited global reasoning: CNNs capture local features effectively but lack the ability to model long-range dependencies between distant layout elements on a page.

The authors propose a bottom-up approach that generates instance masks directly without bounding box detection, using transformers for global context aggregation.

What is the novelty?

Architecture Overview

DocSegTr has three components:

1. CNN backbone with FPN: ResNeXt-101 with Deformable Convolutional Networks (DCN) and Feature Pyramid Networks (FPN) for multi-scale local feature extraction (P2 through P6).
2. Twin-attention transformer: A sparse attention mechanism that decomposes the standard self-attention matrix into sequential row-wise and column-wise attention computations.
3. Layerwise Feature Aggregation Module (LFAM): Combines local FPN features with global transformer features to produce a unified mask feature map.

Twin Attention

The key efficiency contribution is the twin-attention mechanism adapted from Guo et al. Given feature maps $f_i \in \mathbb{R}^{h \times w \times c}$, the approach divides them into $n \times n$ patches, then computes attention along rows and columns separately before combining through a global attention module. This reduces complexity from $\Theta((h \times w)^2)$ to $\Theta(h \times w^2 + w \times h^2)$.

Dynamic Mask Prediction

Instead of predicting masks inside bounding boxes, DocSegTr uses dynamic convolution. The transformer outputs two heads:

• Category head: An MLP that predicts semantic class for each patch ($n \times n \times q_c$, where $q_c$ is the number of classes).
• Kernel head: A linear layer that generates convolution kernels ($n \times n \times b$, where $b = \theta^2 \times c$).

The final mask is produced by convolving the unified feature map with the generated kernels:

$$M_f^{h \times w \times n \times n} = f^{h \times w \times c} \ast k^{n \times n \times b}$$

Instance predictions are post-processed with Matrix NMS. Training uses focal loss for classification:

$$\text{FL}(p_t) = -\alpha_t (1 - p_t)^\gamma \log(p_t)$$

and Dice loss for mask regularization:

$$\mathcal{L}_{\text{Dice}} = 1 - \frac{2 |X \cap Y|}{|X| + |Y|}$$

What experiments were performed?

Datasets

Baselines

DocSegTr is compared against two baselines:

• LayoutParser (Detectron2-based Mask R-CNN)
• Biswas et al. (prior work from the same group, also Mask R-CNN)

Ablation Study (on PRImA)

The ablation on the PRImA dataset demonstrates component contributions:

Note: the DCN row is anomalous. Adding DCN improves AP (32.59 to 33.21) but decreases AP@0.5 (58.62 to 49.13) and AP@0.75 (29.73 to 27.39). The paper does not address this inconsistency. One possible explanation is that DCN shifts the AP distribution across IoU thresholds, improving at mid-range thresholds while degrading at the standard 0.5 and 0.75 cutoffs, but this is speculative.

Key Results (mAP)

What are the outcomes/conclusions?

DocSegTr achieves competitive or better mAP compared to Mask R-CNN baselines on three of four benchmarks. The results suggest that transformer-based global context aggregation is particularly beneficial for larger document objects (tables, figures, lists), where long-range spatial reasoning matters. The model underperforms on smaller regions (e.g., text blocks in PRImA), which the authors acknowledge as a limitation.

Notable observations:

• Larger objects benefit most: Tables, figures, and lists consistently outperform the baselines, while smaller elements like text and separators see less improvement or regressions.
• Efficiency (claimed): The twin-attention mechanism is designed to reduce computational cost relative to standard self-attention, though the paper provides no latency or throughput measurements to confirm this.
• Transfer robustness: Pre-trained PubLayNet weights transferred to PRImA without fine-tuning yield 15% AP, suggesting some cross-domain capability.
• Historical Japanese is close: DocSegTr slightly trails on HJ (83.4 vs. 84.0), likely because the dataset has many small, densely packed elements where CNNs retain an advantage.

Limitations

• Narrow baseline comparison: Only two baselines (both Mask R-CNN variants) are compared. No comparison with other bottom-up or non-Mask-RCNN approaches.
• Small object performance: The authors acknowledge that transformers do not improve segmentation of smaller layout regions.
• No DocLayNet evaluation: The paper predates DocLayNet (2022) and does not evaluate on it, limiting comparisons with later work.
• PRImA dataset is small: The ablation study is conducted on PRImA (~300 pages), which may not generalize well to conclusions about transformer behavior on larger datasets.
• No speed benchmarks: Despite claiming computational efficiency, the paper does not report inference latency or FPS comparisons with the baselines.

Reproducibility

Models

• Architecture: ResNeXt-101-FPN backbone with DCN, twin-attention transformer layers, LFAM, two functional heads (category + kernel), and dynamic convolution for mask prediction.
• Parameter count: Not reported.
• Pre-trained weights: Available on the GitHub repo for PRImA, Historical Japanese, and TableBank. PubLayNet weights are not listed. All three checkpoints are hosted on institutional SharePoint (CVC UAB), which raises long-term durability concerns compared to archival platforms like Zenodo or HuggingFace.
• Framework dependencies: Requires Detectron2 v0.2.1 and AdelaiDet built from source, adding friction to reproduction.

Algorithms

• Optimizer: SGD with learning rate 0.001, Nesterov momentum 0.9, weight decay $10^{-5}$
• Schedule: 300K iterations with warm-up (1,000 steps); LR drops by $10\times$ at 210K and 250K iterations
• Batch size: 8
• Loss: Focal loss (classification) + Dice loss (mask regularization)
• Post-processing: Matrix NMS for instance deduplication

Data

• PubLayNet, PRImA, Historical Japanese, and TableBank are all publicly available datasets.
• No custom data augmentation is described beyond standard practices.
• Train/eval splits follow the original dataset definitions.

Evaluation

• Metric: COCO-style mAP (IoU thresholds 0.5 to 0.95, step 0.05), plus per-category AP
• Baselines: LayoutParser and Biswas et al. (both Mask R-CNN). Comparisons appear fair (same datasets, same metrics).
• No error bars or multi-run statistics reported.

Hardware

• Training: $2\times$ NVIDIA A40 GPUs (48 GB each), training takes 4-5 days
• Framework: PyTorch + Detectron2 v0.2.1
• Inference hardware: Not specified. The paper claims computational efficiency from the twin-attention mechanism but reports no inference latency, FPS, or throughput benchmarks to substantiate this.

BibTeX

@article{biswas2022docsegtr,
  title={DocSegTr: An Instance-Level End-to-End Document Image Segmentation Transformer},
  author={Biswas, Sanket and Banerjee, Ayan and Llad{\'o}s, Josep and Pal, Umapada},
  journal={arXiv preprint arXiv:2201.11438},
  year={2022}
}

TL;DR

A comparative benchmarking study that evaluates LayoutLMv3 (transformer), Paragraph2Graph (GNN), and YOLOv5 (CNN) on document layout analysis across DocLayNet and GROTOAP2 datasets, framing the task as both image-centric (object detection) and text-centric (token classification). The study also tests whether machine translation can improve LiLT’s cross-lingual zero-shot transfer on FUNSD/XFUND forms; the answer is no.

What kind of paper is this?

Dominant: $\Psi_{\text{Evaluation}}$. The paper proposes no new model or method. Its contribution is a head-to-head comparison of three existing architectures (LayoutLMv3, Paragraph2Graph, YOLOv5) across two benchmarks using standard metrics (mAP, F1). The cross-lingual machine translation experiment is also evaluative in nature.

Secondary: $\Psi_{\text{Systematic}}$. The paper organizes the DLA landscape along two axes: architecture type (transformer vs. graph vs. CNN) and task framing (image-centric vs. text-centric), providing a structured overview of related work.

What is the motivation?

The authors identify three gaps in document layout analysis research:

1. No cross-architecture comparison. Prior work developed transformer-based, graph-based, and CNN-based models independently, but no study directly compared their effectiveness on the same benchmarks.
2. Limited dataset coverage. Existing comparisons focused on narrow datasets (often PubLayNet or DocBank). The authors target DocLayNet (diverse, multi-domain) and GROTOAP2 (scientific articles, 22 classes) for broader coverage.
3. Unexplored machine translation for cross-lingual DLA. Language-independent models like LiLT showed promise for zero-shot cross-lingual transfer, but no prior work tested whether translating documents to the training language (English) could improve performance.

What is the novelty?

The novelty is primarily in the experimental design rather than in any algorithmic contribution:

• First direct comparison of LayoutLMv3, Paragraph2Graph, and YOLOv5 on DocLayNet and GROTOAP2 under both image-centric and text-centric framings.
• First use of machine translation (M2M100) as a preprocessing step for cross-lingual document layout analysis with LiLT.

The evaluation metrics are standard:

$$ \text{mAP} = \frac{1}{N} \sum_{i=1}^{N} AP_i $$

for image-centric tasks (COCO-style, IoU thresholds $[0.5, 0.95]$ with step 0.05), and F1-score for text-centric token classification.

What experiments were performed?

Experiment 1: Image-Centric DLA (Object Detection)

GROTOAP2 (first pages only, 22 classes):

YOLOv5 outperformed LayoutLMv3 on 14 of 22 individual classes. LayoutLMv3 was stronger on glossary, tables, figures, unknown elements, and page numbers.

DocLayNet (11 classes): The authors compared LayoutLMv3 against previously reported numbers for YOLOv5 and Paragraph2Graph from prior work. Exact per-class numbers are reported in the paper. The discussion concludes YOLOv5 is most practical for industry use due to speed.

Experiment 2: Text-Centric DLA (Token Classification)

GROTOAP2 (first pages):

LayoutLMv3 outperformed Paragraph2Graph by a large margin, though at substantially higher computational cost.

Experiment 3: Cross-Lingual Transfer with Machine Translation

LiLT (with XLM-RoBERTa tokenizer) was fine-tuned on English FUNSD forms (4 classes), then evaluated zero-shot on XFUND forms in 7 languages. The M2M100 translation model was used to translate non-English forms to English before inference.

Machine translation consistently degraded performance across all languages. The authors attribute this to token-level translation errors disrupting layout alignment.

What are the outcomes/conclusions?

1. YOLOv5 is the most practical choice for image-centric DLA. While LayoutLMv3 achieves marginally higher mAP on GROTOAP2, YOLOv5 trains in hours (vs. days) on a single GPU and runs inference on 1,300 pages in under a minute (vs. ~7 minutes for LayoutLMv3).
2. LayoutLMv3 substantially outperforms Paragraph2Graph in text-centric DLA. With an F1 of 0.866 vs. 0.698 for Paragraph2Graph, the gap is substantial, though the authors note LayoutLMv3’s CC-BY-NC-SA license makes it unsuitable for commercial use. They suggest LiLT as a viable alternative.
3. Machine translation does not help cross-lingual DLA. Translating documents to English before applying LiLT performed worse than direct zero-shot transfer in every language tested. Token-level translation introduces alignment errors that outweigh any linguistic benefit.

Limitations the authors acknowledge:

• Paragraph2Graph was not evaluated on GROTOAP2 for image-centric tasks (left as future work).
• No hyperparameter tuning for YOLOv5; only YOLOv5s (smallest variant) was tested.
• The GROTOAP2 experiments used only first pages of documents, which biases toward front-matter elements.
• Machine translation was only tested at token level; sequence-level classification might yield different results.

Limitations not acknowledged:

• All comparisons use models “as-is” from prior work with minimal tuning, making it unclear whether performance gaps reflect fundamental architectural differences or just different optimization effort.
• The FUNSD/XFUND experiments report F1 scores below 0.50 even in English, suggesting the 4-class setup or fine-tuning protocol may be suboptimal, limiting the interpretability of the cross-lingual comparison.
• DocLayNet results partially rely on numbers from prior work rather than consistent re-evaluation.

Reproducibility

Models

• LayoutLMv3: Pre-trained on IIT-CDIP (11M images). Uses RoBERTa for text, DiT for vision. 133M params (base). CC-BY-NC-SA-4.0.
• Paragraph2Graph: Language-independent GNN. Exact parameter count not reported.
• YOLOv5s: Smallest YOLOv5 variant. Pre-trained on COCO. Open-source (GPL-3.0 via Ultralytics).
• LiLT: Language-independent layout transformer with XLM-RoBERTa tokenizer. MIT license.
• M2M100: 1.2B-parameter many-to-many translation model from Meta.

Algorithms

• Text-centric LayoutLMv3: LR 1e-5 (from original paper), trained until convergence (~6 epochs on GROTOAP2).
• Text-centric Paragraph2Graph: LR 0.001, weight decay 0.005 (from original paper), trained until convergence (~9 epochs).
• Image-centric LayoutLMv3: LR 2e-4, batch size 2, 100 epochs max with early stopping.
• Image-centric YOLOv5s: Default configuration, no hyperparameter tuning, trained until convergence (~75 epochs).
• Convergence determined by monitoring training/validation loss and mAP on validation data.
• The released code consists of Jupyter notebooks for individual experiments, not a unified training framework.

Data

• DocLayNet: 80,863 pages (69,103 train / 6,480 val / 4,994 test), 11 classes. Diverse domains (financial, tenders, laws, manuals, patents, scientific). 95% English.
• GROTOAP2: Originally 13,210 documents / 119,334 pages. Authors used only first pages: 10,492 train / 1,310 val / 1,317 test. 22 classes. Scientific articles only. Converted from XML to COCO format for image-centric tasks and preprocessed for token classification.
• FUNSD: 149 train + 50 test English forms, 4 classes.
• XFUND: Multilingual forms in 7 languages, 4 classes.

Evaluation

• Image-centric: mAP @ IoU [0.5:0.05:0.95] (COCO-style).
• Text-centric: F1-score.
• No error bars, confidence intervals, or multi-run statistics reported.
• GROTOAP2 results are reported on the validation set, not the held-out test set.
• DocLayNet image-centric results for YOLOv5 and Paragraph2Graph taken from prior publications, not re-run.

Hardware

• All experiments on a single GPU (type not specified).
• YOLOv5s: ~2.5 hours for 75 epochs on GROTOAP2.
• LayoutLMv3: ~3 days for 7.5 epochs (image-centric) or ~2 days for 6 epochs (text-centric) on GROTOAP2.
• YOLOv5 inference: <1 min for 1,300 images. LayoutLMv3 inference: ~7 min for 1,300 images.

BibTeX

@article{kastanas2023document,
  title={Document AI: A Comparative Study of Transformer-Based, Graph-Based Models, and Convolutional Neural Networks For Document Layout Analysis},
  author={Kastanas, Sotirios and Tan, Shaomu and He, Yi},
  journal={arXiv preprint arXiv:2308.15517},
  year={2023}
}

TL;DR

BinMakhashen and Mahmoud survey 79 document layout analysis studies, organizing the field around a general DLA framework consisting of preprocessing, analysis parameter estimation, layout analysis (bottom-up, top-down, hybrid), post-processing, and performance evaluation. Bottom-up methods dominate the literature (61% of reviewed papers) due to their flexibility with complex layouts, while top-down methods remain useful for structured documents. The survey predates the shift to deep learning in DLA (PubLayNet, LayoutLM, DETR-based detectors all appeared in 2019-2020), making it a useful historical reference for the classical foundations the field built upon.

What kind of paper is this?

Dominant: $\Psi_{\text{Systematic}}$

This is a survey paper. The headline contribution is a unifying DLA framework that organizes the field into preprocessing, analysis strategies (bottom-up, top-down, hybrid), post-processing, and evaluation. The paper reviews 79 studies and categorizes them by strategy, document type, language, and layout complexity. Most of the paper is devoted to organizing and explaining prior work rather than introducing new methods or datasets.

Secondary: $\Psi_{\text{Evaluation}}$

The survey dedicates a full section to performance evaluation, distinguishing three levels of metrics: pixel-level (PLEF), region-level (REF, IoU), and customizable (CEF, used in ICDAR competitions). It provides formal definitions for each and discusses their strengths and limitations.

What is the motivation?

The authors identify several gaps that motivate the survey:

1. No unified framework. DLA algorithms vary widely in their processing pipelines, making it difficult to compare methods or understand where a new contribution fits. Earlier surveys focused on narrow topics (texture-based methods, skew detection, printed-document DLA) rather than the full pipeline.
2. Historical documents neglected. Most prior surveys concentrated on contemporary printed documents, despite historical manuscripts presenting harder layout challenges (arbitrary layouts, degraded text, multi-script content).
3. Evaluation inconsistency. Many early DLA methods used subjective evaluation or incompatible metrics, hindering progress. The paper aims to consolidate the evaluation landscape.

What is the novelty?

The paper’s primary contribution is the general DLA framework (Figure 2 in the paper), which decomposes DLA into five phases:

1. Preprocessing: Skew detection/correction and binarization.
2. Analysis parameter estimation: Static (predefined thresholds for structured layouts) vs. dynamic (data-driven measurements for heterogeneous documents).
3. Layout analysis: The core segmentation step, divided into three strategies:
   • Bottom-up (61% of 79 papers): Starts from pixels or connected components, grows to regions. Includes connected component analysis (Docstrum), texture analysis (Gabor, autocorrelation), machine learning (both non-deep and deep), Voronoi diagrams, and Delaunay triangulation.
   • Top-down (29%): Starts from page-level and splits into regions. Includes texture-based, RLSA, projection profile (X-Y cut), and whitespace analysis.
   • Hybrid (6%): Combines both strategies.
4. Post-processing: Optional refinement (clustering, morphological cleaning, human interaction).
5. Performance evaluation: Three tiers of metrics.

The survey also adopts and extends Kise’s document layout taxonomy, distinguishing six layout types: regular, Manhattan, non-Manhattan, multi-column Manhattan, arbitrary/complex, and overlapping. The extension to historical manuscripts with arbitrary layouts is the authors’ contribution; the base categorization for printed documents originates from Kise [82].

Evaluation Framework Taxonomy

The paper organizes DLA evaluation into three levels:

Pixel-Level Evaluation (PLEF): Counts pixel matches between segmentation and ground truth. The match score between ground-truth region $G_j$ and segmented region $R_i$ over image foreground $I$ is:

$$ \text{MS}(i, j) = \alpha \frac{\text{T}(G_j \cap R_i \cap I)}{\text{T}((G_j \cup R_i) \cap I)}, \quad \alpha = \begin{cases} 1, & \text{if } g_j = r_i \\ 0, & \text{otherwise} \end{cases} $$

where $\text{T}(\cdot)$ counts foreground pixels and $g_j$, $r_i$ are the labels of the ground-truth and segmented regions, respectively. The match table is aggregated into detection rate (DR) and recognition rate (RR) via weighted one-to-one, one-to-many, and many-to-one correspondences, then combined as $F_{\text{measure}} = 2 \cdot \text{DR} \cdot \text{RR} / (\text{DR} + \text{RR})$.

Region-Level Evaluation (REF): Uses IoU borrowed from computer vision. For each class $i$:

$$ \text{IoU}_i = \frac{G_i \cap R_i}{t_i + (G_i \cup R_i) - (G_i \cap R_i)} $$

The mean IoU averages over $N$ classes ($\text{mIoU} = \frac{1}{N} \sum_i \text{IoU}_i$). A frequency-weighted variant scales each class by its pixel count $t_i$.

Customizable Evaluation Framework (CEF): Introduced by Antonacopoulos and Bridson (ICDAR 2007), this framework transforms regions into interval representations, establishes correspondence, then classifies errors as merge, split, miss, partial miss, or false detection. Each error type $i$ has a weighted error rate $ER_i$ (the affected area multiplied by an application-specific weight $\alpha_i$). The per-type weight is:

$$ w_i = \frac{(N - 1) \cdot ER_i + 1}{N} $$

where $N$ is the number of error types considered. The overall success rate is then:

$$ SR = \frac{\sum_{i=1}^{N} w_i}{\sum_{i=1}^{N} \frac{w_i}{1 - ER_i}} $$

This design allows practitioners to customize error severity by adjusting $\alpha_i$ for different analysis objectives (e.g., penalizing column merges more heavily for OCR applications).

What experiments were performed?

This is a survey; no new experiments are conducted. The paper collects and compares reported results from the literature across multiple axes:

• Strategy distribution: Bottom-up (61%), top-down (29%), hybrid (6%), other (4%).
• Document type: Printed (58%), handwritten (33%), other (9%).
• Language: English (41%), Arabic (16%), German (9%), Latin (9%), French (8%), and others.
• Layout complexity: Manhattan (37%), multi-column (35%), complex (28%).

Table 1 compares skew detection methods by angle range, error, document type, and language. Table 2 catalogues 79 DLA algorithms by strategy, method type, document properties, and output. Table 4 collects quantitative results organized by evaluation metric.

What are the outcomes/conclusions?

The survey reaches several conclusions:

1. Bottom-up dominates but is slow. Connected-component-based analysis is the most flexible approach for complex layouts, but it carries higher computational cost than top-down methods. A complex layout can be analyzed top-down in 0.7 seconds with 78.5% success rate; bottom-up achieves higher accuracy at the cost of longer processing time.
2. Deep learning is emerging. The authors report that deep learning methods (FCNNs, U-Net variants) produced the strongest results among the surveyed papers, but require post-processing (clustering, morphological operations) and large training data. Transfer learning from ImageNet helps with convergence.
3. Hybrid methods are underexplored. Only 6% of reviewed papers use hybrid strategies, despite the complementary strengths of bottom-up (precision on complex layouts) and top-down (speed on structured layouts).
4. Binarization is declining for modern methods but remains critical for historical documents. Deep learning methods use full pixel intensities rather than binarized inputs.
5. Evaluation needs standardization. Many early methods used subjective evaluation. The CEF framework from ICDAR competitions offers the most comprehensive approach, but adoption is inconsistent.

Limitations

• Temporal cutoff. The survey was submitted in July 2018 and published in October 2019, so it predates the rapid adoption of deep learning in DLA. PubLayNet (2019), LayoutLM (2019), Mask R-CNN on documents, and DETR-based detectors are all absent. The “deep learning” methods discussed are early FCNNs and U-Nets, not the object detection or multimodal pre-training approaches that now dominate.
• Narrow language scope. Despite noting the importance of language diversity, the survey focuses primarily on Latin and Arabic scripts. CJK, Indic, and other scripts receive minimal attention.
• No deep learning taxonomy. The bottom-up/top-down/hybrid taxonomy does not cleanly accommodate modern object detection pipelines (Faster R-CNN, YOLO, DETR), which operate at neither the pixel/CC level nor the page-splitting level.
• Dataset coverage. The dataset section (Table 3) lists only 12 datasets, all with fewer than 7K pages. The large-scale weakly-supervised datasets (PubLayNet at 360K, DocBank at 500K) that became widely adopted appeared shortly after this survey.

Reproducibility

Models

Not applicable (survey paper).

Algorithms

Not applicable. The paper reviews but does not implement algorithms.

Data

The survey catalogues 12 DLA datasets (Table 3), including:

• Historical: OHG (596 pages), DIVA-HisDB (150 pages, ICDAR 2017), Parzival (47 pages), George Washington GW20 (20 pages), Saint Gall (60 pages), IMPACT (7K pages)
• Contemporary: UW-3 (1,600 pages), PRImA (305 pages), BCE-Arabic-v1 (1,833 pages), CENIP-UCCP (400 Urdu pages), LAMP (203 mixed pages), MAURDOR (2.5K pages)

These are all small by modern standards. The survey does not discuss dataset licensing.

Evaluation

The paper reviews three evaluation frameworks (PLEF, REF, CEF) and collects quantitative results in Table 4, but does not propose new metrics or benchmarks.

Hardware

Not discussed. This is a survey, so no training or inference hardware is reported.

BibTeX

@article{binmakhashen2019document,
  title={Document Layout Analysis: A Comprehensive Survey},
  author={BinMakhashen, Galal M. and Mahmoud, Sabri A.},
  journal={ACM Computing Surveys},
  volume={52},
  number={6},
  pages={1--36},
  year={2019},
  publisher={ACM},
  doi={10.1145/3355610}
}

TL;DR

Ahuja et al. introduce ETD-OD, a 25K-page dataset of electronic theses and dissertations (ETDs) annotated with bounding boxes across 24 element categories, along with a complete PDF-to-XML parsing pipeline. YOLOv7 achieves 85.3 mAP@0.50, and pre-training on DocBank followed by fine-tuning on ETD-OD provides large gains for Faster R-CNN (+37.1 mAP@0.50).

What kind of paper is this?

Dominant: $\Psi_{\text{Resource}}$ . The headline contribution is the ETD-OD dataset: 25K page images from 200 ETDs with ~100K bounding-box annotations across 24 document element categories. The dataset targets a document type (theses/dissertations) not covered by existing layout datasets.

Secondary: $\Psi_{\text{Method}}$ . The paper also presents an end-to-end PDF-to-XML parsing pipeline (object detection, text extraction, caption-to-figure matching, XML structuring) and benchmarks four detector configurations.

What is the motivation?

Electronic theses and dissertations contain substantial scholarly knowledge but are locked in unstructured PDFs, limiting their use in search, retrieval, summarization, and question-answering systems. Existing document layout datasets and parsing tools focus on shorter research papers (PubLayNet, DocBank, GROBID), which differ from ETDs in several ways:

• ETDs are long-form documents (hundreds of pages) with distinct structural elements: committee pages, degree metadata, lists of contents, and back matter.
• ETDs lack universal formatting; different institutions and disciplines impose different requirements and include domain-specific elements (algorithms in CS, equations in mathematics).
• Layout differences (single vs. double column, wider margins, larger font sizes) mean models trained on research paper datasets do not transfer well.

No existing dataset addressed these gaps at the time of publication.

What is the novelty?

ETD-OD Dataset

The dataset contains 25,073 page images from 200 ETDs sampled across institutions, degree types, and academic domains. Pages are annotated with bounding boxes across 24 categories organized into five groups:

Total annotations: 99,859 bounding boxes. Paragraphs (30,359) and page numbers (24,543) dominate; algorithms (96) are the rarest class.

Annotation was performed by 6 undergraduate students using Roboflow, with each sample validated by 2 graduate students.

PDF-to-XML Pipeline

The parsing pipeline has three stages:

1. Preprocessing: Convert PDF pages to images using pdf2image.
2. Element extraction: Apply an object detection model (Faster R-CNN or YOLO) to produce bounding boxes with class labels. Post-processing rules correct common misclassifications (e.g., abstract heading vs. chapter heading via keyword matching and position constraints).
3. XML structuring: Image-based objects (figures, tables, algorithms, equations) are cropped and stored as files. Text-based objects are extracted via pymupdf (born-digital) or pytesseract (scanned). Captions are matched to figures/tables by Euclidean distance between bounding boxes. Output follows a hierarchical XML schema: front matter, body (chapters with sections), back matter (references).

What experiments were performed?

Four object detection configurations were trained and evaluated on an 80/20 train/validation split:

Faster R-CNN models were trained for 60K iterations using Detectron2 (inference threshold 0.7). YOLO models were trained for 150 epochs.

Per-Category Results (YOLOv7, AP@0.50)

Strong categories (AP > 95): Paragraph (97.4), LOC Text (99.3), Reference Text (99.3), Figure (98.4), Footnote (98.9).

Weak categories: Page Number (51.3), Equation Number (55.0), Algorithm (66.6), Date (68.3), Degree (68.3). The authors attribute low performance to limited training samples (degree, date, algorithm) and small object sizes (page numbers, equation numbers).

Cross-Dataset Transfer

A Faster R-CNN experiment on shared categories between DocBank and ETD-OD tested three training configurations:

• DocBank only: Poor transfer to ETDs due to layout differences.
• ETD-OD only: Reasonable but not optimal.
• DocBank + ETD-OD (pre-train + fine-tune): Best results across all shared categories.

This confirms that domain-specific fine-tuning on ETD data is necessary, and that scholarly document pre-training provides a useful initialization.

What are the outcomes/conclusions?

Key findings:

• YOLOv7 achieves the best overall performance (85.3 mAP@0.50) on ETD-OD, outperforming Faster R-CNN substantially.
• Pre-training on scholarly documents (DocBank) nearly doubles Faster R-CNN’s mAP (39.1 to 76.2), confirming that domain overlap matters for document layout detection.
• The proposed XML schema and parsing pipeline provide a complete end-to-end system for converting ETD PDFs to structured representations.
• ETDs present challenges distinct from shorter research papers: metadata diversity, class imbalance (algorithms appear on < 0.4% of pages), and layout variability across institutions and disciplines.

Limitations and open questions:

• Class imbalance is severe. Algorithm (96 instances) and several metadata classes have very few annotations. The authors acknowledge this but do not apply mitigation strategies (oversampling, class-weighted loss).
• No formal inter-annotator agreement. Although each sample was validated by a graduate student, no IAA metric (e.g., Krippendorff’s Alpha or IoU agreement) is reported.
• Generalization is untested. All data comes from US institutional repositories. Performance on non-English ETDs or ETDs from non-US institutions is unknown.
• The XML pipeline is not evaluated end-to-end. The paper benchmarks object detection separately but does not measure the quality of the final XML output (e.g., text extraction accuracy, caption-to-figure matching precision).
• Small object detection remains a challenge. Page numbers and equation numbers have AP below 56, and the paper does not explore specialized techniques for small objects (e.g., high-resolution inputs, feature pyramid tuning).
• Only standard COCO metrics are reported. No error bars, multiple runs, or significance tests are provided.

Reproducibility

Models

• Faster R-CNN: ResNeXt-101 backbone, implemented via Detectron2.
• Faster R-CNN (DocBank): Pre-trained on DocBank, fine-tuned on ETD-OD. Uses the original DocBank model zoo weights and configurations.
• YOLOv5: Open-source implementation; the paper cites Jocher et al. 2022 (ultralytics/yolov5 v6.2) but does not pin a specific commit or version in the repo.
• YOLOv7: Open-source implementation; specific version not stated.
• Weights availability: The paper claims pre-trained models are available at the GitHub repository. However, as of March 2026, the repository (https://github.com/Opening-ETDs/ETD-OD) is not publicly accessible. The paper uses the URL Opening-ETDS (capital S) while the note uses Opening-ETDs; neither variant resolves to a public repository. Model weights are effectively unavailable.

Algorithms

• Faster R-CNN: 60K iterations, inference threshold 0.7. No learning rate, optimizer, batch size, or augmentation details are reported.
• YOLO models: 150 epochs. No other training hyperparameters (learning rate, optimizer, augmentation, image resolution) are reported.
• Post-processing: Rule-based corrections for abstract headings (keyword matching + position constraint: chapter heading in first 10 pages matching “abstract” keyword) and element-to-caption matching (Euclidean distance between bounding boxes, with a y-coordinate constraint for equation numbers).

Data

• Source: 200 ETDs from publicly accessible institutional repositories. Sampling stratified by degree, domain, and institution, though specific repositories are not named.
• Annotation: Roboflow platform. 6 undergraduate annotators + 2 graduate reviewers. No inter-annotator agreement metrics reported.
• Split: 80/20 train/validation. No separate test set.
• Availability: The paper states that dataset, code, and models are available at a GitHub repository. As of March 2026, this repository is not publicly accessible (neither the Opening-ETDS nor Opening-ETDs URL variant resolves). No dataset files, annotation files, or code have been released publicly.

Evaluation

• COCO-style mAP@0.50 and mAP@0.50:0.95.
• Evaluation on validation split only; no held-out test set.
• No cross-dataset evaluation (e.g., testing on PubLayNet or DocLayNet).
• No statistical rigor: single runs, no error bars, no significance tests.
• Evaluation scripts are not available (no code in the repository).

Hardware

• Not reported. No mention of GPU type, training time, memory requirements, or inference throughput.

BibTeX

@inproceedings{ahuja-etal-2022-parsing,
  title     = {Parsing Electronic Theses and Dissertations Using Object Detection},
  author    = {Ahuja, Aman and Devera, Alan and Fox, Edward Alan},
  booktitle = {Proceedings of the First Workshop on Information Extraction from Scientific Publications},
  month     = nov,
  year      = {2022},
  address   = {Online},
  publisher = {Association for Computational Linguistics},
  doi       = {10.18653/v1/2022.wiesp-1.14},
  pages     = {121--130},
}

TL;DR

GLAM (Graph-based Layout Analysis Model) represents PDF pages as structured graphs built from parsed text-box metadata and frames document layout analysis as joint node classification and graph segmentation. At 4M parameters, GLAM is over 35$\times$ smaller than the SOTA vision models it competes with, outperforming YOLO v5x6 on 5 of 11 DocLayNet classes. A simple ensemble of GLAM and YOLO v5x6 improved the best reported DocLayNet mAP from 76.8 to 80.8.

What kind of paper is this?

Dominant: $\Psi_{\text{Method}}$ : The headline contribution is a new problem formulation (DLA as graph segmentation + node classification on parsed PDF metadata) and a corresponding GNN architecture. Ablations, SOTA comparisons, and an efficiency analysis dominate the experimental sections.

Secondary: $\Psi_{\text{Evaluation}}$ : The paper includes a detailed analysis of where graph-based methods succeed and fail relative to vision-based detectors, with per-class breakdowns and a thorough discussion of bounding-box misalignment artifacts in PubLayNet that systematically degrade parser-based approaches.

What is the motivation?

Most DLA models treat document pages as images and apply standard object detection (Faster R-CNN, YOLO, DETR). This discards the structured metadata available in born-digital PDFs: exact character positions, font information, bounding boxes, and rendering instructions. These metadata features are both precise and cheap to extract relative to image-based inference.

Prior graph-based work in document understanding focused narrowly on table extraction or used CV-derived primitives. The authors argue that a pure-graph formulation operating on PDF-parsed text boxes can be competitive with much larger vision models while being an order of magnitude more efficient.

What is the novelty?

Graph Formulation

Each PDF page is parsed into text boxes (via pdfminer.six), where each box becomes a graph node with a 79-dimensional feature vector (bounding box coordinates, text length, fraction of numerical characters, font type, font size, and other heuristics). Bidirectional edges connect each node to its nearest neighbor in four cardinal directions, plus additional edges approximating reading order (top-to-bottom, left-to-right).

Architecture

GLAM has two components:

1. Graph Network (~1M params): TAG convolutional layers with batch normalization, producing node embeddings. A separate edge embedding is formed by concatenating source/destination node embeddings with edge features. Two classification heads predict node class (segment type) and edge label (same-segment or not).

2. CV Feature Extractor (~3M params): A ResNet-18 renders the page as an image, extracts ROI-pooled visual features per text box, and fuses them with graph features via a three-layer attention encoder. This addresses the graph’s blindness to non-textual visual cues (lines, colors, backgrounds).

Training Objective

The model is trained with a weighted joint loss:

$$L = L_{node} + \alpha L_{edge}$$

where both terms are cross-entropy losses and $\alpha = 4$ upweights the edge classification to improve segmentation quality.

Inference

At inference, negative edges are removed from the graph. Connected components define segments. Each segment is classified by majority vote over its constituent nodes. The bounding box is the minimum spanning box of the segment’s nodes, with confidence equal to the mean class probability.

What experiments were performed?

Datasets

DocLayNet was the primary evaluation target due to its diverse layouts (financial reports, manuals, scientific articles, laws, patents, tenders). PubLayNet’s auto-generated bounding boxes are known to misalign with underlying text boxes, imposing a hard upper bound on parser-based methods.

Baselines

Mask R-CNN (ResNet-50, ResNet-101), Faster R-CNN (ResNet-101), YOLO v5x6, LayoutLMv3, MBC, and VSR.

Ablations

CV feature ablation (Tables 3-4): Removing the ResNet-18 visual branch drops the GNN-only variant (GLAM-CV, ~1M params) to 59.9 mAP on DocLayNet (vs. 68.6 for full GLAM), with the largest drops on visually complex classes like formula ($-22.8$) and list item ($-15.2$). On PubLayNet (visually simple layouts), the drop is minimal ($-1.2$).

Metric

COCO-style mAP at IoU thresholds [0.5:0.95].

What are the outcomes/conclusions?

DocLayNet Results

GLAM (4M params) achieves 68.6 overall mAP vs. 76.8 for YOLO v5x6 (140.7M params). However, GLAM outperforms all baselines on 5 of 11 classes: formula (66.6), page footer (86.2), page header (78.0), section header (79.8), and title (85.1). These are all text-dense, small-region classes where pixel-perfect bounding boxes from parsed text boxes give an advantage.

The ensemble of GLAM + YOLO v5x6 (taking the per-class best) achieves 80.8 mAP, a new SOTA at the time of publication. The complementarity is clear: GLAM excels on text-based classes while YOLO handles vision-rich classes (picture: 77.1 vs. 5.7, table: 86.3 vs. 56.3).

PubLayNet Results

GLAM achieves 72.2 overall mAP (vs. 95.1 for LayoutLMv3 SOTA), significantly handicapped by bounding-box misalignment. At IoU 0.5 only, GLAM exceeds 90 mAP on all text classes, demonstrating that the model correctly identifies segments but cannot match misaligned ground-truth boxes at stricter thresholds.

Efficiency

All benchmarked on NVIDIA Tesla T4 (AWS G4 instance). GLAM processes ~98 pages/sec on GPU and ~64 pages/sec on CPU, making on-device deployment feasible.

Limitations

• No scanned document support: GLAM requires born-digital PDFs with parseable text boxes. It cannot handle scanned documents or image-only PDFs.
• Vision-rich classes fail: Picture mAP is 5.7 on DocLayNet; table mAP is 56.3. Any element not well-represented by text boxes is poorly captured.
• Parser dependency: Performance is bounded by PDF parser quality. Different parsers produce different text boxes, and the graph construction is sensitive to this.
• No text embeddings: GLAM does not use any semantic text features; adding text embeddings is noted as future work.
• PubLayNet misalignment: The auto-generated PubLayNet annotations penalize parser-based methods systematically, making cross-method comparisons on that dataset unfair.

Reproducibility

Models

• GNN: TAG convolutional layers, 512-dim hidden (PubLayNet) or 1024-dim hidden (DocLayNet), scaling down by 2$\times$ per layer. Batch normalization on node features.
• CV branch: ResNet-18 backbone with ROI-average-pooling, followed by a 3-layer attention encoder.
• Total: ~4M parameters (full), ~1M (GNN only).
• No official code or weights released by the authors (Kensho Technologies).

Algorithms

• Joint cross-entropy loss on node and edge classification with edge weight $\alpha = 4$.
• Segment inference via connected components after negative-edge removal, with majority-vote classification.
• No details reported on: optimizer, learning rate, batch size, number of epochs, warmup, gradient clipping, or augmentation.

Data

• DocLayNet: publicly available, human-annotated, text-box-snapped ground truth. Standard train/dev/test splits (69,375 / 6,489 / 4,999).
• PubLayNet: publicly available, auto-generated annotations. Standard splits (~335K / 11K / 11K).
• Graph construction uses 79 node features extracted from parsed text boxes plus heuristic edges (nearest neighbor in four directions + reading order).

Evaluation

• COCO-style mAP at IoU [0.5:0.95], reported per-class and overall.
• No error bars, significance tests, or multi-run statistics.
• The ensemble result (80.8 mAP) is a simple per-class max of two independent models, not a trained fusion.
• The PubLayNet comparison is acknowledged as unfair to parser-based methods due to annotation misalignment.

Hardware

• All inference benchmarks on NVIDIA Tesla T4 (AWS G4 instance).
• Training hardware not reported. No GPU-hours, memory requirements, or training time estimates.

BibTeX

@inproceedings{wang2023glam,
  title={A Graphical Approach to Document Layout Analysis},
  author={Wang, Jilin and Krumdick, Michael and Tong, Baojia and Halim, Hamima and Sokolov, Maxim and Barda, Vadym and Vendryes, Delphine and Tanner, Chris},
  booktitle={International Conference on Document Analysis and Recognition (ICDAR)},
  pages={57--75},
  year={2023},
  publisher={Springer},
  doi={10.1007/978-3-031-41734-4_4}
}

TL;DR

GraphKD is a graph-based knowledge distillation framework for document object detection (DOD) that constructs structured instance graphs from RoI-pooled features and transfers them from a large teacher to a compact student network. By using cosine similarity for node-to-node distillation and Mahalanobis distance for edge-to-edge distillation, the framework enables heterogeneous distillation (e.g., ResNet to EfficientNet) and includes an adaptive text-node sampling strategy to reduce text-class bias.

What kind of paper is this?

Dominant: $\Psi_{\text{Method}}$ : The headline contribution is a graph-based knowledge distillation framework specifically designed for document object detection. The paper proposes a structured instance graph construction from RPN proposals, a new node sampling strategy for text bias reduction, and a combined cosine/Mahalanobis distillation loss. Ablations and SOTA comparisons dominate the experimental sections.

Secondary: $\Psi_{\text{Evaluation}}$ : The paper provides an extensive comparative evaluation across four benchmarks, testing both homogeneous and heterogeneous distillation configurations against three competing KD methods (ReviewKD, NKD, SimKD).

What is the motivation?

Document object detection models have grown increasingly complex (e.g., SwinDocSegmenter at 223M params, LayoutLMv3 at 368M params), making deployment on edge devices impractical. Knowledge distillation offers a path to compact models, but existing KD techniques face three problems in the DOD setting:

1. Feature imbalance: Documents are dominated by text regions, creating a class imbalance that biases standard distillation.
2. Missing instance relations: Logit-based KD loses fine-grained spatial information; feature-based KD suffers from alignment problems across heterogeneous architectures.
3. Homogeneity constraint: Conventional layer-to-layer distillation requires matching teacher/student architectures, preventing cross-architecture transfer (e.g., ViT to CNN).

The authors note this is the first application of knowledge distillation to the DOD task specifically.

What is the novelty?

Structured Instance Graph

Rather than distilling raw feature maps or logits, GraphKD constructs a graph $G = (V, \xi)$ where:

• Nodes ($V$): Vectorized RoI-pooled features from region proposals, categorized into “text” and “non-text” based on feature covariance.
• Edges ($\xi$): Cosine similarity between node pairs, forming a complete, symmetric, undirected graph:

$$e_{pq} = \frac{v_p \cdot v_q}{|v_p| \cdot |v_q|}$$

Teacher and student share the same RPN proposals, ensuring aligned anchor boxes so both backbones extract features from the same regions.

Adaptive Text Node Sampling

To handle the text-dominance problem, the framework applies a mining strategy (Algorithm 1): text nodes with high classification loss in the teacher (likely to be misclassified) are merged with non-text nodes for distillation, while confident text nodes are pruned. This reduces biased edges and improves false-negative regularization.

Graph Distillation Loss

The total loss combines node losses (text and non-text) with an edge loss, all computed via Mahalanobis distance:

$$\mathcal{L}_g = \frac{\lambda_1}{N_{nt}} \sum_{i=1}^{N_{nt}} \left|\frac{v_i^{t,nt} - v_i^{s,nt}}{\sigma_{nt}}\right| + \frac{\lambda_2}{N_{text}} \sum_{i=1}^{N_{text}} \left|\frac{v_i^{t,text} - v_i^{s,text}}{\sigma_{text}}\right| + \frac{\lambda_3}{N^2} \sum_{i=1}^{N} \sum_{j=1}^{N} \left|\frac{e_{ij}^t - e_{ij}^s}{\sigma_\xi}\right|$$

where $\lambda_1 = \lambda_3 = 0.3$ (Optuna search) and $\lambda_2 = \alpha \cdot \frac{N_{nt}}{N_{text}}$ adapts to the class imbalance. The key insight from ablations is that cosine similarity works best for node-to-node alignment (captures orthogonality) while Mahalanobis distance works best for edge-to-edge alignment (sensitive to outliers).

A final GCN + cross-entropy loss classifies the non-text nodes into specific object categories beyond the binary text/non-text split.

What experiments were performed?

Datasets

Four benchmarks covering diverse document types:

Distillation Configurations

• Homogeneous: ResNet152 $\rightarrow$ ResNet101, ResNet101 $\rightarrow$ ResNet50
• Heterogeneous: ResNet50 $\rightarrow$ ResNet18, ResNet101 $\rightarrow$ EfficientNet-B0, ResNet50 $\rightarrow$ MobileNetV2

Baselines

• KD methods: ReviewKD, NKD, SimKD (all adapted to the DOD setting)
• Supervised upper bounds: SwinDocSegmenter (223M), LayoutLMv3 (368M), DocSegTr (168M)

Ablations

1. Component ablation (Table 2): Tested edge-only, edge+non-text, edge+text, and full (edge+non-text+text) on DocLayNet with ResNet50 $\rightarrow$ ResNet18. Full model achieves 42.1 AP vs. 33.1 AP edge-only.
2. Distance function ablation (Table 3): Exhaustive 4$\times$4 grid of L1, L2, Cosine, and Mahalanobis for node and edge losses. Best: Cosine (nodes) + Mahalanobis (edges) at 42.1 AP.

Metric

COCO-style mAP at IoU thresholds 0.5 to 0.95 (step 0.05).

What are the outcomes/conclusions?

Key Results

GraphKD consistently outperforms competing KD methods (ReviewKD, NKD, SimKD) across all four benchmarks and all five teacher-student configurations. The best configuration (ResNet152 $\rightarrow$ ResNet101) achieves:

Noteworthy observations

• On PRImA (a small dataset), the distilled ResNet101 (44.5M params) outperforms LayoutLMv3 (368M) by ~1.6 AP, suggesting large transformers overfit on limited data.
• On DocLayNet, distilled models outperform DocSegTr and LayoutLMv3 on specific classes like “Caption,” “Page-footer,” and “Picture,” where text-bias reduction helps.
• Heterogeneous distillation (e.g., ResNet50 $\rightarrow$ MobileNetV2 at 3.4M params) is functional but incurs substantial performance drops due to double compression of RoI features.

Limitations

• The performance gap to supervised SOTA remains significant: ~7% on PubLayNet/DocLayNet and ~13% on PRImA vs. SwinDocSegmenter.
• Transformer backbones cannot serve as teacher or student due to incompatible data handling at the RPN/RoI level. The authors acknowledge this as a key limitation and future work direction.
• The framework is built entirely on Faster R-CNN with RPN; it does not generalize to anchor-free or DETR-style detectors.
• No latency or inference speed benchmarks are reported, despite efficiency being the stated motivation.
• The “first KD for DOD” claim is plausible but difficult to verify exhaustively.

Reproducibility

Models

• All teacher/student pairs use standard backbones: ResNet18/50/101/152, EfficientNet-B0, MobileNetV2.
• The detection framework is Faster R-CNN with a shared RPN between teacher and student.
• A GCN is appended for final node classification (text $\rightarrow$ specific object categories).
• Code is available on GitHub under MIT license, built on Detectron2.

Algorithms

• Loss penalty coefficients: $\lambda_1 = \lambda_3 = 0.3$ (Optuna), $\lambda_2$ adaptive.
• Node-to-node: cosine similarity loss. Edge-to-edge: Mahalanobis distance loss.
• Logit matching via KL divergence.
• Text node merging threshold $t$ is empirically determined (exact value not reported).
• No details on optimizer, learning rate schedule, batch size, or number of training epochs.

Data

• PubLayNet, PRImA, HJDataset, DocLayNet: all publicly available.
• Standard train/eval splits used (instance counts reported in Table 1).

Evaluation

• COCO-style mAP (IoU 0.5:0.05:0.95), plus AP@50, AP@75, AP_S, AP_M, AP_L.
• Comparisons against three KD baselines (ReviewKD, NKD, SimKD) and three supervised methods (DocSegTr, LayoutLMv3, SwinDocSegmenter).
• No error bars, significance tests, or multi-run statistics reported.
• The supervised baselines use different training setups (pre-training corpora, augmentations) so direct parameter-efficiency comparisons should be interpreted cautiously.

Hardware

• No training hardware, GPU hours, or memory requirements reported.
• No inference latency or throughput measurements, which is a notable gap for a paper motivated by edge deployment.

BibTeX

@inproceedings{banerjee2024graphkd,
  title={GraphKD: Exploring Knowledge Distillation Towards Document Object Detection with Structured Graph Creation},
  author={Banerjee, Ayan and Biswas, Sanket and Llad{\'o}s, Josep and Pal, Umapada},
  booktitle={International Conference on Document Analysis and Recognition (ICDAR)},
  pages={353--370},
  year={2024},
  publisher={Springer},
  doi={10.1007/978-3-031-70543-4_21}
}

TL;DR

This paper extends the DINO detection transformer for document layout analysis with two additions: (1) a query encoding mechanism that fuses RoI-aligned backbone features with decoder queries via cosine similarity, improving small-object detection, and (2) a hybrid matching strategy that trains with one-to-many matching in the first half of training and switches to one-to-one matching in the second half. The authors report 97.3% mAP on PubLayNet, 81.6% on DocLayNet, and 98.6% on PubTables.

What kind of paper is this?

Dominant: $\Psi_{\text{Method}}$. The headline contribution is an improved detection transformer architecture and training strategy. The bulk of the paper describes the query encoding mechanism, the hybrid matching scheme, and ablations isolating their effects.

Secondary: None. No new datasets, benchmarks, or artifacts are released.

What is the motivation?

Transformer-based detectors like DINO have shown strong performance on natural image detection, but the authors argue they underperform on document layout analysis tasks, particularly for small graphical objects (headers, footers, section titles). The core issue: decoder queries in DINO lack direct access to fine-grained backbone features, limiting their ability to detect small or visually subtle layout elements. Additionally, standard one-to-one Hungarian matching produces fewer positive training samples, slowing convergence.

What is the novelty?

Query Encoding

The method extracts features from the ResNet-50 backbone using RoI-Align on bounding box proposals and passes them through an MLP to produce high-level query features:

$$Q_h = \text{MLP}(\text{RoIAlign}(F_h, b_j))$$

These are compared to decoder queries via cosine similarity after self-attention:

$$Q_e = \text{similarity}(Q’_d, Q’_h)$$

The enhanced query features are concatenated with the original decoder queries:

$$Q_t = \text{Concat}(Q_d, Q_e)$$

This combined representation is fed into the decoder:

$$o = \text{Decoder}(Q_t, E \mid A)$$

where $E$ is the encoder output and $A$ is the denoising attention mask.

Hybrid Matching

The training loss combines one-to-many and one-to-one assignment strategies. The one-to-many classification loss is:

$$L_{cls}^{1\text{-}m} = \sum_{i=1}^{N_{obj}} |\hat{g}_i - p_i| \cdot \text{BCE}(p_i, \hat{g}_i) + \sum_{j=1}^{N_{no}} p_j \cdot \text{BCE}(p_j, 0)$$

with regression:

$$L_{reg}^{1\text{-}m} = \sum_{i=1}^{N_{obj}} \hat{g}_i \cdot L_{GIoU}(bx_i, \hat{bx}_i) + \sum_{i=1}^{N_{obj}} \hat{g}_i \cdot L_{L1}(bx_i, \hat{bx}_i)$$

During training, the first half uses one-to-many matching (more positive samples, faster convergence), and the second half switches to one-to-one matching (eliminates duplicate predictions, enabling NMS-free inference).

What experiments were performed?

The method is evaluated on three benchmarks:

• PubLayNet (360K pages, 5 classes): 97.3% mAP, improving over DINO baseline (95.5%) and the prior best (Zhong et al., 96.5%).
• DocLayNet (80K pages, 11 classes): 81.6% mAP, improving over DINO (74.3%) and Zhong et al. (81.0%). Particularly strong on Page-footer (+37.1 over DINO), Page-header (+14.1), and Title (+3.5).
• PubTables (table detection): 98.6% mAP (AP50: 99.8%, AP75: 99.1%).

Baselines include Faster R-CNN, Mask R-CNN, YOLOv5, DINO, DiT-L, LayoutLMv3, VSR, and others.

Ablations

1. Query encoding ($Q_d + Q_e$ vs. $Q_d$ alone): +1.78 mAP on PubLayNet validation.
2. Matching strategy: One-to-many alone reaches 98.4 mAP but generates duplicates. The hybrid scheme (one-to-many then one-to-one) achieves 97.3 mAP while remaining NMS-free.
3. Number of queries: 300 queries is optimal. Performance degrades at 400 due to overfitting.

What are the outcomes/conclusions?

The two proposed modifications yield consistent improvements over the DINO baseline across all three benchmarks. The query encoding mechanism is particularly effective for small layout elements. The hybrid matching strategy provides a practical trade-off between detection quality and duplicate elimination without requiring NMS at inference.

The gains over DINO are substantial on DocLayNet (+7.3 mAP) where documents are structurally diverse, but more modest on PubLayNet (+1.8 mAP) where layouts are relatively uniform. No code or model weights are released, limiting independent verification.

Reproducibility

Models

• Backbone: ResNet-50 pre-trained on ImageNet, with multi-scale feature maps (1/4, 1/8, 1/16, 1/32, 1/64), each projected to 256 channels via 1$\times$1 convolutions
• Architecture: DINO transformer with deformable attention, augmented with the proposed query encoding module
• Parameter count: Not reported
• Queries: 300 learnable queries (optimal per ablation)
• Initialization: DINO weight initialization source (pretrained or from scratch) not specified
• Weights: Not released

Algorithms

• Optimizer: AdamW
• Batch size: 16
• Epochs: 12 (PubLayNet, PubTables), 24 (DocLayNet)
• Base learning rate: Not reported
• Learning rate schedule: Reduced by factor of 10 at a later stage (exact milestone not specified)
• Training tricks: Warmup, gradient clipping, and mixed precision not mentioned
• Loss functions: BCE (classification) + GIoU + L1 (regression), combined in one-to-many and one-to-one branches. Loss weighting coefficients not reported.
• Multi-scale training: Images resized to various lengths with a max size limit (exact dimensions not specified)
• Other augmentation: Not described beyond multi-scale resizing
• Test-time: Shorter side resized to 640
• Matching switch: One-to-many for first half of training epochs, one-to-one for second half

Data

• PubLayNet: 360K pages, 5 classes (Text, Title, List, Table, Figure). Standard train/val/test splits. CDLA-Permissive-1.0 (annotations); underlying PDFs non-commercial only.
• DocLayNet: 80K pages, 11 classes. Standard splits. CDLA-Permissive-1.0 (annotations); underlying doc licenses vary by domain.
• PubTables: Table detection benchmark. Standard splits. CDLA-Permissive-1.0.
• All three are established benchmarks with published annotation protocols.
• No additional or proprietary training data used.

Evaluation

• Metric: COCO-style mAP at IoU thresholds 0.50 to 0.95 (step 0.05), plus AP@50 and AP@75
• Baselines: Comparison against CNN-based (Faster R-CNN, Mask R-CNN, YOLOv5) and transformer-based (DINO, DiT-L, LayoutLMv3) methods
• Baseline provenance: DocLayNet DINO numbers are sourced from Zhong et al. [62], not from the original DINO paper or the authors’ own runs. PubLayNet DINO numbers cite the original DINO paper [19]. Provenance of other baselines varies by table.
• Statistical rigor: No error bars, significance tests, or multi-run statistics reported. Number of runs and seed sensitivity not mentioned.
• Author-acknowledged limitations: The paper’s conclusion does not explicitly discuss limitations. It focuses on summarizing contributions without qualifying scope or failure cases.

Hardware

• GPU: NVIDIA “RTXA600” GPUs (plural; likely RTX A6000). Exact GPU count not specified.
• Training time: Not reported
• Inference speed: Not reported
• Memory requirements: Not reported
• Cost estimates: Not reported
• Deployment considerations: Not discussed. Given the DINO-based architecture with ResNet-50, CPU-only inference is likely impractical for real-time use.

BibTeX

@inproceedings{shehzadi2024hybrid,
  title={A Hybrid Approach for Document Layout Analysis in Document Images},
  author={Shehzadi, Tahira and Stricker, Didier and Afzal, Muhammad Zeshan},
  booktitle={International Conference on Document Analysis and Recognition (ICDAR)},
  pages={20--37},
  year={2024},
  organization={Springer}
}

TL;DR

M2Doc is a pluggable multi-modal fusion approach for document layout analysis that adds two lightweight modules to existing object detectors: an early-fusion module that gates pixel-level textual features into backbone outputs, and a late-fusion module that injects block-level BERT embeddings into candidate bounding boxes via IoU matching. Applied to DINO and Cascade Mask R-CNN, it achieves 89.0 mAP on DocLayNet (+11.3 over unimodal DINO) and 69.9 mAP on M6Doc (+1.9).

What kind of paper is this?

Dominant: $\Psi_{\text{Method}}$ : The core contribution is a multi-modal fusion architecture designed specifically for document layout analysis. The paper introduces two pluggable fusion modules (early and late), demonstrates them across multiple detector families, and validates with extensive ablations and SOTA comparisons. The majority of the paper describes architectural design choices and ablation experiments.

Secondary: None. The paper does not introduce new datasets, benchmarks, or evaluation protocols. Code is released, but as a companion to the method, not as an independent infrastructure contribution.

What is the motivation?

Document layout analysis has traditionally been treated as a vision-only object detection problem, even when using multimodal pre-trained models. The authors identify three specific gaps:

1. Unimodal pipelines from multimodal pre-training: Models like DiT and LayoutLM undergo multimodal pre-training but are reduced to unimodal (image-only) backbones when fine-tuned for layout detection. The textual modality learned during pre-training is discarded at inference time.
2. Weak multi-modal baselines: The primary multi-modal method (VSR) uses a complex two-backbone architecture with Chargrid/Wordgrid/Sentencegrid inputs and Transformer-based relation modeling, yet it performs worse than unimodal detectors on complex logical layout datasets like DocLayNet.
3. Detector-specific solutions: Existing DLA enhancements (TransDLANet, SwinDocSegmenter, SelfDocSeg) modify specific detector internals, making them non-transferable across detector families.

The authors argue that documents are inherently multimodal (rich text + visual structure) and that layout analysis should exploit textual semantics, particularly for semantically distinct categories like page headers, footers, and section headers.

What is the novelty?

Textual Grid Representation

M2Doc uses a BERTgrid-style approach to create pixel-aligned textual features. Given a document image $\mathbf{I} \in \mathbb{R}^{H \times W \times 3}$ with $N$ OCR-detected words, each word $w_i$ with bounding box $b_i$ is encoded by BERT:

$$(T_1, \dots, T_N) = \text{BERT}(w_1, \dots, w_N)$$

The sequential embeddings $T_i \in \mathbb{R}^{d \times 1}$ (where $d = 768$) are placed into a 2D grid $\mathbf{G} \in \mathbb{R}^{H \times W \times d}$:

$$G_{x,y} = \begin{cases} T_i, & \text{if } (x, y) \in b_i \\ 0, & \text{otherwise} \end{cases}$$

Unlike VSR’s two-backbone approach, M2Doc feeds both the image and the textual grid through a single shared backbone (ResNet), using convolutions to align channel dimensions before the first block.

Early Fusion (Pixel-Level)

After feature extraction, each scale $\theta \in \{1, 2, 3, 4\}$ produces visual features $P_\theta$ and textual features $S_\theta$. A gated mechanism adaptively weights the textual contribution:

$$\alpha_\theta = \eta(S_\theta)$$

$$F_\theta = \text{LayerNorm}(\alpha_\theta \odot S_\theta + P_\theta)$$

where $\eta(\cdot)$ is two $1 \times 1$ convolutions with ReLU and Tanh activations (restricting scores to $(0, 1)$), and $\odot$ is element-wise multiplication. LayerNorm handles the zero-valued regions of the textual grid.

Late Fusion (Block-Level)

After the RPN or Transformer encoder generates candidate bounding boxes $r_j$, late fusion matches each candidate to OCR boxes via IoU:

$$\text{IoU}_{i,j} = \frac{|r_j \cap b_i|}{|r_j \cup b_i|}$$

Words with IoU above a threshold are aggregated into a block-level textual feature:

$$E_j = \Gamma(T \cdot J_j)$$

where $J_j$ is a binary inclusion vector and $\Gamma$ is an MLP mapping BERT dimensions to the decoder’s channel dimensions. For end-to-end detectors (DINO), these features are added to content queries:

$$\text{Query}_j = \text{Query}_j + \lambda_1 E_j$$

For two-stage detectors (Cascade Mask R-CNN), block-level textual features are added to RoI features scaled by $\lambda_2$ before the R-CNN head.

Design Insight

The authors find that gated fusion works best for early fusion (where textual features are backbone-extracted and need adaptive weighting), while simple summation works best for late fusion (where textual features come directly from the pre-trained language model and are already high-quality).

What experiments were performed?

Datasets

Detectors Evaluated

• End-to-end: DINO (DINO-4Scale, 900 queries)
• Two-stage: Cascade Mask R-CNN
• Pluggability test: Mask R-CNN, Faster R-CNN, Deformable DETR

All use ResNet-101 + FPN as the visual backbone and BERT-Base-Multilingual-Cased as the language model. Detectors are initialized from COCO 2017 pre-trained weights.

Training Configuration

• DocLayNet and M6Doc: 36 epochs
  • Cascade Mask R-CNN: SGD, lr = 2e-2, decayed to 2e-3 at epoch 27 and 2e-4 at epoch 33
  • DINO: AdamW, lr = 1e-4, decayed to 3.3e-5 at epoch 27 and 1e-5 at epoch 33
• PubLayNet: 6 epochs, same initial lr, decayed by 10$\times$ at epoch 5
• Custom anchor scales for Cascade Mask R-CNN: [0.01, 0.02, 0.05, 0.1, 0.2, 0.5, 1.0, 2, 5, 10]
• Built on MMDetection

Key Results

DocLayNet (Table 1): DINO + M2Doc achieves 89.0 mAP, up from 77.7 for unimodal DINO. This is reported as the first method to significantly exceed the human baseline of 83 mAP. Cascade Mask R-CNN + M2Doc reaches 86.7 mAP (+11.1). The largest per-category gains come from semantically distinct classes: Page-footer (+27.3), Page-header (+14.4), Section-header (+18.6), Formula (+17.1).

M6Doc (Table 2): DINO + M2Doc achieves 69.9 mAP (+1.9). Gains are more modest due to M6Doc’s 74 fine-grained categories and diverse input scales limiting recall.

PubLayNet (Table 3): DINO + M2Doc achieves 95.5 mAP (+0.1 over unimodal DINO). Minimal improvement because PubLayNet’s 5-category physical layout is already near-saturated and semantically simple.

Ablations (M6Doc)

• Module ablation (Table 4): Both early and late fusion contribute. Together they yield +1.9 mAP for DINO and +2.1 mAP for Cascade Mask R-CNN. Individual modules contribute roughly equally (+0.4 to +0.5 for DINO).
• Fusion strategy ablation (Table 5): Gate mechanism is best for early fusion; summation is best for late fusion. Concatenation and other combinations underperform.
• Pluggability (Table 6): All five detectors improve with M2Doc. Gains range from +1.6 mAP (Deformable DETR) to +2.9 mAP (Mask R-CNN). The authors note that their reproduced baselines are higher than M6Doc’s originally reported numbers due to different experimental settings, so absolute mAP values should be compared with that caveat in mind.

Notable Observation

The only category where multimodal fusion hurts performance is “Figure” (which mostly lacks text content). Text-overlaid pictures can also degrade boundary detection for multimodal models.

What are the outcomes/conclusions?

Strengths

• The pluggable design is detector-agnostic, demonstrated across five architectures spanning two-stage and end-to-end families.
• The DocLayNet results (+11.3 mAP) are substantial and indicate that multimodal fusion addresses a real gap in logical layout analysis where semantic categories require textual understanding.
• The single-backbone design is simpler and more parameter-efficient than VSR’s two-backbone approach.
• Ablations are thorough, testing both module contributions and fusion strategy alternatives.

Limitations

• OCR dependency: The method requires pre-computed OCR results (word boxes + text). DocLayNet provides human-annotated OCR, but real-world OCR introduces noise. The paper does not evaluate robustness to OCR errors.
• Marginal gains on simple datasets: On PubLayNet (+0.1 mAP), the multimodal overhead (BERT inference + fusion modules) provides almost no benefit, suggesting the approach is only worthwhile for complex logical layout tasks.
• No latency or throughput analysis: Adding BERT inference and fusion modules has a computational cost that is never quantified. Inference time comparisons with unimodal baselines are absent.
• Missing comparison with pre-trained multimodal models: The paper compares against unimodal detectors and VSR, but does not compare with LayoutLMv3 or DiT fine-tuned for DLA, which also leverage multimodal features (through pre-training). LayoutLMv3 appears in the PubLayNet table but not DocLayNet or M6Doc.
• Figure category degradation: The authors acknowledge that multimodal fusion hurts text-overlaid image detection but offer no mitigation strategy.

Reproducibility

Models

• Visual backbone: ResNet-101 + FPN, initialized from COCO 2017 pre-trained weights.
• Language model: BERT-Base-Multilingual-Cased (110M params), from HuggingFace.
• Early fusion: two $1 \times 1$ conv layers with ReLU + Tanh per feature scale ($\theta \in \{1,2,3,4\}$).
• Late fusion: IoU matching + MLP mapping BERT dim (768) to detector channel dim.
• Pre-trained weights for R50 and R101 variants (4 checkpoints: Cascade Mask R-CNN and DINO, each with R50 and R101) available via BaiduNetDisk and Google Drive (per the GitHub repo). Weights are only released for DocLayNet; no PubLayNet or M6Doc checkpoints are provided. Full training and inference code is provided, including multi-GPU scripts.

Algorithms

• Cascade Mask R-CNN: SGD, lr = 2e-2, step decay at epochs 27 and 33, 36 epochs.
• DINO: AdamW, lr = 1e-4, step decay at epochs 27 and 33, 36 epochs.
• PubLayNet: 6 epochs, lr decayed by 10$\times$ at epoch 5.
• IoU threshold for late fusion: not stated in the paper text. Config files in the GitHub repo (mmdetection/m2doc_config/) likely contain the actual values used.
• $\lambda_1$ and $\lambda_2$: described as adjustable hyperparameters; exact values not stated in the paper text but may be recoverable from the repo config files.
• Framework: MMDetection.
• Environment: Python 3.8.0, CUDA 10.2, PyTorch 1.8.1, mmcv, mmengine, transformers.

Data

• PubLayNet: 360K pages, word-level OCR from PDFMiner. Publicly available (CDLA-Permissive-1.0).
• DocLayNet: 80K pages, human sentence-level OCR annotations included. Publicly available (CDLA-Permissive-1.0).
• M6Doc: 9K pages, sentence-level OCR obtained via OCR engine (unspecified which). Restricted access (CC BY-NC-ND 4.0); requires application and approval from HCIILAB.
• Standard train/val/test splits used for all three datasets.

Evaluation

• Metric: COCO-style mAP @ IoU [0.50:0.95:0.05].
• Per-category AP reported for DocLayNet (11 classes) and PubLayNet (5 classes).
• AP50, AP75, and Recall additionally reported for M6Doc.
• No error bars, significance tests, or multi-run statistics.
• Baselines reproduced in the same MMDetection framework with consistent settings.

Hardware

• Not explicitly reported in the paper. The GitHub repo’s distributed training script (dist_train.sh) defaults to 8 GPUs, suggesting an 8-GPU setup was used, but GPU type, training time, and memory requirements are unspecified.
• No inference speed benchmarks.

BibTeX

@inproceedings{zhang2024m2doc,
  title={M2Doc: A Multi-Modal Fusion Approach for Document Layout Analysis},
  author={Zhang, Ning and Cheng, Hiuyi and Chen, Jiayu and Jiang, Zongyuan and Huang, Jun and Xue, Yang and Jin, Lianwen},
  booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
  volume={38},
  number={7},
  pages={7233--7241},
  year={2024},
  doi={10.1609/aaai.v38i7.28552}
}

TL;DR

RoDLA introduces the first systematic robustness benchmark for Document Layout Analysis (DLA), comprising ~450K perturbed document images across three datasets (PubLayNet-P, DocLayNet-P, M6Doc-P) with 12 perturbation types at 3 severity levels. The authors propose two metrics (mPE for perturbation assessment and mRD for robustness evaluation) and a robust model that integrates channel attention and average pooling into a DINO-based detector to resist perturbation-induced attention shifting.

What kind of paper is this?

Dominant: $\Psi_{\text{Evaluation}}$. The headline contribution is a robustness benchmarking framework: a perturbation taxonomy, two new evaluation metrics (mPE and mRD), and a systematic assessment of 10+ DLA methods under controlled perturbations. The paper’s center of gravity is measuring how existing models fail under realistic document corruptions.

Secondary: $\Psi_{\text{Resource}}$. The perturbed benchmark datasets (PubLayNet-P, DocLayNet-P, M6Doc-P) and perturbation generation code are released as reusable community assets.

Secondary: $\Psi_{\text{Method}}$. The RoDLA model itself is a methodological contribution (channel attention + average pooling in a DINO encoder), though it serves primarily to demonstrate the benchmark’s utility.

What is the motivation?

DLA models are typically evaluated on clean digital documents, but real-world documents suffer from scanning artifacts, camera distortions, noise, and other degradations. Prior work had not systematically studied how these perturbations affect DLA performance. The authors observed that existing high-performing models (e.g., SwinDocSegmenter achieving 93.7% mAP on clean PubLayNet) can drop drastically under perturbation (down to 29.2% mAP under speckle noise). This gap between clean-data performance and real-world robustness motivates both the benchmark and the robust model.

Existing robustness metrics (borrowed from ImageNet-C) conflate perturbation severity with model robustness because they rely on a single baseline model’s performance as reference. The authors argue this introduces metric uncertainty from model randomness.

What is the novelty?

Perturbation Taxonomy

A hierarchical taxonomy of 12 document-specific perturbations organized into 5 groups, each with 3 severity levels:

1. Spatial Transformation: Rotation (P1), Warping (P2), Keystoning (P3)
2. Content Interference: Watermark (P4), Complex Background (P5)
3. Inconsistency Distortion: Non-uniform Illumination (P6), Ink-bleeding (P7), Ink-holdout (P8)
4. Blur: Defocus (P9), Vibration (P10)
5. Noise: Speckle (P11), Texture (P12)

Mean Perturbation Effect (mPE)

A model-independent metric for quantifying perturbation severity that combines IQA metrics with degradation scores:

$$\text{mPE}_p = \frac{1}{NMK} \sum_{s=1}^{N} \left(\sum_{i=1}^{M} f_{s,p}^i + \sum_{g=1}^{K} D_{s,p}^g\right)$$

where $f_{s,p}^i$ are IQA scores (MS-SSIM, CW-SSIM) and $D_{s,p}^g$ is the degradation of a reference model. By combining multiple assessment methods, mPE reduces sensitivity to any single metric’s blind spots (e.g., MS-SSIM is insensitive to warping; CW-SSIM is insensitive to defocus at fine severity levels).

Mean Robustness Degradation (mRD)

A robustness metric that normalizes a model’s performance drop by the perturbation’s inherent difficulty:

$$\text{RD}_p = \frac{1}{N} \sum_{s=1}^{N} \frac{D_{s,p}^g}{\text{mPE}_{s,p}}$$

The overall mRD averages $\text{RD}_p$ across all 12 perturbation types. Values above 100 indicate the model degrades more than expected; below 100 indicates better-than-expected robustness. Lower is better.

RoDLA Model

Built on DINO with an InternImage backbone, the model adds:

• Channel-wise Attention (CA) in the encoder: performs self-attention along the channel dimension to aggregate correlated feature channels, reducing attention shifting on perturbed tokens
• Average Pooling Layers (APL) with a dilation predictor: leverages neighboring token attention to mitigate overemphasis on individual (potentially corrupted) tokens

The channel-wise attention operates as:

$$Y = \sigma\left(\frac{\text{Softmax}(Q) \cdot \text{Softmax}(K^T)}{\sqrt{d}} \cdot \text{MLP}(V)\right)$$

where $Q, K, V \in \mathbb{R}^{d \times n}$.

What experiments were performed?

Benchmark Setup

• Datasets: PubLayNet (360K pages, 5 classes), DocLayNet (80.8K pages, 11 classes), M6Doc (9K pages, 74 classes)
• Perturbed versions: Each dataset $\times$ 12 perturbations $\times$ 3 severity levels = ~450K total perturbed images
• Baselines: 10+ methods including LayoutParser, Faster R-CNN, Mask R-CNN, Cascade R-CNN, DocSegTr, SwinDocSegmenter, DINO, Co-DINO, and multi-modal models (DiT, LayoutLMv3)
• Backbones: ResNet, ResNeXt, Swin Transformer, DiT, LayoutLMv3, InternImage
• Framework: All models reproduced in MMDetection with consistent training settings
• Training: Models trained on clean data only; perturbations applied only at test time
• Metrics: Clean mAP, P-Avg (average mAP across all perturbations), mRD

Key Results

PubLayNet-P:

RoDLA achieves the best clean mAP (96.0%) and the best mRD (116.0) among single-modal methods, with +3.8% P-Avg over Faster R-CNN. Among all methods including multi-modal ones, DiT + Cascade R-CNN achieves the best mRD (95.8), benefiting from document-specific pre-training on IIT-CDIP.

DocLayNet-P: RoDLA achieves 82.8% clean mAP (+1.9% over DINO baseline), 62.5% P-Avg (+7.1%), and mRD of 135.4.

M6Doc-P: RoDLA achieves 67.5% clean mAP, 52.9% P-Avg (+12.1%), and mRD of 150.4.

Ablation Studies

• Backbone: InternImage outperforms Swin Transformer, but RoDLA narrows the gap (only 0.1% P-Avg difference and 8.2 mRD difference between backbones)
• CA placement: Encoder-only CA placement is optimal; adding CA to the decoder hurts robustness
• APL placement: Encoder-only APL is optimal; the combined CA (encoder) + APL (encoder) configuration yields the best mRD of 115.7
• Multiple perturbations: Stacking perturbations degrades performance progressively (P-Avg drops from 64.9 with 2 perturbations to 41.9 with 5 on M6Doc-P)

What are the outcomes/conclusions?

Key findings:

• Existing DLA models are fragile under realistic perturbations, with some experiencing 60%+ mAP drops under severe blur or noise
• Document-specific pre-training (DiT, LayoutLMv3 on IIT-CDIP) substantially improves robustness, particularly against texture and noise perturbations
• Multi-modal models are not uniformly more robust; LayoutLMv3 shows vulnerability to content interference (watermark, background) despite strong overall numbers
• The RoDLA model demonstrates that attention mechanism modifications can improve both clean performance and robustness simultaneously
• mPE provides more balanced perturbation assessment than individual IQA metrics, and mRD decouples perturbation difficulty from model robustness measurement

Limitations the authors acknowledge:

• The benchmark focuses on vision-based single-modal DLA models; multi-modal robustness testing is limited
• No human-in-the-loop evaluation for deployment readiness
• Perturbations are applied only at test time (no robustness-aware training or augmentation strategies explored)

Limitations not discussed:

• The 12 perturbations, while comprehensive, do not cover all real-world degradations (e.g., partial occlusion, page folding, mixed-resolution scanning)
• mPE still depends on a baseline model (Faster R-CNN) for the degradation component, partially undermining the claim of model independence
• The benchmark uses existing datasets that are predominantly English and scientific/business documents; robustness for non-Latin scripts and diverse document types is not evaluated
• Severity levels are somewhat arbitrary; real-world degradation distributions may not align with the L1/L2/L3 bins

Reproducibility

Models

• RoDLA: DINO-based detector with InternImage backbone (~323M parameters), pre-trained on ImageNet
• Channel-wise Attention module and Average Pooling Layers added to the encoder
• Pre-trained checkpoints for PubLayNet, DocLayNet, and M6Doc are released via the GitHub repo

Algorithms

• Optimizer: AdamW (lr=2e-4, weight decay=1e-4)
• Schedule: Step-based with warmup at epochs {16, 22}, warmup ratio 1e-3
• Training: 24 epochs, batch size 2 per GPU
• Augmentation: Horizontal flip (0.5 ratio), random crop (384, 600)
• All models trained on clean data only; perturbations applied at test time only
• 38 total models trained for the benchmark (24 on PubLayNet, 7 each on DocLayNet and M6Doc)

Data

• PubLayNet-P: PubLayNet test set + 12 perturbations $\times$ 3 levels
• DocLayNet-P: DocLayNet test set + 12 perturbations $\times$ 3 levels
• M6Doc-P: M6Doc test set + 12 perturbations $\times$ 3 levels
• Perturbation generation code and pre-generated benchmarks available via the GitHub repo
• Background perturbation (P5) uses images from ILSVRC (ImageNet)

Evaluation

• Primary metrics: mAP (COCO-style), P-Avg (average mAP across all perturbations/levels), mRD
• mPE uses MS-SSIM + CW-SSIM + Faster R-CNN degradation as components
• All baselines reproduced in MMDetection for fair comparison
• No error bars or multiple-run variance reported
• Rotation perturbation (P1) causes the largest performance drops across all models due to annotation misalignment at high severity

Hardware

• Training: 4$\times$ NVIDIA A100 (40 GB each), 300 GB CPU memory per node
• No GPU-hours or cost estimates reported
• No inference latency or throughput measurements

BibTeX

@inproceedings{chen2024rodla,
  title={RoDLA: Benchmarking the Robustness of Document Layout Analysis Models},
  author={Chen, Yufan and Zhang, Jiaming and Peng, Kunyu and Zheng, Junwei and Liu, Ruiping and Torr, Philip and Stiefelhagen, Rainer},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  pages={15556--15566},
  year={2024},
  doi={10.1109/CVPR52733.2024.01473}
}

TL;DR

SwinDocSegmenter is a unified end-to-end transformer architecture for document instance segmentation that combines a Swin Transformer backbone with a DETR-style encoder-decoder, contrastive denoising training, and hybrid bipartite matching. It reports competitive results on PubLayNet (93.7 mAP), TableBank (98.0), and Historical Japanese (84.6), while providing the first transformer-based instance segmentation baseline on DocLayNet (76.9). The key finding is that generic MS-COCO pre-training combined with contrastive learning and adaptive matching reduces the domain bias caused by document-specific pre-training.

What kind of paper is this?

Dominant: $\Psi_{\text{Method}}$ : The headline contribution is a unified transformer architecture for document instance segmentation with three components: mixed query selection with contrastive projection heads, contrastive denoising training adapted for masks, and hybrid bipartite matching for domain shift. Ablations and SOTA comparisons dominate the experimental sections.

Secondary: $\Psi_{\text{Evaluation}}$ : The paper provides extensive benchmarking across five datasets (PubLayNet, PRImA, HJDataset, TableBank, DocLayNet) and includes detailed ablations on backbone choice, resolution, query count, loss functions, and pre-training bias.

What is the motivation?

Document layout analysis has traditionally been addressed by detection models (bounding boxes), but instance segmentation (pixel-level, class-aware and instance-aware labels) provides richer structure for downstream document understanding tasks. The authors identify three specific gaps:

1. Domain shift: Transformer models pre-trained on domain-specific document datasets (e.g., PubLayNet for scientific articles) fail when applied to different document types (e.g., magazines, historical texts). The pre-training introduces bias toward common classes.
2. No unified detection + segmentation: Existing approaches treat detection and segmentation as separate or sequential tasks rather than jointly optimizing both.
3. Low-frequency class penalty: Transformers struggle with document element categories that have few training examples (e.g., separators, “other” regions), because standard matching and loss functions are biased toward high-frequency classes like text.

What is the novelty?

Unified Architecture

The model combines a Swin Transformer Large (SwinL) backbone with a transformer encoder-decoder pair and a segmentation branch. The segmentation branch constructs a pixel embedding map (PEM) by fusing multi-scale features:

$$M = Q_e \otimes \delta(\Gamma(S_b) + \psi(T_e))$$

where $Q_e$ is the query embedding from the decoder, $S_b$ is the 1/4-resolution backbone feature map, $T_e$ is the 1/8-resolution encoder feature map (upsampled via $\psi$), $\Gamma$ is a channel projection convolution, and $\delta$ is a segmentation head.

Mixed Query Selection with Contrastive Projection

The encoder output passes through three parallel heads (classification, detection, segmentation). Top-ranked features (by classification confidence) are selected as content queries. Two projection heads apply contrastive learning:

Low-level projection (shallow MLP) for contrastive feature alignment:

$$\mathcal{L}_{low} = \sum_{i=1}^{n} \sum_{j=1}^{n’} -\log \frac{\exp(f_i \cdot f_j / \tau)}{\sum_{c=1}^{k} \exp(f_c \cdot f_j / \tau)}$$

High-level projection (deep MLP) with learned prototypes $P = (p_1, p_2, \ldots, p_m)$:

$$\mathcal{L}_{high} = \sum_{i=1}^{n} \sum_{j=1}^{n’} -\log \frac{\exp(f_i \cdot p_j / \phi_j)}{\sum_{c=1}^{k} \exp(f_c \cdot p_j / \phi_j)}$$

where $\tau$ is a dataset-specific temperature and $\phi_j$ is a concentration estimate per prototype. Content queries remain learnable (not initialized from encoder features), which the authors argue aids domain shift.

Contrastive Denoising Training (CDN)

Adapted from DINO-style denoising but extended to the segmentation task. Ground-truth boxes are randomly noised, and:

• Positive queries (noise scale $< \lambda_p$) are trained to reconstruct ground truth
• Negative queries (noise scale between $\lambda_p$ and $\lambda_e$) are suppressed via focal loss

The paper states $\lambda_e > \lambda_p$ but reports $\lambda_p = 0.1$ and $\lambda_e = 0.02$, which is an apparent contradiction in the text. The numerical values are reproduced here as published.

For segmentation, the noised boxes serve as the denoising target, treating masks as the clean signal and boxes as a noised version. Noised queries are added during training and removed at inference.

Hybrid Bipartite Matching

The standard Hungarian matching is augmented with an additional mask prediction loss alongside L1 and focal loss. During domain shift from pre-trained weights, the mask prediction penalty is initially weighted more heavily, then decreased as the model converges. This addresses the inconsistency between masks predicted from different heads.

What experiments were performed?

Datasets