Paper: PaddleOCR 3.0: Advancements in Open-Source OCR and Document AI
Code: PaddlePaddle/PaddleOCR
Models: PaddlePaddle on HuggingFace
Docs: paddlepaddle.github.io/PaddleOCR

TL;DR

PaddleOCR 3.0 is an open-source OCR and document AI toolkit (Apache 2.0) that ships three main solutions: PP-OCRv5 (lightweight multilingual OCR), PP-StructureV3 (end-to-end document parsing), and PP-ChatOCRv4 (OCR + LLM for KIE/QA). The system includes deployment infrastructure (high-performance inference, Triton/FastAPI serving, on-device, MCP integration). On OmniDocBench OCR, PP-OCRv5 ranks first on average across 17 scenarios. On document parsing, PP-StructureV3 achieves Edit distance 0.145 (EN) and 0.206 (ZH), outperforming MinerU and Docling.

What kind of paper is this?

Primarily $\Psi_{\text{Resource}}$, with substantial $\Psi_{\text{Method}}$ components.

• Dominant: $\Psi_{\text{Resource}}$: The paper delivers a production-ready toolkit with Apache 2.0 licensing, pre-trained model zoo (PP-OCRv5 server/mobile variants, PP-StructureV3, PP-ChatOCRv4), layered API/CLI interfaces, and comprehensive deployment infrastructure (HPI with automatic backend selection, FastAPI/Triton serving, on-device via Paddle-Lite, MCP server). The resource framing is explicit: enabling reproducible OCR and document understanding for the research and practitioner communities. Performance optimizations (73% latency reduction on T4 for mobile recognition) and serving options are presented as reusable tooling rather than operational validation.
• Secondary: $\Psi_{\text{Method}}$: PP-OCRv5 introduces a single multilingual model supporting Simplified Chinese, Traditional Chinese, Pinyin, English, and Japanese. PP-StructureV3 integrates layout analysis, table recognition, and specialized document items (seals, formulas, charts). PP-ChatOCRv4 combines pipeline OCR outputs with vector retrieval, VLM-based answer extraction (PP-DocBee2), and LLM reasoning (ERNIE-4.5) with result fusion.

What is the motivation?

The report frames OCR and document parsing as foundational for downstream document understanding, with LLM and RAG adoption increasing demand for high-quality text extraction, structure recovery, and semantic interpretation across diverse document types.

• Multilingual and layout complexity: Documents span handwriting, multilingual text, complex layouts, tables, formulas, and charts. Prior OCR systems often specialize narrowly or require multiple models for different languages/scripts.
• Production barriers: Deploying OCR at scale requires optimized inference (low latency, resource efficiency), robust serving infrastructure, and on-device support. Existing open-source OCR tools may lack these deployment features or use restrictive licenses.
• Key information extraction and QA: Document workflows increasingly involve extracting structured information and answering questions over document content, motivating integrated OCR + LLM/VLM pipelines.

What is the novelty?

PP-OCRv5 single multilingual model: A unified recognition model handling Simplified Chinese, Traditional Chinese, Pinyin, English, and Japanese, with server (GPU-optimized) and mobile (CPU/resource-constrained) variants. The pipeline includes preprocessing, text detection, text-line orientation classification, and recognition.

PP-StructureV3 end-to-end parsing: Integrates layout analysis, table recognition, and structure extraction into a unified document parsing system. Extends to specialized items including seal text recognition, formula recognition, and chart analysis. Reports Edit distance 0.145 (EN) and 0.206 (ZH) on OmniDocBench parsing, outperforming MinerU (0.333 EN / 0.350 ZH) and Docling (0.538 EN / 0.569 ZH).

PP-ChatOCRv4 retrieval-augmented KIE/QA: Combines PP-StructureV3 OCR outputs with vector retrieval, a 3B VLM (PP-DocBee2) for prompt-based answer extraction from document images, and a 300B LLM (ERNIE-4.5-300B-A47B) for reasoning. Result fusion merges text-based and image-based answers. Reports 85.55% Recall@1 on a custom benchmark (638 documents, 1,196 QA pairs), outperforming GPT-4o (63.47%), PP-ChatOCRv3 (70.08%), and Qwen2.5-VL-72B (80.26%).

Deployment infrastructure: Redesigned inference library (PaddleX 3.0) with layered API/CLI, high-performance inference (HPI) with automatic backend selection (Paddle Inference, OpenVINO, ONNX Runtime, TensorRT), built-in optimizations (multi-threading, FP16), FastAPI/Triton serving, on-device support (Paddle-Lite), and an MCP server exposing OCR/parsing as tools with stdio and Streamable HTTP transports.

What experiments were performed?

PP-OCRv5 on OmniDocBench OCR (17 scenarios, 1-EditDist metric): PP-OCRv5 ranks first on average across all scenarios, compared against multiple VLM baselines with reported parameter sizes.

PP-StructureV3 on OmniDocBench parsing (Edit distance, lower is better): Evaluated on English and Chinese document parsing. PP-StructureV3 achieves Edit 0.145 (EN) and 0.206 (ZH), outperforming MinerU-1.3.11 (0.333 EN / 0.350 ZH) and Docling-2.14.0 (0.538 EN / 0.569 ZH).

PP-ChatOCRv4 on custom KIE/QA benchmark (638 document images, 1,196 QA pairs, Recall@1 metric): The dataset spans financial reports, research papers, contracts, manuals, and regulations. PP-ChatOCRv4 achieves 85.55% Recall@1, compared to GPT-4o (63.47%), PP-ChatOCRv3 (70.08%), and Qwen2.5-VL-72B (80.26%).

HPI latency reduction (NVIDIA Tesla T4): Enabling HPI reduces latency for PP-OCRv5_mobile_rec by 73.1% and PP-OCRv5_mobile_det by 40.4%.

What are the outcomes/limitations?

Outcomes:

• OmniDocBench OCR: PP-OCRv5 ranks first on average across 17 scenarios using the 1-EditDist metric.
• OmniDocBench parsing: PP-StructureV3 achieves Edit 0.145 (EN) and 0.206 (ZH), leading among open-source pipeline systems.
• Custom KIE/QA benchmark: PP-ChatOCRv4 achieves 85.55% Recall@1, a +5.29 point improvement over Qwen2.5-VL-72B (80.26%) and +22.08 points over GPT-4o (63.47%).
• Inference optimization: HPI reduces mobile recognition latency by 73.1% and mobile detection latency by 40.4% on T4 hardware.
• Deployment surface: The toolkit provides FastAPI/Triton serving, on-device deployment via Paddle-Lite, and an MCP server with Local, AI Studio, and Self-Hosted modes supporting stdio and Streamable HTTP transports.

Limitations and open questions:

• Benchmark scope: OmniDocBench OCR and parsing evaluations are publicly documented, but the custom KIE/QA benchmark (638 documents, 1,196 QA pairs) has limited external validation. The authors do not release this benchmark publicly, limiting reproducibility of the PP-ChatOCRv4 results.
• Multilingual coverage: PP-OCRv5 supports Simplified Chinese, Traditional Chinese, Pinyin, English, and Japanese. Generalization to other languages (Arabic, Cyrillic, Indic scripts, etc.) is not addressed. The single multilingual model design may face capacity constraints when scaling to dozens of languages.
• VLM dependency for KIE/QA: PP-ChatOCRv4 relies on PP-DocBee2 (3B VLM) and ERNIE-4.5 (300B LLM). The paper does not ablate the contribution of each component (pipeline OCR vs. VLM-based answer extraction vs. LLM reasoning vs. result fusion). It is unclear whether the Recall@1 gains come primarily from the fusion strategy, the quality of PP-StructureV3 outputs, or the VLM/LLM model choices.
• Deployment overhead: The HPI latency reductions are reported for mobile variants on T4 hardware. Server variants on GPU hardware (A100, H100) are not profiled. Throughput and cost-per-page estimates are not provided, limiting comparison to olmOCR ($176/M pages on L40S) or MinerU2.5 (1.224 pages/s on A100).
• License and model availability: The toolkit is Apache 2.0, but the report does not clarify the licensing or availability of ERNIE-4.5-300B-A47B (used in PP-ChatOCRv4). PP-DocBee2 weights are not linked in the paper. This limits reproducibility of the KIE/QA results.

Model

PP-OCRv5 pipeline and variants

Two variants:

• Server: GPU-optimized for throughput and accuracy.
• Mobile: CPU/resource-constrained for edge deployment.

Pipeline stages:

1. Image preprocessing: Handles rotation correction, noise reduction, and normalization.
2. Text detection: Identifies text regions with bounding boxes.
3. Text-line orientation classification: Corrects text-line rotation (0°, 90°, 180°, 270°).
4. Text recognition: Converts detected text regions to Unicode strings.

Key architectural ingredients (Figure 3 in paper, selected examples):

• Backbones: PP-HGNetV2 (server), PP-LCNetV3 (mobile).
• Detection enhancements: PFHead, DSR (Dynamic Scale Regression), Lite-Neck.
• Recognition enhancements: GTC-NRTR (Guided Training of CTC with NRTR), multi-scale training.
• Data strategy: Pretrain distillation, synthetic data generation, label refinement using ERNIE 4.5.

PP-StructureV3 modules

Core capabilities:

• Layout analysis: Identifies and classifies document regions (text blocks, tables, figures, formulas, etc.).
• Table recognition: Extracts table structure and cell content, supporting rowspan/colspan via HTML output.
• Structure extraction: Recovers reading order and hierarchical document structure.

Specialized document items:

• Seal text recognition: Handles circular and non-rectangular text (stamps, official seals).
• Formula recognition: Converts mathematical notation to LaTeX.
• Chart analysis: Extracts data from charts and graphs (details not provided in the report).

PP-ChatOCRv4 architecture

Pipeline stages (Figure 9 in paper):

1. Prompt engineering: User query + optional image input.
2. PP-StructureV3 extraction: OCR and document parsing to recover text, layout, and structure.
3. Vector retrieval: Index OCR outputs for semantic search.
4. PP-DocBee2 (3B VLM): Prompt-based answer extraction from document image regions.
5. ERNIE-4.5-300B-A47B (LLM): Reasoning over retrieved text and VLM outputs.
6. Result fusion: Combines text-based (OCR + LLM) and image-based (VLM) answers.

Key design choices:

• Dual-stream reasoning: Separate text-based (pipeline OCR $\rightarrow$ vector retrieval $\rightarrow$ LLM) and image-based (VLM) pathways. Fusion merges results to handle cases where OCR fails (e.g., complex tables, handwriting) or VLM hallucinates.
• PP-DocBee2: A 3B VLM fine-tuned for document understanding tasks. The paper does not provide architectural details, training data, or standalone evaluation results for PP-DocBee2.

Data

PP-OCRv5 training data

The report does not specify the size, composition, or sourcing of the PP-OCRv5 training dataset. Key data strategies are described qualitatively:

• Synthetic data generation: Augments training with rendered text images.
• Label refinement using ERNIE 4.5: Corrects noisy labels in existing datasets.
• Multilingual coverage: Training data includes Simplified Chinese, Traditional Chinese, Pinyin, English, and Japanese text.

PP-ChatOCRv4 evaluation benchmark

Size: 638 document images, 1,196 QA pairs.

Document types: Financial reports, research papers, contracts, manuals, regulations, and other unspecified categories.

Metric: Recall@1 (exact match or fuzzy match against ground-truth answers).

Limitations: The benchmark is not publicly released, limiting external validation and reproducibility. The authors do not specify the annotation protocol, inter-annotator agreement, or difficulty distribution of the QA pairs.

OmniDocBench datasets

PP-OCRv5 evaluation: 17 scenarios covering diverse document types, scripts, and layouts. The paper cites OmniDocBench but does not detail the dataset composition or size.

PP-StructureV3 evaluation: English and Chinese document parsing subsets of OmniDocBench. The Edit metric measures character-level edit distance between predicted and ground-truth document structure.

Evaluation

PP-OCRv5 (OmniDocBench OCR)

The report states PP-OCRv5 ranks first on average across 17 scenarios but does not provide absolute 1-EditDist values or the names/sizes of competing VLM baselines. The table in the paper includes parameter counts for baselines but omits numeric results.

PP-StructureV3 (OmniDocBench parsing)

Edit distance is computed at the character level between predicted and ground-truth document structure. Lower is better. PP-StructureV3 leads by a substantial margin (2.3$\times$ better than MinerU on EN, 1.7$\times$ on ZH).

PP-ChatOCRv4 (custom KIE/QA benchmark)

Recall@1 measures whether the system’s top-ranked answer matches the ground-truth (exact or fuzzy match). PP-ChatOCRv4 achieves a +5.29 point gain over Qwen2.5-VL-72B and +22.08 points over GPT-4o.

Limitations: The benchmark is not publicly released. The authors do not ablate the contribution of PP-StructureV3 OCR quality vs. VLM answer extraction vs. LLM reasoning vs. result fusion. It is unclear whether the gains come from superior OCR, better VLM/LLM models, or the fusion strategy.

Hardware / Production

High-performance inference (HPI)

Key features:

• Automatic backend selection: Paddle Inference, OpenVINO, ONNX Runtime, TensorRT. The system selects the optimal backend based on hardware and model architecture.
• Built-in optimizations: Multi-threading, FP16 precision, on-demand ONNX conversion.
• Enabled via API: enable_hpi=True in the Python API.

Latency reduction on NVIDIA Tesla T4:

The report does not provide absolute latency values (ms/page) or throughput estimates (pages/s).

Serving options

Basic Serving (FastAPI):

• Lightweight REST API for OCR and document parsing.
• Multi-language client examples (Python, C++, etc.) provided in documentation.

High-Stability Serving (Triton):

• Nvidia Triton Inference Server integration for production deployments.
• Supports dynamic batching, model versioning, and ensemble inference.

On-device deployment:

• Paddle-Lite tooling for Android, iOS, and edge hardware.
• Mobile variants of PP-OCRv5 designed for CPU/resource-constrained environments.

MCP server

Model Context Protocol integration:

• Exposes OCR (PP-OCRv5) and document parsing (PP-StructureV3) as tools for LLM agents.
• Modes: Local (runs models on local hardware), AI Studio (cloud-hosted inference), Self-Hosted (user-managed servers).
• Transports: stdio (standard input/output) and Streamable HTTP (streaming responses).

Example configuration (Local mode, stdio transport):

{
  "mcpServers": {
    "paddleocr-mcp": {
      "command": "python",
      "args": ["-m", "paddleocr_mcp.server"],
      "env": {
        "MODE": "local"
      }
    }
  }
}

Implementation sketch

Python API for PP-StructureV3

from paddlex import create_pipeline

pipeline = create_pipeline(pipeline="PP-StructureV3")

# Single-page inference
output = pipeline.predict("document.pdf")

# Batch inference
for result in pipeline.predict(["doc1.pdf", "doc2.pdf"], batch_size=4):
    result.save_to_img("output/")
    result.save_to_json("output/")

Python API for PP-ChatOCRv4

from paddlex import create_pipeline

pipeline = create_pipeline(pipeline="PP-ChatOCRv4Doc")

# Build vector index from documents
pipeline.build_vector(data_root="documents/")

# Query the indexed documents
result = pipeline.chat(
    key_list=["contract_2023.pdf"],
    query="What is the contract termination date?"
)

print(result["result"])

Enabling HPI

pipeline = create_pipeline(
    pipeline="PP-OCRv5",
    enable_hpi=True  # Automatic backend selection + optimizations
)

Notes and open questions

Observations:

• PaddleOCR 3.0 is a comprehensive toolkit (OCR + parsing + KIE/QA + deployment) rather than a single model. The Apache 2.0 license and deployment infrastructure (HPI, serving, on-device, MCP) target production adoption.
• PP-StructureV3 leads on OmniDocBench parsing with substantial margins (2.3$\times$ better than MinerU on EN Edit). This suggests strong layout analysis and structure recovery, though the paper does not ablate the contribution of individual modules (layout detector, table recognizer, reading-order recovery, etc.).
• PP-ChatOCRv4’s +22 point Recall@1 gain over GPT-4o is striking, but the custom benchmark is not publicly released and the ablation is incomplete. It is unclear whether the improvement comes from superior OCR (PP-StructureV3), better VLM/LLM models (PP-DocBee2 + ERNIE-4.5), or the result fusion strategy.

Open questions:

• PP-OCRv5 training data: The report describes data strategies (synthetic generation, label refinement with ERNIE 4.5) but does not specify dataset size, composition, or sourcing. How much training data is required to achieve the reported OmniDocBench performance?
• OmniDocBench OCR results: The paper states PP-OCRv5 ranks first on average but omits numeric 1-EditDist values and baseline names/sizes. External validation on other OCR benchmarks (e.g., olmOCR-Bench, FoxBench) is absent.
• PP-ChatOCRv4 ablations: What is the contribution of each component (PP-StructureV3 OCR vs. PP-DocBee2 VLM vs. ERNIE-4.5 LLM vs. result fusion)? Does the fusion strategy generalize to other VLM/LLM combinations (e.g., GPT-4o + Qwen2.5-VL)?
• Deployment cost and throughput: The HPI latency reductions (73% for mobile recognition on T4) are impressive, but absolute latency (ms/page) and throughput (pages/s) are not reported. Cost-per-page estimates (e.g., olmOCR’s $176/M pages on L40S) are absent, limiting production planning.
• PP-DocBee2 details: The paper does not provide architectural details, training data, or standalone evaluation results for the 3B VLM used in PP-ChatOCRv4. Is PP-DocBee2 a general-purpose VLM or a document-specialized model? Are the weights publicly available?
• Multilingual scaling: PP-OCRv5 supports five languages/scripts (Simplified Chinese, Traditional Chinese, Pinyin, English, Japanese). How would the single multilingual model design scale to dozens of languages (Arabic, Cyrillic, Indic scripts, etc.)? Would capacity constraints require larger models or ensemble architectures?

Paper: olmOCR: Unlocking Trillions of Tokens in PDFs with Vision Language Models
Code: allenai/olmocr
Demo: olmocr.allenai.org
Model: olmOCR-7B-0825

TL;DR

olmOCR is a PDF-to-text linearization toolkit centered on a 7B vision-language model fine-tuned from Qwen2-VL-7B-Instruct. The system converts PDFs (both born-digital and scanned) into clean plain text suitable for language model development. The core innovation is “document anchoring,” a prompt technique that injects PDF-extracted text blocks with coordinates to reduce reading-order errors and hallucinations. On olmOCR-Bench (1,402 PDFs, 7,010 unit tests), the anchored model achieves 75.5% overall pass rate with an estimated cost of \$176 per million pages on L40S hardware.

What kind of paper is this?

Primarily $\Psi_{\text{Resource}}$, with meaningful $\Psi_{\text{Method}}$ and $\Psi_{\text{Evaluation}}$ components.

• Dominant: $\Psi_{\text{Resource}}$: The headline deliverables are (1) a training dataset (olmOCR-mix-0225, 260k pages), (2) a benchmark suite (olmOCR-Bench with 7,010 unit tests), and (3) a released fine-tuned model with inference code. The paper emphasizes enabling reproducible PDF processing for the research community.
• Secondary: $\Psi_{\text{Method}}$: Document anchoring (layout-aware prompt engineering with extracted text blocks and bounding boxes) is presented as a novel technique for reducing VLM hallucinations in document contexts.
• Secondary: $\Psi_{\text{Evaluation}}$: The benchmark design itself represents a substantial contribution: deterministic unit-test-style pass/fail rules that avoid LLM-as-judge biases.

What is the motivation?

PDFs encode rendering primitives (glyphs with positions) rather than semantic text structure or ground-truth reading order, making them challenging inputs for language model training and document-grounded inference.

• Low-fidelity extraction hurts LM training: Poor OCR quality can degrade training stability and downstream task performance. Cascading errors in document workflows compound the problem.
• Proprietary solutions are expensive: The authors cite GPT-4o API costs exceeding \$6,200 per million pages as a barrier to large-scale PDF processing for open research.
• Existing open tools lack validation: Pipeline systems (Marker, MinerU) and open VLMs have no standardized, reproducible benchmark for comparing PDF linearization quality.

What is the novelty?

Document anchoring: The system extracts text blocks with bounding boxes from the PDF using pypdf, then injects a subset of these blocks (with coordinates and image placeholders) into the VLM prompt alongside the page image. This provides layout hints that help the model maintain correct reading order and reduce hallucinated content.

Silver-data recipe: The authors generate approximately 260k page OCR targets with GPT-4o using structured JSON output, then fine-tune Qwen2-VL-7B-Instruct into olmOCR-7B-0225-preview. The training data is filtered for English-only content and sampled up to 3 pages per PDF.

Unit-test benchmark: olmOCR-Bench uses deterministic binary checks (presence/absence/order/table structure/math formula accuracy) designed to be evaluator-model-independent, explicitly addressing concerns about LLM-judge bias in document evaluation.

What experiments were performed?

olmOCR-Bench evaluation (Table 4): 1,402 PDFs spanning 7,010 unit tests across categories including text presence/absence, natural reading order, table accuracy, and math formula rendering. Baselines include open pipeline tools (Marker v1.7.5, MinerU v1.3.10), proprietary OCR systems (Mistral OCR, GPT-4o, Gemini Flash), and Qwen2-VL variants with and without document anchoring.

Cost analysis (Table 6): Estimates tokens/sec, pages/USD, and cost per million pages for multiple systems under specified hardware assumptions (L40S at \$0.79/hr, H100 at \$2.69/hr), including GPT-4o API vs. batch pricing.

Downstream LM pretraining ablation (Table 5): Continues pretraining OLMo-2-1124-7B for 50B tokens using PDFs linearized by olmOCR vs. Grobid+rules (peS2o baseline), then evaluates on standard LM benchmarks (MMLU, ARC-Challenge, DROP, etc.).

Ablations on anchoring: Compares Qwen2-VL variants and GPT-4o with and without document anchoring prompts to isolate the contribution of layout-aware context.

What are the outcomes/limitations?

Key results:

• olmOCR-Bench overall pass rate: 75.5% (95% CI via bootstrap) for the anchored model, outperforming Marker (70.1%), MinerU (61.5%), Mistral OCR (72.0%), GPT-4o (68.9%), and Qwen2.5-VL (65.5%).
• Cost estimate: \$176 per million pages on L40S hardware (assumes 12% retry rate for JSON parsing failures and degenerate repetition), compared to \$6,240 for GPT-4o batch API and \$596 for MinerU on L40S.
• Downstream pretraining impact: +1.3 percentage points average improvement across benchmark suite when replacing Grobid+rules PDF processing with olmOCR (53.9% $\rightarrow$ 55.2% average score).

Limitations and open questions:

• English-only training data: Explicit filtering via Lingua removes non-English documents, limiting multilingual generalization. The 3-page-per-PDF sampling may bias coverage toward short documents.
• Teacher model inheritance: Fine-tune targets are GPT-4o outputs, so the student model inherits teacher quirks and evaluation aligns to teacher behavior rather than purely human preference.
• Reliability challenges: The paper describes retries for JSON parsing failures and degenerate repetition (mitigated with higher temperature $\tau = 0.8$). The 12% retry rate impacts throughput in production.
• Benchmark generalization: olmOCR-Bench is carefully engineered but remains an in-house suite. The document distribution (55.9% academic papers) may not reflect other production use cases. External validation on held-out benchmarks is not the core claim.
• Long document handling: Sampling only a few pages per PDF during training raises questions about failure modes on documents with complex multi-page structure.

Contrast to olmOCR 2: This initial olmOCR release (February 2025) focuses on GPT-4o distillation with document anchoring prompts, whereas olmOCR 2 (October 2025) shifts to RL-based policy improvements using unit-test rewards from the same benchmark suite. olmOCR 2 reports 81.2% pass rate (vs. 75.5% here) and eliminates structured JSON output in favor of direct plain-text generation, improving robustness and reducing retry overhead.

Model

Base model: Qwen2-VL-7B-Instruct fine-tuned into olmOCR-7B-0225-preview (7B parameters).

Output format: The model generates structured JSON during training (matching GPT-4o’s synthetic target schema), though the inference prompt can be simplified to return “plain text representation” directly. The JSON schema includes fields for primary_language, rotation validity/correction, boolean flags for tables/formulas/diagrams, and natural_text.

Document anchoring representation

The prompt construction pipeline:

1. Extract PDF text blocks and image blocks with bounding boxes using pypdf.
2. Sample a subset of blocks (preferring start/end of document) to inject into the prompt.
3. If the prompt exceeds 8,192 tokens, regenerate with reduced character limits per block.
4. Concatenate the page image, selected text blocks (with RAW_TEXT_START/END delimiters), and generation instructions.

The authors describe this as “layout-aware retrieval” that provides the VLM with content hints and rough ordering cues, reducing hallucinations and reading-order errors.

Data

Training set: olmOCR-mix-0225

Size: 102,825 unique PDFs / 258,641 pages total (Table 1).

Source breakdown:

• Web PDFs: 96,929 documents / 240,940 pages (drawn from internal crawl of over 240 million PDF documents).
• Internet Archive books: 5,896 documents / 17,701 pages.

Filtering and sampling:

• Remove non-English documents (Lingua language detector).
• Filter parsing failures, spam keywords (explicit list), fillable forms, and documents with insufficient extractable text.
• Sample up to 3 pages per PDF.

Document type distribution (Table 2, manual annotation):

• Academic papers: 55.9%
• Brochures: 11.2%
• Legal documents: 10.2%
• Books: 6.8%
• Table-heavy: 5.6%
• Diagram-heavy: 4.7%
• Slideshows: 1.9%
• Other: 3.7%

Synthetic labeling with GPT-4o

The teacher model (GPT-4o) receives prompts with rules including:

• Preserve natural reading order.
• Use LaTeX for equations, Markdown for tables.
• Remove headers/footers when appropriate.
• Handle handwritten annotations if present.
• Do not hallucinate content.
• Output null if no text is present.

The structured JSON schema captures language, rotation metadata, content flags, and the final natural_text field.

Algorithms / Training

Fine-tuning configuration

• Effective batch size: 4
• Optimizer: AdamW
• Learning rate: $1 \times 10^{-6}$
• Schedule: Cosine annealing
• Steps: 10,000 (approximately 1.2 epochs over 258k pages)
• Hardware: 8 $\times$ H100 80GB
• Runtime: 16 node-hours
• Context truncation: Training examples truncated to 8,192 tokens; loss masked to final response tokens only.

Inference-time prompt

The production prompt for olmOCR-7B-0225-preview is minimal:

• Page image
• Prior extracted raw text blocks (with RAW_TEXT_START/END delimiters)
• Instruction: “return the plain text representation of this document as if you were reading it naturally”

No explicit JSON schema enforcement at inference (though the model was trained to produce JSON).

Evaluation

olmOCR-Bench construction

Design philosophy: Deterministic unit tests with binary pass/fail outcomes, avoiding LLM-as-judge and fuzzy reference matching. The authors explicitly cite concerns about evaluator bias and non-reproducibility in document benchmarks.

Size: 1,402 PDFs / 7,010 tests.

Test categories (Table 3, selected examples):

• Text presence (TP): Fuzzy string matching to verify key content appears.
• Text absence (TA): Verify headers/footers are removed.
• Natural reading order (NR): Check relative ordering of text segments.
• Table accuracy (TT): Validate neighbor-cell constraints in Markdown/HTML tables (HTML required for rowspan/colspan).
• Math formula accuracy (MF): KaTeX-rendered symbol layout matching.

Document type distribution in benchmark (Table 3):

• arXiv Math (AR): 2,927 formula tests
• Multi-Column (MC): 884 reading-order tests
• Tables (TT): 1,020 table structure tests
• Legal-Tabular-Tiny (LTT): 213 combined tests
• Open Street Map (OSM): 123 specialized tests
• Table Accuracy (TA): (additional table tests, count not specified in draft)

Benchmark results

Overall pass rate (95% CI, Table 4):

Category-wise highlights:

• Anchored olmOCR leads on AR (74.9%), MC (78.3%), and LTT (73.3%).
• Math-heavy OSM and table-heavy TA categories show more competitive performance across baselines.

Cost comparison

Cost per million pages (Table 6, selected systems):

Assumptions: L40S at \$0.79/hr, H100 at \$2.69/hr, 12% retry rate for JSON parsing failures and repetition handling.

Downstream LM pretraining ablation

Setup: Continue pretraining OLMo-2-1124-7B for 50B tokens on PDFs linearized with olmOCR vs. Grobid+rules (peS2o baseline). Evaluate on standard benchmarks.

Results (Table 5):

• Baseline (Grobid+rules) average score: 53.9%
• olmOCR average score: 55.2%
• Improvement: +1.3 percentage points

This suggests OCR/linearization quality can measurably impact LM performance at the 50B-token pretraining scale.

Hardware / Production

Inference pipeline and robustness

Orchestration: The system uses SGLang for serving and chunks work into batches of approximately 500 pages, coordinated through shared cloud storage (S3).

Reliability heuristics:

• Prompt regeneration: If token count exceeds 8,192, reduce character budget for sampled text blocks and rebuild prompt.
• Retry on JSON parse failure: Re-run generation with adjusted parameters.
• Rotation correction: Use PDF metadata to detect and correct page orientation.
• Degenerate repetition mitigation: Retry with higher temperature ($\tau = 0.8$) if the model produces repetitive output; fall back to alternative strategies if retries fail.

The reported 12% retry rate indicates non-trivial overhead in production deployments, though the authors frame this as necessary for quality assurance.

Throughput estimates

The paper provides pages-per-hour and cost-per-page calculations based on measured token throughput on L40S and H100 hardware, factoring in the retry rate. Specific throughput numbers are embedded in the cost estimates (Table 6).

Implementation sketch

# Conceptual pipeline based on paper description
for page in pdf:
    blocks = pypdf.extract_text_and_images_with_bboxes(page)
    selected = sample_blocks_preferring_doc_start_end(blocks)
    prompt = build_prompt(page_image, selected)
    
    if prompt_tokens > 8192:
        selected = regenerate_selected_blocks_with_lower_char_budget()
        prompt = build_prompt(page_image, selected)
    
    output = vlm.generate(prompt)
    
    if parse_fail_or_repetition(output):
        output = retry_with_adjustments(prompt, tau=0.8)
    
    plaintext_pages.append(output)

return concatenate(plaintext_pages)

This matches the described use of pypdf for block extraction, selective sampling, prompt regeneration under token limits, and retry logic for robustness.

Notes and open questions

Observations:

• Document anchoring is essentially layout-aware retrieval: injecting PDF primitives (text + bboxes) into the VLM prompt. The technique is straightforward to implement but depends heavily on PDF parser quality and block selection policy.
• The benchmark design deliberately avoids LLM-judge evaluators, addressing real reproducibility concerns in document evaluation. The unit-test approach is deterministic but may miss nuanced quality issues that human evaluators would catch.
• The downstream pretraining ablation (+1.3 points at 50B tokens) provides evidence that OCR quality matters for LM training, at least in this experimental setup.

Open questions:

• Robustness beyond English academic PDFs: How do the reported gains generalize to document distributions that are not 56% academic papers, and to multilingual documents (given explicit English-only filtering)?
• Anchoring vs. model quality: How much of the Table 4 advantage comes from model weights vs. prompt engineering (anchoring) vs. post-processing heuristics (rotation, retries, truncation)? Ablations on GPT-4o and Qwen show modest anchoring gains (~1 point), but the fine-tuned model may benefit more.
• Long document failure modes: Sampling only 3 pages per PDF during training may leave gaps in handling complex multi-page structures (cross-references, continued tables, section numbering). Production use cases likely encounter these patterns frequently.
• Structured output overhead: The JSON schema adds tokens and introduces parsing failures (12% retry rate). olmOCR 2’s shift to plain-text generation suggests this was a recognized pain point.

Paper: NovaChart: A Large-scale Dataset towards Chart Understanding and Generation of Multimodal Large Language Models (ACM MM ‘24)
Code: Elucidator-V/NovaChart
Data: ympan/novachart
License: MIT (code), Apache-2.0 (dataset)

TL;DR

NovaChart is a chart-focused instruction-tuning dataset designed to improve MLLMs on both chart understanding and chart generation. The authors report 47K high-resolution chart images and 856K instruction-response pairs, spanning 18 chart types and 15 tasks, backed by per-chart metadata including data points, visual elements, source tables, and rendering code.

What kind of paper is this?

Dominant: $\Psi_{\text{Resource}}$

• Headline contribution is a large-scale dataset + tooling for chart instruction tuning, including metadata designed for scalability.

Secondary: $\Psi_{\text{Method}}$ (data engine/pipeline) + $\Psi_{\text{Evaluation}}$ (task suite + metrics + model comparisons)

Rough superposition:

• $\approx 0.6\,\Psi_{\text{Resource}} + 0.25\,\Psi_{\text{Method}} + 0.15\,\Psi_{\text{Evaluation}}$

What is the motivation?

The authors identify three claimed limitations in existing chart datasets used for training chart-capable MLLMs:

Chart type coverage is narrow and imbalanced

• Many datasets focus on bar/line/pie; long-tail types (histograms, radar, word clouds, etc.) are underrepresented.

Task diversity is restricted

• Prior work often centers on extraction/QA/summarization; practical tasks like conditional extraction, visual element recognition, and chart-type conversion are missing or rare.

Scalability is limited by sparse annotations

• Many datasets provide only data points (sometimes colors), but not source tables and visualization code, which blocks easy task/instruction expansion via LLMs.

What is the novelty?

Dataset scale + coverage

47K chart images, 856K instruction-response pairs

18 chart types (explicitly enumerated in the paper’s overview figures/tables):

• Table (as a “chart type”), single/multi-class line plot, single/multi-class scatter plot, single/multi-class bar plot, univariate/bivariate histogram, correlation heatmap, pie, ring, rose, radar, box, sankey, knowledge graph, word cloud.

15 tasks, grouped into 4 capability buckets:

• Chart Data Understanding: data identification, data comparison, conditional data extraction, data referring
• Chart Visual Understanding: color recognition, style detection, chart classification, visual elements identification/retrieval, text extraction
• Chart Summarization and Analysis: chart pattern recognition, chart analysis, chart summarization
• Chart Generation: chart blueprint, chart type conversion, table-to-chart generation

“Scalable metadata” design

For each chart, NovaChart includes four kinds of annotations:

1. Data points (numeric/statistical values shown)
2. Visual elements (e.g., colors, style choices)
3. Source data (the originating sub-table)
4. Visualization code (Python rendering code; exception noted for knowledge graphs where code is unavailable)

Data generation engine (pipeline novelty)

They emphasize an end-to-end engine that produces: raw tables → curated subtables + stats → styled chart images + code → instruction-response pairs.

What experiments were performed?

Dataset comparisons

Compares NovaChart against ChartQA, PlotQA, Chart-to-text, SimChart9K, UniChart, MMC, ChartLlama.

Key table claims:

• NovaChart: 18 types, 47K images, 856K instruction pairs, 15 tasks, covering understanding + generation.
• Metadata comparison table shows NovaChart is the only listed dataset providing data points + visual elements + source data + visualization code (others lack source data/code).

Fine-tuning + evaluation

Fine-tune 3 open MLLMs:

• LLaVA-v1.5, InternLM-XComposer, Qwen-VL-Chat

Evaluate on an independent evaluation set spanning all 15 tasks.

Metrics (by task type):

• Exact Match (EM) for classification/QA
• RNSS for numerical results in data-referring tasks
• Levenshtein distance + SCRM for multi-point extraction
• GPT-Score for open-ended summarization/analysis and chart generation tasks

Reported outcomes

Across tasks, fine-tuning yields large relative improvements reported as 35.47%–619.47% (range across tasks/models).

Qualitative examples show improvements in:

• Correctly interpreting axes/distributions for analysis
• Producing executable chart conversion code (baseline sometimes outputs “not directly executable”).

What are the outcomes/limitations?

Outcomes the paper emphasizes

• The authors report gains across all 15 tasks after tuning, including generation tasks. Relative improvement ranges are reported as 35.47% to 619.47%, though baseline performance levels vary significantly by task.
• Chart classification and text extraction reportedly reach “excellent” performance in their evaluation setup.
• Improvements are claimed across chart types beyond common bar/line/pie charts, attributed to broader type coverage in the training data. Per-type breakdowns are not provided.

Limitations / open questions

Dataset design and coverage:

• Chart type distribution is not reported; balance across the 18 claimed types is unclear, which may affect model generalization.
• Knowledge graph charts explicitly lack visualization code, breaking the “full metadata” claim for at least one chart type.
• Source data for the 47K charts is not specified; provenance, diversity, and potential biases are unclear.

Evaluation concerns:

• Open-ended task evaluation relies on GPT-Score (LLM judge), introducing model-judge dependence and sensitivity to prompt formulation. No inter-rater reliability or correlation with human judgment is reported.
• The paper acknowledges improvements are “limited” for certain analysis and generation tasks, suggesting the dataset may not sufficiently address those capabilities.
• Accuracy drop magnitude when moving from extracted tables to predicted tables is not quantified for the fine-tuned models.

Reproducibility gaps:

• Critical hyperparameters (fine-tuning learning rates, batch sizes, epochs) are deferred to appendices not included in the provided excerpt.
• Chart-type-specific attribute rules and full metric definitions are also appendix-only.
• Training compute (GPU hours, hardware specs) is not reported.

Reproducibility Details

Model

NovaChart itself is a dataset, but the paper fine-tunes 3 MLLMs: LLaVA-v1.5, InternLM-XComposer, Qwen-VL-Chat. Hyperparameters and fine-tuning details are stated as being in Appendix 2 (not included in the visible content).

Data

Dataset size and structure:

• 47K chart images (high-resolution)
• 856K instruction-response pairs
• Built from:
  • 1.3K raw tables (post-filtering)
  • 28K curated sub-tables (“source data”) used to derive chart statistics

Chart types (18):

From the overview and distribution figure, the chart types include:

• Table
• Single-class and multi-class: line plot, scatter plot, bar plot
• Univariate and bivariate histogram
• Correlation heatmap
• Pie, ring, rose, radar
• Box plot
• Sankey
• Knowledge graph
• Word cloud

Tasks (15):

Grouped as:

• Chart Data Understanding: data identification; data comparison; conditional extraction; data referring
• Chart Visual Understanding: color recognition; style detection; chart classification; visual elements identification/retrieval; text extraction
• Chart Summarization and Analysis: pattern recognition; analysis; summarization
• Chart Generation: blueprint; chart type conversion; table-to-chart generation

Metadata fields (per chart):

The paper’s “chart metadata” is explicitly:

• Data points
• Visual elements
• Source data (sub-table)
• Visualization code (rendering code)

Algorithms / Training (Data Engine)

The paper describes a 4-stage pipeline:

1. Raw Data Acquisition

• Source: Kaggle relational tables with high user votes.
• Filtering/preprocess:
  • Remove non-English tables
  • Remove tables with too few rows (< 50)
  • Remove unnamed columns
  • Remove columns with too many missing values (> 90%)
  • Remove columns with overly long contents (example: movie reviews)

2. Data Curation

• Goal: sample attributes + rows to produce many sub-tables and then compute chart statistics.
• Attribute type schema (used for choosing columns):
  • Numeric: numeric dependent variables (e.g., y-axis)
  • Unique-Numeric: numeric, mostly unique (e.g., year), usable as independent variable (x-axis for lines)
  • Categorical: string with <= 5 unique values (class labels for multi-class charts)
  • Enumerable: <= 25 unique values (qualitative variable for bar/pie, etc.)
• Attribute classification model:
  • Uses GPT-turbo-3.5 with in-context learning; prompt includes the attribute-type definitions, number of unique values, example values, and an instruction to classify the attribute.
• Sub-table sampling:
  • For each chart type: randomly select the required attribute types and sample 30–50 rows from the raw table
  • If a raw table cannot yield a sub-table meeting requirements, skip it
  • Compute chart statistics from the sampled sub-table to derive chart data points

3. Image Styling and Visualization

• Rendering libraries mentioned: Matplotlib, Seaborn, Pyecharts.
• Visual diversity: randomize visual elements (example: colors, shadow, style) and render multiple images per data-point instance.
• The paper states this stage yields ~40K charts initially, then expanded to 47K after later extensions.

4. Instruction Formulation

• Task design: 15 tasks, including “new” ones the authors emphasize (conditional extraction; some visual element checks like fitting-curve existence in histograms; chart-to-chart conversions).
• LLM usage:
  • For tasks with answers obtainable from metadata: use GPT-4 to generate diverse instruction templates and convert to instruction-following format.
  • For more open-ended tasks (summarization/analysis): use GPT-turbo-3.5, providing prompt with task instruction, in-context demos, and the relevant chart metadata; the model output becomes the response.

Data expansion steps:

• Add line-chart source data from Statista due to shortage of suitable line-chart subtables from Kaggle.
• Add HTML/CSS-rendered tables as an additional “chart” type.
• Manually collect some chart types (explicitly mentioned: knowledge graphs and word clouds); knowledge-graph visualization code unavailable.

Evaluation

Metrics and protocols:

• EM for classification/QA tasks, RNSS for numeric data-referring, Levenshtein + SCRM for multi-point extraction, GPT-Score for open-ended and generation tasks.
• Comparisons shown in figures:
  • Per-task spider/radar charts before vs after tuning for the three models
  • Per-chart-type performance plots on selected tasks (example shown for InternLM-XComposer).

Reported improvement magnitude:

• Across tasks: 35.47%–619.47% relative improvements after fine-tuning (paper-reported range).

Hardware / Production

The main text does not list GPU types, wall-clock training time, or serving throughput; it points to appendices for fine-tuning details.

Data Availability

Is all the data publicly shared?

They claim the dataset is publicly available, and they point to a public GitHub repo as the distribution entry point.

That said, “all the data” depends on what you mean:

Released (public):

• GitHub repo (code + toolkit; also points to where to download the dataset): Elucidator-V/NovaChart
• Hugging Face dataset linked from the GitHub repo as the download location for the “full NovaChart dataset”: ympan/novachart

Potentially not fully released / ambiguous:

• The paper emphasizes rich chart metadata (data points, visual elements, source data, visualization code).
• But a public GitHub issue reports that only the instruction-tuning JSONL files were found and asks whether metadata will be released, suggesting at least some parts may be missing or not obvious in the current release.

Underlying raw sources are likely not fully redistributable as-is:

• Their pipeline uses Kaggle tables and additionally Statista tables for line charts.
• The paper doesn’t specify whether the original Kaggle/Statista tables are redistributed (and those sources typically have their own licenses/ToS).

Where is it shared?

• GitHub: Elucidator-V/NovaChart (paper and repo both point here)
• Hugging Face (dataset download): ympan/novachart — contains a ~10 GB novachartv1.rar file

What license?

• GitHub repository license: marked as MIT on the repo page (this generally covers the code and repo contents, unless the repo says otherwise)
• Hugging Face dataset license: listed as Apache-2.0 on the dataset page/README metadata
• Paper itself: does not appear to state a dataset license in the main text; it only states availability and links

GOT-OCR2.0 — Notes

• Paper: arXiv:2409.01704v1
• Code: Ucas-HaoranWei/GOT-OCR2.0
• Models: stepfun-ai/GOT-OCR2_0

TL;DR

GOT-OCR2.0 is a unified 580M parameter encoder-decoder OCR model that treats diverse optical signals (plain text, formulas, tables, charts, sheet music, geometric shapes) as a single “character” space. The system introduces interactive region OCR via box/color prompts, dynamic resolution through multi-crop tiling, and multi-page OCR via decoder post-training. On the Fox benchmark for dense document OCR, GOT achieves edit distance 0.035 (EN) and 0.038 (ZH) with F1 scores of 0.972 and 0.980 respectively. The model supports only English and Chinese with high quality.

What kind of paper is this?

Dominant: $\Psi_{\text{Method}}$ (new unified model architecture + three-stage training recipe + task unification framework).

Secondary: $\Psi_{\text{Resource}}$ (extensive synthetic data engines for formulas, molecules, tables, sheet music, geometry, and charts; open-source code and model release).

The paper introduces “General OCR Theory” (OCR-2.0) as a conceptual framework for treating diverse optical signals as characters, with a simple encoder-decoder architecture trained via multi-stage optimization. The resource component is substantial: the authors describe detailed synthetic data engines totaling $>$10M image-text pairs across stages, with open-source code and Apache 2.0 model weights provided.

What is the motivation?

• OCR-1.0 pipeline brittleness and cost: Traditional multi-module systems (detection/cropping/recognition) are prone to local optima, cascading errors, and high maintenance overhead.
• Task fragmentation: Different specialized models exist for text detection, scene OCR, formula recognition, table extraction, etc. Practitioners must choose among many models for different subtasks, increasing deployment complexity.
• Broadened “OCR” demand: The authors frame the target as “intelligent processing of man-made optical signals” beyond plain text—including formulas, tables, charts, sheet music, molecular structures, and geometric diagrams.
• Limited multilingual support in end-to-end systems: Prior unified OCR models often focus narrowly on English or require separate models for different languages.

What is the novelty?

General OCR Theory (“OCR-2.0”) framing: Treats diverse optical signals (plain text, mathematical notation, tables, charts, sheet music, molecular structures, geometric shapes) as “characters” in a unified space, aiming for a single end-to-end model that handles multiple OCR tasks.

Simple encoder-decoder OCR architecture: Vision encoder (VitDet base with local attention, $\sim$80M params) + linear connector + language decoder (Qwen-0.5B, total 580M params). The encoder compresses $1024 \times 1024$ images to $256 \times 1024$ tokens, with the decoder supporting up to 8K context length.

Feature extensions via decoder post-training: Fine-grained region OCR (box and color prompts), dynamic resolution (multi-crop sliding window with max 12 tiles following InternVL-1.5), and multi-page OCR are added in Stage 3 without modifying the vision encoder. This preserves encoder pretraining while expanding capabilities.

Staged training strategy with data mixing: Three-stage pipeline (encoder pretrain with tiny OPT-125M, joint training with Qwen-0.5B on formatted/general OCR, decoder post-train for features) with 80% mixing of previous stage data to reduce regression.

Extensive synthetic data engines: Stage 1 uses 5M pure text pairs from LAION/Wukong/PDFs; Stage 2 adds 1M formulas (arXiv LaTeX $\rightarrow$ Mathpix format, $>$20$\times$ faster rendering), 1M molecules (ChEMBL SMILES), 0.3M tables, 1.2M full-page formatted docs, 0.5M sheet music (GrandStaff + Verovio), 1M geometry (TikZ), and 2M charts (Matplotlib/Pyecharts); Stage 3 constructs 60w fine-grained, 50w multi-crop, and 20w multi-page samples.

What experiments were performed?

The authors evaluate across five OCR task families:

1. Plain document OCR: Fox benchmark (dense multi-page documents); word-level segmentation; metrics include edit distance, F1, precision, recall, BLEU, METEOR. Compares GOT (580M) against larger LVLMs.

2. Scene text OCR: 400 natural scene images (200 English, 200 Chinese) with manually corrected ground truth; character-level segmentation.

3. Formatted document OCR: 90 pages (English + Chinese) with Mathpix pseudo-labels manually corrected; evaluates single-scale vs multi-crop inference on formula and table metrics.

4. Fine-grained OCR: Box-guided and color-guided referential OCR evaluation in English and Chinese; compares GOT to Fox on region extraction accuracy.

5. General OCR: Chart OCR using ChartQA (structure-extraction version) and PlotQA benchmarks; reports AP@strict/slight/high for chart structure accuracy.

Test data filtering: The authors apply “strict text similarity filtering” to reduce overlap between training and test text, though details are not provided.

What are the outcomes/limitations?

Outcomes

Plain document OCR (Fox benchmark):

GOT achieves very strong dense document OCR metrics. The paper reports GOT ranks favorably against larger LVLMs in their comparison table, though specific baseline names and parameter sizes are included in the paper but numeric results are only provided for GOT.

Scene OCR (400 images):

Scene OCR edit distance is higher than document OCR (0.112 EN vs 0.035), reflecting the increased difficulty of natural scene text with perspective distortion, occlusion, and varied fonts.

Formatted document OCR (90 pages, single vs multi-crop):

Multi-crop inference provides substantial gains for formula recognition (+11.6 F1 points) and table extraction (+5.1 METEOR points), demonstrating the effectiveness of dynamic resolution for high-detail regions.

Chart OCR:

GOT shows stronger performance on ChartQA (0.747 strict AP) compared to PlotQA (0.133 strict AP), suggesting better generalization to the ChartQA structure extraction task.

Fine-grained OCR: The authors report GOT outperforms Fox on both box-guided and color-guided referential OCR in English and Chinese (Table 4 in paper), though specific numeric metrics are not transcribed in the rough draft.

Limitations and open questions

Language coverage: The authors explicitly state they “mainly support English and Chinese” and “cannot guarantee OCR quality for other languages” even if some appear in crawled PDFs. Scaling to dozens of languages (Arabic, Cyrillic, Indic scripts, etc.) is not addressed. The single multilingual model design may face capacity constraints beyond 2-3 languages.

Geometry scope: The authors describe geometry rendering as “preliminary” and state the model “can only recognize basic geometry at present.” Complex geometric diagrams, 3D figures, and advanced TikZ constructs are likely beyond current capabilities.

Ablation gaps: The paper does not ablate the contribution of individual components:

• How much gain comes from VitDet local attention vs other encoders?
• What is the impact of the 80% data mixing strategy vs training on new data only?
• How much do synthetic data engines contribute vs real data?

No throughput or latency data: Training hardware is reported (64 L40s), but the paper omits inference latency (ms/page), throughput (pages/s), and cost-per-page estimates. This limits comparison to olmOCR ($176/M pages on L40S) or MinerU2.5 (1.224 pages/s on A100).

Test set construction and leakage: The authors mention “strict text similarity filtering” to reduce train/test overlap but do not specify the threshold, filtering method, or amount of data removed. External validation on established benchmarks beyond Fox and ChartQA is limited.

Multi-page OCR evaluation: The paper describes 20w multi-page training samples (2-8 pages each, <8K tokens total) but does not evaluate multi-page OCR performance separately. It is unclear how well page-breaking and cross-page references are handled compared to single-page inference.

Comparison baseline details: The paper compares GOT to “larger LVLMs” on Fox but omits model names, parameter sizes, and numeric results for baselines in the extracted text. This limits reproducibility of the comparison.

Model

Architecture

High-level design: Vision encoder + linear connector + language decoder.

Vision encoder:

• Backbone: VitDet (base variant), chosen for local attention mechanism to reduce compute on high-resolution images.
• Parameters: $\sim$80M.
• Tokenization: Final layers compress $1024 \times 1024 \times 3$ input image to $256 \times 1024$ image tokens.
• Projection: Linear layer ($1024 \times 768$) projects image tokens into language model dimension during encoder pretraining (described in Stage 1).

Language decoder:

• Stage 1 (encoder pretrain): OPT-125M serves as a “tiny decoder” to efficiently pass gradients to the encoder.
• Stage 2-3 (main model): Qwen-0.5B replaces OPT-125M for broader OCR-2.0 training.
• Context length: Supports up to 8K tokens (increased from 4K in Stage 1 to 6K in Stage 2 to 8K in Stage 3).

Total parameters: 580M (encoder ~80M + decoder ~500M).

Supported inputs and outputs

Inputs:

• Scene and document images (single-page or multi-page).
• Fine-grained region specification via bounding box coordinates (normalized $\times 1000$) or color-coded frames (red/green/blue).
• Multi-crop tiling for ultra-high-resolution documents (max 12 tiles, $1024 \times 1024$ per tile, InternVL-1.5 cropping strategy).

Outputs:

• Plain text: Character sequences for scene and document OCR.
• Formatted outputs: Mathpix-markdown (formulas, tables), TikZ (geometry), SMILES (molecules), custom notation (sheet music, charts) via prompting.

Aspect ratio handling: Input images of various shapes are resized to $1024 \times 1024$ via a “compromise” strategy (details not specified in paper).

Design rationale

VitDet local attention: Reduces computational cost on high-resolution images compared to global attention mechanisms. The authors note this is critical for $1024 \times 1024$ inputs.

High compression ratio: $1024 \times 1024$ image $\rightarrow$ $256 \times 1024$ tokens represents $\sim$16$\times$ spatial compression (256 tokens for $1024^2$ pixels), enabling efficient processing of dense document pages.

Decoder post-training for features: Stage 3 freezes the vision encoder and only trains the decoder on fine-grained, multi-crop, and multi-page data. This design preserves encoder pretraining while adding capabilities, reducing compute cost vs end-to-end retraining.

Contrast to olmOCR2: GOT uses VitDet with local attention for high-resolution processing, while olmOCR2 employs SigLIP with global attention and dynamic tiling. GOT compresses to 256 tokens per image vs olmOCR2’s variable token count (up to 1984 tokens for high-detail regions).

Contrast to Nougat: Nougat uses Swin Transformer encoder with academic document specialization, while GOT adopts VitDet for broader “OCR-2.0” coverage including charts, sheet music, and geometry. GOT’s 8K context supports multi-page OCR, while Nougat processes single pages.

Data

Stage 1: Pure text recognition (encoder pretraining)

Total: $\sim$5M image-text pairs (3M scene OCR + 2M document OCR).

Scene OCR sources (3M):

• Image sources: LAION (English), Wukong (Chinese).
• Pseudo ground truth: PaddleOCR extracts text from images.
• Processing:
  1. Remove bounding boxes and concatenate text top-to-bottom, left-to-right.
  2. Crop text regions into slices, yielding additional $\sim$1M slice pairs.

Document OCR sources (2M):

• PDFs: Collected from Common Crawl.
• Text extraction: Fitz library extracts text.
• Outputs: $\sim$1.2M full-page pairs + 0.8M line/paragraph slice data.

Preprocessing: Images of various aspect ratios are resized to $1024 \times 1024$ via a “compromise” strategy (details not specified).

Stage 2: Formatted OCR + general characters

Formatted data sources:

Formulas (1M):

• Source: arXiv LaTeX files.
• Processing: Extract formula fragments, convert to Mathpix-markdown format.
• Rendering: Mathpix-markdown-it library (HTML $\rightarrow$ SVG $\rightarrow$ PNG), claimed $>$20$\times$ faster than LaTeX rendering.

Molecules (1M):

• Source: ChEMBL_25 dataset with 2M SMILES strings.
• Processing: Produce $\sim$1M molecular structure image-text pairs.
• Rendering: Mathpix-markdown-it + rdkit.Chem library.

Tables (0.3M):

• Source: LaTeX table code.
• Rendering: LaTeX rendering (authors prefer this over Mathpix-markdown-it for tables).

Full-page formatted documents (1.2M):

• English: $\sim$0.5M markdown PDF-text pairs via Nougat method, converted to Mathpix format.
• Chinese: $\sim$0.5M markdown pairs via Vary-style processing, converted to Mathpix format.
• In-house labeled: $\sim$0.2M Mathpix-labeled data from books, papers, and financial reports.

General OCR sources:

Sheet music (0.5M):

• Source: GrandStaff dataset.
• Additional rendering: Verovio library.
• Total: $\sim$0.5M samples after rendering.

Geometry (1M):

• Source: TikZ text outputs.
• Total: $\sim$1M geometric TikZ data.

Charts (2M):

• Rendering: Matplotlib and Pyecharts libraries.
• Composition: 1M Matplotlib + 1M Pyecharts chart image-text pairs.

Stage 3: Feature-specific data engines

Fine-grained OCR (60w samples):

• Scene sources: RCTW, ReCTS, ShopSign, COCO-Text for natural fine-grained OCR.
• Document sources: Parse PDFs (filter scanned format), use Fitz/PDFminer to record page images, line/paragraph boxes, and text.
• Coordinate normalization: Bounding boxes normalized then multiplied by 1000.
• Color-guided variant: Red/green/blue frame colors drawn on image to specify region.

Ultra-large images via multi-crop (50w pairs):

• Tiling strategy: Sliding window $1024 \times 1024$, InternVL-1.5 cropping method, max 12 tiles.
• Synthetic data: Horizontal and vertical stitching of single-page PDFs to create ultra-large documents.

Multi-page OCR (20w pairs):

• Motivation: Page-breaking is difficult for some formatted PDFs (e.g., arXiv LaTeX); training on multi-page pairs directly improves cross-page handling.
• Construction: Sample 2-8 pages; each page <650 tokens; ensure total length <8K context.
• Language mixing: Often mixes Chinese and English pages in single multi-page samples.

Data quality and leakage control

Pseudo-labeling quality: Stage 1 uses PaddleOCR for pseudo ground truth. Stage 2 formatted data uses Mathpix-markdown-it rendering and manual correction for evaluation sets. The quality of pseudo-labels is not quantified.

Test set filtering: The authors apply “strict text similarity filtering” to reduce overlap between training and test text, but do not specify the threshold, filtering method, or amount of data removed.

Language coverage: Training data primarily covers English and Chinese. The authors state they cannot guarantee OCR quality for other languages even if some appear in crawled PDFs.

Algorithms / Training

Three-stage training strategy

Stage 1: Encoder pretraining (pure text recognition)

• Objective: Pretrain vision encoder using tiny OPT-125M decoder for efficient gradient propagation.
• Data: 5M pure text pairs (scene + document OCR).
• Optimization: Global batch size 128; 3 epochs; AdamW optimizer; cosine annealing schedule; starting LR $1e{-}4$; max token length 4096.

Stage 2: Joint training (formatted + general OCR)

• Objective: Connect encoder to Qwen-0.5B and train on broader OCR-2.0 “characters” (formulas, charts, geometry, sheet music, molecules, tables).
• Data: Stage 2 formatted/general data (1M formulas, 1M molecules, 0.3M tables, 1.2M full-page, 0.5M sheet music, 1M geometry, 2M charts) + 80% of Stage 1 data (4M pure text samples).
• Optimization: 1 epoch; max token length 6000; same optimizer settings as Stage 1 (AdamW, cosine annealing, LR $1e{-}4$).

Stage 3: Decoder post-training (feature extensions)

• Objective: Freeze vision encoder; train decoder on fine-grained, multi-crop, and multi-page OCR capabilities.
• Data: 60w fine-grained samples + 50w multi-crop pairs + 20w multi-page pairs + 80% of Stage 1-2 data.
• Optimization: 1 epoch; max token length 8192; starting LR $2e{-}5$ (lower than Stage 1-2).

Data mixing strategy

80% mixing across stages: During each stage, the authors sample 80% of previous stage(s) data to reduce catastrophic forgetting when adding new features or data types. For example:

• Stage 2: 80% of Stage 1 pure text data (4M samples) + 100% of Stage 2 formatted/general data.
• Stage 3: 80% of Stage 1-2 data + 100% of Stage 3 feature-specific data.

Hardware and compute

Training hardware: 64 L40s GPUs (described as “8$\times$8 L40s”).

Training time: Not reported in the paper.

Throughput: Not reported in the paper.

Evaluation

Plain document OCR (Fox benchmark)

The Fox benchmark evaluates dense multi-page document OCR with word-level segmentation. Metrics include edit distance (character-level), F1/precision/recall (word-level), BLEU, and METEOR.

The paper includes a comparison table with larger LVLMs but the rough draft does not transcribe baseline names or numeric results. GOT is reported to achieve competitive or superior performance despite its smaller size (580M).

Scene text OCR (400 images)

A custom evaluation set of 400 natural scene images (200 English, 200 Chinese) with manually corrected ground truth. Character-level segmentation is used for metrics.

Scene OCR edit distance is 3$\times$ higher than Fox document OCR (0.112 vs 0.035 for English), reflecting the increased difficulty of natural scene text with perspective distortion, occlusion, complex backgrounds, and varied fonts.

Formatted document OCR (90 pages)

A custom evaluation set of 90 pages (English + Chinese) with Mathpix pseudo-labels manually corrected. Compares single-scale ($1024 \times 1024$) vs multi-crop inference.

Multi-crop inference provides +11.6 formula F1 points and +5.1 table METEOR points. This demonstrates the effectiveness of dynamic resolution for high-detail regions (small fonts, dense tables, complex formulas).

Fine-grained OCR (box- and color-guided)

The paper evaluates referential OCR using bounding box coordinates and color-coded frames in English and Chinese. Results are compared to Fox.

The rough draft notes “GOT is reported as better than Fox on both region and color referential OCR in both languages (see Table 4)” but does not transcribe specific numeric metrics.

Chart OCR (ChartQA-SE and PlotQA-SE)

Structure extraction variants of ChartQA and PlotQA benchmarks. Metrics are Average Precision at strict/slight/high thresholds.

GOT shows substantially stronger performance on ChartQA (0.747 strict AP) compared to PlotQA (0.133 strict AP). This gap suggests dataset-specific generalization—ChartQA may have chart types or structure patterns more similar to GOT’s training data (2M Matplotlib/Pyecharts charts).

Metric definitions

Edit distance: Levenshtein distance at character level, normalized by ground-truth length. Lower is better.

F1/Precision/Recall: Word-level (Fox document OCR) or character-level (scene OCR) token matching. Higher is better.

BLEU: Bilingual Evaluation Understudy score, n-gram overlap between prediction and reference. Higher is better.

METEOR: Metric for Evaluation of Translation with Explicit ORdering, incorporates synonyms and paraphrasing. Higher is better.

AP@strict/slight/high: Average Precision at different IoU or tolerance thresholds for chart structure extraction. Higher is better.

Hardware / Production

Training infrastructure

Hardware: 64 L40s GPUs (described as “8$\times$8 L40s” setup).

Training time: NR (not reported in the paper).

Throughput: NR (pages/second or tokens/second not reported).

Cost: NR (no cost estimates provided).

Inference performance

Latency: NR (milliseconds per page not reported).

Throughput: NR (pages/second not reported).

Cost per page: NR (no cost estimates provided for inference).

Hardware requirements: The paper does not specify minimum GPU memory or recommended inference hardware.

Deployment

Model availability: Open-source weights released on HuggingFace (stepfun-ai/GOT-OCR2_0).

License: Apache 2.0.

Code: GitHub repository (Ucas-HaoranWei/GOT-OCR2.0) provides inference scripts and examples.

Serving infrastructure: The paper does not describe production serving infrastructure, batching strategies, or optimization techniques for deployment.

Notes and open questions

Observations:

• GOT’s “OCR-2.0” framing is ambitious—treating formulas, charts, sheet music, geometry, and molecules as “characters” in a unified space. The single-model approach is simpler than multi-module pipelines, but the capacity constraints of a 580M model may limit depth in each task.
• The three-stage training strategy with 80% data mixing is a practical approach to reducing catastrophic forgetting. However, the paper does not ablate the mixing ratio or compare to alternative continual learning strategies.
• Multi-crop inference provides substantial gains for formatted documents (+11.6 formula F1, +5.1 table METEOR), validating the dynamic resolution design. The max 12 tiles limit balances detail vs context length constraints (8K tokens).
• Scene OCR edit distance is 3$\times$ higher than document OCR (0.112 vs 0.035 for English), suggesting GOT is optimized primarily for document-style inputs. Natural scene text with perspective distortion and complex backgrounds remains challenging.

Open questions:

• Ablation studies: What is the contribution of VitDet local attention vs other encoders (e.g., SigLIP, Swin)? How much gain comes from the 80% data mixing strategy vs training on new data only? How do synthetic data engines compare to real data?
• Geometry and sheet music quality: The authors state geometry recognition is “preliminary” and limited to “basic geometry.” How much of the 1M geometric TikZ training data does the model successfully learn? What is the error rate on GrandStaff sheet music notation?
• Multilingual scaling: GOT supports only English and Chinese with high quality. How would the 580M parameter budget scale to dozens of languages (Arabic, Cyrillic, Indic scripts, etc.)? Would capacity constraints require larger models or ensemble architectures?
• Inference efficiency: No latency or throughput data is reported. How does GOT compare to olmOCR ($176/M pages on L40S) or MinerU2.5 (1.224 pages/s on A100) in cost and speed?
• Test set leakage: The “strict text similarity filtering” method is not detailed. What threshold is used? How much test data is removed? External validation on held-out benchmarks (e.g., olmOCR-Bench, DocVQA) would strengthen confidence in generalization.
• Multi-page OCR evaluation: The paper describes 20w multi-page training samples but does not evaluate multi-page performance separately. How well does GOT handle page-breaking, cross-page references, and consistent formatting across pages compared to single-page inference?
• Comparison baseline details: The Fox benchmark comparison table in the paper includes larger LVLMs, but the rough draft omits model names, parameter sizes, and numeric results. Access to the full table would enable better positioning of GOT’s performance.

Paper: Kim et al., SIMPLOT: Enhancing Chart Question Answering by Distilling Essentials (arXiv:2405.00021v3, NAACL 2025 Findings)
Code: sangwu99/Simplot
License: No explicit license stated in the paper

TL;DR

SIMPLOT improves chart-to-table extraction (and downstream chart QA) by (1) training on essential-only “simple charts” to avoid chart noise, (2) distilling an encoder so original charts map into the simple-chart representation space, (3) adding row/column rendering supervision, and (4) prompting an LMM with a human-oriented chart instruction so it uses both the extracted table + original image at inference. Reported gains include better table extraction on ChartQA (RDF1) and higher QA relaxed accuracy than Deplot in their table-plus-LMM setting.

What kind of paper is this?

Dominant: $\Psi_{\text{Method}}$

• Main contribution is an end-to-end method/training recipe (preprocess + 2-phase training + inference prompting) to improve chart-to-table extraction and chart QA.

Secondary: $\Psi_{\text{Evaluation}}$

• Proposes a modified metric (Relative Distance, RD) derived from RMS to better evaluate chart-to-table extraction when header strings differ but reasoning is unaffected.

What is the motivation?

• Prior “chart-to-table then LLM reasoning” approaches (e.g., Deplot) struggle because charts mix essential and irrelevant information, and this “noise” degrades table extraction (examples include missed units and inclusion of irrelevant text like attribution).
• Table-only reasoning also misses purely visual questions that require chart geometry or visual attributes (e.g., “third bar from the top”, “what year does the orange line represent?”).

What is the novelty?

1) Essential-only training signal via “simple chart generation”

• Uses existing datasets’ chart-to-CSV table annotations to re-plot a simplified chart (Matplotlib) from the ground-truth table, excluding irrelevant chart artifacts; training on these “simple charts” improves extraction.

2) Row-column rendering (explicit supervision)

• Renders row/column header info onto the image during training (since ground-truth table exists).
• For inference, since ground-truth is unavailable, an LMM is used to extract row/column text from the chart and render it onto the image (Appendix B with an illustrative example).

3) Two-phase distillation to map original charts into the “simple chart” representation space

• Phase 1: train a teacher encoder + table decoder using essential-only (simple chart) pairs.
• Phase 2: train a student encoder on original charts with a triplet loss (anchor = original chart; positive = simple chart; negative = value-shuffled simple chart) plus a table cross-entropy loss, to generate accurate tables from original charts.

4) Human-oriented chart instruction prompt (for LMM reasoning using image + table)

• A chart-specific prompt that encodes stepwise “how humans interpret charts” (universal rules + chart-type-specific rules for bar/line/pie). It is used so the LMM can resolve questions that require aligning the predicted table with chart positions and visual cues.

What experiments were performed?

Benchmarks / tasks

• ChartQA: pie/bar/line charts; QA includes human-authored + LLM-augmented questions; evaluation reports Relaxed Accuracy on 2,500 test questions.
• PlotQA: dot line/line/bar; large synthetic QA set; they sample 10% of images for training/inference to reduce cost.
• Additional evaluations discussed: OpenCQA (open-ended chart QA; BLEU) and MultiChartQA (comparative/sequential multi-chart reasoning).

Comparisons / baselines (high level)

• Vision-language pretraining baselines and supervised chart models are compared for ChartQA QA. Table-based reasoning baselines include Deplot and Unichart (table extraction variant), with GPT-4V used for reasoning in the “extract table then LMM” setting for fairness.

Key ablations

• Impact of (a) row-column rendering and (b) distillation from simple charts on table extraction, plus (c) the human-oriented prompt on QA.
• Effect of using image + table vs table-only for Unichart/Deplot reasoning and for failure cases where both methods have low table extraction quality.
• Negative-sample (triplet) benefit is evaluated (small but positive).

What are the outcomes and limitations?

Outcomes (reported)

• ChartQA table extraction (RDF1): SIMPLOT reported higher overall RDF1 than Deplot and UniChart in their setup.
• ChartQA QA (RA): In the “table + LMM” category, SIMPLOT reports substantially higher overall relaxed accuracy than Deplot in their table-plus-LMM comparison table.
• Ablations: row/column rendering and simple-chart distillation each improve table extraction; adding the human-oriented prompt improves QA.
• Harder questions: when questions require referencing multiple rows/columns, SIMPLOT’s reported gap vs a “Deplot + image + prompt” baseline increases, supporting the claim that table extraction quality matters more under harder compositional queries.

Limitations / failure modes (reported)

• Expected reduced performance on unseen OOD charts; they argue one round of table-extraction training can adapt to OOD better than per-task training, but still flag OOD as future work.
• Practical error cases: row/column extraction can split entities due to line breaks; LMM can fail at fine-grained color identification (example: confusing orange vs red).
• Misuse risk: high-performing chart interpretation could be used to mislead; authors recommend critical verification.

Model

Backbone and components

• Built on Deplot-style chart-to-table: an image encoder and a text/table decoder (they follow Deplot and use ViT encoders).
• Naming: chart encoder $E_{\text{chart}}$ and table decoder $D_{\text{table}}$; Phase 1 produces a teacher encoder $E^{\text{teacher}}{\text{chart}}$; Phase 2 trains a student encoder $E^{\text{student}}{\text{chart}}$.

Tokenization / table linearization details

• Table sequence uses | to separate cells and <0x0A> as line break, following Deplot.

Data

Training inputs created by preprocessing

• Simple charts: generated by re-plotting from ground-truth CSV tables using Matplotlib; intended to contain only “essential” information needed for reasoning.
• Row-column rendering: in training, rows/columns from the ground-truth table are rendered onto the paired chart image; in inference, an LMM extracts the row/column strings for rendering.

Dataset stats (as reported)

• ChartQA split counts are provided (train/val/test counts by chart type; 2,500 QA pairs in test).
• PlotQA is very large; they sample 10% of PlotQA images stratified by type for training/inference, and use one QA pair per image in their reduced setting.

Data release

• Not explicitly released. The authors do not describe releasing a new SIMPLOT dataset or any additional data artifacts; instead, they use existing public datasets (ChartQA and PlotQA) and note those are “publicly accessible for research purposes.”
• They describe generating “simple charts” offline from the datasets’ existing CSV tables, which suggests their pipeline can produce derived artifacts, but they don’t state they publish those derived data outputs.

Algorithms / Training

Preprocessing stage (offline)

1. Simple Chart Generation: re-render chart from CSV table using Matplotlib.
2. Row-Column Rendering: render row/column header strings onto images; inference uses an LMM to extract these headers.

Training stage: chart-to-table extraction

Phase 1 (teacher): fine-tune Deplot backbone on (simple chart, table) pairs.

• Output: teacher encoder representations focus on essential info; $D_{\text{table}}$ and an FC layer are later frozen.

Phase 2 (student): train $E^{\text{student}}_{\text{chart}}$ on original charts to match teacher-space outputs.

• Define original chart $A$, simplified positive $P$, negative $N$ (shuffled values).
• Representations:
  • $z_a = D_{\text{table}}(E^{\text{student}}_{\text{chart}}(A))$
  • $z_p = D_{\text{table}}(E^{\text{teacher}}_{\text{chart}}(P))$
  • $z_n = D_{\text{table}}(E^{\text{teacher}}_{\text{chart}}(N))$
• Triplet loss: $$ L_{\text{triplet}}(A,P,N)=\max{d(z_a,z_p)-d(z_a,z_n)+m,0} $$ with $d$ as $\ell_2$ distance and margin $m$.
• Table generation cross-entropy $L_{\text{table}}$ over the linearized table tokens, and final loss: $$ L_{\text{final}}=\lambda L_{\text{triplet}}+(1-\lambda)L_{\text{table}} $$ with $\lambda=0.1$ reported.

Inference stage: reasoning

• LMM receives (original chart image + extracted table), plus the human-oriented chart instruction prompt so it can answer both numeric table queries and visual-attribute queries.

Evaluation

Metrics

• Chart QA: Relaxed Accuracy (RA) on 2,500 ChartQA test questions.
• Chart-to-table extraction: they discuss RMS and introduce Relative Distance (RD) with RDF1 as the harmonic mean of precision/recall using numeric relative distance after minimal-cost matching over header strings.
• Open-ended QA: BLEU for OpenCQA comparisons.

Notable reported result patterns (qualitative)

• Table-based reasoning (table + LMM) generally outperforms image-only supervised models on ChartQA in their comparison, and SIMPLOT is reported best within their table-based group.
• Adding the image at inference helps table-based methods answer questions that need positional/visual interpretation, and the prompt further improves this.

Hardware / Production

• GPU: NVIDIA RTX A6000.
• Training time per epoch: Phase 1 about 2 hours/epoch, Phase 2 about 4 hours/epoch; they train Phase 1 for 7 epochs and Phase 2 for 9 epochs.
• Parameter count: 374M parameters (SIMPLOT model); GPT-4 parameter count is unknown (as noted).
• LMM prompting: temperature 0.1 is mentioned.

Paper: TinyChart: Efficient Chart Understanding with Visual Token Merging and Program-of-Thoughts Learning
Code: X-PLUG/mPLUG-DocOwl
Data: mPLUG/TinyChartData
License: Apache-2.0

TL;DR

TinyChart is a 3B-parameter multimodal LLM for chart understanding that targets efficiency via (1) Visual Token Merging inside the vision transformer to reduce high-res visual sequence length, and (2) Program-of-Thoughts (PoT) learning to offload numeric reasoning into executable Python programs. Authors report strong results on ChartQA and other chart benchmarks, plus higher inference throughput than larger (7B–13B) chart-focused MLLMs.

What kind of paper is this?

• Dominant: $\Psi_{\text{Method}}$
  • Introduces an end-to-end model recipe combining visual token merging in ViT + PoT learning and validates via ablations and benchmark comparisons.
• Secondary: $\Psi_{\text{Resource}}$
  • Builds ChartQA-PoT (PoT-augmented supervision) and describes its construction and statistics.
• Secondary: $\Psi_{\text{Evaluation}}$ (light)
  • Emphasizes efficiency metrics (throughput, OOM behavior at high resolution) alongside accuracy.

What is the motivation?

Chart understanding requires: (a) robust OCR/text retrieval, (b) numerical reasoning, and (c) high-resolution vision encoding. Authors identify three constraints of existing chart MLLMs:

1. Large parameter counts hinder deployment/training (13B chart models need substantial VRAM).
2. Numerical errors are common for calculation-heavy questions.
3. High-resolution images produce long ViT token sequences that are expensive for the LLM to process.

What is the novelty?

Visual Token Merging inside the ViT

• Observation: Charts contain large regions of blank space / uniform color blocks, yielding many similar patches.
• Mechanism: In each transformer layer, tokens are split into two disjoint sets, then bipartite matching selects the top-$r$ most similar token pairs across the sets (similarity = cosine distance between self-attention key vectors), and merges each paired feature by average pooling.
• Not limited to spatial neighbors: non-adjacent tokens can merge if similar and in opposite sets.
• Proportional attention fix: Since merging reduces multiplicity of a feature, they add a patch-count term $s$ into attention:

$$ \text{Attention}=\text{softmax}\left(\frac{QK^\top}{\sqrt{d}}+\log s\right)V $$

where $s$ is the number of original patches represented by the merged token.

Program-of-Thoughts learning for numeric chart QA

• Instead of learning arithmetic implicitly, model is trained to output executable Python assignment statements (with comments), then a Python interpreter produces the final numeric answer.
• Authors construct ChartQA-PoT from ChartQA training split using two pipelines:
  • Template-based PoT: 40 question templates (from PlotQA) + manually written numpy-style code templates; placeholders filled from chart tables, then rule-based filtering removes bad fills.
  • GPT-based PoT: gpt-3.5-turbo generates PoT code from (question + chart data table text); they execute generated code and keep only samples where execution succeeds and matches the annotated ChartQA answer.

What experiments were performed?

• Benchmarks: ChartQA, Chart-to-Text (Pew), Chart-to-Table, OpenCQA; plus ChartX cognition generalization.
• ChartQA evaluation settings:
  • Direct: short answer.
  • PoT: emit Python program; interpreter gives answer.
  • Combine (default): keyword rule decides if question is “calculative”; PoT for calculative, Direct otherwise; fallback to Direct if program has syntax errors.
  • Oracle: choose the correct one between Direct and PoT after the fact (upper bound).
• Ablations:
  • With/without PoT data; template-only vs template+GPT PoT.
  • Resolution changes ($384 \rightarrow 512 \rightarrow 768$) with/without token merging; varying merge rate $r$.
  • Tracks visual sequence length and inference throughput as efficiency metrics.
• Qualitative case studies: QA, chart-to-table, chart-to-text, and chart redrawing; includes error examples.

What are the outcomes/limitations?

Outcomes (as reported)

• TinyChart@512 and TinyChart@768 (3B) are reported to outperform or match larger chart MLLMs on several benchmarks, while achieving higher throughput.
• ChartQA setting breakdown: Combine improves over Direct; Oracle is notably higher than Combine, implying combination policy is not optimal.
• Calculative vs non-calculative: PoT is much better on “calculative” subset than Direct, while Direct remains strong on non-calculative; Combine captures gains from both.
• Efficiency: Token merging enables high resolution (notably 768) without OOM on 32GB V100.
• Key numbers:
  • Throughput (ChartQA test, batch size 1 on V100 32GB): TinyChart@512 3.65 it/s, TinyChart@768 3.14 it/s
  • ChartQA results: TinyChart@768 Direct 76.36, PoT 80.84, Combine 83.60, Oracle 89.12
  • Ablation: 768×768 without merging produces visual length 2916 and OOM; with merging r=84 produces visual length 732 at 3.14 it/s

Limitations / failure modes (from paper)

Extraction and OCR brittleness:

• Chart-to-table extraction struggles when values must be inferred from axes without explicit OCR labels near data points. This is a fundamental limitation when text merging relies heavily on visible annotations.
• The paper does not quantify how often this failure mode occurs in their test sets.

Generation quality issues:

• Chart-to-text generation can hallucinate mismatched content even when some extracted values are correct, suggesting weak grounding between vision and language outputs.
• Chart redrawing struggles with unseen chart types (example given: 3D bar charts), indicating limited generalization beyond training distribution.

PoT combination policy gap:

• The simple keyword-based routing between Direct and PoT modes leaves a substantial gap to Oracle performance (83.60 vs 89.12 on ChartQA).
• The paper does not explore learned routing or confidence-based selection, which could close this gap.
• Error analysis on when the policy fails is not provided.

Evaluation and reproducibility concerns:

• ChartQA-PoT construction relies heavily on GPT-3.5 generation (21K of 140K pairs), which may introduce style biases and limit diversity of reasoning patterns.
• Template-based PoT generation uses 40 question templates from PlotQA; coverage of reasoning types beyond these templates is unclear.
• The paper does not report inter-annotator agreement or validation rates for the generated PoT programs beyond “execution succeeds and matches answer.”
• Full training cost breakdown (GPU hours, energy, cost) is not provided beyond “3 days on 32 V100s.”

Model

Architecture

• Standard MLLM structure: Vision Transformer encoder → vision-language connector → LLM decoder.
• Vision Transformer: Token merging reduces sequence length by approximately $r$ per layer; parameter-free merging module inserted into each ViT layer.
• For image resolution $N \times N$ and patch size $P \times P$, number of vision tokens is $(\lfloor N/P \rfloor)^2$; without reduction, this long sequence is passed downstream.
• Vision-language connector: Implemented as an MLP with one hidden layer and GeLU activation, mapping vision features into the LLM embedding space.
• LLM: Transformer decoder with causal mask; supervised fine-tuning objective is LM loss over response tokens only.

Training recipe

• Initialization: from TinyLLaVA, using SigLIP as vision encoder and Phi-2 as LLM.
• Vision resolution changes: base encoder at 384×384; extend to 512×512 and 768×768 with token merging.
• Token merging rates:
  • 512×512: $r=20$ (authors also ablate $r=12,15,20$).
  • 768×768: $r=84$ (enables training vs OOM without merging).
• Train entire model for 3 epochs, batch size 512, learning rate $1\times 10^{-4}$, warmup 3% of steps, then decay to 0.
• Training cost: 3 days on 32 Tesla V100 GPUs (32 GB VRAM).

Data

ChartQA-PoT (PoT supervision)

• Built from ChartQA training split; total 140,584 (question, PoT answer) pairs.
• Construction counts:
  • Template-based: 119,281 pairs over 17,498 images.
  • GPT-based: 21,303 pairs over 15,521 images.
• GPT-based generation constraints: assignment statements only; no loops/branches; one-line comment before each statement; last variable must be Answer; restricted operator/function list (numpy ops + simple arithmetic/comparators).

Multitask supervised fine-tuning mix

• Total training samples: 1,364,921 across tasks (QA, chart-to-text, chart-to-table, instruction-following), using datasets including ChartQA, PlotQA, DVQA, OpenCQA, Pew/Statista/Vistext/ChartSumm/Chart2Text-8k, and ChartLlama instruction data.

Evaluation

Metrics and protocols

• ChartQA: “relaxed accuracy” allowing numerical error within 5%.
• Chart-to-Text (Pew): BLEU4.
• Chart-to-Table: RMSF1 metric.
• OpenCQA: BLEU4.
• ChartX cognition: GPT-Accuracy for QA, GPT-score for summary/description/redrawing.

Routing rule for “calculative” questions

Uses a keyword detector; list includes terms like “sum, mean, average, ratio, subtract, divide, times, absolute, minus, greater, lowest, number, how many” etc.

• UniMERNet: A Universal Network for Real-World Mathematical Expression Recognition
• Code
• Models (base, small, tiny)
• Dataset (OpenDataLab)

TL;DR

UniMERNet introduces UniMER-1M (1,061,791 image-LaTeX pairs) and UniMER-Test (23,757 samples across four subsets) to address real-world mathematical expression recognition beyond clean rendered formulas. A Swin-Transformer + mBART encoder-decoder architecture with fine-grained embedding, local convolution modules, and decoder attention compression achieves stronger BLEU scores and throughput than Pix2tex and Texify baselines on complex printed, screen-captured, and handwritten expressions.

What kind of paper is this?

• Dominant: $\Psi_{\text{Resource}}$ — Releases UniMER-1M (1M training pairs), UniMER-Test benchmark (four subsets covering diverse real-world conditions), and model weights.
• Secondary: $\Psi_{\text{Method}}$ — Proposes UniMERNet architecture modifications (fine-grained embedding, convolutional enhancement, decoder attention compression) for formula-specific recognition.
• Secondary: $\Psi_{\text{Evaluation}}$ — Provides comprehensive baseline comparisons (Pix2tex, Texify) and ablations across architecture, training data, pretraining, and augmentation.

What is the motivation?

Prior mathematical expression recognition (MER) benchmarks emphasize simple printed or handwritten formulas with limited complexity and clean backgrounds. Real-world deployments encounter long expressions, noisy screen captures, font inconsistencies, geometric distortions, and mixed capture modalities. Existing models trained on small clean datasets fail to generalize to these conditions. The paper addresses this by constructing a large-scale dataset reflecting practical distributions and an architecture tuned for formula-specific challenges such as local spatial relations and similar-looking symbols.

What is the novelty?

Dataset:

• UniMER-1M: 1,061,791 training pairs; maximum formula length 7,037 tokens; average length 79.48 tokens. Sourced from arXiv (89%), Wikipedia (9%), StackExchange (2%); rendered with multiple math fonts at 80-350 DPI.
• UniMER-Test: 23,757 samples across four subsets:
  • SPE (short printed expressions): rendered LaTeX, clean backgrounds
  • CPE (complex printed expressions): long/complex rendered LaTeX, clean backgrounds
  • SCE (screen-captured expressions): extracted from 1,000 PDF pages, annotated via Mathpix + manual correction
  • HWE (handwritten expressions): CROHME + HME100K combined into 6,332 test samples

Model: UniMERNet modifies a Swin + mBART baseline with:

• FGE (Fine-Grained Embedding): Overlapping 3×3 convolution stack (stride 2, padding 1) for patch embedding to reduce character fragmentation from non-overlapping patches.
• CE (ConvEnhance): Depthwise 3×3 convolution + GELU before attention/MLP blocks for local perception of superscripts, subscripts, and adjacent symbols.
• RSW (Remove Shift Window): Eliminates Swin shift-window mechanism when FGE+CE already enlarge receptive field, improving speed and accuracy.
• SA (Squeeze Attention): Low-dimensional projection of query/key tensors in decoder attention to improve throughput without significant accuracy loss.

What experiments were performed?

Data scaling (Table 2): Compared Pix2tex-only, Pix2tex+HWE, and UniMER-1M training on UniMER-Test subsets, measuring BLEU and edit distance.

Architecture ablations (Table 3): Evaluated baseline vs. incremental addition of FGE, CE, SA, and RSW, measuring BLEU by subset and throughput (images/s at batch size 128, max sequence length 1536 on A100).

SOTA comparisons (Table 5): Compared Pix2tex, Texify, Texify* (Texify trained on UniMER-1M with their augmentations), and UniMERNet-T/S/B variants, reporting BLEU, edit distance, FPS, and parameter counts.

Pretraining ablation (Table 6): Tested with/without 16M arXiv-derived image-text pretraining corpus for Texify and UniMERNet-B.

Augmentation ablation (Table 7): Evaluated with/without Albumentations + custom augmentations (erosion, dilation, degradation, geometric transforms) for Pix2tex and UniMER-1M training.

Depth sweeps (Tables 8-9): Varied encoder depth per Swin stage and decoder depth, tracking BLEU and FPS.

What are the outcomes/limitations?

Outcomes:

• UniMER-1M training improves generalization on complex and real-world subsets:
  • CPE BLEU: 0.724 (Pix2tex+HWE) → 0.925 (UniMER-1M); edit distance: 0.225 → 0.056
  • SCE BLEU: 0.529 (Pix2tex+HWE) → 0.626 (UniMER-1M); edit distance: 0.309 → 0.224
• UniMERNet-B (325M parameters) achieves the strongest reported results:
  • SPE BLEU 0.915, CPE BLEU 0.925, SCE BLEU 0.626, HWE BLEU 0.895
  • Throughput: 5.06 img/s (batch 128, max seq len 1536, A100)
• SA (Squeeze Attention) is the primary speed improvement: 4.07 img/s → 5.04 img/s with minimal BLEU change.
• Pretraining improves printed/screen-captured more than handwritten: SCE BLEU 0.601 → 0.626 with 16M arXiv pretraining.

Limitations:

• Inconsistent counts: UniMER-Test size described as 23,789 in text but Table 1 lists 23,757; CROHME test described as 3,332 but Table 1 lists 3,233.
• Sequence length mismatch: Dataset reports max formula length 7,037 tokens while training/evaluation uses max sequence length 1,536; handling of longer sequences (truncation vs. filtering) is not specified.
• SCE labels via Mathpix: Requires proprietary API for initial annotation followed by manual correction; reproducibility implications and licensing considerations are not discussed.
• Pretraining data availability: 16M arXiv-derived corpus is described as “in-house”; public release status is not clarified.

Model

Architecture

UniMERNet follows the encoder-decoder paradigm used in Donut, Nougat, and Texify:

• Encoder: Swin Transformer with modifications
• Decoder: mBART

Encoder Modifications

Fine-Grained Embedding (FGE):

• Two convolutional layers, kernel size 3, stride 2, padding 1
• Each followed by LayerNorm + GELU
• Produces overlapping patches to reduce character feature fragmentation

ConvEnhance (CE):

• Depthwise 3×3 convolution + GELU inserted before attention and MLP modules
• Provides local perception for spatial relations (superscripts, subscripts)
• Alternates local (conv) and global (attention) feature aggregation

Remove Shift Window (RSW):

• Eliminates Swin shift-window mechanism
• FGE and CE already expand receptive field sufficiently
• Improves both speed and accuracy in ablation studies

Decoder Modifications

Squeeze Attention (SA):

• Projects query $Q$ and key $K$ to lower-dimensional space before attention computation
• Reduces computational cost in decoder attention, which becomes the throughput bottleneck
• Value $V$ remains full-dimensional

Model Variants

Data

UniMER-1M Composition

• Total: 1,061,791 LaTeX-image pairs
• Sources: arXiv (89%), Wikipedia (9%), StackExchange (2%); approximately 4M LaTeX expressions collected
• Length distribution:
  • Maximum formula length: 7,037 tokens
  • Average formula length: 79.48 tokens
  • Initial long-formula proportion: 2.3%; rebalanced by extracting longest formulas as CPE subset

Rendering:

• XeLaTeX with multiple math fonts: Asana Math, Cambria Math, XITS Math, TeX Gyre (Bonum, Pagella, Schola, Termes), Latin Modern Math
• DPI range: 80-350
• Latin Modern Math used in ~22% of samples (default)

Normalization:

• Based on Deng et al. (2017) LaTeX normalization
• Adjusted for multi-line environments (align, cases, etc.)
• Filters unsupported syntax

UniMER-Test Subsets

SCE construction:

• 1,000 PDF pages (Chinese + English)
• Formula detection and bounding box annotation
• Initial labels from Mathpix API
• Manual correction pass
• Perceptual hashing deduplication

HWE construction:

• Training: 83,338 samples from CROHME + HME100K
• Test: 6,332 samples (note: text describes CROHME test as 3,332, but Table 1 lists 3,233)

Training

Framework: PyTorch

Maximum sequence length: 1,536 tokens

Optimization:

• Loss: Cross-entropy language modeling
• Learning rate schedule: Linear warmup + cosine decay
  • Initial LR: $1 \times 10^{-4}$
  • Minimum LR: $1 \times 10^{-8}$
  • Warmup LR: $1 \times 10^{-5}$
• Weight decay: 0.05
• Total iterations: 300,000

Augmentation:

• Albumentations library + custom transforms
• Morphological: erosion, dilation
• Degradation: fog, frost, rain, snow, shadow simulation
• Geometric: rotation, distortion

Pretraining (optional):

• 16M image-text pairs from arXiv
• Constructed via layout detection + OCR on text blocks, matched to source LaTeX
• Pretraining improves printed/screen-captured performance more than handwritten

Hardware:

• NVIDIA A100 80GB
• Batch size: 64
• Training setup description includes both “single GPU” and “eight such GPUs” references (apparent inconsistency in source material)

Evaluation

Metrics

• BLEU: Measures token-level similarity between predicted and ground-truth LaTeX
• Edit Distance (Levenshtein): Character-level edit operations normalized by ground-truth length
• ExpRate: Exact match rate (expression-level accuracy)

Key Results

Data Scaling (Table 2)

Training on UniMER-1M vs. Pix2tex+HWE, evaluated on UniMER-Test:

Architecture Ablation (Table 3)

Impact of UniMERNet modifications on SCE subset (batch 128, A100):

Model Comparison (Table 5)

UniMERNet-B vs. baselines on UniMER-Test:

Texify* = Texify trained on UniMER-1M with their augmentation pipeline

Pretraining Impact (Table 6)

UniMERNet-B with/without 16M arXiv pretraining:

Hardware

Training:

• NVIDIA A100 80GB
• Batch size: 64 per GPU
• Setup: references both “single GPU” and “eight such GPUs” (inconsistent in source)

Inference throughput tests:

• Batch size: 128
• Max sequence length: 1536
• Hardware: A100 80GB
• UniMERNet-B: 5.06 img/s

License & Availability

Code: Apache-2.0 (GitHub)

Models: Apache-2.0 (Hugging Face)

Dataset: Apache-2.0 (Hugging Face, OpenDataLab)

Important caveat: The dataset card notes that HME100K portion must be downloaded manually “for copyright compliance,” and the dataset incorporates multiple upstream sources (Pix2tex, CROHME, HME100K). The Apache-2.0 tag on the Hugging Face dataset repository may not override upstream licensing terms for these components. Verify upstream licenses before commercial use.

Paper: ChartThinker: A Contextual Chain-of-Thought Approach to Optimized Chart Summarization (arXiv:2403.11236v2, 25 Apr 2024)
Authors/Affiliations: Sun Yat-sen University; Alibaba Group; Guangdong Provincial Key Lab; Pazhou Lab
Code/Data: GitHub: Notonion/ChartThinker, HuggingFace: ChartThinker/Chart-Sum-QA
License: MIT (dataset on HuggingFace); see notes below for source dataset licenses

TL;DR

ChartThinker targets chart summarization failure modes in VLMs: (1) poor chart-text matching degree (omissions, hallucinated values) and (2) reasoning errors about the chart’s intended message. It introduces (a) a large chart-caption + instruction QA dataset (Chart-Sum-QA) and (b) a method that combines chart parsing (OCR + DePlot), chain-of-thought (CoT) style staged generation, and a small retrieval library for in-context examples.

What kind of paper is this?

Dominant: $\Psi_{\text{Method}}$ (new end-to-end approach: chart parsing + context-enhanced CoT generation + retrieval integration; extensive ablations).
Secondary: $\Psi_{\text{Resource}}$ (release of Chart-Sum-QA: 595,955 chart-caption pairs + 8.17M instruction QA pairs).
(Using the taxonomy defined in the provided guidelines.)

What is the motivation?

The authors argue that large vision-language models often fail at chart summarization because:

• Insufficient matching degree: summaries omit small-but-critical numbers/text, or fabricate chart content due to pretraining priors.
• Reasoning errors: models misread trends/patterns or infer incorrect “takeaways” from complex charts.

Figure 1 illustrates both error types via examples comparing LLaMA-Adapter-v2 and MiniGPT-4 outputs to a chart, showing both hallucinations and incorrect inferences.

What is the novelty?

1. Chart-Sum-QA dataset (scale + supervision breadth)

   • 595,955 chart-caption pairs (pretraining) and 8,170,000 instruction question-answer pairs (fine-tuning).
   • Covers chart types: scatter, line, bar, pie (per Table 1).

2. Context-Enhanced CoT Generator (retrieval + staged “thoughts”)

   • The model generates multiple thought steps (example steps: identify chart type; understand legend/axes; observe trends/proportions), and for each thought retrieves top-$k$ chart-text exemplars from a small retrieval library to use as context examples, then consolidates outputs into a final summary (overview shown in Figure 2).
   • Retrieval uses cosine similarity between encoded chart features and stored example features, then uses an order-based weighting where example weight is proportional to $1/i$ for rank $i$ (Equations 3–4), and conditions generation on a weighted context function (Equation 5).

3. Chart parsing module that fuses OCR + DePlot outputs

   • OCR extracts textual/numeric strings; DePlot converts chart to a table (text-number alignment), and the module outputs:
     • text-number pairs and
     • other text (title/notes/etc.).
   • The workflow is depicted in Figure 3.

What experiments were performed?

Benchmarks and baselines

They compare against two baseline groups:

• Classic (text-generation) transformer baselines with OCR-augmented inputs: T5, Chart2text, Field-Infuse, BART, plus their OCR-ChartThinker variant (Table 2).
• Large VLM baselines (encoder-decoder style): LLaMA-Adapter-v2, MiniGPT-4, mPLUG-Owl, LLaVA, plus ChartThinker (Table 3).

Metrics

Automatic evaluation uses:

• BLEU, BLEURT (base-128), CIDEr, Content Selection (CS), and perplexity using GPT-2 Medium; plus a normalized aggregate score $S_{\text{norm}}$ over five indicators.

Human evaluation:

• 200 generated summaries, scored by 3 evaluators on Matching Degree and Reasoning Correctness (1–5).

Ablations (Table 4)

Ablations remove:

• chart parsing module,
• context retrieval (“No Context-Enhanced”),
• CoT (“No CoT”),
• entire context-enhanced CoT generator,
• fine-tuning on caption dataset,
• fine-tuning on instruction dataset.

What are the outcomes/limitations?

Key results (as reported)

• OCR-ChartThinker vs OCR baselines (Table 2): reported best BLEU (11.81), best CIDEr (2.21), best PPL (9.23), best $S_{\text{norm}}$ (0.948), though not best CS (32.72% vs OCR-T5 40.87%).
• ChartThinker vs VLM baselines (Table 3): reported best across listed metrics among those VLM baselines (BLEU 5.82, CIDEr 1.58, CS 21.68%, PPL 11.43).
• Human evaluation (Table 5): ChartThinker rated highest on Matching Degree (4.32) and Reasoning Correctness (4.27) among compared models in their study.

Limitations / caveats (from the paper + inferred risks)

Evaluation metric concerns:

• CS (Content Selection) metric shows counterintuitive behavior: the authors argue it can penalize longer correct summaries that include additional valid detail beyond the reference (Table 6 example). This raises questions about whether CS is a reliable primary metric.
• Heavy reliance on automatic metrics (BLEU, CIDEr, BLEURT) may not capture summary quality aspects like clarity, coherence, or usefulness to end users.
• Human evaluation is limited to 200 samples with 3 evaluators; inter-rater agreement is not reported.

Data generation pipeline concerns:

• A large portion of instruction QA (exact ratio not specified) is generated using ChatGPT-4 from human-written summaries, which may introduce:
  • Style biases toward GPT-4’s generation patterns
  • Potential data leakage if GPT-4 was trained on overlapping sources
  • Limited diversity in question types and reasoning patterns
• The paper mentions “a subset” is manually validated but does not specify validation coverage, criteria, or pass rates.
• Prompt templates and filtering heuristics for QA generation are not fully detailed.

Retrieval library limitations:

• Library contains only 1,000 chart-text pairs split across 4 stages (250 each).
• Coverage of unusual chart designs, rare chart types, or domain-specific visualizations may be insufficient.
• The paper does not analyze retrieval failure modes or cases where no good match exists.

Architectural and scope constraints:

• Chart parsing module combines OCR + DePlot but may inherit limitations from both (e.g., DePlot’s known weakness with color-dependent questions).
• Context-enhanced CoT is stage-specific, but the paper does not justify the choice of 4 stages or explore alternative decompositions.
• Generalization to chart types beyond scatter, line, bar, and pie is not evaluated.

Reproducibility gaps:

• The paper provides LoRA hyperparameters but omits:
  • Full hardware specifications (GPU type, memory, count)
  • Wall-clock training time
  • Inference throughput (tokens/sec, images/sec)
  • Total training cost or energy consumption
• Dataset mixing ratios and sampling strategies for the multi-source training corpus are not detailed.

Research questions (explicitly answered)

The paper poses three RQs; below are the answers supported by the reported experiments:

• RQ1: Can answer reasoning benefit from introducing a chain of thought?
  Yes, per Table 4: removing CoT (“No CoT”) reduces BLEU (5.10 vs 5.82) and lowers human score (3.92 vs 4.25), indicating CoT contributes to both automatic metrics and judged quality.

• RQ2: How can context retrieval and chain of thought effectively interact with each other?
  The reported synergy claim is supported by ablations: removing context enhancement reduces performance (“No Context-Enhanced”: BLEU 5.45 vs 5.82; human 4.11 vs 4.25), and removing the entire combined module drops more (“No Context-Enhanced CoT Generator”: BLEU 4.59; human 3.85). The method’s interaction mechanism is “retrieve exemplars per thought step” (Figure 2).

• RQ3: How does instruction fine-tuning improve the chart-summary matching degree?
  Table 4 shows dropping instruction fine-tuning degrades BLEU (4.52 vs 5.82) and BLEURT (–0.63 vs –0.45), consistent with the claim that directive QA tuning improves chart-specific grounding.

Model

High-level architecture (from Figure 2)

Inputs: chart image + prompt. The pipeline includes:

• Image encoder (CLIP-based encoder) producing chart feature vector $V$.
• Text encoder for the prompt producing token sequence/features $K$ (paper text labels it “encoder” but describes generation with a decoder; see below).
• Chart parsing module produces underlying data features by merging OCR + DePlot outputs and concatenating with prompt features.
• Context-Enhanced CoT Generator generates a sequence of “thoughts”; for each thought it retrieves top-$k$ example chart-text pairs and uses them as in-context examples, then integrates outputs into the final summary.

Generator backbone

• Paper states the final generation is done with Idefics and also mentions a LLaMA2 decoder (the description mixes terms: “text encoder” vs “decoder”); the operational takeaway is: a VLM generator (Idefics) plus LoRA is used for conditioned generation.

Retrieval library design (Appendix B summary in-text)

• 1,000 chart-text pairs, split into 4 stages of 250 each:
  1. chart type,
  2. chart overview (title-driven),
  3. axes meanings,
  4. numerical trend description.

Data

Chart-Sum-QA composition (Table 1)

• Aggregates multiple existing chart datasets (Autochart, Linecap, DVQA, PlotQA, Chart-to-text, FigureQA), yielding:
  • 595,955 images and
  • 8,170,000 QA pairs total (their constructed dataset).

Construction steps (Section 3.1)

1. Data collection from six datasets.
2. Preprocessing: resize, standardize formats, clean titles/descriptions, ensure alignment.
3. Generate QA pairs: create ~400k QA from summaries (Chart-to-text, Autochart, Linecap), merge with other QA datasets, filter, finalize 8.17M; use ChatGPT-4 to generate questions from human-written summaries; manually validate a subset.
4. Splits: 80% train, 10% validation, 10% test.

Algorithms / Training

Chart parsing module (Figure 3)

• OCR output: text strings and numbers (without positions).
• DePlot output: a table of text-number correspondences (can be affected by irrelevant chart elements; may have numeric extraction errors).
• Fusion output: separate text-number pairs (key aligned entities) and other text.

Retrieval + weighting

• Similarity: cosine similarity between input chart feature vector $V$ and example feature $T_i$ (Eq. 3).
• Context weighting: rank-based weight $1/i$; aggregated weighting function $W_\ell(\cdot)$ conditions generation at token $\ell$ (Eq. 4–5).

Fine-tuning implementation (LoRA)

• LoRA rank 16, applied to qproj/kproj/vproj.
• LoRA scaling factor $\alpha = 32$, dropout 0.05.
• Optimizer: paged_adamw_8bit, learning rate $2\times 10^{-4}$.
• Gradient accumulation: 8; eval every 20 steps.

Evaluation

Automatic metrics

• BLEU, BLEURT-base-128, CIDEr, CS, PPL (GPT-2 Medium), plus normalized aggregate score $S_{\text{norm}}$.

Human evaluation protocol

• 200 summaries; 3 annotators; criteria:
  • Matching Degree (data fidelity: minimal omissions/fabrications)
  • Reasoning Correctness (intended message inferred correctly)
• Rated 1–5; randomized order to reduce bias.

Hardware / Production

Not fully specified. The paper provides the optimizer choice and LoRA settings, but does not give:

• GPU type/count, training duration, batch size (beyond gradient accumulation), or inference throughput.

Dataset release + license

Do they share their dataset online?

Yes. The paper explicitly says their “dataset and codes are publicly accessible” and that they “release our dataset and codes at OpenChartThinker.” A public repo associated with the project also points to a Hugging Face dataset upload (GitHub: Notonion/ChartThinker).

Under what license?

On Hugging Face, the ChartThinker/Chart-Sum-QA dataset is labeled MIT (“License: mit”). (HuggingFace: ChartThinker/Chart-Sum-QA)

Detail to be aware of

In the dataset construction section, the authors note the source datasets they aggregate are under various licenses (examples given: CC BY-NC-SA 4.0, MIT, GPL3.0). So, even though the HF card says MIT, it’s worth double-checking provenance/constraints for any subset you plan to use (especially commercial use).

Paper: OneChart: Purify the Chart Structural Extraction via One Auxiliary Token (arXiv:2404.09987; MM ‘24; DOI: 10.1145/3664647.3681167)
Project page: onechartt.github.io
Code: LingyvKong/OneChart
Models: kppkkp/OneChart
License: Apache-2.0 (code/model); research use only (with upstream constraints from Vary/OPT)

TL;DR

OneChart is a chart-to-Python-dict structural extraction model that adds a single auxiliary token (<Chart>) plus a small auxiliary number decoder trained with an $L_1$ loss to improve numeric reliability. It also uses a self-consistency distance between “numbers implied by the text dict” and “numbers predicted by the auxiliary decoder” as an optional reliability score to filter (“purify”) outputs at inference. Authors report strong structural-extraction AP across several chart benchmarks with a relatively small model ($\approx$200M params) and gains when feeding the extracted dict into downstream ChartQA systems.

What kind of paper is this?

Dominant: $\Psi_{\text{Method}}$ (new architecture element: auxiliary token + decoder + training/inference recipe; includes ablations and SOTA-style tables).

Secondary:

• $\Psi_{\text{Resource}}$ (introduces ChartY benchmark; also describes large synthetic data engine).
• $\Psi_{\text{Evaluation}}$ (self-consistency reliability scoring and “purified output” evaluation protocol; additional comparisons/ablations).

What is the motivation?

Chart parsing is hard because charts vary widely in style, values, text, and layouts, and even large VLMs can struggle, especially when charts lack explicit numeric annotations. Authors attribute failures mainly to:

• “CLIP bias”: CLIP-ViT encoders are trained on natural-image captioning and may miss chart-local details; English-heavy pretraining can also hurt non-English charts.
• Cross-entropy numerics problem: token-level CE does not strongly distinguish “close-looking” numbers (example: “7008” vs “70.8” can have deceptively similar loss), which can slow convergence and reduce numeric accuracy.

Public benchmarks are described as limited in type/style/language diversity (e.g., ChartQA and PlotQA skew heavily to bar/line; many synthetic sets have limited styling).

What is the novelty?

Auxiliary token + auxiliary number decoder:

• Prefix a special token <Chart> at the start of the output sequence; its hidden state embedding is routed to an auxiliary MLP decoder trained to predict a fixed-length vector of normalized chart numbers using masked $L_1$ loss.
• Because the main LM is causal, later text tokens can attend to the <Chart> token embedding, aiming to improve numeric fidelity in the generated Python-dict.

Self-evaluation / “purification”:

• At inference, parse the raw Python-dict output, extract numbers, normalize them, and compare to the auxiliary decoder’s numeric vector via a mean absolute distance $S \in [0,1]$; filter outputs by a threshold (example $\delta=0.1$) to keep only “reliable” predictions and report improved AP on the retained subset.

Data engine + benchmark:

• Large synthetic chart generation (Matplotlib + Pyecharts) with style randomization and bilingual (English/Chinese) content; and the ChartY chart-to-dict benchmark (reported as $\sim$6K charts in the intro, bilingual and stylistically broader).

What experiments were performed?

• Structural Extraction (SE) evaluation on multiple sources (ChartQA-SE, PlotQA-SE, ChartX-SE, ChartY-en/zh), using SCRM mean AP under strict/slight/high tolerances; plus textual OCR accuracy using Reverse Edit distance (RE) for fields like title/source/axes.
• Comparisons to chart parsing baselines (e.g., UniChart, DePlot, ChartVLM, ChartAst), emphasizing performance when charts do not have explicit numeric labels.
• Ablations: auxiliary token presence and position (front vs behind), and training strategy (Stage2 vs Stage3 vs both).
• Downstream QA: combine OneChart’s extracted dict with LLM/VLM QA systems on ChartQA; report accuracy changes when giving models chart image, dict, or both.

What are the outcomes/limitations?

Outcomes (as reported):

• The authors report OneChart at 0.2B parameters achieves competitive or leading AP across multiple SE benchmarks in their evaluation (Table 2).
• Filtering by self-consistency distance threshold $\delta=0.1$ is reported to increase AP on the retained subset (example: ChartQA-SE AP@strict 72.02 $\rightarrow$ 81.97), though this comes at the cost of reduced coverage.
• Token position matters: placing <Chart> at the front performs better than at the end, consistent with causal attention (Table 4).
• The authors report QA accuracy improvements when feeding the extracted dict into LLaVA variants (Table 6).

Limitations / open questions (based on what’s in the paper):

Dataset and generalization scope:

• Synthetic data generator focuses primarily on bar/line (“barline”) and pie charts, with explicit constraints (e.g., barline charts “up to three legends”). Coverage of other chart types (scatter, area, stacked variants, multi-panel figures) is limited.
• ChartY benchmark is reported as $\sim$6K charts, but exact split sizes and per-type distributions are not provided.
• Generalization to chart styles outside the synthetic generation distribution is not systematically evaluated.

Architectural constraints:

• The auxiliary number decoder outputs a fixed-length 256 vector with padding. Behavior when charts contain many series or data points (e.g., dense scatter plots, time series with hundreds of points) is unclear.
• The paper does not analyze failure modes when the fixed-length representation is insufficient.

Evaluation methodology concerns:

• “Purified” results are reported on filtered subsets with reduced coverage. Full risk-coverage curves are not presented; only snapshots at specific thresholds ($\delta=0.1$).
• This shifts evaluation from “overall accuracy” to “accuracy at coverage,” but the trade-off is not fully characterized.
• The self-consistency distance may be biased toward certain chart types or value ranges; this is not analyzed.

Reproducibility gaps:

• The paper reports three training stages but does not provide full hardware specifications, wall-clock training time, or GPU hours.
• Stage-specific data mixing ratios and sampling strategies are not detailed.
• License complexity: while code is Apache-2.0, the model inherits constraints from Vary/OPT upstream dependencies, making the effective license “research use only” despite the stated Apache-2.0 for code.

Model

Task and output format

• Input: a chart image (resized to 1024$\times$1024).
• Output: a serialized Python-dict containing at least: title, source, x_axis, y_axis, and value.

Backbone architecture

• Built on a VLM-style encoder–decoder; authors choose Vary-tiny: a vision encoder “from SAM-base” paired with an autoregressive OPT-125M decoder, connected via a linear projection for channel alignment.
• Vision features occupy 256 tokens in the conversation template.

Auxiliary token and decoder

• Add special token <Chart> at the start of the token sequence (Figure 3 pipeline depiction).
• Let the <Chart> hidden embedding be $t \in \mathbb{R}^{768}$. Feed $t$ into an auxiliary decoder $F$ described as a 3-layer MLP with 2 ReLU activations.
• Auxiliary output: $F(t) \in \mathbb{R}^{256}$ representing min–max normalized numeric values for the chart.

Data

Synthetic generation (“data engine”)

• Uses both Matplotlib and Pyecharts to render charts of varied styles/types (Figure 2 summarizes the pipeline).
• Adds a “chart source” field (beyond typical title/axes/body) to better match real-world charts; includes a two-stage rendering variant that stitches title/source after rendering the main body to increase layout diversity.
• Style randomization: random 16-bit color codes for text/graphics (beyond standard palettes), “hundreds” of fonts, variability in size/direction/quantity of visual elements.
• Scale: about 10M synthetic chart images with labels for pretraining.
• Chart types emphasized:
  • Barline charts: single/multi column, single/multi line, combo; balanced between with/without numeric labels; up to 3 legends.
  • Pie charts: labeled pies and legend-based pies in equal proportion.
• Text/content generation: random corpora for pretraining; GPT-3.5 prompts for more “logical/practical” themed data across domains (finance/education/technology/etc.).

Fine-tuning (SFT) data

• Total SFT data reported: 2.7M samples (Table 1).
• Mix includes ChartQA (real), PlotQA (real), and multiple GPT-3.5-driven synthetic sets in English and Chinese rendered via Matplotlib/Pyecharts.

ChartY benchmark

• Bilingual (English/Chinese) chart-to-dict benchmark with broader style/type diversity than prior benchmarks.
• Reported as $\sim$6K charts total (split between ChartY-en and ChartY-zh).
• Available via project page and GitHub repository.

Algorithms / Training

Conversation template and tokens

Uses Vicuna v1-style prompt template:

USER: <img> [image] </img> Convert the key information of the chart to a Python-dict. ASSISTANT: <Chart> [texts output] </s>

Adds <img>, </img>, <Chart> to the OPT tokenizer as special tokens.

Objectives

Text generation loss: standard causal LM cross-entropy:

$$L_{\text{text}}(\theta, w) = -\mathbb{E}_{(w,v)\sim D}\log P_\theta(w_m \mid w_{<m}, v)$$

Auxiliary numeric loss: masked $L_1$ on non-padded entries:

$$L_{\text{num}}(\theta, u) = \mathbb{E}_{(u,t)\sim D}\left|F(t) - u\right|_{\text{masked}}$$

with ground-truth values min–max normalized and padded with “nan” to length 256.

Three-stage training recipe (with reported compute)

Stage 1 (Pretraining):

• 10M synthetic charts
• Batch size 16; lr $10^{-4}$; 3 epochs
• Trains vision encoder + language model
• Loss: $L_{\text{text}}$
• Compute: 32 A100 (80G) for $\sim$12 hours

Stage 2 (Warmup aux decoder):

• 2.7M SFT samples
• Freeze vision encoder; train LM + auxiliary decoder
• Batch size 16; lr $5\times10^{-5}$; 1 epoch
• Loss: $L_{\text{text}} + L_{\text{num}}$
• Compute: 16 A100 (80G) for $\sim$3 hours

Stage 3 (SFT all params):

• Same SFT pool
• Train all parameters
• Batch size 16; lr $5\times10^{-5}$; 1 epoch
• Loss: $L_{\text{text}} + L_{\text{num}}$
• Compute: 24 A100 (80G) for $\sim$4 hours

Evaluation

Metrics

Textual OCR for title, source, x_axis, y_axis: uses normalized edit distance, reported as Reverse Edit distance (RE) = 1 - normalized edit distance (higher is better).

Structural extraction for value dict: compute tuples of (key, item) pairs and evaluate via SCRM mean Average Precision with tolerances:

• strict: $J_{thr}=0$, $e_{thr}=0$
• slight: $J_{thr}=2$, $e_{thr}=0.05$
• high: $J_{thr}=5$, $e_{thr}=0.1$

Benchmarks described

• ChartQA-SE / PlotQA-SE: derived from ChartQA and PlotQA test sets; PlotQA-SE emphasizes charts without explicit numeric annotations.
• ChartX-SE: Matplotlib-rendered bar/line/pie variants, with and without numeric labels.
• ChartY-en / ChartY-zh: authors’ added benchmark (Pyecharts, bilingual, partial numeric annotations).

Key reported results (selected)

Table 2 (SE AP, OCR RE): OneChart (“Ours”, 0.2B) reports higher AP than compared baselines across many settings, with notable gains on no-numeric-annotation scenarios (authors highlight this in the text).

Table 3 (raw vs purified): with threshold $\delta=0.1$, AP@strict increases after filtering (example: ChartQA-SE 72.02 $\rightarrow$ 81.97) while the number of evaluated samples decreases.

Table 4 (token position): <Chart> at sequence front outperforms “behind” and “no token” for ChartQA-SE and PlotQA-SE AP under all tolerances.

Table 6 (ChartQA QA): authors report that providing the extracted dict can substantially improve LLaVA1.5/1.6 accuracy in their setting (e.g., LLaVA1.5 avg 17.5 $\rightarrow$ 50.1 when given figure + dict; improvements are also shown for other configurations).

Hardware / Production

• Inference preprocessing: resize to 1024$\times$1024, scale pixels to [0,1], embed to $v \in \mathbb{R}^{256\times768}$, and decode until </s> token.
• Reliability scoring (optional): parse the text dict (authors mention using json.loads()), extract numbers, min–max normalize to $u_r$, compare to auxiliary decoder output $u_c$ via:

$$S=\frac{1}{N}\sum_{i=1}^{N}|u_{r_i}-u_{c_i}|$$

and threshold $S$ to decide whether to “trust”/retain outputs.

• Efficiency note (authors’ claim): model size is $\approx$200M parameters and they report token-time comparisons (example: 1.3 ms vs 5.7 ms for “one token” compared to a much larger baseline), but the exact measurement setup is not fully detailed in the excerpted text.

Paper: SciGraphQA: A Large-Scale Synthetic Multi-Turn Question-Answering Dataset for Scientific Graphs (arXiv 2023)
Authors: Shengzhi Li, Nima Tajbakhsh
Code: findalexli/SciGraphQA
Data: GitHub, HuggingFace
License: Research only (Palm-2/GPT-4 terms)

TL;DR: SciGraphQA is a large synthetic multi-turn QA dataset grounded in real scientific graphs extracted from ArXiv papers: the images are real, but the dialogues (questions/answers) are generated (Palm-2) from text context around each figure. The authors report current MLLMs perform poorly zero-shot on this benchmark (low CIDEr), but prompt augmentation with DePlot-extracted tables and fine-tuning a LLaVA-13B baseline materially improves scores.

What kind of paper is this?

• Dominant: $\Psi_{\text{Resource}}$ 0.65
• Secondary: $\Psi_{\text{Evaluation}}$ 0.20, $\Psi_{\text{Method}}$ 0.15

The primary contribution is a large-scale dataset/benchmark (SciGraphQA) with construction pipeline + release framing. The paper also evaluates multiple MLLMs on the dataset and introduces a baseline fine-tuning recipe plus a prompt-augmentation technique (DePlot tables).

What is the motivation?

Scientific papers communicate key results via graphs/figures that require interpretation in context (abstract + surrounding paragraph), often via interactive multi-turn explanation in real life (reading groups, presentations). Existing chart/graph VQA datasets either rely heavily on synthetic charts and/or template questions, or are much smaller in scale; the authors aim to scale “ChartQA-style” real-chart VQA to the academic graph domain. They also position “dialogue QA” as more useful than “caption prediction” for graphs because captions can be short/underspecified and not very helpful to a user.

What is the novelty?

Dataset construction (core novelty):

• Uses ~290,000 CS/ML ArXiv papers (2010–2020) and extracts figures (PDFFigures 2.0 is referenced)
• Focuses on graphs (not all figure types) and generates multi-turn QA dialogues about each graph using Palm-2 prompted with text context: title, abstract, figure caption, and the first paragraph mentioning the figure; OCR text is also included in the context input
• Scale: 295K multi-turn samples and 657K QA pairs/turns (multi-turn, average 2.23 turns per sample)

Prompt augmentation idea (secondary novelty):

• Prepend the question with a serialized data table extracted from the chart using DePlot (plot-to-table model) to mitigate weak OCR/text-reading in MLLMs (Figure 4 shows the “table then serialize” concept with newline token <0x0A>)

What experiments were performed?

Dataset quality checks:

• GPT-4 is used as a judge to rate QA matching quality on a 3K test subset; authors report average 8.7/10, with distribution heavily skewed high (Figure 2, p.6)

Zero-shot MLLM evaluation:

• Zero-shot evaluation on 3K test samples with NLP overlap metrics: BLEU-4, ROUGE, CIDEr
• Models discussed include LLaVA variants, mPLUG-owl, BLIP-2, OpenFlamingo, and DePlot+LLM pipelines

Fine-tuning:

• Fine-tune LLaVA-13B on SciGraphQA: (1) 1 epoch on full dataset, then (2) additional fine-tuning on a 30K DePlot-augmented subset
• Study effect of training set size on fine-tuning performance (Figure 5, p.9)

What are the outcomes/limitations?

Outcomes (as reported):

• Baseline zero-shot scores are very low; prompt augmentation with DePlot tables improves some models; fine-tuning improves further (Table 2, p.8; Figure 5, p.9)
• Authors argue this indicates: (a) scientific graphs are out-of-distribution and hard for current MLLMs, and (b) external “chart-to-table” extraction can meaningfully help

Limitations / caveats:

• Synthetic answers: images are real graphs, but QA text is model-generated (Palm-2). So evaluation is against synthetic “ground truth,” and overlap metrics may reward stylistic similarity rather than correctness
• Filtering heuristic: they drop QA turns not containing any of: graph/diagram/figure/chart/axis/plot/table/image/visual/illustrat (string match), which can remove legitimate conceptual follow-ups
• Metric choice: they mostly report BLEU/ROUGE/CIDEr for reproducibility; they discuss GPT-4 evaluation issues (inconsistency with chain-of-thought prompting; resource limits; nondeterminism)
• Internal inconsistency to note: the abstract text claims LLaVA-13B is “most performant” zero-shot at CIDEr 0.08, but Table 2 lists OpenFlamingo v2-7B at CIDEr 0.12, higher than 0.08. Also the paragraph mentions prompting GPT-3.5, while Table 2 labels DePlot+GPT-3. (This may be a reporting mismatch, but it is present in the paper text/tables)

Reproducibility Details

Model

Baselines evaluated:

• MLLMs: BLIP2-2.7B; mPLUG-owl-7B; LLaVA-7B; LLaVA-13B; OpenFlamingo v2-7B
• “Expert system” style: DePlot + (mPLUG-owl / LLaVA-13B / GPT-3.x)

Vision and language backbones (as described):

• Evaluations generally use a fixed CLIP/ViT vision encoder; mPLUG-owl is noted as an exception (encoder unfrozen in its own training, per related-work discussion)
• Fine-tuning: uses a LLaVA checkpoint with LLaMA-2-13B-chat as base model

Prompt augmentation (DePlot):

• DePlot produces a text table with chart title/legends and interpolated values; they serialize it into a single string and prepend to the question prompt (Figure 4, p.8)

Data

Source corpus:

• ~290,000 ArXiv papers (CS or stat.ML), 2010–2020
• Figures extracted (mentions PDFFigures 2.0); of ~2.1M figures, top categories include tables (23.6%), graphs (19.2%), flowcharts (8.5%); graphs chosen as focus

SciGraphQA dataset stats:

• Generation target: multi-turn dialogues about graphs using Palm-2, with context: title, abstract, caption, first paragraph referencing the figure, OCR text; in-context examples used in prompting (Figure 1, p.5; Appendix prompt on p.11–12)
• Pipeline yields 350K initial samples; after filtering, 295K multi-turn entries
• Total size: 59.1M tokens (byte-wise BPE). Avg turn count 2.23; 111K samples have 3+ turns
• Per-turn verbosity: avg 143 chars per question and 775 chars per answer; avg 39 tokens per question and 164 tokens per answer; tokens per sample 199 ± 98

Quality rating:

• GPT-4 judge on 3K test subset: average 8.7/10; authors report 86% are rated 8.5+; small tail of low ratings (Figure 2, p.6)

Dataset availability:

• Publicly available via GitHub repository and Hugging Face datasets
• GitHub repo contains multiple dataset artifacts: 295K training set and 3K test set, sizes “excluding images”
• Licensing note from repo README: “data, code and checkpoint is intended and licensed for research use only” with uses restricted to those that follow Palm-2, LLaMA and GPT-4 license agreements
• Practical guidance: treat as publicly accessible for research-only use; commercial applications should seek clarification from authors

Algorithms / Training

Question filtering:

• Removes questions deemed “not graph-related” via keyword list: graph, diagram, figure, chart, axis, plot, table, image, visual, illustrat

Fine-tuning recipe (LLaVA-13B baseline):

• Two-step: (1) 1 epoch on full 295K dataset; (2) fine-tune on 30K subset with DePlot-augmented prompts
• Optimization: cosine LR schedule, warmup ratio 3%, learning rate $5 \times 10^{-6}$ (explicitly lower than prior LLaVA instruction-tuning LR)
• PEFT: LoRA rank 64, LoRA dropout 0.05

Evaluation

Metrics and setup:

• Test set: 3K (image, question, answer) samples; compute BLEU-4, ROUGE, CIDEr between generated and reference answers

Key result table (Table 2, p.8):

Additional evaluation notes:

• OpenFlamingo: authors attribute underperformance to not reproducing the retrieval-based in-context selection (RICE); random 3/6/9-shot in-context examples did not help in their trials
• Dataset-size scaling: performance improves with more training data; biggest gains in the first ~50% of dataset; authors speculate LoRA (vs full fine-tuning) may limit gains from 50% to 100% (Figure 5, p.9)

Hardware / Production

• Fine-tuning run uses DeepSpeed ZeRO-2 with 4× A100-80GB GPUs (Azure)
• Batch sizing: per-device batch size 16, gradient accumulation 2, effective global batch size 128

Paper: VisText: A Benchmark for Semantically Rich Chart Captioning
Authors: Benny J. Tang, Angie Boggust, Arvind Satyanarayan
Venue: ACL 2023 (Toronto, July 9–14, 2023)
Code + Dataset: mitvis/vistext
License: GNU GPL v3.0 (data and code)

Core artifact: VisText dataset with 12,441 charts, each paired with (a) rasterized image, (b) data table, and (c) scene graph
Chart types covered: area, bar, line
Task: generate semantically rich chart captions (including higher-level and trend statements, not just “what the chart shows”)

TL;DR

VisText is a chart-captioning benchmark designed to push beyond surface-level captions by including human-written, semantically rich statements (notably “value/trend” style content) and multiple machine-consumable chart representations (image, data table, scene graph). Their experiments suggest text-only representations (scene graph or table) outperform image-based approaches, and semantic prefix-tuning is mainly useful for controlling semantic level rather than improving standard metrics.

What kind of paper is this?

Dominant: 0.55 $\Psi_{\text{Resource}}$ (new benchmark + dataset + standardized inputs)
Secondary: 0.25 $\Psi_{\text{Method}}$ (semantic prefix-tuning to control caption semantic level)
Secondary: 0.20 $\Psi_{\text{Evaluation}}$ (systematic comparisons across representations, models, and metrics)

What is the motivation?

Prior chart captioning benchmarks and methods often emphasize surface descriptions (or omit deeper semantics), while assistive technologies and visualization authoring workflows benefit from captions that include derived insights (values, comparisons, trends). The paper frames semantically rich captions as more useful and closer to human-written chart descriptions.

What is the novelty?

1. Dataset design for richer semantics: Each chart has multiple representations (image, data table, scene graph) and captions intended to be more semantically informative than older datasets.
2. Multi-level captioning framing: Uses a semantic “level” lens (L1–L4) adapted from accessibility guidance and prior work, focusing on the first three levels for modeling.
3. Semantic prefix-tuning: A fine-tuning approach that aims to generate different semantic levels without training separate models per level.

What experiments were performed?

• Quantitative benchmarking comparing:
  • text-only models from scene graphs vs data tables
  • multimodal/image-guided models (image-only and image+text hybrids)
  • prior baseline (Kantharaj et al. 2022) using BLEU, perplexity, ROUGE-L, WMD, TER, and a relation-generation check.
• Ablations over different language-model backbones (ByT5-small, T5-small, BART-base) and prefixes.
• Qualitative error analysis categorizing common failure modes (identity/direction/value/etc.).

What are the outcomes/limitations?

• Outcome: Image-based captioning performed worst; text-only encodings (scene graph or table) performed best, with scene graphs often competitive with data tables.
• Outcome: L1 captions are much easier than L2/L3 (models score far higher on L1 than on richer statements).
• Limitation: Dataset focuses on three univariate chart types; generalization to more complex charts and real-world, messy charts remains open.
• Limitation: Models are evaluated via proxies; the paper calls out the need for work on interactive authoring and better grounding/linking of text to chart regions.

Model

Inputs (three chart representations)

Each example includes:

• Rasterized image of the chart,
• Underlying data table, and
• Scene graph derived from the rendered visualization.

Baseline families evaluated

• Text-only seq2seq: ByT5-style models mapping a text linearization of scene graph or data table to captions.
• Image-guided: VL-T5 variants using image features (alone or paired with text representations).

Semantic prefix-tuning

They propose prefix-tuning to generate captions at different semantic levels without training separate models; importantly, they note it does not necessarily improve standard metrics, but supports semantic-level control.

Data

Source data and cleaning

• Data tables come from the Statista public dataset; they convert tables to structured form, handle missing values, and normalize some date-like formats (e.g., week to datetime).

Chart generation (synthetic but structured)

• For each table, they iterate through field pairs to create univariate charts (quantitative measure vs categorical/temporal dimension), using Vega-Lite/Altair-like constraints (e.g., max rows/cols for certain field types).
• Final dataset size and composition: 12,441 charts across 3,189 area, 6,238 bar, 3,014 line, split approximately 80:10:10 train/val/test.

Caption construction (semantic levels)

They use a 4-level caption taxonomy (L1–L4) adapted from accessibility guidance and prior work, focusing on L1–L3 for this benchmark.

• L1 generation (synthetic): Templates produce a first sentence (3 templates) plus follow-on sentences (26 combinations), with randomized synonym phrasing and optional omissions (e.g., dropping scale words or outlier mentions).
• L2/L3 collection (crowdsourced): Captions are written by crowd workers; they are instructed to produce multiple “non-repeating” captions per chart and are screened for vision/location/approval criteria.

Dataset analysis of semantic content

On a 2% sample (230 captions), they find most semantic statements are L2 or L3, with L3 appearing about 1.4× as often as L2 in that coding.

Algorithms / Training

Representation preprocessing (critical implementation detail)

Before training, they minimize and linearize both scene graphs and tables to reduce token length:

• Scene graphs are reduced by preserving specific elements (title, axis labels/ticks, marks, mark coordinates/sizes) and removing other details, then linearized with a depth-first traversal.
• The paper reports average representation lengths (for the “reduced” forms) on the order of 948 characters for scene graphs vs 426 characters for data tables.

Training setup (reported)

• Text models (ByT5): 50 epochs, batch size 1024, learning rate 5e-5, dropout 0.1.
• Image-guided VL-T5: 50 epochs, batch size 512, learning rate 2e-5.

Evaluation

Metrics

They use BLEU, ROUGE-L, WMD, TER, plus GPT-2-medium perplexity as a fluency proxy, and a “relation generation” check based on matching chart fields (title, axis names, etc.) mentioned in captions.

Main quantitative findings (Table 1)

On combined captions (L1+L2+L3), representative rows include:

• Prior baseline (Kantharaj et al. 2022): BLEU 0.30, perplexity 28.51.
• Text-only (scene graph, no PT): BLEU 0.34, perplexity 17.04.
• Text-only (data table, no PT): BLEU 0.34, perplexity 17.86.
• Image-only: BLEU 0.13, perplexity 51.65 (worst).

Overall narrative: image models underperform; scene graphs and data tables do well and are close to each other.

L1 vs L2/L3 difficulty

They explicitly report that models do much better on L1 than on L2/L3. For example, scene-graph model BLEU is 0.71 on L1 but about 0.06 on L2/L3 in their breakdown table.

Ablation observations

Backbone comparison suggests performance differences across ByT5-small, T5-small, and BART-base, with notes about prefix-tuning feasibility (they mention not being able to prefix-tune BART in their setup).

Qualitative error analysis (common failure modes)

In manual analysis of generated captions, they report broad error categories such as:

• Identity errors (wrong entity/label): 86 errors (22.93%).
• Direction errors (wrong trend direction): 32 errors (8.53%).
• Value errors (wrong numeric values): 12 errors (3.20%).
• Stability errors: 4 errors (1.07%).
• Repetition: 117 errors (31.2%).
• Nonsensical: 9 errors (2.4%).

Hardware / Production

Reported training hardware + runtime

They trained on 8× Titan XP (12 GB) GPUs, with 16 CPU cores and 128 GB RAM, reporting per-epoch training times on the order of minutes (varying by model and prefix-tuning).

Production status

No deployed system is claimed; the paper positions this as a benchmark and a step toward mixed-initiative authoring workflows and richer accessibility outputs rather than a production-ready captioning pipeline.

Paper: DePlot: One-shot visual language reasoning by plot-to-table translation
Authors: Fangyu Liu, Julian Eisenschlos, Francesco Piccinno, Syrine Krichene, Chenxi Pang, Kenton Lee, Mandar Joshi, Wenhu Chen, Nigel Collier, Yasemin Altun
Venue: Findings of ACL 2023 (July 2023)
Code: google-research/google-research (DePlot path)
Models: google/deplot
License: Apache-2.0 (code/models); mixed dataset licenses (GPL-3.0 for ChartQA, CC-BY-4.0 for PlotQA data; see detailed notes below)

TL;DR

DePlot decomposes chart question answering into (1) plot-to-table translation (image $\rightarrow$ linearized table) and (2) LLM reasoning over the translated table using one-shot prompts. They train an image-to-text Transformer (initialized from MatCha) for plot-to-table, introduce a table-matching metric (RMSF1) meant to be more structure-aware than prior number-set matching, and report large gains on human-written ChartQA when combining DePlot with LLM prompting.

What kind of paper is this?

• Dominant: $\Psi_{\text{Method}}$. The headline contribution is a new modality conversion module (DEPLOT) plus a plug-and-play pipeline to use LLMs for chart QA with one-shot supervision, with extensive benchmark results.
• Secondary: $\Psi_{\text{Evaluation}}$. They propose and validate a new metric (RMSF1) and run a human-correlation study to argue it better reflects extraction quality than RNSS.
• Secondary: $\Psi_{\text{Resource}}$ (weaker). They standardize task formats/metrics for plot-to-table and train on a combined corpus, but the core novelty is not a new dataset release.

(These labels follow the provided taxonomy.)

What is the motivation?

• End-to-end chart QA systems need large task-specific finetuning sets and still struggle on complex human-written questions (example given: MatCha at 38.2% on ChartQA human).
• LLMs have strong few-shot reasoning, but the missing piece is getting chart content into a form LLMs can reliably use without bespoke multimodal training.
• Prior chart extraction systems are often pipeline-based (OCR + rules + detection) and chart-type-specific, with inconsistent evaluation metrics.

What is the novelty?

• Two-stage decomposition:
  1. Convert chart image to a linearized table (markdown-style table text).
  2. Use an off-the-shelf LLM with one-shot prompting to answer questions from the table.
• DEPLOT model: an end-to-end image-to-text Transformer finetuned specifically for plot-to-table translation, intended to be chart-type-agnostic (line, dot, bar, pie).
• RMSF1 metric: a table similarity metric that (a) accounts for row and column headers plus values, (b) supports approximate matching for numeric/text errors, (c) exposes precision vs. recall, and (d) is invariant to row/column permutations and transposition.
• Prompting stack: evaluates Chain-of-Thought, Self-Consistency, and Program-of-Thought prompting (including executing generated Python for arithmetic).

What experiments were performed?

• Plot-to-table evaluation: PlotQA plot-to-table reconstruction compared against ChartOCR (pipeline), PaLI-17B finetuned variants, MatCha off-the-shelf, and DePlot (new finetune). Metrics: RNSS and RMSF1.
• Downstream QA: ChartQA (augmented + human) and PlotQA (v1 + v2) QA accuracy with 5% numeric tolerance; compares fully supervised models vs. DePlot+LLM one-shot variants.
• Metric validation: human study on 50 plot-table pairs with 6 annotators; correlates RNSS vs. RMSF1 with human ratings.
• OOD check: annotate 10 TaTa charts (excluding choropleths) to estimate generalization; report average RMSF1 and qualitative failure modes.

What are the outcomes and limitations?

Key outcomes

• ChartQA human: best reported DePlot+LLM setup reaches 67.6%, compared with MatCha 38.2% (reported as +29.4 absolute points).
• ChartQA augmented: DePlot+FlanPaLM+Codex PoT SC reported at 91.0%, comparable to strong supervised baselines (MatCha 90.2).
• PlotQA QA: DePlot+LLM underperforms MatCha on synthetic PlotQA (example: 66.6 vs. 91.5 average in their table), despite doing well on ChartQA human.
• Metric: RMSF1 correlates better with human judgments than RNSS (Pearson’s $r$ 0.87 vs 0.46; Spearman’s $\rho$ 0.96 vs 0.84).

Limitations and failure modes

• Loss of visual attributes: plot-to-table drops information like color, which breaks questions referencing “gray bar”, “red line”, etc. They show a concrete ChartQA failure where the LLM answers “Yes” for the wrong reason because the table has no color encoding.
• Synthetic-query bias: they argue PlotQA’s templatic synthetic questions can be exploited by supervised finetuning, which one-shot prompting cannot leverage.
• OOD robustness unclear: in a small TaTa sample, they observe distraction by adjacent text and trouble with arrow-linked labels; they mention cropping helps in that setting.
• Scope: they note DEPLOT is not suited to visual language without a clear latent textual structure (example: some textbook figures).

Reproducibility Details

Model

DEPLOT architecture and representation

• Model type: image-to-text encoder-decoder Transformer, trained autoregressively to emit a table left-to-right.
• Initialization: starts from MatCha weights and continues finetuning specifically for plot-to-table conversion.
• Table format: markdown-like linearization, using | separators between cells and newline between rows.
• Parameter count: DEPLOT 282M parameters (LLMs used are much larger: FlanPaLM 540B; GPT-3 and Codex reported around 175B).

DEPLOT + LLM interface

• Output table is appended with the question and a one-shot demonstration; they evaluate natural-language reasoning (CoT) and code-generation reasoning (PoT) plus self-consistency. Figure 1 illustrates the overall flow.

Data

Plot-to-table finetuning corpus

Three sources mixed 1:1:1 (only training splits used to reduce leakage into downstream eval):

• Synthetic plots from Liu et al. (MatCha pretraining pipeline): 270K plot-table pairs.
• ChartQA plot-table pairs: 22K.
• PlotQA plot-table pairs: 224K.

QA benchmarks and test sizes

• ChartQA: human and machine (augmented) sets, each reported as 1,250 QA pairs with 625 (human) and 987 (machine) tables.
• PlotQA: v1 and v2 each with 33K tables; QA pairs reported as 1.2M (v1) and 4.3M (v2).

Algorithms and training

Training

• Training steps: 10k steps.
• Max sequence length: 512 tokens (they note MatCha used 192; they extend to fit longer tables).
• Inference temperature for DePlot: 0 (deterministic decoding).

Prompting and decoding for LLMs

• LLM prompting temperature: 0.4 in their experiments.
• Self-consistency: majority vote across 10 samples (they also combine CoT and PoT samples in a joint voting scheme for best ChartQA results).
• Program-of-Thought: prompt Codex to emit Python snippets, then execute to compute the final answer; motivated by more reliable arithmetic.
• Prompts are shown in the appendix figures (Figure 3: CoT-style table reasoning; Figure 4: Python-code variant).

Evaluation

Plot-to-table metrics

RNSS (baseline)

• Treats predicted and target tables as unordered sets of numbers and matches them with a relative-distance cost, then normalizes by max set size.

They define relative distance: $$ D(p, t)=\min\left(1, \frac{|p-t|}{|t|}\right) $$ and compute a minimal-cost matching matrix $X$ over predicted set $P$ and target set $T$, with: $$ RNSS = 1 - \frac{\sum_i\sum_j X_{ij} D(p_i, t_j)}{\max(N, M)} $$ as reported in the paper.

RMSF1 (proposed)

• Represents a table as an unordered set of (row header, column header, value) triples.
• Matches entries by header similarity (normalized Levenshtein with thresholding) and value similarity (relative error with thresholding), then computes precision/recall and $F1$.
• Handles transposition by scoring both table orientations and taking the better score.

The core idea is that similarity is high only when both header keys and values align; unlike RNSS, this penalizes “right numbers, wrong structure”.

Plot-to-table results

On PlotQA plot-to-table reconstruction (their Table 4):

• ChartOCR: RNSS 81.0, RMSF1 60.1
• PaLI-17B (224 res): RNSS 77.2, RMSF1 24.8
• PaLI-17B (588 res): RNSS 90.5, RMSF1 74.9
• MatCha: RNSS 95.4, RMSF1 92.3
• DePlot: RNSS 97.1, RMSF1 94.2

The PaLI 224 vs 588 contrast is used to argue input resolution matters for chart extraction.

Downstream QA results

Key rows from their main results table (Table 5):

• MatCha: ChartQA aug 90.2, human 38.2, avg 64.2; PlotQA avg 91.5
• DePlot+FlanPaLM+Codex PoT SC: ChartQA aug 91.0, human 67.6, avg 79.3; PlotQA avg 66.6
• They also report intermediate variants (GPT-3 CoT/SC, FlanPaLM CoT/SC, Codex PoT SC) showing the effect of prompting and tool-use.

Error analysis highlights

• “Visual attribute” questions (color, shape, orientation) are a recurring failure because the table encoding omits those attributes. Their Table 7 example centers on not being able to identify “highest value of the gray bar” after translation.
• Another failure mode: imperfect alignment between plotted points and x-axis labels can cause incorrect table reconstruction (their Table 11 example).

Hardware and production

• Training hardware: 64 TPUv3 on GCP.
• Training time: roughly 5 hours for DEPLOT.

Release & Licensing Notes

What “other (synthetic) data” DePlot uses

• For plot-to-table training, DePlot trains on a 1:1:1 mix of:
  1. synthetic data generated by Liu et al. (2023a),
  2. synthetic data generated by Methani et al. (2020) (the same synthetic source used in PlotQA), and
  3. real-world data crawled by Masry et al. (2022) (the same source used in ChartQA, sourced from Statista / Pew Research / Our World in Data / OECD).
• In their Table 2 training stats, they explicitly list “synthetic (by us)” = 270K alongside ChartQA and PlotQA portions.
• They also note they use only training-set charts from ChartQA/PlotQA for training (to avoid leakage).

Was that synthetic data shared publicly, or just code/concept?

DePlot authors’ explicit release statement

• The DePlot paper explicitly points to “Code and models” (a repo link in footnote), but does not explicitly say “we release the 270K synthetic plot–table pairs” as a downloadable dataset.
• Their ethics statement says training/eval data are synthetic via rules or publicly available web data with permissive licenses—but that’s about the inputs’ provenance, not a clear statement that they redistribute the full synthetic corpus.

Best-supported takeaway: from the paper text alone, it looks like they publicly shared code + model artifacts, and they describe the synthetic data + sources, but they don’t clearly claim a public release of the “synthetic (by us) 270K” dataset itself.

If shared, what licenses apply?

ChartQA

• Availability: ChartQA dataset is distributed via the repo and also via a Hugging Face dataset.
• License: Repo lists GPL-3.0.

PlotQA

• License breakdown (explicitly stated):
  • Dataset: CC-BY-4.0
  • Models + code: MIT

DePlot (code/models)

• Paper statement: “Code and models” are provided via the google-research repo path referenced in the paper.
• Model (Hugging Face): Apache-2.0
• google-research repository (where DePlot code lives): repo license is Apache-2.0

Questions answered

1. Was the (synthetic) data shared publicly?

   • ChartQA / PlotQA: yes, they’re publicly distributed.
   • DePlot’s “synthetic (by us) 270K”: the paper does not clearly state that this synthetic corpus is released as a dataset download; it only explicitly calls out code + models.

2. Or just the code?

   • For DePlot: code + models are explicitly shared.

3. Or just the concept?

   • DePlot also fully describes the idea + training mixture, but the explicit “released artifact” callout is code + models.

4. If anything was shared, what license was attached to the data/code?

   • ChartQA: GPL-3.0
   • PlotQA: Dataset CC-BY-4.0; Models/code MIT
   • DePlot: Code in google-research under Apache-2.0; HF model card shows Apache-2.0

Open point (what I couldn’t verify from the paper text alone)

• Exact license (if any) for DePlot’s newly-generated “synthetic (by us) 270K” plot–table pairs isn’t stated in the paper sections we have—so I can’t responsibly claim a data license for that specific synthetic corpus without checking the repo/docs that accompany the code release.

Paper: Masry et al., UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoning (arXiv:2305.14761v3, 10 Oct 2023)
Code: vis-nlp/UniChart
Models: UniChart checkpoints referenced in the paper (via the GitHub repository)
License (corpus): Varies by source—see Section 3 for per-source licensing details

TL;DR

UniChart is an end-to-end chart vision-language pretrained model (chart image encoder + text decoder) trained on a large real-world chart corpus (611K charts) with multiple chart-specific pretraining objectives (table extraction, reasoning, open-ended QA, summarization). The authors use knowledge distillation and GPT-based summary generation to address the lack of high-quality chart summaries in real-world data. The model reports strong results across ChartQA, OpenCQA, Chart-to-Text, and Chart-to-Table, with efficiency gains of 11× speedup versus MatCha while using 28% fewer parameters.

What kind of paper is this?

$\Psi_{\text{Method}}$ 0.50, $\Psi_{\text{Resource}}$ 0.35, $\Psi_{\text{Evaluation}}$ 0.15

Dominant: Method (0.50) — The paper introduces a chart-specific vision-language pretraining approach with novel objectives (notably, bootstrapped summarization via GPT distillation) and demonstrates an end-to-end OCR-free architecture for chart comprehension.

Secondary: Resource (0.35) — The authors release a 611K chart pretraining corpus assembled from multiple real-world sources, along with ~470K GPT-distilled summaries and synthetic reasoning QA pairs (5.3M examples). The paper explicitly states the corpus and code are publicly available.

Tertiary: Evaluation (0.15) — The paper evaluates on four established benchmarks (ChartQA, OpenCQA, Chart-to-Text, Chart-to-Table) and includes human + ChatGPT-based evaluation for summarization quality, plus error analysis. However, it does not introduce new evaluation protocols or benchmarks.

Key Questions

1. What is the motivation?

The authors identify a gap: while chart understanding requires both low-level data extraction (table generation) and high-level reasoning/text generation (QA, summarization), existing work either (a) relies on external OCR engines, (b) trains primarily on synthetic charts, or (c) focuses narrowly on specific tasks. MatCha, a notable prior chart pretraining model, was trained largely on textual reasoning datasets (e.g., DROP, FeTaQA), which may limit visual reasoning capacity. UniChart aims to build a single “universal” chart model that handles multiple chart tasks without external OCR, trained on diverse real-world charts rather than synthetic-only data.

2. What is the novelty?

Key insight: The authors treat the lack of high-quality chart summaries as a data acquisition problem. They bootstrap summaries using GPT models—directly prompting ChatGPT for some sources, and distilling a Flan-T5 XL summary generator (trained on 3,700 GPT-generated examples) to produce ~470K summaries for charts lacking captions. This GPT-augmented pretraining corpus is then used for chart-to-text objectives.

Technical contributions:

1. Real-world chart corpus: 611K charts from diverse sources (OWID, OECD, Pew, Statista, PlotQA, etc.), explicitly prioritizing real over synthetic charts.
2. Four-task pretraining framework: data table generation, numerical/visual reasoning (90 templates), open-ended QA (T5-generated questions from summaries), and chart summarization (GPT-distilled).
3. Summary generation pipeline: 3,700-sample GPT dataset → finetune Flan-T5 XL → generate ~470K summaries; additionally use ChatGPT + OCR text for Pew charts without data tables.
4. End-to-end OCR-free architecture: Swin Transformer encoder + BART decoder, following Donut’s design principles but specialized for charts.

3. What experiments were performed?

The authors evaluate UniChart on four benchmarks:

1. ChartQA (relaxed accuracy): factoid question answering over bar, line, and pie charts.
2. OpenCQA (BLEU): open-ended question answering over similar chart types.
3. Chart-to-Text (BLEU, human eval, ChatGPT eval): summarization on Pew Research and Statista charts.
4. Chart-to-Table (RNSS, RMS): data extraction on ChartQA and WebCharts (zero-shot).

Additional evaluations:

• Human evaluation for summarization informativeness (4-level taxonomy: visual encoding, statistical/relational, perceptual/cognitive, contextual/domain) and factual correctness.
• ChatGPT-based evaluation as a proxy for human judgment.
• Efficiency comparison: inference speed and parameter count versus MatCha.

4. What are the outcomes?

Main results (Table 2 in paper):

Human evaluation (Table 3): UniChart zero-shot summaries scored highest in informativeness among model outputs (both human and ChatGPT ratings). Finetuned UniChart further reduces factually incorrect sentences versus MatCha (Table 7).

Efficiency: The authors report UniChart is 11× faster than MatCha with 28% fewer parameters (though absolute numbers are not provided in the cited sections).

5. What are the limitations and a good follow-up?

Limitations acknowledged by the authors:

• Overpopulated charts: Charts with many elements (e.g., dense legends, many bars) can confuse the model and reduce summary quality.
• Factual errors: Generated summaries can contain factual inaccuracies (error analysis example provided).
• OCR dependency for some data: Pew charts lack data tables, so the authors rely on layout-preserving OCR text fed to ChatGPT for summary generation, introducing potential OCR errors.

Assumptions baked into the approach:

• Pretraining uses large amounts of automatically constructed supervision (template-based reasoning QAs, synthetic open-ended QA from summaries, and GPT-generated summaries), assuming these transfer to real downstream distributions.
• The corpus-building stance explicitly excludes some synthetic chart datasets, which reduces control over chart type/style coverage but prioritizes realism.

Concrete follow-up experiment:

Ablate “summary source quality” versus downstream gains: Keep UniChart architecture fixed, but pretrain the summarization objective with (a) original dataset summaries, (b) ChatGPT-generated summaries, (c) Flan-T5-distilled summaries, then measure downstream changes on Chart-to-Text and OpenCQA, plus factual error rates (Table 7-style analysis). This directly tests whether the LLM-bootstrapped summaries are the causal driver of gains and where they help/hurt. Motivation: The authors explicitly replace some original summaries with ChatGPT-generated ones and produce ~470K via Flan-T5 XL distillation—isolating this design choice would clarify its impact.

Reproducibility Details

Model

Architecture:

• Chart image encoder: Swin Transformer with patch embeddings and shifted-window attention, following Donut’s encoder design.
• Text decoder: BART decoder; task-specific textual prompts are fed to the decoder, and output is generated conditioned on the image encoding + prompt (Figure 1 in paper).
• OCR-free: End-to-end design with no external OCR preprocessing.

Parameter count: The paper claims 28% fewer parameters than MatCha but does not state the absolute count in the cited sections.

Data

Pretraining corpus (Table 4): 611,934 charts from diverse sources:

• Our World in Data (OWID): CC-BY license
• Web Data Commons (WDC): Apache License (software)
• Pew Research Center: permissive license with attribution required
• OECD: downloadable/publishable with appropriate credit
• Statista: permissive license for scientific purposes
• Other publicly available datasets: PlotQA, Beagle, ChartInfo, ExcelChart400K, LineCap, Neural Captions (licenses per original releases)

Important: The paper does not specify a single unified license for the released corpus package; users must comply with terms of each underlying data source.

Pretraining supervision scale (Table 1):

Summary generation pipeline:

1. Created 3,700-sample dataset using in-context “table → caption” prompting.
2. Finetuned Flan-T5 XL on this dataset.
3. Used finetuned model to generate ~470K summaries for PlotQA, augmented charts, OWID, OECD.
4. Prompted ChatGPT (gpt-3.5-turbo) to generate summaries for Statista and Pew charts.
5. For Pew charts (no data tables), extracted layout-preserving OCR text and fed to ChatGPT.

Algorithms / Training

Pretraining schedule (Table 6):

• Stage 1: 300K steps at 512×512 resolution, LR 1e-4, batch size 160, checkpoint every 50K steps.
• Stage 2: 100K steps at 960×960 resolution, LR 1e-4, batch size 80, checkpoint every 50K steps.

Hardware: One 4×A100 (40GB), one 4×A100 (80GB), and one 4×V100 (32GB) machine.

Finetuning (per benchmark):

• ChartQA: 20 epochs, LR 5e-5
• Pew: 200 epochs, LR 5e-5
• Statista: 100 epochs, LR 5e-5
• OpenCQA: 200 epochs, LR 5e-5

Task-specific batch sizes and GPU configurations are provided in Table 6.

Evaluation

Metrics:

• ChartQA: Relaxed Accuracy (RA)
• OpenCQA, Chart-to-Text: BLEU (authors note limitations of BLEU for summarization)
• Chart-to-Table: RNSS (Relative Number Set Similarity), RMS (Relative Mapping Similarity)

Human evaluation criteria:

• Informativeness: 4-level semantic taxonomy (visual encoding, statistical/relational, perceptual/cognitive, contextual/domain)
• Factual correctness: count of factually incorrect sentences

ChatGPT evaluation: Used as a proxy for human judgment on summarization quality.

Reproducibility assessment:

• ✅ Datasets and code: Authors state corpus + code are publicly available via GitHub.
• ✅ Architecture: Encoder (Swin) and decoder (BART) clearly described; end-to-end design specified.
• ✅ Training hyperparameters: Pretraining/finetuning details (steps/epochs, LR, batch size, GPU type) provided in Table 6.
• ⚠️ Summary generation pipeline: High-level process described, but exact prompts for GPT models and OCR tool specifics not fully detailed.
• ✅ Compute requirements: GPU machines listed (A100 40GB/80GB, V100 32GB) with staged pretraining resolutions.

• Optimized Table Tokenization for Table Structure Recognition
• Code: Not publicly released
• Models: Not publicly released

TL;DR

OTSL (Optimized Table Structure Language) reduces table structure vocabulary from 28+ HTML tokens to 5 tokens (C, L, U, X, NL) with backward-only syntax rules enabling on-the-fly validation during autoregressive decoding. Evaluated in TableFormer on PubTabNet, FinTabNet, and PubTables-1M, OTSL achieves approximately $2\times$ inference speedup versus HTML while maintaining or improving tree edit distance (TEDs) and cell mAP@0.75, with particularly large gains on FinTabNet (all-TEDs 0.959 vs 0.920; mAP 0.862 vs 0.722).

What kind of paper is this?

• Dominant: $\Psi_{\text{Method}}$ — Proposes a novel tokenization language with syntactic constraints for table structure recognition.
• Secondary: $\Psi_{\text{Evaluation}}$ — Compares HTML versus OTSL representations across multiple datasets, model configurations, and metrics (TEDs, mAP, latency).
• Minor: $\Psi_{\text{Resource}}$ — States intent to release popular TSR datasets converted to OTSL format.

What is the motivation?

Image-to-Markup-Sequence (Im2Seq) table structure recognition typically reuses general-purpose HTML tokenization, which was not designed for autoregressive decoding efficiency. HTML presents several challenges:

• Large vocabulary: Requires at least 28 tokens to cover common rowspan/colspan attributes; skewed token frequency distribution complicates learning.
• Variable row lengths: Rows with complex spanning produce longer token sequences, making positional encoding and attention less effective.
• Late error detection: Invalid HTML outputs are difficult to detect early during generation; partial sequences often violate structural consistency but remain syntactically valid markup.
• Attention drift: Long sequences on large tables cause output misalignment, particularly in later rows; bounding box predictions degrade.

What is the novelty?

OTSL representation:

• 5-token vocabulary representing a rectangular grid:
  • C: new cell (anchor for cell region top-left)
  • L: merge with left neighbor (horizontal span continuation)
  • U: merge with upper neighbor (vertical span continuation)
  • X: merge with both left and upper (2D span interior)
  • NL: end-of-row marker
• Fixed-width rows: All rows have equal token count, each terminated with NL, regardless of spanning complexity.
• Backward-only syntax rules: Each token can be validated using only previously generated tokens, enabling incremental constraint enforcement during decoding:
  1. Left neighbor of L must be C or L
  2. Upper neighbor of U must be C or U
  3. Left neighbor of X must be U or X; upper neighbor must be L or X
  4. First row allows only C and L
  5. First column allows only C and U
  6. All rows have equal length, terminated by NL

Error mitigation: Invalid token predictions signal decoding errors; one proposed heuristic replaces the highest-confidence invalid token with the next-highest valid candidate until syntax rules are satisfied.

Efficiency claim: Example comparison (Figure 1) reports 12 HTML tokens versus 5 OTSL tokens for vocabulary; 55 HTML tokens versus 30 OTSL tokens for sequence length on the same table structure.

What experiments were performed?

Architecture: TableFormer (encoder-decoder transformer for Im2Seq TSR) with separate structure and bounding box decoders.

Hyperparameter sweep (Table 1): Compared HTML versus OTSL on PubTabNet with encoder/decoder layer variations:

• enc=6, dec=6, heads=8
• enc=4, dec=4, heads=8
• enc=2, dec=4, heads=8
• enc=4, dec=2, heads=8

Metrics: TEDs (split by simple/complex/all tables), mAP@0.75 for cell boxes, single-core CPU inference time (AMD EPYC 7763 @ 2.45 GHz).

Cross-dataset evaluation (Table 2): Selected best configuration (enc=6, dec=6, heads=8) and trained/evaluated on:

• PubTabNet: 395K samples
• FinTabNet: 113K samples
• PubTables-1M: ~1M samples

Same metrics as hyperparameter sweep; OTSL outputs converted back to HTML for TEDs computation.

Qualitative analysis: Figures 5-6 show bounding box predictions on sparse and many-row complex tables, comparing token counts and visual alignment quality.

What are the outcomes/limitations?

Outcomes:

Latency: Approximately $2\times$ speedup across configurations:

• PubTabNet (enc=6, dec=6): 2.73s (OTSL) vs 5.39s (HTML)
• FinTabNet: 1.85s (OTSL) vs 3.26s (HTML)
• PubTables-1M: 1.79s (OTSL) vs 3.26s (HTML)

Accuracy:

• PubTabNet: Similar all-TEDs (0.955 for both); improved mAP (0.880 vs 0.857)
• FinTabNet: Large gains reported — all-TEDs 0.959 vs 0.920; mAP 0.862 vs 0.722
• PubTables-1M: Gains on both metrics — all-TEDs 0.977 vs 0.966; mAP 0.896 vs 0.889

Qualitative: Reduced bounding box drift and overlap on long/sparse tables compared to HTML; HTML sometimes fails to terminate correctly or shows misalignment in later rows.

Limitations:

• Syntactic validity ≠ structural correctness: Valid OTSL sequences can still represent incorrect table structures; syntax rules only enforce grid consistency.
• Heuristic error correction: Token replacement strategy is not formally evaluated; no comparison with beam search or constrained decoding alternatives.
• Single architecture family: Evidence limited to TableFormer; transfer to object detection + graph neural network TSR pipelines not demonstrated.
• Dataset release unconfirmed: OTSL-converted datasets stated as “will be made publicly available” but release details not provided in paper.
• Training details omitted: GPU type/count, training duration, hyperparameter search methodology not specified; limits reproducibility.

Model

Task Framing

• Input: Table image
• Output: Autoregressive token sequence representing table structure (HTML baseline vs OTSL)

TableFormer Integration

• Structure decoder generates structure tags (HTML or OTSL)
• Separate decoder predicts table cell bounding boxes
• Architecture: encoder-decoder transformer with configurable depth (2-6 encoder layers, 2-6 decoder layers, 8 attention heads in reported experiments)

Data

Datasets

Representation Conversion

Ground truth from all datasets converted to OTSL format for training/evaluation. Predicted OTSL sequences converted back to HTML to compute tree edit distance metrics against original HTML ground truth.

Benchmark Licensing

Evaluation benchmarks use publicly available datasets with permissive licensing:

All three benchmarks distribute annotations under Community Data License Agreement (CDLA) Permissive terms, which allow broad use, modification, and sharing. Source images may have separate licensing (particularly PubMed Central Open Access images in PubTabNet); users should review upstream provenance for complete rights assessment.

Algorithms / Training

OTSL Language Definition

Tokens: {C, L, U, X, NL}

Semantics: Each table cell region has a C token at its top-left anchor position. Other grid locations within the cell’s span are filled with L (horizontal continuation), U (vertical continuation), or X (2D interior) according to adjacency.

OTSL Syntax Rules

1. Left-looking: Left neighbor of L must be C or L
2. Up-looking: Upper neighbor of U must be C or U
3. Cross rule: Left neighbor of X must be U or X; upper neighbor must be L or X
4. First row: Only C and L allowed
5. First column: Only C and U allowed
6. Rectangular: All rows equal length, terminated by NL

Error Detection and Mitigation

• During generation, validate each token against backward-checkable syntax rules
• Invalid token indicates decoding error
• Proposed heuristic: if highest-confidence token violates rules, replace with next-highest confidence valid token
• No beam search or formal constrained decoding comparison provided

Evaluation

Metrics

• TEDs (Tree Edit Distance): Structural accuracy, reported separately for simple/complex/all tables after OTSL-to-HTML conversion
• mAP@0.75: Mean average precision at 0.75 IoU threshold for cell bounding boxes
• Inference time: Single-core CPU latency (AMD EPYC 7763 @ 2.45 GHz)

Quantitative Results

Table 1: PubTabNet hyperparameter sweep (OTSL vs HTML)

Table 2: Cross-dataset evaluation (enc=6, dec=6, heads=8)

Qualitative Observations

• Figure 5 (sparse table): OTSL produces cleaner bounding box alignment with less overlap; HTML shows drift; token count 258 (HTML) vs 135 (OTSL)
• Figure 6 (many-row complex table): OTSL captures repeating horizontal merge pattern and completes sequence correctly; HTML misses merges, ends with incorrect termination, shows drift/overlap

Hardware / Production

• Inference timing: Single CPU core (AMD EPYC 7763 @ 2.45 GHz) for all reported experiments
• Training compute: GPU type, count, and wall-clock training duration not specified

Paper: Rahman et al., ChartSumm: A Comprehensive Benchmark for Automatic Chart Summarization of Long and Short Summaries (arXiv:2304.13620v3)
Code/Dataset: pranonrahman/ChartSumm, Google Drive
License: Unspecified (no LICENSE file found in repository; treat reuse rights as not granted)

TL;DR

ChartSumm is a chart-to-text benchmark dataset with 84,363 chart samples, each with chart images, metadata, and paired summaries spanning short system-generated summaries (Knoema) and longer descriptive human-written summaries (Statista). The authors benchmark T5 and BART variants and report that while models can produce fluent summaries, they often fail on factual correctness, trend description, and exhibit hallucination.

What kind of paper is this?

$\Psi_{\text{Resource}}$ 0.70, $\Psi_{\text{Evaluation}}$ 0.30

Dominant: Resource (0.70) — The headline contribution is a new large-scale benchmark dataset for chart summarization with defined splits, dual summary regimes (short vs. long), and documented chart types/topics. The authors explicitly release the dataset and code.

Secondary: Evaluation (0.30) — The paper includes baseline benchmarking with T5 and BART variants, cross-dataset generalization comparisons (Chart-To-Text), manual error analysis of 100 sampled generations, and multilingual exploration (Bengali). However, it does not introduce novel evaluation protocols or metrics.

Key Questions

1. What is the motivation?

Chart summarization benefits visually impaired users and improves information retrieval by converting chart/tabular insights into natural language. The field is data-constrained: prior datasets are limited in size, coverage, or summary quality/type (e.g., captioning-only or template-generated descriptions). ChartSumm aims to address this by providing a larger, more varied benchmark with both short and long summaries from diverse chart types and topics.

2. What is the novelty?

Dataset scale and dual-summary regimes:

• 84,363 charts with images, metadata, and summaries from two sources:
  • Knoema: 43,179 charts with short descriptions generated by its “digital data assistant” (Yodatai), primarily year-indexed line charts
  • Statista: 41,184 charts with human-written descriptive summaries; includes multiple chart types with “simple vs. complex” categorization

Test set design for summary length:

• test-k: From Knoema, featuring “precise and well structured” shorter summaries
• test-s: From Statista, featuring longer descriptive summaries

Multilingual exploration:

• Bengali expansion via machine translation with human-translated test set and mT5 baseline

Key distinction from prior work: The dual-summary regime allows evaluation of models across different summary styles (short/precise vs. long/descriptive), which prior datasets did not systematically address.

3. What experiments were performed?

Baselines:

• T5-Base
• BART-Base, BART-Large-CNN, BART-Large-XSUM

Training regimes:

• Fine-tune on ChartSumm full dataset, ChartSumm-K (Knoema subset), and ChartSumm-S (Statista subset)
• Compare against models fine-tuned on Chart-To-Text (Kantharaj et al., 2022)
• Evaluate on multiple test sets including Chart-To-Text Statista test split

Metrics:

• BLEU, BLEURT (base-128), CIDEr
• Perplexity (using pretrained GPT-2)
• Content Selection (CS; Wiseman et al., 2017)

Error analysis:

• Manual review of 100 sampled generations, categorizing common failures: factual errors (wrong numbers/units), wrong trend descriptions, uninformative summaries, hallucinated irrelevant facts

4. What are the outcomes?

Main results (as reported):

• BART-Large variants tend to lead on BLEURT, CIDEr, and CS metrics
• T5 tends to achieve best perplexity across tests
• Models fine-tuned on ChartSumm-S reportedly outperform Chart-To-Text trained baselines even on the Chart-To-Text test set, suggesting stronger generalization
• Models trained on Chart-To-Text transfer poorly to ChartSumm-K (short/precise style), indicating Chart-To-Text is less suited for structured summaries

Limitations and failure modes:

• Critical issue: Even fluent outputs often contain wrong facts/units (e.g., “million” vs. “billion”), incorrect trend interpretation, and hallucinated attributes not grounded in chart metadata
• Evaluation gap: Primarily automatic-metric-driven; qualitative error analysis reveals that BLEU/BLEURT scores do not correlate well with factual correctness
• Generalization concerns: The authors acknowledge that models trained on one summary style (short vs. long) do not transfer well to the other

Contrast to UniChart: While UniChart (2023) also addresses chart-to-text tasks, ChartSumm focuses on providing a benchmark with dual summary styles and explicit error categorization, whereas UniChart emphasizes pretraining on diverse chart tasks (table extraction, reasoning, QA, summarization) with GPT-distilled summaries.

5. What are good follow-ups?

Concrete experiment: Factual consistency as a primary metric

Establish a factual consistency evaluation protocol (e.g., using structured claim extraction + verification against chart data tables) and re-rank model outputs by factual correctness rather than fluency-oriented metrics. This directly addresses the paper’s core finding that automatic metrics miss critical factual errors.

Motivation: The manual error analysis reveals that the highest-scoring outputs by BLEU/BLEURT often contain factually incorrect numbers or trends. A follow-up that prioritizes factual grounding would better align evaluation with the downstream task requirements (especially for accessibility applications where accuracy is critical).

Reproducibility Details

Model

Task formulation:

• Table-to-text summarization using chart metadata
• Input: chart metadata (title + data table + labels)
• Table is flattened row-wise and concatenated with caption/title using separator tokens
• T5-style prompting uses prefix: “Summarize chart: "

Baselines:

• T5-Base
• BART-Base
• BART-Large-CNN
• BART-Large-XSUM

Data

Sources and composition:

ChartSumm contains 84,363 charts from two sources:

1. Knoema (43,179 charts):

   • Crawled ~110,000 statistics, filtered to publicly available sources
   • Charts are year-based line charts
   • Summaries: short descriptions from Knoema’s digital assistant (Yodatai)

2. Statista (41,184 charts):

   • Crawled ~750,000 pages
   • Charts categorized as simple vs. complex based on column count
   • Summaries: descriptive human-written text

Chart type distribution (Statista):

• Bar: 64.70%
• Line: 33.76%
• Pie: 1.54%

Topic coverage (via LDA):

• Economy & Politics: 21.60%
• Society & Science: 13.03%
• Internet & Media: 11.43%
• Public life & Health: 10.42%
• Sports & Entertainment: 9.14%
• Consumer Goods: 7.71%
• Retail & Trade: 5.35%
• Education: 5.32%

Preprocessing:

• Tokenize title/caption, remove whitespace/newlines via stemming
• Normalize numeric entities
• Missing x-axis labels assigned heuristically (Year/Month/Day/Quarter/Country/City/Area)
• NER-based types for companies, social media, etc.
• Chart type classification uses ChartReader

Splits (80/10/10):

Overlap with Chart-To-Text:

• Overlap defined as >90% token similarity in captions
• Reported overlaps (Statista samples): 5,338 total (4,144 simple, 1,194 complex)

Dataset statistics:

Algorithms / Training

Fine-tuning setup (English baselines):

• Epochs: 3
• Batch size: 8
• Initial learning rate: $1 \times 10^{-6}$
• Optimizer: AdamW
• Loss: Cross-entropy
• Implementation: HuggingFace Transformers
• Compute: Google Colab (specific GPU type not specified)

Bengali experiment (multilingual):

• Translation:
  • Train/validation: machine-translated using NLLB
  • Test: human-translated by undergraduate students proficient in English and Bengali
• Model: mT5 pretrained on multilingual XL-SUM
• Fine-tuning:
  • Epochs: 4
  • Batch size: 8
  • Initial learning rate: $1 \times 10^{-6}$
  • Optimizer: AdamW
  • Loss: Cross-entropy
• Evaluation: BLEU only (BLEURT/CIDEr not used due to lack of Bengali-specific models)

Evaluation

Metrics (English):

• BLEU (n-gram overlap)
• BLEURT (base-128)
• CIDEr
• Perplexity (via GPT-2)
• Content Selection (CS; Wiseman et al., 2017)

Manual error analysis (100 samples):

Categories identified:

• Wrong facts/units (e.g., “million cubic meters” vs. gold “billion cubic meters”)
• Incorrect trend interpretation
• Uninformative outputs (restates framing without key numbers/trends)
• Hallucinated unrelated details (e.g., company headquarters not in chart/metadata)

Key finding: Automatic metrics (BLEU, BLEURT) do not reliably correlate with factual correctness. High-scoring outputs can contain critical factual errors.

Hardware / Production

• English experiments conducted in Google Colab
• Detailed hardware specs (GPU type, wall-clock training time) not specified in paper

Reproducibility Assessment

• ✅ Dataset available: Released via GitHub + Google Drive with documented structure
• ⚠️ License unclear: No LICENSE file in repository; reuse rights not explicitly granted
• ✅ Baseline code: Available in GitHub repository
• ✅ Training hyperparameters: Fully specified (epochs, batch size, LR, optimizer)
• ⚠️ Hardware details: Limited (only “Google Colab” mentioned; no GPU type or training time)
• ✅ Evaluation protocol: Metrics clearly documented; error analysis methodology described
• ⚠️ Preprocessing details: High-level description provided; exact heuristics for label assignment may require code inspection

Paper: ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning (Findings of ACL 2022)
Authors: Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, Enamul Hoque
Code: vis-nlp/ChartQA
Data: vis-nlp/ChartQA
License: GPL-3.0

TL;DR: ChartQA is a benchmark for question answering over real-world charts requiring visual references (color, position, size) and multi-step logical or arithmetic reasoning. It contains 9,608 human-written questions and 23,111 machine-generated questions over 20,882 charts from Statista, Pew Research, OWID, and OECD. The paper evaluates a pipeline combining chart data extraction (extended ChartOCR) with TableQA-style transformers augmented with image features (VL-T5, VisionTaPas), showing that VisionTaPas achieves 61.12% accuracy on human questions with gold tables but drops to 44.33% with extracted tables.

What kind of paper is this?

• Dominant: $\Psi_{\text{Resource}}$ 0.60
• Secondary: $\Psi_{\text{Method}}$ 0.25, $\Psi_{\text{Evaluation}}$ 0.15

The primary contribution is a new large-scale benchmark with real-world charts and human-authored questions. The paper also introduces models (VisionTaPas) and provides detailed evaluation, but these serve to establish baselines for the benchmark.

What is the motivation?

Existing chart question answering datasets and models under-serve:

• Complex reasoning questions requiring multiple operations (difference, sum, max, etc.)
• Questions that refer to visual attributes (e.g., “orange line”, “rightmost bar”)
• Human-authored language with natural variation, not small template sets
• Real-world chart styles with diverse layouts and visual complexity

Prior datasets (DVQA, PlotQA, FigureQA) are largely synthetic and/or template-based, limiting their ability to evaluate systems on realistic chart understanding tasks.

What is the novelty?

Benchmark contributions:

• ChartQA-H (human-authored): 9,608 questions collected via Amazon Mechanical Turk, with workers explicitly asked to write compositional and visual questions requiring multi-step reasoning or visual references
• ChartQA-M (machine-generated): 23,111 questions generated from human-written Statista chart summaries using a two-stage T5 pipeline (answer extraction + answer-aware question generation), then filtered for answerability
• Real-world chart diversity: 20,882 charts from four sources (Statista, Pew, OWID, OECD) covering multiple domains and chart types
• Open-vocabulary answers: Unlike prior work with closed-vocabulary or template-based answers, ChartQA requires generating free-form text or numeric answers

Modeling contributions:

• VisionTaPas: extends TaPas with a cross-modality encoder that fuses ViT image features with TaPas table encodings via cross-attention blocks
• Operation extension: adds SUBTRACT and DIVIDE operations to TaPas (which originally supported SUM, COUNT, AVERAGE) to handle difference and ratio questions
• Extended ChartOCR: adapts ChartOCR to output fully-structured data tables (not just mark values) by adding text recognition with CRAFT and associating values to labels using positional and color information

What experiments were performed?

Dataset construction:

• ChartQA-H annotation: AMT workers write 2 questions + answers per chart; second annotator answers the same questions; manual resolution on disagreements
• ChartQA-M generation: fine-tune T5 on SQuAD, apply to Statista summaries; filter questions whose answers don’t appear in chart data; manual check of 1,250 pairs found 86.64% valid

Models evaluated:

• T5: flatten table + question into text sequence; generate answer
• TaPas: table encoder with row/column embeddings; heads for aggregation + cell selection
• VL-T5: T5 with visual mark features from Mask R-CNN (36 objects)
• VisionTaPas: TaPas extended with ViT image encoder and cross-modality fusion via cross-attention

Experimental conditions:

• Gold tables: use ground-truth data tables extracted from chart sources
• Extracted tables: use tables predicted by their extended ChartOCR pipeline

Ablations:

• Impact of operation extension (SUBTRACT + DIVIDE) on TaPas and VisionTaPas
• Comparison of gold vs. extracted tables to isolate extraction quality impact

What are the outcomes/limitations?

Main results (relaxed accuracy: non-numeric exact match, numeric within 5% relative error):

Key findings:

• VisionTaPas outperforms all baselines on both human and machine-generated questions when using gold tables
• Accuracy drops significantly with extracted tables: VisionTaPas goes from 61.12% to 44.33% on ChartQA-H, indicating data extraction is a major bottleneck
• Operation extension matters: adding SUBTRACT + DIVIDE improves VisionTaPas from 65.19% to 74.47% on ChartQA-M (gold table)
• Data extraction accuracy: overall extraction accuracy on ChartQA reported as 83.85% using a normalized distance metric with linear assignment

Question type distribution (sample of 300 human questions):

76.33% of questions require compositional reasoning or combine visual + compositional reasoning.

Limitations called out by the authors:

• Extraction brittleness: modular pipeline breaks when table extraction fails; motivates end-to-end approaches
• Separate representations: current approach combines table and vision “separately then combine”; authors propose semantic graph representations to better capture chart structure
• Nested reasoning difficulty: even with correct extraction, models struggle with multi-step computations (e.g., subtraction then sum over multiple years)
• Extraction metric limitations: their adapted ChartOCR metric ignores noisy chart text (tick labels) and needs refinement

Reproducibility Details

Data

Chart sources:

• Statista: crawled publicly available charts
• Pew Research: crawled charts (images only, no underlying tables available)
• Our World in Data (OWID): crawled charts with underlying data
• OECD: crawled charts with underlying data

What they store:

• Chart images
• Underlying data tables (when available)
• Metadata: title, type, source
• SVG files (when available, used to extract bounding boxes for training extraction models)
• Text descriptions (for machine-generated question synthesis)

Dataset splits:

Chart type distribution (Statista-M subset, Table 3):

• Bar: 15,223
• Line: 1,768
• Pie: 150

Human annotation procedure (ChartQA-H):

1. AMT workers write 2 questions + answers per chart, focusing on compositional and visual questions
2. Second annotator answers the same questions independently
3. Manual resolution on disagreements
4. Agreement metrics: exact-match 61.04%; manual check on 500 samples yields 78.55% when accounting for typos/lexical variation
5. Compensation: $0.6 per task (estimated 3–5 minutes), relative to US minimum wage ($7.25/hour at time of study)

Machine augmentation procedure (ChartQA-M):

1. Fine-tune T5 on SQuAD
2. Apply to Statista chart summaries:
   • Answer extraction model: generate candidate answers from summary
   • Answer-aware question generation model: conditioned on (answer + summary)
3. Filter: remove questions whose answer is not found in chart data table
4. Quality check: manual verification of 1,250 generated pairs found 86.64% valid
5. Test set manually cleaned to ensure quality

Model

VisionTaPas Architecture

Extends TaPas with cross-modal fusion:

• ViT encoder: processes image patches (standard Vision Transformer)
• TaPas encoder: processes question + flattened table with row/column embeddings
• Cross-modality encoder: 4 blocks, each containing:
  • Bidirectional cross-attention (image ↔ table)
  • Self-attention layers
  • Feed-forward layers
  • Residual connections
• TaPas heads: aggregation head (selects operation: SUM, COUNT, AVERAGE, SUBTRACT, DIVIDE) and cell selection head

VL-T5 Visual Features

• Train Mask R-CNN (ResNet-101 backbone) to detect chart marks
• Object categories: xAxisTitle, yAxisTitle, legend, label, bar, pie, line, etc.
• Pad object features to 36 objects (fixed size)
• Feed visual features alongside text into T5

Operation Head Extension

• Original TaPas supports: SUM, COUNT, AVERAGE
• Extension adds: SUBTRACT, DIVIDE
• Supervision: heuristic rules for selecting operand cells
• Noise level: manual check of 100 questions found 24% noisy labels

Data Extraction Pipeline

Extends ChartOCR to output structured tables:

1. Value detection: detect numeric values on chart (ChartOCR baseline)
2. Text recognition: add CRAFT detector to recognize axis labels, legend text, tick labels
3. Value-to-label association: use positional information (x/y coordinates) and color matching to link values to semantic labels
4. Table construction: assemble into structured rows and columns

Training

All training on 1 Tesla P100 GPU.

All models fine-tuned from pretrained checkpoints (TaPas-Base, T5-Base, VL-T5-Base).

Evaluation

QA metric (relaxed accuracy):

• Non-numeric answers: exact string match after normalization (lowercase, strip whitespace, remove articles)
• Numeric answers: correct if within 5% relative error of ground truth

Data extraction metric (adapted from ChartOCR):

• Compute cost based on normalized distance between ground-truth and predicted values
• Solve minimum-cost linear assignment between gt and predictions
• Overall score: 1 minus average normalized cost
• Limitation noted by authors: metric ignores noisy chart text (tick labels, axis titles); better metrics needed

Error analysis (by question type):

The authors perform manual error analysis on a sample of failures, identifying:

• Extraction errors: incorrect or missing values in extracted table
• Reasoning errors: model fails to perform correct operation sequence even with correct extraction
• Visual grounding errors: model cannot resolve visual references (e.g., “rightmost bar”)

Ethical Considerations

Data sourcing:

• Used publicly available charts under source terms (Statista, Pew, OECD, OWID)
• Authors state they complied with each source’s terms of service
• AMT annotators anonymized

Annotator compensation:

• Estimated 3–5 minutes per task
• Paid $0.6 per task
• Authors frame relative to US minimum wage at time ($7.25/hour)

Potential for misuse:

• Authors explicitly note models could be abused to mislead about chart content or manipulate chart-based arguments
• Recommend caution in deployment for public-facing applications without human oversight

Practical Takeaways

If we were to use ChartQA:

• Best for: evaluating chart QA systems on real-world language with explicit visual references and multi-step reasoning
• ChartQA-H is the gold standard: human-authored questions with natural language variation, typos, synonyms
• ChartQA-M is larger: useful for pretraining or augmentation, but slightly lower quality (86.64% valid)
• Extraction quality is critical: performance drops ~17 absolute points when using extracted vs. gold tables, so any deployment would need robust extraction
• Operation space matters: if your questions involve arithmetic (differences, ratios), extending operation heads significantly improves TaPas-style models (+9 points for VisionTaPas on ChartQA-M)

Debugging strategy:

• Separate extraction errors from reasoning errors
• The authors’ results show extraction accounts for roughly half the error gap
• Test with gold tables first to isolate model reasoning capabilities

Chart types:

• Benchmark heavily skewed toward bar charts (>88% of Statista-M)
• Line and pie charts underrepresented
• Results may not generalize to other chart types (scatter, heatmap, etc.)

License considerations:

• GNU GPL-3.0 applies to code and dataset files
• Underlying charts from third-party sources have their own terms
• For commercial use, review source terms (Statista especially restrictive)

• Syntax-Aware Network for Handwritten Mathematical Expression Recognition
• HME100K Dataset (download portal)
• Not publicly released

TL;DR

Grammar-constrained decoder for handwritten mathematical expression recognition using syntax-aware attention and stack-based tree expansion. Achieves 53-56% exact match on CROHME benchmarks and 67% accuracy on HME100K, a new 100K-image dataset with complex camera-captured handwriting.

What kind of paper is this?

• Dominant: $\Psi_{\text{Method}}$ — Proposes grammar-driven parse tree decoding with syntax-aware attention and stack-based traversal for handwritten mathematical expression recognition.
• Secondary: $\Psi_{\text{Resource}}$ — Introduces HME100K, a 100k-image dataset collected from approximately 10,000 writers.
• Secondary: $\Psi_{\text{Evaluation}}$ — Defines expression-level accuracy and structure-only protocol (ESPR) with ablations on grammar and attention.

What is the motivation?

Encoder-decoder systems for handwritten mathematical expression recognition (HMER) often predict character-by-character, which produces structural errors on 2D math layouts and messy handwriting. Even tree-based decoders can behave like sequential models without explicit grammar constraints. The paper addresses this by embedding syntax constraints directly into the decoding process, predicting components and subtrees according to syntactic relationships rather than next-token likelihood.

What is the novelty?

• Grammar formulation: Converts LaTeX into a parse tree with constraints following reading order (left-to-right, top-to-bottom) and spatial relations between symbols.
• Tree expansion decoding: Predicts production rules to expand non-terminals while traversing the tree with a stack-based algorithm.
• Syntax-aware attention: Accumulates attention only along the path from root to current node, reducing drift between unrelated components.
• Attention self-regularization: Uses a reversed decoder to predict parent nodes and regularizes forward vs. reversed attention with a KL term during training.

What experiments were performed?

The model is evaluated on CROHME 2014, 2016, 2019 (offline images rendered from InkML strokes) and HME100K (camera-captured handwriting). Metrics include ExpRate (exact expression match), relaxed ExpRate $\leq 1$ and $\leq 2$ (tolerating one or two symbol-level errors), and ESPR (structure correct regardless of symbol labels). Comparisons are made against prior HMER systems including DWAP-TD and BTTR. Ablations isolate the effects of grammar syntax and syntax-aware attention.

What are the outcomes/limitations?

Outcomes:

• On CROHME 2014/2016/2019, SAN achieves the highest ExpRate among non-augmented methods and reports further gains with data augmentation.
• On HME100K, SAN outperforms DWAP, DWAP-TD, and BTTR on overall accuracy and the hard subset (51.5% vs. 45.4-46.0%), with higher inference speed (23.9 FPS vs. 6.9-23.3 FPS on V100).
• Ablations show grammar rules contribute the majority of improvement, with additional gains from syntax-aware attention.

Limitations:

• Distorted or overlapping components can cause under-translation or over-translation errors.

Model

Grammar Formulation

SAN defines a grammar $G = (N, \Sigma, R, S, \Gamma, C, D)$ with non-terminals $N$, terminals $\Sigma$, production rules $R$, start symbol $S$, spatial relations $\Gamma$, encoder $C$, and decoder $D$.

Spatial relations: The system uses 7 relations (right, above, below, low right, upper left, upper right, inside) derived from 9 base relations by removing redundant directions under reading-order constraints.

Non-terminals: $S$ (expression) and $E$ (extendable structure with relation slots).

Production rules:

1. $S \rightarrow \sigma S \mid E \mid \epsilon$ where $\sigma \in \Sigma$
2. $E \rightarrow [((\gamma_1)S \mid \epsilon), \ldots, ((\gamma_7)S \mid \epsilon)]$ where $\gamma_i \in \Gamma$

The grammar generates parse trees where leaves are terminals or relations and internal nodes are non-terminals. LaTeX is recovered via preorder traversal.

Encoder

A DenseNet backbone processes grayscale input $X \in \mathbb{R}^{1 \times H \times W}$ to produce a feature map of size $C \times \frac{H}{\zeta} \times \frac{W}{\zeta}$, flattened to $E(X) = [e_1, \ldots, e_L]$ where $e_i \in \mathbb{R}^{C}$ and $L = \frac{H}{\zeta}\frac{W}{\zeta}$. Implementation uses $C = 684$ and $\zeta = 16$.

Decoder

Two GRUs with syntax-aware attention process the parse tree. For each node $\alpha$, the context state includes:

• Historical state $c_h^\alpha$: how $\alpha$ was produced
• Partner state $c_p^\alpha$: embedding of the latest terminal symbol or relation

GRU-$\alpha$ computes $c_o^\alpha = \text{GRU}(c_p^\alpha, c_h^\alpha)$. Attention produces compact visual feature $\Omega = \text{Att}(E(X), c_o^\alpha, \text{att}_\alpha(X))$. GRU-$\beta$ computes $c_\beta^\alpha = \text{GRU}(\Omega, c_o^\alpha)$.

Two output branches:

• Symbol branch: Softmax over $|\Sigma| + 2$ (terminals, $E$, empty)
• Relation branch: Sigmoid over 7 relations

Decoding logic:

1. If terminal $\sigma$ selected: apply $S \rightarrow \sigma S$
2. If $E$ selected: apply relation branch; keep relations with probability $> 0.5$; create corresponding $(\gamma)S$ children
3. If empty selected: apply $S \rightarrow \epsilon$

Syntax-Aware Attention

Standard attention weights:

$$ \xi^\alpha = \text{softmax}\left(W_w \tanh(W_o c_o^\alpha + W_\alpha \text{att}_\alpha(X) + W_e E(X))\right) $$

Key modification: instead of summing over all past steps, accumulate only along the root-to-node path:

$$ \text{att}\alpha(X) = \sum{i \in \text{path}_\alpha} \xi^i $$

This reduces attention drift between structurally unrelated components.

Inference

Stack-based traversal:

1. Encode image and initialize stack with start symbol $S$
2. While stack non-empty:
   • Pop node and context
   • Compute production rule probabilities
   • Select highest-probability rule
   • Push produced non-terminals/relations with updated contexts
   • Update parse tree

Data

CROHME (2014, 2016, 2019)

The paper uses CROHME as the primary benchmark, converting InkML strokes to offline images. Training set: 8,836 expressions with 101 symbol classes. Test sizes: 986 (2014), 1,147 (2016), 1,199 (2019).

HME100K

A new dataset of 74,502 train + 24,607 test images with 245 symbol classes, collected from approximately 10,000 writers via camera-captured uploads. Characteristics include color variation, blur, complex backgrounds, perspective distortion, and illumination issues. Statistics:

• Max sequence length: 184 (vs. 96 for CROHME 2019)
• Average sequence length: 17.62 (vs. 15.79 for CROHME 2019)
• Writer count: ~10,000 (vs. ~100 for CROHME 2019)

Dataset availability: Download links provided via GitHub repository and official portal. No explicit license specified; verify terms before use.

Training

Objective

Multi-task loss combining symbol prediction, relation prediction, reversed symbol prediction, and attention regularization:

$$ L = L_{\text{symbol}} + L_{\text{relation}} + L_{\text{rev. symbol}} + L_{\text{reg}} $$

Attention regularization uses a reversed decoder to predict parent nodes and enforces consistency:

$$ L_{\text{reg}} = -\sum_\beta \hat{\xi}^\eta \log \frac{\hat{\xi}^\eta}{\xi^\alpha} $$

The reversed decoder is removed at inference.

Supervision

Ground-truth LaTeX is parsed into a parse tree via depth-first search, generating parent-child training samples processed in preorder with teacher forcing.

Implementation

• Framework: PyTorch
• Hardware: Single NVIDIA Tesla V100 (32GB)
• Batch size: 8
• GRU hidden size: 256
• Word/relation embedding dimension: 256
• Optimizer: Adadelta ($\rho = 0.95$, $\epsilon = 10^{-6}$)
• Learning rate: Linear warmup from 0 to 1 over first epoch, then cosine decay to 0

Results

CROHME Benchmarks

Without data augmentation:

A variant trained with data augmentation achieves higher scores across all benchmarks.

HME100K

Ablations

ExpRate improvements on CROHME 2019:

• Baseline → SAN-GS (with grammar): significant gain
• SAN-GS → SAN (with syntax-aware attention): additional improvement

Grammar syntax contributes the majority of gains, with syntax-aware attention providing further refinement.

Paper: Chart-to-Text: Generating Natural Language Descriptions for Charts by Adapting the Transformer Model (ACL 2022)
Code: vis-nlp/Chart-to-text
Data: vis-nlp/Chart-to-text
License: GPL-3.0 (+ source restrictions)

TL;DR

The paper introduces Chart-to-Text, a benchmark for generating natural-language summaries of charts, with two datasets (Statista and Pew) totaling 44,096 charts across multiple chart types and topics, plus baselines spanning image captioning, table-to-text, and OCR-to-text settings. Results suggest pretrained seq2seq models (BART/T5) are strongest, but hallucinations, factual errors, and trend/pattern reasoning remain key failure modes, especially when the underlying data table is unavailable.

What kind of paper is this?

• Dominant: $\Psi_{\text{Resource}}$

  • Primary contribution is a large-scale benchmark (two datasets, construction + analysis) and public release framing.

• Secondary: $\Psi_{\text{Evaluation}}$

  • Strong baseline suite + automatic metrics + human evaluation + qualitative error analysis.

• Tertiary: $\Psi_{\text{Method}}$

  • Some method engineering (OCR pipelines, chart-text role classifier, ChartOCR extension), but not the main novelty.

What is the motivation?

• Charts are widely used for communicating quantitative information, but extracting key insights can require substantial cognitive/perceptual effort; summaries can help readers, authors, accessibility use cases, and retrieval/indexing.

• Prior work is limited by:

  • Small datasets, narrow chart coverage (often just bar/line), and single-source collections.
  • Template-heavy systems that describe how to read charts rather than synthesizing insights.
  • Lack of baselines leveraging modern large-scale pretraining for generation.

What is the novelty?

• Benchmark scale + breadth: two sources, many topics, multiple chart types, and two task settings:

  1. Table-available chart-to-text: input includes chart image + underlying table + metadata.
  2. Image-only chart-to-text: underlying table unavailable, requiring extraction from chart images (OCR-based).

• Pew pipeline for chart-summary alignment when pages contain many paragraphs and charts:

  • OCR extraction, chart-text role classification, candidate paragraph heuristics, and crowd labeling for relevance.

• Baseline sweep spanning:

  • image captioning (vision-only),
  • data-to-text (table-to-text),
  • OCR+text hybrids.

What experiments were performed?

• Dataset analysis: chart-type distribution, linguistic stats (tokens/sentences), semantic content categories.

• Automatic evaluation: BLEU, CIDEr, BLEURT, Content Selection (CS), and perplexity using GPT-2 Medium.

• Baseline comparisons across Statista and Pew, with variants that use:

  • ground-truth tables (TAB-*),
  • OCR text (OCR-*),
  • automatically extracted tables (TAB_OCR-*).

• Human evaluation: pairwise comparisons among TAB-T5, OCR-T5, and gold summaries on factual correctness, coherence, fluency.

• Qualitative error analysis on sampled outputs to categorize failures (hallucination, factual errors, reasoning over trends).

What are the outcomes/limitations?

• Best-performing family: pretrained seq2seq (TAB-T5 / TAB-BART when tables exist; OCR-T5 on Pew), but performance drops sharply in Pew (image-only, diverse styles, missing tables).
• Persistent issues: hallucinations and factual errors, especially in OCR-based settings where value-to-label association is brittle, plus difficulty describing complex trends/patterns that humans perceive easily.
• Measurement gap: good fluency and reasonable overlap metrics can coexist with factual mistakes; human evaluation highlights factual correctness deficits relative to gold summaries.

Reproducibility Details

Data

Sources and scale

• Statista

  • Crawled 34,810 publicly accessible webpages (Dec 2020), yielding 34,811 charts.
  • Collected: chart screenshot, downloaded data table (when available), title, axis labels, and human-written description text.

• Pew Research

  • Scraped 3,999 publicly accessible pages (Jan 2021), yielding 9,285 charts.
  • Underlying data tables are usually unavailable (only 143 charts had tables).
  • Collected: chart image, surrounding paragraphs, and alt text (if present).

• Total: 44,096 charts across both datasets.

Chart complexity definition

• Statista: “simple” charts have data tables with two columns; “complex” charts have $\ge 3$ columns (e.g., grouped/stacked bars, multiple-line charts).
• Pew: complexity labeled manually because tables are generally missing.

Summary selection and annotation

• Statista summary selection: used the first part of the webpage text (from chart icon to next heading) as the summary; remaining text often contains background.

• Statista x-axis label completion:

  • Many charts lacked explicit x-axis labels.
  • Used regex heuristics over cell values to detect common entity types; remaining missing labels handled via Wikidata-based entity typing, then manual annotation when labels were too generic.

Pew: 3-stage alignment pipeline (Figure 2)

(i) Data extraction from chart images

• OCR text extracted using CRAFT.
• Extracted bounding boxes and geometric features; trained gradient boosting classifiers to categorize recognized text into: title, axis labels, legends, data labels.
• Separate classifier per chart type.
• Manual labels: 319 examples (171 bar, 68 line, 80 pie) split 8:1:1 into train/val/test.
• Reported performance: 95.0% precision overall, 97.6% precision for title classification on test.
• Title choice: if alt text exists, take the longer of (alt text, OCR-title); otherwise use OCR-title.

(ii) Identification of candidate paragraphs

• Candidate set: paragraph adjacent to the chart plus five before and five after (max 11).

• Heuristic relevance score:

  • Sentence relevance: $$s_i = 0.58 l_i + 1.4 n_i - 0.5 u_i$$ where $l_i$ is lexical matches, $n_i$ numerical matches (excluding years), $u_i$ numerical tokens in sentence not in chart.
  • Content score: $$\text{content}=\frac{1}{1+\exp(0.3(-\max_i(s_i)+1.7))}$$
  • Proximity score (distance $dist \in [-5,5]$): $$\text{proximity}=0.4\exp(-0.1|dist|^2)+0.6$$
  • Paragraph relevance: $$rel = \text{content} \times \text{proximity}$$
  • Paragraph considered relevant if constraints hold, including $rel > 0.72$, lexical matches sum $>3$, numerical/year matches $>0$, and no extra numerical tokens ($\sum u_i = 0$). (Figure 7)

• Heuristic evaluation (random sample): recall 21.1%, precision 100%, chosen to prioritize precision.

(iii) Selection of relevant paragraphs via crowdsourcing

• Annotated 5,478 charts and 13,237 paragraphs.
• Two annotators per chart; agreement used when both label irrelevant vs relevant; disagreements (2,888 paragraphs) resolved internally.
• Reported overall agreement: 78.2%.

Splits and descriptive stats

• Train/val/test split: 70% / 15% / 15%.
• Chart types (Table 1): bar charts dominate both datasets; line charts second; Pew includes additional types (area/scatter).
• Linguistic stats (Table 2): Pew summaries are ~2× longer than Statista by characters/tokens/sentences; complex charts tend to have longer summaries.
• Semantic content analysis (Table 3): “statistical/comparative” content most common; Pew has more “perceptual/cognitive” sentences than Statista.

Model

Task formulation

• Dataset instance: $\langle C, T, M, S \rangle$ where

  • $C$ chart image,
  • $T$ data table (when available),
  • $M=(C_{title}, C_{type}, C_{labels})$ metadata,
  • $S$ reference summary.

• Two input settings:

  • table-available: $X=\langle C, T, M\rangle$
  • image-only: $X=\langle C, M\rangle$ (must recover info from chart image)

Baseline families (Section 4)

1) Image captioning (vision-only)

• Show, Attend, and Tell style: ResNet50 encoder + uni-directional LSTM decoder.
• ResNet50 pretrained via Barlow Twins self-supervision (separate pretraining per dataset) because ImageNet-pretrained ResNet transferred poorly to charts.

2) Data-to-text (table-to-text)

• Chart2text (adapted transformer, with auxiliary content selection objective, plus templating strategy to reduce hallucination).
• Field-Infusing model: LSTM encodes cell values, concatenated with row index + column heading embeddings, then a Transformer encoder-decoder generates text.
• BART / T5: flatten table row-by-row; input includes title + table content; T5 uses prefix "translate Chart to Text:" to mimic pretraining style.

3) Vision+text hybrids (OCR-to-text)

• Use CRAFT OCR to extract chart text, then feed to text generation models (Chart2text, Field-Infuse, BART, T5).
• OCR-T5 has a variant that injects bounding-box positional embeddings (inspired by Tan and Bansal style spatial encoding).

Algorithms / Training

Training setup (Appendix A.3)

• Hardware: CPU Intel Xeon Gold 6240 @ 2.60GHz, GPU 4× NVIDIA GTX 2080 Ti.
• Training time note: T5 fine-tuning is reported as the most expensive, ~16–20 hours on 4 GPUs.

Model-specific training details

• Chart2text: 1 encoder layer, 6 decoder layers, dropout 0.1, 80 epochs, batch size 6; beam size 4 at inference.
• Field-Infusing: 10 epochs, dropout 0.1, batch size 1.
• BART-Base: ~140M params, 6 layers; fine-tune 500K iterations, batch size 4, LR 0.0005; validate every 2,000 iters; beam size 4 at inference.
• T5-Base: ~220M params, 12-layer encoder-decoder; fine-tune 500K iterations, batch size 4, LR 0.0005; validate every 2,000 iters; beam size 4 at inference.

Evaluation

Automatic metrics (Section 5.1)

• BLEU and CIDEr: n-gram overlap (CIDEr TF-IDF weighted).
• BLEURT-base-128: learned metric for grammaticality/semantic similarity (sentence-level averaged).
• Content Selection (CS): overlap in selected records relative to gold (sentence-level averaged).
• Perplexity (PPL): computed with GPT-2 Medium for fluency proxy.

Main quantitative results (Table 4)

• Statista (table-available is strongest setting):

  • Best BLEU among reported baselines: TAB-T5 37.01, TAB-BART 36.36.
  • OCR-only variants are slightly worse (example: OCR-T5 35.29) but still competitive, with generally lower PPL.
  • Image captioning baseline has relatively low CS even if PPL is low.

• Pew (mostly no tables, more diverse styles):

  • Best BLEU among reported baselines: OCR-T5 10.49 (OCR-T5* 10.42).
  • Vision-only image captioning collapses (BLEU ~4).

Human evaluation (Section 5.2, Table 5)

• Setup: 150 Statista charts; 4 internal annotators (native English); 450 pairwise comparisons:

  • TAB-T5 vs OCR-T5
  • Gold vs TAB-T5
  • Gold vs OCR-T5

• Criteria: factual correctness, coherence, fluency.

• Agreement on a subset (excluding ties): 74.3%.

• Outcome: TAB-T5 beats OCR-T5 strongly on factual correctness (and also coherence/fluency), while gold summaries still win more often than either model, especially on factual correctness/coherence.

Error analysis themes (Section 5.3)

• Perceptual and reasoning failures: models struggle with trends/relationships that are visually salient but not trivial from raw extracted text (examples shown in Figure 4).
• Hallucinations: fluent but irrelevant tokens/statements.
• Factual errors: especially OCR-based, due to missing data labels or mis-association between values and entities (Figure 4 example where follower counts swap entities).
• Computer vision constraints: charts often omit explicit numeric labels, and OCR alone does not recover mark-to-label alignment.
• Proposed direction (as an aspiration): richer representations such as semantic graphs encoding numerical/logical relations among chart objects.

Hardware / Production

• Reported training environment: Xeon Gold 6240 CPU, 4× GTX 2080 Ti, with T5 fine-tuning taking ~16–20 hours in their setup.
• No serving/latency benchmarks; evaluation is offline.

Appendix-specific implementation detail: automatic table extraction (Appendix A.5)

• They extend ChartOCR to recover fully-structured tables:

  • Keypoint detection for chart elements and marks; extend detector to include textual labels and legend marks.
  • OCR (CRAFT) recognizes x-axis/legend labels; associate values to nearest labels and series by color; estimate scale using y-axis labels.

• Reported automatic extraction accuracy: 77.31% (used to create TAB_OCR-* model inputs).

Notes on ethics and misuse (brief)

• Dataset collection constrained to publicly available charts with publication rights considerations (Statista free studies; Pew attribution/terms).
• AMT compensation targeted to minimum wage rates; per-chart payments 0.10–0.15 USD depending on candidate paragraphs.
• They explicitly call out a misuse risk: fluent outputs with hallucinations/factual errors could misinform if published uncorrected.

Data Availability & Licensing Notes

TL;DR

Yes: the benchmark datasets are publicly available online (GitHub), and the repository is licensed under GNU GPL v3. But: the charts come from third-party sites (Statista, Pew), and your rights to redistribute/reuse those chart images can be constrained by their Terms of Use, even if the repo itself is GPL.

Questions answered

1) Is there data publicly available?

Yes. The paper explicitly says the “code and benchmark datasets are publicly available” (via their GitHub).

2) Online? Where?

Yes (online). The stated primary location is the public GitHub repository vis-nlp/Chart-to-text. (There are also public mirrors/derivatives in the ecosystem, but GitHub is the canonical source per the paper.)

3) With what licensing?

Repository license: The repo includes a GNU General Public License v3.0 license file.

Important nuance (data vs. code):

• GPL v3 clearly applies to “software and other kinds of works” released under it.
• However, the dataset contains/derives from third-party chart content, and that content may carry separate legal/contractual restrictions (see below).

Third-party terms that affect what you can do with the data

Statista (as used by the dataset)

The authors say they use only “free studies” from Statista and cite Statista’s Terms: free content comes with “publication rights” for academic purposes, but paid content does not—and they claim to use only the free portion.

The Statista Terms document explicitly discusses redistribution limits and conditions, including that some redistribution is only allowed for “free material” and typically requires leaving materials unchanged and referencing Statista. It also prohibits “crawlers/spiders” (relevant because the dataset was constructed via crawling).

Pew Research Center

Pew’s Terms grant a license to use Pew content with attribution, and include a specific requirement to “provide proper attribution… in accordance with the citation below.” They also restrict reuse of content attributed to another party that is not Pew.

Practical implications (what this means for you)

• If you just need a benchmark to reproduce research results, the GitHub release being public + GPL’d is straightforward.
• If you plan to redistribute the dataset, host it, or use it in a commercial product, you should treat the third-party chart/image rights (Statista/Pew) as potentially more restrictive than the repo’s GPL label.
• If your use-case needs clean licensing, you may prefer datasets built from explicitly open-licensed charts (or datasets whose creators confirm redistribution rights unambiguously).

Paper • Data • Code • License

TL;DR

GNHK introduces a camera-captured, in-the-wild English handwriting dataset (687 images) with word-level quadrilateral annotations, line grouping, and handwritten-versus-printed tags. The paper provides baseline results for text localisation (Mask R-CNN / Faster R-CNN) and cropped-word recognition (Clova scene-text recognizer variants), with best reported recognition reaching CAR 0.861 and WAR 0.502 under TPS + BiLSTM + attention decoding.

What kind of paper is this?

Dominant: $\Psi_{\text{Resource}}$ (dataset release + benchmark baselines)
Secondary: $\Psi_{\text{Evaluation}}$ (establishes baseline protocols and metrics for localisation and recognition)

Justification: The headline contribution is the dataset itself (“we created a dataset…”) plus accompanying baselines, which matches the taxonomy guideline that dataset/tool releases are typically $\Psi_{\text{Resource}}$-dominant.

What is the motivation?

• Existing widely-used handwriting datasets (e.g., IAM, RIMES) are largely flatbed-scanned and do not reflect camera-captured “in the wild” conditions.
• Prior scene-text benchmarks are mostly printed text; the paper argues there is a gap for offline English handwriting captured via cameras under unconstrained conditions.
• English handwriting varies across regions (lexicon and styles), motivating collection across Europe, North America, Asia, and Africa.

What is the novelty?

• A camera-captured English handwriting dataset modeled after scene-text datasets, explicitly including “in the wild” document types (shopping lists, sticky notes, diaries) and non-handwriting content (printed text, images).
• Annotation schema designed for both detection/localisation and recognition:
  • Per-image JSON annotations containing multiple objects with (text, polygon, line_idx, type).
  • Text values drawn from ASCII printable characters plus the British pound sign, with no whitespace characters included.
  • Three special tokens for polygon annotations: %math% (math expressions), %SC% (illegible scribbles), %NA% (no characters/math symbols).
  • Quadrilateral polygons listed clockwise starting at the top-left point.
  • line_idx groups texts that belong to the same line; type indicates handwritten vs printed.

What experiments were performed?

Text localisation (detection/segmentation)

• Baselines: Mask R-CNN (instance segmentation) and Faster R-CNN (object detection), implemented in detectron2.
• Evaluation: recall, precision, and $F$-measure at IoU $> 0.5$.

Text recognition (cropped word recognition)

• Benchmark setup: use ground-truth word boxes to crop word images, then recognize text from crops (segmented recognition).
• Model family: Clova AI deep text recognition framework with four components (transformation, feature extraction, sequence modelling, prediction), evaluating eight configurations.
• Data filtering for recognition eval: keep words and punctuation; remove unknown characters, scribbles, math symbols, and words that contain only punctuation.

What are the outcomes/limitations?

Key outcomes

• Localisation: both Mask R-CNN and Faster R-CNN achieve $F$-measure $> 0.86$ (IoU $> 0.5$), with high precision in both cases.
• Recognition: best reported configuration (TPS + BiLSTM + attention) reaches CAR 0.861 and WAR 0.502; attention decoding substantially outperforms CTC, and TPS improves CAR/WAR in the reported comparisons.

Limitations and open ends (from the paper’s setup)

• Recognition benchmark is not end-to-end: it assumes ground-truth word crops rather than predicted boxes.
• For localisation, the dataset lacks pixel-level masks separating word vs non-word; the baseline uses polygon-to-box conversion (min/max over polygon points) for the R-CNN box regression.
• The paper explicitly points to future work on end-to-end approaches that do localisation and recognition sequentially in a single framework.

Model

Localisation

• Mask R-CNN baseline; bounding boxes derived from polygon min/max in $x$ and $y$, with the polygon used as the segmentation mask.
• Implementation: detectron2; backbone ResNet-50 with FPN, pretrained on ImageNet; network pretrained on MS COCO for segmentation.
• Comparator: Faster R-CNN in the same detectron2 framework.

Recognition

• Clova AI deep text recognition framework with configurable components:
  • Transformation: TPS vs none
  • Sequence modelling: BiLSTM vs none
  • Prediction: attention vs CTC

Data

Dataset size and composition

• Total: 687 images, 172,936 characters, 39,026 texts, 9,363 lines.
• “Texts” include words, ASCII symbols, and math expressions (via tokenization rules).
• Regional sourcing across Europe, North America, Asia, and Africa.
• Region-level counts (chars/texts/lines): EU 58,982 / 13,592 / 3,306; NA 47,361 / 10,967 / 2,099; AS 39,593 / 8,586 / 2,780; AF 27,000 / 5,881 / 1,178.
• Text statistics: 39,026 total texts, 12,341 unique; median texts per image 57; mean 44.1.
• Character set: 96 unique characters; median character count 486; mean 1,801; max per-character frequency up to 17,887.
• Collection constraint: no more than 5 images per writer.

Annotation format

• Per-image JSON file with objects keyed by text, polygon, line_idx, type.
• polygon is a quadrilateral with four $(x,y)$ points in clockwise order starting at top-left.
• type indicates handwritten or printed.

Splits

• Train/test split: 75% training, 25% testing.

Download and license

• Official download: GoodNotes hosts the dataset with links (Google Drive / Baidu Netdisk) available after agreeing to terms and conditions on the GNHK webpage.
• Pointer repository: The GoodNotes/GNHK-dataset GitHub repo points to the official dataset page.
• License: Creative Commons Attribution 4.0 (CC BY 4.0). You can use, share, and adapt the dataset (including commercially) as long as you provide attribution and include the license notice (and indicate if you made changes).
• Note: Third-party mirrors exist (e.g., on Hugging Face), but the GoodNotes page is the authoritative source.

Algorithms / Training

• Localisation: Mask R-CNN trained in detectron2 with ResNet-50 + FPN backbone; initialization includes ImageNet pretraining and MS COCO pretraining for segmentation.
• Recognition: segmented recognition using ground-truth crops, evaluated across eight Clova framework configurations (TPS/none × BiLSTM/none × attention/CTC).

Training hyperparameters such as learning rate, batch size, and epochs are not specified in the paper.

Evaluation

Localisation metrics and results

• Metric: Recall, precision, and $F$-measure with IoU $> 0.5$.
• Results (Table 5):
  • Mask R-CNN: recall 0.8237, precision 0.9079, $F$-measure 0.864
  • Faster R-CNN: recall 0.8077, precision 0.9215, $F$-measure 0.860
• Qualitative note: $F$-measure differences are described as coming from high precision (fewer false positives).

Recognition metrics and results

• Metrics:
  • Character accuracy rate (CAR): average over words of $1 - \frac{\text{edit distance}(gt_i, pred_i)}{N_i}$
  • Word accuracy rate (WAR): fraction of words with zero edit distance
• Results (Table 6): best configuration TPS + BiLSTM + attention gives CAR 0.861 and WAR 0.502.
• Comparative findings:
  • Attention decoding outperforms CTC across the reported configurations.
  • TPS improves CAR and WAR in like-for-like comparisons.
  • BiLSTM generally improves CAR; the paper notes a case where WAR decreases when adding BiLSTM without TPS (0.377 vs 0.430).

Hardware / Production

Not specified.

Paper: ChartOCR: Data Extraction from Charts Images via a Deep Hybrid Framework (WACV 2021)
Code: soap117/DeepRule
Data: HuggingFace, GitHub
License: BSD-3-Clause (code), MIT (dataset on HuggingFace)

TL;DR

ChartOCR is a deep + rules hybrid pipeline for extracting the underlying numeric table from bar, line, and pie chart images by (1) detecting chart keypoints with a shared deep model, then (2) applying type-specific geometric rules to reconstruct chart elements and map pixels to values. The authors report strong results on their large ExcelChart400K dataset (scores: 0.919 bar, 0.918 pie, 0.962 line) and show that with ground-truth keypoints the remaining rule module becomes near-perfect, suggesting keypoint localization is the dominant error source.

What kind of paper is this?

• Dominant: $\Psi_{\text{Method}}$. The core contribution is a unified extraction method that uses keypoint detection plus chart-type rules to support multiple chart types in one framework.
• Secondary: $\Psi_{\text{Resource}}$ (they introduce ExcelChart400K, a large annotated dataset) and $\Psi_{\text{Evaluation}}$ (they propose new chart-type-specific metrics).

What is the motivation?

• Prior rule-based chart extractors struggle to generalize across chart styles and layouts, since rules are brittle and style-specific.
• Pure end-to-end deep approaches can be accurate but often specialize to a single chart type and provide less controllable intermediate structure (plot area, range, components).
• The paper targets a middle ground: use deep learning for what generalizes (keypoint detection) and rules for what is structured (geometry and grouping).

What is the novelty?

• Key idea: reduce chart extraction across types to a keypoint detection problem, then reconstruct components with type-specific post-processing rules.
• A shared “common information extraction” network predicts (a) keypoints and (b) chart type; then downstream modules handle range estimation and object grouping per type.
• New dataset: ExcelChart400K (386,966 images) for training deep models on diverse chart styles.
• New metrics: separate evaluation metrics for bar, line, and pie that better match the “read the data values” objective than borrowed detection metrics.

What experiments were performed?

• Datasets:

  • FQA (100 synthetic chart images across bar/pie/line) and WebData (100 web-crawled images) from prior work.
  • ExcelChart400K (large-scale dataset introduced here).

• Baselines compared (as reported):

  • Rule-based: Revision.
  • Deep: Vis, ResNet+Faster-RCNN (bar), RotationRNN (pie), ResNet+RNN (end-to-end).
  • Commercial: think-cell (bar only, qualitative).

• Ablations:

  • Replace predicted keypoints with ground-truth keypoints to estimate an upper bound for the rule module.
  • Replace OCR with ground-truth OCR for FQA (since WebData lacks GT OCR).

What are the outcomes/limitations?

• Main outcomes (authors’ reported):

  • ExcelChart400K scores: 0.919 (bar), 0.918 (pie), 0.962 (line); with GT keypoints: 0.989/0.996/0.991, implying post-processing is strong when keypoints are accurate.
  • Public datasets (mean error): ChartOCR improves substantially over Vis, especially on pie and line.
  • Runtime: ChartOCR averages 0.206s (bar), 0.193s (pie), 0.507s (line); rule-based Revision is much slower.

• Limitations called out:

  • Line charts: hard cases with multiple entangled line segments remain challenging; the QUERY network struggles in these settings.

• Practical assumptions (implicit in method description, potential fragility):

  • Data range extraction assumes y-axis labels are on the left of the plot area and uses only top/bottom OCR numbers to infer scale. This may break for unconventional layouts, multi-axis charts, or non-linear scales (not addressed in the paper text).

Model

Common information extraction: keypoints + chart type

• Backbone: modified CornerNet with Hourglass Net backbone (104 layers).
• Output: a pixel-level probability map with 3 channels (top-left, bottom-right, background). Map size matches input image.
• Uses corner pooling (from CornerNet) on the penultimate keypoint branch layer to expand receptive field along horizontal/vertical directions.
• Loss: “CornerNet-style” combination of probability-map loss plus smooth L1 for keypoint coordinates (the paper references CornerNet’s settings rather than fully re-deriving them).

Chart type classification head

• Adds a conv layer from Hourglass output to downsample features (example given: $(32 \times 32)$), then max-pools to a 1D vector and feeds FC layers to a softmax classifier.
• Loss: cross-entropy (explicit formula provided).

Data

FQA and WebData

• FQA: 100 synthetic images total across chart types; limited style diversity per authors.
• WebData: 100 images crawled from the web; larger style variation than FQA.

ExcelChart400K

• Size: 386,966 chart images, collected by crawling public Excel sheets; chart images captured with Excel APIs and underlying data extracted directly from the source spreadsheets.

• Privacy: authors overwrite chart text with random characters for anonymization.

• Splits (Table 1):

  • Bar: train 173,249, val 6,935, test 6,970
  • Line: train 116,745, val 3,073, test 3,072
  • Pie: train 73,075, val 1,924, test 1,923

Algorithms / Training

Data range extraction (bar + line)

• Uses Microsoft OCR API to extract text from chart images (legend/title/axis labels).

• Locates plot area via a similar keypoint detection routine (top-left + bottom-right corners).

• Assumption: y-axis numbers are on the left-hand side of the plot area; filter OCR results accordingly.

• Algorithm 1 (Data Range Estimation) (high-level):

  • Find nearest OCR number near bottom-left of plot area as $r_{\max}$ and near top-left as $r_{\min}$ (using a left-of-plot constraint like r.r < Left - 4).
  • Convert text to numbers, compute a pixel-to-value scale: $$Y_{\text{scale}} = \frac{r_{\max}.num - r_{\min}.num}{r_{\min}.t - r_{\max}.t}$$
  • Estimate $Y_{\min}$ and $Y_{\max}$ using scale and vertical offsets relative to plot area bounds.

• For pie charts, they skip this step since the total is assumed to be 100% by default.

Type-specific chart object detection

Bar charts

• Threshold keypoint heatmap at $s = 0.4$, then match each top-left point to the nearest bottom-right point using a weighted distance: $$dist = \gamma , dist_x + \nu , dist_y$$
• For vertical bars, set $\gamma > \nu$ and constrain search to the right side of plot area; for horizontal bars, $\nu > \gamma$.

Pie charts

• Modify keypoint net: replace corner pooling with center pooling (from CenterNet-style methods) to capture 360-degree context.

• Threshold at $s = 0.3$.

• Algorithm 2 (Sector Combining) handles:

  1. Tight pies (single center): sort arc points clockwise; pair consecutive arc points with the center.
  2. Exploded pies (multiple centers): run Pie Radius Estimation (details in supplement per authors) and only connect center-to-arc pairs whose distances fall within a threshold of the estimated radius.

Line charts

• Adds an embedding layer (after an early conv) to encourage points on the same line to have similar embeddings; uses pull/push losses (CornerNet-style associative embedding).

• Total keypoint loss for line: $loss’{\text{point}} = loss{\text{point}} + \lambda \cdot loss_{\text{embedding}}$, with $\lambda = 0.1$.

• Groups points into lines via hierarchical clustering using union-find.

• Handles points shared by multiple lines (“intersection points”) with a QUERY network:

  • For an intersection point $s$ and closest assigned point $e$, sample $K$ points evenly along segment $s \rightarrow e$; obtain features via linear interpolation; classify whether $s$ and $e$ belong to the same line.

Training setup (as reported)

• Optimizer: Adam, LR $2.5 \times 10^{-4}$, reduced to $2.5 \times 10^{-5}$ for last 5,000 batches.
• Batch size: 27.
• Mentions $\alpha = 2$, $\beta = 4$ (consistent with focal-loss-style params, though the paper does not re-explain them in detail here).
• Postproc: Soft-NMS to merge keypoints.
• Early stopping used; validation set from ExcelChart400K used for hyperparameter tuning.

Evaluation

Proposed metrics (Section 6)

Bar

• Defines a custom distance between predicted and GT bar boxes $p$ and $g$ emphasizing $(x,y,h)$ (width is treated as less relevant for reading): $$D(p,g) = \min\left(1, \frac{|x_p-x_g|}{w_g} + \frac{|y_p-y_g|}{h_g} + \frac{|h_p-h_g|}{h_g}\right)$$
• Computes assignment cost via a minimum-cost matching (job assignment) over predictions vs GT; final score is $1 - cost/K$.

Line

• Treats a line as continuous and evaluates via interpolation-based error with a precision/recall and an $F1$ formula; weights segments by point intervals (larger gaps count more).
• For multi-line charts, they enumerate combinations to find best matching score.

Pie

• Treats extraction as a sequence matching problem in clockwise order; uses dynamic programming with a per-element match reward of $1 - \left|\frac{x_i-y_j}{y_j}\right|$, then normalizes.

Key results (as reported)

ExcelChart400K (Table 2, higher is better)

• ChartOCR: 0.919 (bar), 0.918 (pie), 0.962 (line)
• ChartOCR + GT keypoints: 0.989, 0.996, 0.991
• ResNet+Faster-RCNN: 0.802 (bar)
• Revision: 0.582 (bar), 0.838 (pie)
• RotationRNN: 0.797 (pie)
• ResNet+RNN: 0.000 (bar), 0.411 (pie), 0.644 (line)

FQA + WebData (Table 3, mean error, lower is better)

• FQA: ChartOCR 0.185 (bar), 0.038 (pie), 0.484 (line); with GT OCR: 0.093, 0.038, 0.496.
• WebData: ChartOCR 0.285 (bar), 0.439 (pie), 0.740 (line).
• The authors note OCR quality is a major contributor to bar-chart extraction error on these sets.

Hardware / Production

• Training environment: 4 $\times$ Tesla P100 GPUs.

• Average runtime (Table 4):

  • ChartOCR: 0.206s (bar), 0.193s (pie), 0.507s (line)
  • Revision: 20.032s (bar), 5.423s (pie)
  • ResNet+Faster-RCNN: 0.120s (bar)
  • RotationRNN: 0.421s (pie)

• Authors suggest merging QUERY with the keypoint backbone if runtime is critical (currently QUERY does not share parameters with keypoint net).

Data availability + licensing (ExcelChart400K / “CHARTEX Data”)

• Publicly available? Yes — the paper explicitly states “the code and the dataset are publicly available” and points to the DeepRule GitHub repo.

• Online? Yes — the DeepRule repo is public, and its README links to a hosted dataset download (“Downloading CHARTEX Data”) on Hugging Face. (GitHub)

• What license is it under?

  • Repo (code) license: The DeepRule GitHub repository is labeled BSD-3-Clause. (GitHub)
  • Dataset (as hosted) license: The Hugging Face dataset page for DeepRuleDataset declares the license as MIT. (Hugging Face)
  • Important caveat: The paper does not state a dataset license (it only says it’s publicly available), and the dataset itself was collected by “crawling public Excel sheets” (with text anonymization). So if you need high confidence for downstream use, rely on the license/terms shipped with the dataset distribution you download (and treat GitHub’s BSD-3-Clause as applying to the repo contents unless the dataset package states otherwise).

DSD — Notes

Paper: Decoupled Style Descriptors (ECCV 2020)
Code & Models: github.com/brownvc/decoupled-style-descriptors

TL;DR

Proposes Decoupled Style Descriptors (DSDs) that explicitly factor handwriting into a writer style vector $w$ and a content-conditioned character matrix $C_{ct}$, producing writer-character descriptors $w_{ct}=C_{ct}w$. Human preference studies show 88% preference over DeepWriting, with writer identification accuracy reaching 99.70% on 50-word samples.

Five Key Questions

What kind of paper is this?

Dominant: $\Psi_{\text{Method}}$. Introduces a new modeling construction for handwriting generation: $w_{ct}=C_{ct}w$ with an explicit invertible content matrix to factor out writer style.

Secondary: $\Psi_{\text{Evaluation}}$. Runs human similarity judgments against DeepWriting and analyzes invertibility, writer ID performance.

Secondary: $\Psi_{\text{Resource}}$. Introduces the BRUSH dataset with design and collection details.

What is the motivation?

Classic encoder-decoder handwriting models learn writer-dependent and character-dependent latent vectors $w_{ct}$ but do not provide a mechanism to represent character-independent writer style directly. They argue explicit structure is needed to extract writer style cleanly.

What is the novelty?

Linear latent factorization: $$w_{ct} = C_{ct} w,\quad w = C_{ct}^{-1} w_{ct}$$ assuming $C_{ct}$ is invertible.

Sequence-aware $C$ construction: $C_{ct}$ is predicted from character substrings using a character encoder $g_\theta$ that embeds characters, passes them through an LSTM to capture history, then maps to a $256\times256$ matrix.

Writer style estimation by averaging per-prefix inversions: for a sample producing multiple $w_{ct}$ across prefixes, they estimate $$w = \frac{1}{M}\sum_{t=1}^{M} C^{-1}{ct} w{ct}.$$

Multiple synthesis routes: producing $w_{ct}$ via the encoder, or via “Method $\alpha$” (mean writer style) and “Method $\beta$” (sampling from a database of writer-character DSDs).

Unsupervised character segmentation ($k_\theta$): trains a segmentation network with a modified CTC-style objective so stroke points can be attributed to characters and end-of-character indices can be identified.

What experiments were performed?

Human similarity preference on IAM vs DeepWriting: 25 participants on MTurk, 40 sentence-level target samples, 15 assessments per case totaling 600, with randomized ordering.

Writer identification: reports 89.38% from a single word and 99.70% from 50 words on holdout writers.

Invertibility checks: explicitly evaluates matrix ranks for $C_{ct}$ for 1-, 2-, and 3-character strings, noting a small number of rare non-invertible 3-character cases.

What are the outcomes/limitations?

Outcomes

Generated handwriting achieves 88% human preference over the baseline. The sampling variant is chosen 5.22x more often as most similar to target. Strong writer-identification performance demonstrates the learned style vectors capture meaningful writer characteristics.

Structural decoupling via $C$ performs better than style-transfer baselines where the network must implicitly separate content and style.

Limitations / failure modes

Delayed strokes, such as dotting an i after completing the letter body, are challenging for substring-based representations. Missed delayed strokes shown as a limitation case.

Relies on segmentation quality (predicting end-of-character indices); errors propagate into $w_{ct}$ extraction and synthesis.

Reproducibility Details

Model

Input stroke representation: each point $p_t$ stores $(\Delta x_t, \Delta y_t)$ and an end-of-stroke flag $eos\in{0,1}$, giving $x\in\mathbb{R}^{(N,3)}$.

Outputs (decoder): MDN parameters $(\pi_t,\mu_x,\mu_y,\sigma_x,\sigma_y,\rho)$ plus probabilities for $eos$ and $eoc$ (end-of-character).

Core latent variable shapes: $w\in\mathbb{R}^{256}$, $C_{ct}\in\mathbb{R}^{256\times256}$, $w_{ct}\in\mathbb{R}^{256}$.

Character encoder $g_\theta$ for producing $C_{ct}$: FC layer $\rightarrow$ LSTM $\rightarrow$ FC mapping to 65,536 dims and reshape to $256\times256$; FC2 layer is approximately one-third of total parameters.

Segmentation network $k_\theta$: bidirectional LSTM taking 23 features per point; trained with a modified CTC-style loss to label each point with a character and find eoc indices.

Data

BRUSH dataset design: baseline shown in every drawing box so the initial action includes the shift from baseline to start point; 170 individuals; 488 common words across 192 sentences.

Character inventory: 86 characters (space + 85 listed including digits, upper/lowercase, punctuation/symbols).

Collection constraints: 170 writers on MTurk; 60-minute time limit; word selection based on 3,036 Gutenberg books to cover character pairs.

Coverage: selection achieves 99.9% coverage of character pairs with 3,894 pairs.

Cleaning / ordering: correcting missed delayed strokes by adding them back and sorting strokes left-to-right.

License: Dataset may only be used for non-commercial research purposes; contact Prof. James Tompkin for other uses.

Download: Google Drive ZIP (566 MB) linked from repository README.

Algorithms / Training

Training uses teacher forcing in the decoder: during training feeds true point sequences; at runtime feeds predicted points.

Loss (decoder location term): negative log-likelihood of the next point under the MDN mixture distribution, with additional EOS/EOC terms.

Invertibility enforced implicitly: $L_{wct}$ terms penalize failures where $CC^{-1}\neq I$, discouraging singular $C$.

Backprop through matrix inverse: uses $\frac{dC^{-1}}{dx}=-C^{-1}\frac{dC}{dx}C^{-1}$ to enable end-to-end training.

Optimization + key hyperparameters: Adam, learning rate 0.001, gradient clipping to $[-10,10]$, 5 sentence-level samples per batch, 3-layer stacked LSTMs.

Sampling Method $\beta$ (database-driven): procedure to build/query a database $D$ of writer-character DSDs keyed by substrings and sample $w_{ct}$ for synthesis.

Evaluation

Human study protocol: 25 MTurk participants; 40 sentence-level targets; 15 assessments per case; randomized ordering; total 600 assessments.

Primary reported preference signal: sampling method selected 5.22x more often as “most similar.”

Writer ID metrics: 89.38% from a single word and 99.70% from 50 words.

Hardware / Production

Compute constraints: baseline comparisons adjusted training settings due to 8GB VRAM GPU constraint, requiring batch size reduction.

License & Availability

Code license: Brown University copyright; non-commercial use only. Grants permission for use/copy/modify/distribute except incorporation into a commercial product or service.

Dataset license: BRUSH dataset restricted to non-commercial research purposes per repository README. Commercial or other uses require contacting the authors.

Paper: ICDAR 2019 Competition on Harvesting Raw Tables from Infographics (CHART-Infographics) (ICDAR 2019)
Data: Synthetic (CHART2019-S), PMC
License: CC-BY-NC-ND 3.0 (synthetic), CC-BY-NC-SA 3.0 (PMC)

TL;DR

This paper reports the setup and results of the first CHART-Infographics competition: a task decomposition for chart understanding plus a large synthetic training set and a smaller, manually annotated real-chart test set from PubMedCentral. Main takeaway: systems do very well on synthetic data but drop substantially on real charts; the benchmark, annotation tooling, and evaluation scripts are meant to standardize comparisons.

What kind of paper is this?

• Dominant: $\Psi_{\text{Resource}}$

  • Primary “artifact” is a benchmark ecosystem: synthetic dataset generation, real-chart dataset sampling/annotation tooling, and released evaluation scripts.

• Secondary: $\Psi_{\text{Evaluation}}$

  • Defines task-specific metrics (notably for text detection+OCR scoring, axis tick localization scoring, legend marker localization scoring) and analyzes cross-domain generalization (synthetic to real).

What is the motivation?

• Chart understanding has many proposed methods, but the field lacked common benchmarks and tools to compare approaches fairly.
• The authors aim to (1) provide a large-scale synthetic training regime to enable deep models and (2) test robustness on real charts from scientific literature, where layout and rendering variability is higher.

What is the novelty?

• Task decomposition of “extract raw tables from charts” into a pipeline of independently evaluable sub-tasks (Tasks 1–7).

• Synthetic dataset generation from real-world data tables, rendered into 10 chart types with randomized stylistic/layout variations, and annotated automatically via Matplotlib API.

• New evaluation formulations for chart-specific subtasks:

  • Text detection+recognition: combines IoU-based matching with OCR string similarity via normalized character error rate (NCER).
  • Axis analysis: weighted F-measure with partial credit based on tick-location accuracy as a function of distance relative to image diagonal.
  • Legend analysis: weighted F-measure with partial credit based on legend-marker bounding box IoU.

What experiments were performed?

• Competition evaluation on two domains:

  • Synthetic charts (large-scale; used for training and testing).
  • PMC (real charts) from PubMedCentral Open Access, manually annotated for evaluation (with different annotation depths per task).

• Tasks actually receiving submissions: Tasks 1–5 (no submissions for Tasks 6–7).

• Reported results include:

  • Task 1 confusion matrices for top methods on PMC (Figure 4).

  • Task-level summary tables:

    • Task 1: average F-measure on synthetic vs PMC (Table II).
    • Task 2: text detection/recognition scores (Table III).
    • Tasks 3–5: average/weighted F-measures (Table IV).