The Matter vs. Meaning Framework: Taxonomy and Annotation Guide

Why This Framework Exists

The Core Disconnect

A major challenge in Document Layout Analysis (DLA) is the mismatch between what a computer vision model can detect and what a downstream system actually needs:

1. Logical Structure: The hierarchical semantic tree (e.g., Article $\to$ Section $\to$ Nested Section $\to$ Paragraph) defined in document standards like Tagged PDF or JATS.
2. Visual Structure: The flat list of bounding boxes (e.g., “Text Block”, “Title”, “Table”) detected by computer vision models.

This gap means that even a top-scoring CV model can fail to reconstruct a valid document because it misses the relationships between elements (e.g., which caption belongs to which figure, or that a text block continues a paragraph from the previous page).

The Granularity Problem

Models and datasets often disagree on what constitutes an “object”:

• Annotation Level: Does a bounding box wrap a whole paragraph (PubLayNet), a single line (common in OCR), or individual tokens (DocBank)?
• Geometry: Are regions defined by bounding boxes (COCO-style) or polygons (PRImA/PAGE-XML)?
• Scope: Do structures persist across pages? Most models are single-page only.

Computer vision datasets typically “flatten” the document hierarchy into a single layer of bounding boxes. The choice of classes dictates what the model can see: coarse buckets like Text, Table, Figure (PubLayNet) work for chunking but lose semantic detail; fine-grained roles like SectionHeader, Caption, Footnote (DocLayNet) capture more meaning but introduce annotation disagreement.

The Framework in One Sentence

The Matter vs. Meaning framework decouples the physical nature of what is on the page (what an annotator can identify at a glance, without reading) from the semantic function of that content (what it means in context, which requires understanding the document).

This is a practical framing, not a perfect schema. The goal is minimal inter-annotator disagreement and maximum composability across large teams, diverse document types, and evolving use cases. New document types do not require redefining the primitive set; they extend the role layer.

This page defines what to label, why, and how. The framework, operational heuristics, and annotation workflow are integrated here as a single reference.

The Framework

1. Visual Primitives (The “Matter”)

What kind of content is this, visually?

An annotator should be able to assign a primitive without reading the text or understanding the document’s purpose. The question is purely perceptual: “Is this a block of text? A grid? An image? A formula?” This is the fast, low-disagreement pass.

2. Logical Roles (The “Meaning”)

What semantic function does this primitive serve in the document?

Assigning a role requires understanding context: reading the text, recognizing its position in the document’s structure, or knowing the document type. This is the slower, higher-cognition pass where inter-annotator disagreement is more likely.

A note on multi-primitive roles: Some roles support multiple primitives because the same semantic function can be realized through different visual forms. A PageHeader can be text or a logo image. A Sidebar can contain prose, a data table, or an inset figure. In each case, the primitive determines which downstream processor handles extraction (OCR, TSR, VLM), while the role determines reading-order treatment and structural placement.

3. Attributes (The “Nuance”)

Properties that modify the handling of a primitive without changing its fundamental type.

4. Relations (The “Structure”)

Edges that connect nodes to reconstruct the document graph.

Annotation Workflow

To avoid cognitive overload and ensure high-velocity annotation, we strongly recommend splitting the task into Two Separate Label Studio Projects. Mixing geometry (drawing boxes) with semantics (classifying roles) in a single interface destroys the flow state needed for precise bounding box creation and offers poor visual feedback on completion.

Project 1: The “Matter” Scan (Geometry and Primitives)

• Goal: Pure visual segmentation.
• Action: Rapidly draw bounding boxes around every distinct layout element.
• Classification: Assign only one of the 10 Visual Primitives (Text, Image, Table, etc.).
• Mindset: “Do not read the text. If it is a distinct block, box it.”
• Speed: Fast (~2-3 sec/box).
• Output: A dataset of high-quality bounding boxes with generic labels (e.g., all paragraphs are just Text).

Project 2: The “Meaning” Refinement (Semantic Roles)

• Input: The “Matter” annotations are imported as pre-annotations.
• Goal: Logical understanding.
• Action: Click existing boxes and change their label.
• Gamification: The goal is to turn “Raw” primitives (e.g., gray Text boxes) into specific roles (e.g., red Title, blue Caption).
• Visual Feedback: You can instantly see which boxes are still generic and need attention.
• Constraint: Do not move the boxes. The geometry was fixed in Project 1.

The Matter-First Mental Model

When annotators try to do both steps at once, they often subconsciously alter the geometry to fit their semantic opinion (e.g., excluding “Figure 1:” from a caption box because they think it is “meta-data,” or splitting a paragraph because it contains a variable).

Rule: Geometry is objective. Meaning is subjective. Determine the geometry first.

Level 1: Visual Primitives

Task: Draw bounding boxes and classify only the 10 Primitives.

Overlap Rule: Generally, primitives should not overlap (they are competing hypotheses).

• Exception: Inlines (The Span vs. Block Model). Small Formula, OpticalCode, or Image (icon) regions inside a paragraph are treated as “children.” Draw the parent Text block around the whole paragraph and the child boxes inside it.

Goal: High-precision bounding boxes (IoU > 0.90). This step is purely objective and high-agreement.

Conflict Resolution Rules

When in doubt between primitives during Pass 1, apply these tests in order:

1. The “Rich Text vs. Rendered Art” Rule (Text vs. Image)

   • Test: Is the text rendered as standard selectable fonts?
   • Yes: Text.
   • No: If rendered as vector shapes/outlines (WordArt), heavily distorted, or heavily stylized (e.g., a “30% OFF” burst) $\to$ Image.

2. The “Brand Identity” Rule (Logo vs. Text)

   • Test: Is this text a company logo or wordmark?
   • Result: Always Image.
   • Why: Logos imply specific geometry, color, and branding that standard OCR cannot capture.

3. The “Recursion” Rule (Image vs. Document)

   • Test: Is this an image of a document (screenshot, photo of a receipt) or a contained graphic?
   • Document: Annotate the inner content (Text, Tables, etc.).
   • Graphic: Image (stop and caption).

4. The “Visual Grid” Rule (Table vs. Text)

   • Test: Can the content be read linearly (left-to-right, wrapping) without losing meaning?
   • Yes: Text (e.g., a simple 2-column key-value list like “Name: John”).
   • No: If spatial position (X,Y) defines the relationship (e.g., a matrix of prices), or strict columnar alignment is required to understand it $\to$ Table.
   • Heuristic: 3+ columns usually force a Table classification.
   • Reading order test: Would a single bounding box over this region produce interleaved OCR output because of multiple side-by-side columns? If yes, split into one box per column rather than promoting to Table.

5. The “Interaction” Rule (Field vs. Structure)

   • Test: Is a human expected to physically write on or alter this line/box?
   • Yes: Field. No: Structure.

6. The “Info-Content” Test (Structure vs. Image)

   • Test: Does the graphical element communicate meaning?
   • Yes (e.g., an icon, specific brand pattern): Image.
   • No (purely visual separation: lines, solid blocks, generic gradients): Structure (will be ignored downstream).

7. The “Explicit Geometry” Rule (Selection vs. Text)

   • Test: Does the bounding box for a checkbox include the label text (e.g., “[ ] Yes”)?
   • Result: No. The Selection box must strictly wrap the geometric toggle (square/circle). The label “Yes” is a separate Text box (Role: Option or Label).

8. The “Notation” Rule (Formula vs. Image)

   • Test: Does it use a standardized notation system (math/chem)?
   • Yes: Formula. No: Image (use for creative illustrations).

9. The “Machine” Rule (OpticalCode vs. Image)

   • Test: Is it for machine vision?
   • Yes: OpticalCode. No: Image.

10. The “Key-Value” Rule (Text vs. Table for labeled pairs)

    • Context: Invoice headers, form summaries, financial totals – any block of “Label: Value” pairs.
    • Single inline pair (“Invoice #: 12345”): one Text box.
    • Single-column block (stacked pairs): one Text box. Reading order is linear.
    • Multi-column grid (pairs arranged in two or more side-by-side columns): one Text box per column. Do not draw one large box over the whole grid; do not promote to Table.
    • When is it actually a Table? Only when row-wise relationships across columns carry meaning: “Widget A”, “2”, “$10.00” belong together as a row and the X-axis position encodes that grouping.

Level 2: Semantic Roles

Task: Assign the final semantic label to the primitives. This happens in Project 2.

Goal: Full understanding. Turn every gray box into a colored box if a specific role applies.

Categories to label:

• Headings: Title, Subtitle, SectionHeader
• Content: Abstract, Body, Blockquote, ListItem, Figure, Chart, Diagram, Sketch, Code, ChemScheme, DisplayEquation, InlineEquation, Score
• Meta: Author, Affiliation, Keywords, Caption, TableNote, Credit, Dateline, PageHeader, PageFooter, PageNumber, LineNumber, JumpLine, Footnote, BibEntry, FormulaNumber, Address, Watermark, Annotation, Correction, Logo, Icon, Stamp, Advertisement, Barcode1D, Barcode2D, TOC, Index, Calendar
• Forms: Label, Instruction, Value, Option, Signature, Input, Checkbox, Radio, FormGroup

Guidance: If a Text box is truly just standard body text, label it Body or leave as Text if Body is implied.

Level 3: Attributes

Task: Select a box and toggle any relevant attributes.

• is_handwritten: The text is handwritten/script (for ICR).
• orientation: The content is rotated (0/90/180/270).
• state: checked / unchecked (for Checkbox/Radio).
• is_filled: True for Input Fields/Signatures that have content.
• is_header: True for Table cells that act as headers.
• row_span / col_span: (Integer) Defines cell topology for Tables.

Level 4: Relations

Task: Create directed edges between boxes to reconstruct the logical graph.

Tool: Use the Relation Mode (Alt + Click Source $\to$ Click Target).

Relations to label:

• group_with: Links a Figure/Table to its Caption or TableNote.
• label_for: Links a form Label to its blank Input/Checkbox field.
• value_of: Links a form Label to the filled Value/Option selection.
• header_of: Links a SectionHeader to the Body content it introduces.
• row_of / col_of: Explicitly links a cell to its parent Table (if geometry is ambiguous).

Operational Heuristics

These heuristics expand on the conflict resolution rules with the reasoning behind them.

The “Span vs. Block” Model (Handling Inline Elements)

A major source of ambiguity is the treatment of inline elements like mathematical variables ($x$), small icons ($\to$), or chemical formulas embedded within a sentence.

The problem: Is the variable $x$ a Formula or part of the Text paragraph? If we label it Formula, do we split the paragraph into three boxes (Text, Formula, Text)? This fragmentation destroys semantic coherence.

The solution: Treat these elements as a separate Enrichment Layer on top of the base text layer.

1. Base Layer (The “Block”): The entire paragraph is detected as a single Text block. Action: run standard OCR. Result: “The value of x is 10.” (OCR may misread complex math, but captures sentence structure.)
2. Enrichment Layer (The “Span”): The inline equation is detected as a Formula span inside the text block. Action: run specialized LaTeX OCR on the span crop. Result: $x$.
3. Reconciliation: The pipeline matches the span’s geometry to the text’s character coordinates and injects the high-fidelity result. Final output: “The value of $x$ is 10.”

Key Rule: Any primitive fully contained within a Text block is exempt from NMS. Instead of competing for existence, it is treated as an inline child span of the parent paragraph.

The Overlap Exceptions (Stamps and Watermarks)

Some macro-elements physically exist on top of content but are semantically distinct:

• Stamp (Image): Extracts the visual seal (e.g., “APPROVED”), ignoring text underneath.
• Watermark (Text): Extracts the overlay text (“CONFIDENTIAL”), identifies Role: Watermark, and removes it from the reading order.

Drop Capitals

A drop cap is a large, often decorative first letter spanning multiple lines at the start of a paragraph. We do not define a separate role for it because the drop cap is semantically the first character of the paragraph: its role is Body.

The primitive depends on the Rich Text vs. Rendered Art rule:

• Plain enlarged letter (just a bigger font): Text. Standard OCR can extract it.
• Illustrated or decorated initial (gold leaf, figural illustration): Image. The letterform carries visual information that OCR cannot capture.

In both cases, the drop cap’s bounding box should be treated as a child span of the parent paragraph (the same Span vs. Block model used for inline equations).

Key-Value Pairs and Reading Order

A common document pattern is a block of labeled values: a left column of keys (“Invoice #:”, “Customer:”) and a right column of values (“12345”, “Acme Corp”). These appear in invoice headers, form summaries, and financial totals.

The primitive assignment depends on the spatial structure, not the content:

• Single inline pair (“Invoice #: 12345” on one line): one Text box. A text renderer handles it cleanly and NLP can parse the key and value without layout information.
• Single-column block (stacked pairs, one column): one Text box. Linear reading order is preserved and each pair reads naturally as prose.
• Multi-column grid (two or more side-by-side columns of pairs): one Text box per column. A single bounding box over a multi-column layout produces interleaved OCR output (“Invoice #: 12345 Date: 2026-06-30 Customer: Acme Corp Due Date: 2026-07-30”), which garbles reading order. The column boundary is the natural annotation unit.

These blocks are rarely Table. A Table is warranted only when the row-wise relationship between columns carries meaning (e.g., “Widget A”, “2 units”, “$10.00” belong together as a row and the X-axis position is load-bearing for that grouping). In key-value grids, each pair is semantically independent; no cross-column row relationship exists.

Rule of Thumb:

• Single pair or single column? $\to$ One Text box.
• Multi-column grid? $\to$ One Text box per column.
• Row-wise relationships across columns? $\to$ Table.

The “Grid vs. Group” Distinction

A common failure mode is either (A) forcing the model to read text to distinguish Lists from Tables, or (B) being so permissive that an entire form (like ACORD 125) is swallowed as one giant Table.

The compromise: “Visual Atomicity”

1. The “Stop at Header” Rule: A Table or FormGroup must not cross a SectionHeader or a full-width Structure line. An ACORD form is not one big grid; it is five separate grids (“Agency”, “Insured”, “Coverages”, etc.), separated by their headers. This ensures downstream parsers receive manageable chunks, not a whole page.
2. The “Visual Grid” Rule (inner content): Inside those chunks, if it looks aligned, label it Table. Do not worry whether it is actually a list of records or a matrix of numbers. The downstream parser can distinguish them once it has the clean chunk.
3. Vital Inline Elements: Tables can feature an inline overlay of key primitive types like Selection, Field, and Image. This allows capturing checkboxes and input fields inside a table without needing to split it into separate boxes.

Rule of Thumb:

• Is it 2D aligned? $\to$ Table.
• Does it cross a Section Header? $\to$ SPLIT IT. Two Tables.

Why There Is No “ListContainer”

Many datasets (PubLayNet, M6Doc, DocGenome, OmniDocBench) annotate lists as a single bounding box around the entire block. We intentionally omit a ListContainer role for several reasons:

1. Ambiguous primitive: A list container is visually just Text. An annotator cannot reliably distinguish “a block of list items” from “a paragraph” at a glance; there is no clear visual signature separating them.
2. The useful unit is the item: Downstream tasks (chunking, hierarchy recovery) need individual ListItem boundaries, not the outer wrapper. Annotating the container encourages skipping the harder, more valuable step.
3. Geometry problems: In multi-column layouts, a list can span columns or wrap in ways that make a single rectangular bounding box misleading or impossible.

When mapping datasets that annotate list containers, we map them to ListItem and note the granularity mismatch.

Cross-Page and Cross-Column Continuation

The continues_from relation handles cases where a paragraph, table, or list splits across a page or column break. The second fragment continues_from the first, signaling that they are parts of the same logical element and should be merged during reconstruction.

Several datasets annotate this explicitly. HRDoc provides cross-page parent-child relations for hierarchical document reconstruction. LADaS uses a -Continued suffix on region labels to mark fragments that span page boundaries. DocGenome annotates Identical relations between regions that belong to the same logical entity across pages.

Most single-page detection models ignore continuation entirely. For downstream reconstruction pipelines that stitch pages into a coherent document, continues_from is essential for producing correct output. Without it, a paragraph split across two pages becomes two unrelated text blocks.

Global Decision Precedence

Since most CV models require mutually exclusive classes, these rules handle objects with dual natures. They override specific primitive/role guidelines.

1. Navigation trumps Content (The “Crop Rule”)

   • Scenario: A company logo inside the top margin.
   • Classification: PageHeader (not Logo).
   • Why: In RAG pipelines, we usually want to “crop out” headers to avoid repetition. If we label it Logo, it survives the crop and pollutes the stream.

2. Function trumps Semantics (The “Usage Rule”)

   • Scenario: A handwritten signature on a line.
   • Classification: Signature (not just Text).
   • Why: Its role triggers a specific downstream action (Signature Verification) that generic Text does not. The printed text “Sign Here:” is an Instruction.

3. Specific trumps Generic (The “Specificity Rule”)

   • Scenario: A standard chart in a financial report.
   • Classification: Chart (not Figure).
   • Why: Chart implies numeric data extraction is possible; Figure implies visual description only.

4. Content trumps Chrome (The “Exclusion Rule”)

   • Scenario: A running header “Page 4 of 20” or “CONFIDENTIAL” watermark.
   • Classification: PageNumber (or PageHeader) / Watermark.
   • Why: These labels act as “stop words.” Any region with these roles is deleted from the final text extraction stream. If you mislabel them as Text, they will pollute the output.

5. Layout trumps Logic (The “Visual Grid Rule”)

   • Scenario: A resume “Experience” section or invoice address block arranged in columns.
   • Classification: Table (Primitive) $\to$ FormGroup (Role).
   • Why: Reading order is ambiguous in 2D space (row-major vs. column-major). If you cannot read it top-to-bottom, left-to-right without losing structure, it is visually a Table. We resolve the semantic logic later.

6. Scope trumps Continuity (The “Stop at Header Rule”)

   • Scenario: A long form (e.g., ACORD 125) with multiple sections (“Agency”, “Insured”, “Coverages”) that all look like grids.
   • Classification: Multiple Table boxes, split by the headers.
   • Why: A Table or FormGroup must not cross a SectionHeader or a full-width Structure line. This ensures downstream parsers receive manageable chunks rather than one giant, mixed-schema blob.

Domain Case Study: Interactive Forms

Forms are documents with a high density of interaction primitives. This section details how the framework handles user input zones and states.

Form Interaction Primitives

Critical Distinctions

1. Signature Lines vs. Input Lines:

   • Visual (Primitive): Both are Field with format="line". They are physically just lines waiting for ink.
   • Semantic (Role): Context determines the role. A field next to “Sign:” is Role: Signature. A field next to “Name:” is Role: Input.
   • Downstream: Signature fields trigger Verification/Matching Models. Input fields trigger HTR Models.

2. Explicit vs. Implicit Selections:

   • Explicit (Primitive: Selection): A physical box or circle exists. The model detects the geometry and classifies the interior pixel state.
   • Implicit (Primitive: Text + Role: Option): The user circles or strikes through the text “Strongly Agree.”
   • Comb processing: A series of small adjacent boxes, one per character. Requires Split-and-Merge: the image is sliced by cell vertical lines; each cell is recognized individually as a single character; results are concatenated.

Binding Relations

To reconstruct a form, link these primitives:

• group_with(Figure, Caption): Groups an image and its description.
• value_of(Label, Text): Links a question to its answer (e.g., “Name:” $\to$ “John Doe”).
• header_of(SectionHeader, Body): Hierarchical grouping.

Implementation Strategy

The taxonomy above is designed around human annotation: primitives are what an annotator can label quickly by sight, while roles require deeper understanding. This same principle carries over to model design.

The “Visual Salience” Hierarchy

Some roles are visually distinctive enough that both annotators and models can assign them without reading the text. We use this to decide which roles to target directly in Stage 1.

1. High Salience (Target Directly): These roles have unique geometric signatures (font size, position, texture).

   • Title (vs. Body)
   • ListItem (bullets/indents)
   • Caption (proximity to image + small font)
   • PageHeader / PageFooter (absolute position)
   • Footnote (separator line + bottom location)

2. Low Salience (Target as Primitive): These roles look identical to generic text blocks.

   • Address $\to$ detect as Text
   • Author $\to$ detect as Text
   • Body $\to$ detect as Text
   • Instruction $\to$ detect as Text

Recommended Detection Classes

For a robust layout model (e.g., RT-DETR), we recommend a simplified class list (~15 classes) that mixes primitives and high-salience roles:

1. Textual Anchors: Title, SectionHeader, Caption, ListItem, PageHeader, PageFooter, Footnote.
2. Visual Primitives: Table, Figure, Formula, OpticalCode, Text (catch-all).
3. Form Primitives: Input, Checkbox, Signature.

The Two-Stage Pipeline

• Stage 1 (Vision): Specific classes (like Title) are accepted as-is. Generic classes (like Text) are passed to Stage 2.
• Stage 2 (Logic): A Multimodal Model (LayoutLM) or graph heuristic acts on the generic Text boxes to assign specific roles (Author, Abstract, Address) based on content and context.

The Recall Advantage: “Surface Everything”

A common failure mode in fine-grained detection is premature rejection: a model fails to detect a “Section Header” because it looks slightly ambiguous, so it outputs nothing at all.

By simplifying the visual targets to “Matter” (e.g., just Text), we significantly lower the complexity of the visual decision boundaries. This allows the detector to optimize for High Recall, ensuring every patch of ink is captured as a distinct segment.

It shifts the failure mode from “missing data” (vision failure, unrecoverable) to “mislabelled data” (logic failure, recoverable). In production, a mislabelled paragraph is easier to fix than a missing one.

Label Studio Configuration

To support the two-project workflow, we use two different configurations.

Config A: Project 1 (Matter)

Simple, fast, no nested choices. Focus on drawing.

<View>
  <Image name="image" value="$image"/>
  <RectangleLabels name="label" toName="image">
    <!-- Visual Primitives Only -->
    <Label value="Text" background="#FFAABB" />
    <Label value="Image" background="#BBFFAA" />
    <Label value="Table" background="#BBAAFF" />
    <Label value="Formula" background="#00FF00" />
    <Label value="Structure" background="#CCCCCC" />
    <Label value="Music" background="#FF00FF" />
    <Label value="OpticalCode" background="#0000FF" />
    <Label value="Field" background="#FFCC00" />
    <Label value="Selection" background="#FF5500" />
    <Label value="Redaction" background="#000000" />
  </RectangleLabels>
</View>

Config B: Project 2 (Meaning)

Designed for refining labels and adding attributes. We flatten the taxonomy here to make the “Color Change” explicit (changing Text to Title) but use Choices for boolean attributes.

<View>
  <Image name="image" value="$image"/>

  <RectangleLabels name="label" toName="image">
    <!-- Visual Primitives (Gray - To be refined) -->
    <Label value="Text" background="#CCCCCC" />
    <Label value="Image" background="#CCCCCC" />
    <Label value="Table" background="#CCCCCC" />
    <Label value="Structure" background="#CCCCCC" />
    <Label value="Formula" background="#CCCCCC" />
    <Label value="Music" background="#CCCCCC" />
    <Label value="OpticalCode" background="#CCCCCC" />
    <Label value="Field" background="#CCCCCC" />
    <Label value="Selection" background="#CCCCCC" />
    <Label value="Redaction" background="#CCCCCC" />

    <!-- Role: Headings (Red) -->
    <Label value="Title" background="#FF0000" />
    <Label value="Subtitle" background="#FF4444" />
    <Label value="SectionHeader" background="#FF8800" />

    <!-- Role: Content (Blue / Teal / Lime) -->
    <Label value="Body" background="#4444FF" />
    <Label value="Abstract" background="#0000FF" />
    <Label value="Blockquote" background="#8888FF" />
    <Label value="ListItem" background="#00AAFF" />
    <Label value="Code" background="#000088" />
    <Label value="Figure" background="#00FFFF" />
    <Label value="Chart" background="#00AAAA" />
    <Label value="Diagram" background="#008888" />
    <Label value="Sketch" background="#006666" />
    <Label value="ChemScheme" background="#44AA00" />
    <Label value="DisplayEquation" background="#88FF00" />
    <Label value="InlineEquation" background="#AAFF44" />
    <Label value="Score" background="#CC44FF" />

    <!-- Role: Meta (Green / Navy / Orange) -->
    <Label value="Author" background="#008800" />
    <Label value="Affiliation" background="#008844" />
    <Label value="Keywords" background="#008888" />
    <Label value="Caption" background="#00FF00" />
    <Label value="TableNote" background="#00AA00" />
    <Label value="Credit" background="#006600" />
    <Label value="Dateline" background="#004400" />
    <Label value="PageHeader" background="#AAFF00" />
    <Label value="PageFooter" background="#AAFF00" />
    <Label value="PageNumber" background="#CCFF00" />
    <Label value="LineNumber" background="#CCFF66" />
    <Label value="JumpLine" background="#AAFFAA" />
    <Label value="Footnote" background="#555500" />
    <Label value="BibEntry" background="#888800" />
    <Label value="FormulaNumber" background="#BBBB00" />
    <Label value="Address" background="#666600" />
    <Label value="Watermark" background="#444444" />
    <Label value="Annotation" background="#FF0088" />
    <Label value="Correction" background="#FF0055" />
    <Label value="Logo" background="#002222" />
    <Label value="Icon" background="#00AAAA" />
    <Label value="Stamp" background="#008888" />
    <Label value="Advertisement" background="#00FF88" />
    <Label value="Barcode1D" background="#000088" />
    <Label value="Barcode2D" background="#0000AA" />
    <Label value="TOC" background="#A0522D" />
    <Label value="Index" background="#D2691E" />
    <Label value="Calendar" background="#CD853F" />

    <!-- Role: Forms (Purple / Brown) -->
    <Label value="Label" background="#880088" />
    <Label value="Instruction" background="#FF00FF" />
    <Label value="Value" background="#CC88CC" />
    <Label value="Option" background="#BB66BB" />
    <Label value="Signature" background="#440044" />
    <Label value="Input" background="#660066" />
    <Label value="Checkbox" background="#FF5500" />
    <Label value="Radio" background="#FF8800" />
    <Label value="FormGroup" background="#8B4513" />

  </RectangleLabels>

  <!-- Mocked for now; still reviewing how to best implement in Label Studio -->
  <!-- <Choices name="attributes" toName="image" perRegion="true">
    <Choice value="is_handwritten" />
    <Choice value="rot_90" />
    <Choice value="rot_180" />
    <Choice value="rot_270" />
    <Choice value="checked" />
    <Choice value="unchecked" />
    <Choice value="is_filled" />
    <Choice value="is_header" />
    ...
  </Choices>

  <RelationLabels name="relation" toName="image">
    <Label value="group_with" />
    <Label value="label_for" />
    <Label value="value_of" />
    <Label value="header_of" />
    <Label value="row_of" />
    <Label value="col_of" />
    ...
  </RelationLabels> -->
</View>

Note: Depending on your Label Studio version, you may prefer to use a single RectangleLabels block containing both sets, or use the LayoutLMv3 template if you are doing strict taxonomy mapping.

Known Limitations

• No spatial relations. The framework captures semantic and logical edges (group_with, header_of) and reading order (follows), but not spatial adjacency (above, below, left, right). GraphDoc defines these, and they matter for form understanding and table reconstruction. We may add spatial relations in a future revision.
• refers_to requires text understanding. Cross-references (“see Figure 3”) cannot be resolved from layout geometry alone, making them a Stage 2 task. GraphDoc reports only 16.8% AP for reference relations using visual features, underscoring the difficulty.
• Heritage-specific visual roles are thin. Decorative elements (vignettes, friezes, illuminated initials) map to Image or Structure, but the role layer does not distinguish art-historically significant ornament from layout chrome. Heritage DLA practitioners should extend the role set for their domain.
• Single-page scope. Most primitives and roles are defined for a single page. Cross-page reconstruction depends on continues_from and follows relations, which few datasets annotate consistently.

Conclusion

The core insight of the Matter vs. Meaning framework is that the two hardest problems in document layout analysis require different kinds of thinking, and forcing annotators (or models) to do both at once is where most taxonomies break down.

Matter asks: “What kind of content is this?” An annotator can answer quickly, by sight, with high agreement. Text looks like text. A table looks like a grid. A formula looks like spatial notation. These judgments are fast, reliable, and largely independent of document type or domain.

Meaning asks: “What role does this content serve?” That requires reading, context, and sometimes domain knowledge. Is this text block an abstract or an introduction? Is this image a figure or a logo? These judgments are slower, more subjective, and more likely to vary across annotators and document types.

By separating these two passes, we get practical benefits at every stage:

1. Annotation scales. The fast visual pass can be done by any annotator across any domain. The slower semantic pass can be done by specialists, deferred to a second stage, or handled by rules and models. New document types do not require redefining the primitive set.
2. Models get simpler targets. A vision model optimizing for “find all text regions” has far simpler decision boundaries than one trying to distinguish “Abstract” from “Body.” This trades classification errors (recoverable) for detection misses (not recoverable).
3. Teams share a common language. OCR engineers, data curators, and downstream NLP researchers can coordinate on a shared primitive vocabulary even when they disagree on roles, because the visual layer is stable across domains.

The taxonomy is deliberately not exhaustive. We expect roles to grow as new document types are encountered, and we expect the relations to deepen as cross-page and cross-document reconstruction become better understood. The primitives, by contrast, should remain relatively stable: the ways humans put marks on a page change slowly.