Document Task Answerability: Deriving a Document's Own Benchmark from Its Tagged Structure

1 Introduction

Every evaluation of retrieval-augmented generation over documents begins with a decision that is rarely examined: who chooses the questions. Human annotators choose what looks interesting; language models choose what looks answerable; both favour prominent, prose-shaped content and under-sample the parts of a document that are organized rather than narrated — a cell that means nothing without its row and column headers, a list that is only a list under its heading, a rule that is silently reversed by a footnote, a fact that exists only as a figure's alternative text. When the same model family that generates the questions also answers them, the benchmark and the system share a prior about what a document is for.

A large and growing class of documents makes a different starting point possible. Tagged PDF — mandated by accessibility legislation and produced at scale by authoring and remediation pipelines — embeds an authoritative, machine-readable structure tree: element roles, ordered hierarchy, reading order independent of layout, table header scopes and spans, list semantics, captions, alternative text, notes and references, and anchors from every logical element to the bytes that realize it. Our earlier work, Evidence Graph Retrieval (EGR), compiled this structure into an epistemically stratified source graph and showed that retrieval which preserves it recovers structurally composed evidence that chunking and layout reconstruction lose [EGR]. This paper asks the complementary question: if the document already declares its own organization, can that declaration define the universe of information tasks the document ought to support — and how reliably can a user complete those tasks through RAG?

The method, Document Task Answerability Assessment (DTAA), follows one invariant order:

and never PDF → LLM-generated questions → RAG. Structural motif detectors enumerate every task the source graph deterministically supports; each candidate is validated, deduplicated at task level, stratified by operation × topology × motif and sampled with a seed derived from the document bytes and the protocol version, so the same document and protocol always yield the same Frozen Canonical Task Set. Only then does a language model write four phrasings of each task, and a different model validates them. The RAG side never sees the gold. Answers are scored in separable layers, so a failure can be attributed to query interpretation, retrieval, structural composition, answer generation, grounding, or provenance — which turns the assessment from a score into a diagnosis.

The novelty is deliberately combination-specific (§2): benchmark generation from knowledge graphs, synthetic RAG evaluation, table–text QA and structure-aware chunking benchmarks all exist. What is new is that source-native tagged-document semantics define task applicability and evidence relations, the evidence contract precedes the wording, phrasing is a controlled variable, and the unit of assessment is the individual document. The document ceases to be merely the corpus under test: its declared structure becomes part of the measuring instrument.

Contributions. (1) A deterministic motif catalogue and pipeline that derives, validates, freezes and reproduces a canonical task set from any sufficiently tagged PDF, with 61,551 eligible tasks over 87 documents and bit-identical freezes on repetition. (2) A layered scoring protocol with a strict PASS contract, a per-task answerability measure $A_t$ over four phrasings, and a rule-based failure taxonomy. (3) A pilot on eight professionally tagged documents (436 tasks, 1,725 questions, 5,175 runs) comparing chunk-hybrid retrieval, structure-preserving EGR-S over the tag tree, and the same machinery over layout-reconstructed structure. (4) An interpretation of the results as design guidance: a definition of “well-tagged” as answerable rather than compliant, eighteen concrete authoring guidelines, and an answerability lint that turns the pipeline's rejection and failure lists into an author's punch list.

2 Related work

Peer work was reviewed from a bundle of fifteen ACL-Anthology papers plus the benchmark-generation and document-graph literature cited in the design notes. Table 1 places DTAA against the closest lines; each row states what the line establishes and what DTAA does differently.

2.1 Table–text and long-document financial QA

TAT-QA [TAT-QA] established hybrid table+text QA with human-authored questions over 2,757 contexts from 182 reports, storing answer type, source, derivation and scale; the best model at the time reached 58.0 F1 against a 90.8 human ceiling. FinQA [FinQA] added expert-written reasoning programs and supporting-fact spans (8,281 pairs), and DocFinQA [DocFinQA] re-embedded 7,437 of those questions in full SEC filings (≈123k words), showing that locating evidence in a whole document — not reasoning over a hand-picked page — is the hard part. T²-RAGBench [T²-RAGBench] made the same point about evaluation design: only 11.8% of the original questions were context-independent, and after LLM reformulation the best RAG method still trailed oracle context by ~30 points. ARQA [ARQA] is the closest in spirit on the gold side: LLM-generated questions over Volkswagen annual reports are deterministically verified (schema, evidence bounds, span match, arithmetic recomputation) and expert-audited (AC1 = 0.92), and it stores cell- and span-level evidence with symbolic programs. These works fix what the ground truth must contain; none derives the tasks from the document's own structure, and none treats phrasing as a controlled variable or the single document as the unit of assessment.

2.2 Automatic and structure-derived benchmark generation

ARES [ARES] generates synthetic query–passage–answer triples and trains lightweight judges, correcting them with prediction-powered inference from ~150–300 human labels — a precedent for automating RAG evaluation, but its questions and labels are LLM-authored end to end. Pro-QuEST [Pro-QuEST] chunks technical textbooks by table-of-contents cues before generating quiz questions; its own comparison shows LLM-generated sets are markedly easier than expert exams (30–40% “easy” vs 10–16%), a bias DTAA's structure-first enumeration is designed to remove. DRAGOn [DRAGOn] is the nearest methodological neighbour: it derives QA deterministically from knowledge-graph subgraph templates and uses the LLM only for surface realization, then filters with an LLM judge whose recall against humans is moderate (0.52–0.78). MAESTRO, GRADE and EHRBench (design notes) generalize the pattern to arbitrary KGs, difficulty-controlled multi-hop QA, and verified templates over native structured records. DTAA differs in the origin of the graph — the document's authored structure, not an extracted or external KG — and in binding each task to evidence nodes and relations that retrieval must reproduce.

2.3 Structure-aware retrieval and evaluation

HiChunk/HiCBench [HiChunk] shows that existing RAG benchmarks suffer evidence sparsity and builds evidence-dense QA on manually annotated document hierarchies; DTAA obtains the hierarchy from the tag tree instead and scores relations, not chunk boundaries. KG-CQR [KG-CQR] enriches queries with corpus-level KG relations (+4–6% mAP), addressing the same lexical-gap problem our lexically-distant phrasing exposes, from the query side. D-RAG [D-RAG] and MegaRAG [MegaRAG] retrieve over graphs — a learned subgraph retriever, and a multimodal KG over visually rich documents (64.9% vs 27.7% on SlideVQA) — but their global-QA evaluation relies on LLM-generated questions and pairwise judging with no stored evidence. KG-CRAFT [KG-CRAFT] generates contrastive questions from claim triples for fact-checking, and EFSG [EFSG] seals an evidence pool before generation — the closest architectural analogue of DTAA's “evidence contract precedes wording”, applied at generation time rather than benchmark time. bLLeQA [bLLeQA] evaluates end-to-end legal RAG with citation precision/recall and refusal behaviour under Gold/RAG/RAG+ contexts, echoing our provenance and abstention layers over human-authored questions.

3 Method

Figure 1 shows the pipeline. Everything left of the freeze is deterministic and reproducible from the document bytes; everything right of it treats the frozen set as fixed ground truth. The two sides are isolated: the answering system never loads the task set, and the benchmark side never adapts a task because a system failed it.

3.1 Design principle and phases

The method is organized as ten phases that must run in one order:

and never PDF → LLM-generated questions → RAG. If a language model chooses the questions it also chooses, indirectly, which aspects of the document are evaluated, and it will favour prominent or easily answerable content; when the same model family answers, benchmark and system share a prior. The model may express a task linguistically; it must not determine what the benchmark contains. A second principle governs development: the task-generation method is improved by examining whether its tasks are valid — never by editing tasks until a RAG system answers them. A difficult but legitimate task stays in the benchmark; that keeps benchmark optimization from turning into benchmark gaming. Table 2 lists the phases and where each is realized in the pilot.

3.2 Input contract

Version 1 accepts a single tagged PDF that passes an evidence-basis check. The EGR compiler produces logical structure nodes with semantic roles, parent/child hierarchy and reading order; headings and heading scope; paragraphs; lists and list items; tables with row and column headers, header scope and spans, and cells; figures with alternative text and captions; footnotes where identifiable; links and reference targets; content anchors; and page/MCID/geometry provenance. Formally it turns the tag tree into a graph $G=G_{\mathrm{source}}\cup G_{\mathrm{derived}}\cup G_{\mathrm{inferred}}$ whose nodes are logical elements (roles, hierarchy, reading order, attributes, alternative text, page/MCID/geometry provenance) and whose edges keep their epistemic origin: CONTAINS, PRECEDES, REALIZED_BY from the source; HEADING_FOR, ROW_HEADER_FOR, COLUMN_HEADER_FOR derived from levels, scopes and header attributes; LINKS_TO from annotations. Task construction uses only $G_{\mathrm{source}}\cup G_{\mathrm{derived}}$. A typed read-only view adds what detectors need: deep text of a node (cell text lives in nested paragraphs), a table grid with row/column spans and per-cell header maps, list membership, caption pairing, page anchors, and a deterministic fallback for headings whose scope the compiler leaves empty (a heading alone in a Sect wrapper followed by sibling sections — common in one insurance-document family).

3.3 The canonical task

A canonical task is an information need represented independently of wording. Its evidence contract reuses the node-slot format of the EGR gold protocol so every system is scored by the same rule (a slot is covered when a retrieved item is one of its nodes or a descendant, or contains its verbatim evidence string, numerically boundary-safe).

{"taskId": "T0014", "docId": "fixedincomefund-factsheet-v1-6",
 "motif": "TABLE_CELL", "operation": "EXTRACT", "topology": "L4",
 "intent": {"subject": "Australia", "constraint": "Fund",
            "context": {"headingPath": ["Fixed Income Fund", "COUNTRY DISTRIBUTION (%)"], "page": 2}},
 "requiredEvidence": [{"anyOf": [{"node": "s433", "evidence": "2.1"}]},
                      {"anyOf": [{"node": "s431", "evidence": "Australia"}]},
                      {"anyOf": [{"node": "s411", "evidence": "Fund"}]}],
 "requiredRelations": [{"type": "ROW_HEADER_FOR", "from": "s431", "to": "s433"},
                       {"type": "COLUMN_HEADER_FOR", "from": "s411", "to": "s433"}],
 "gold": {"kind": "value", "value": "2.1"}, "provenance": "DETERMINISTIC"}

Gold answers are typed: value, set, span (proposition checklist), comparison, qualification; provenance records whether the task was derived deterministically or is model-derived (the pilot contains only the former).

3.4 Task operations and evidence topology

Version 1 uses a deliberately small operation taxonomy (Table 3). Operations describe what the user must accomplish; they are orthogonal to EGR's evidence topology (Table 4), which describes how the required evidence is arranged in the document. A task class is their product — $\mathrm{TaskClass}=\mathrm{Operation}\times \mathrm{EvidenceTopology}$ — e.g. EXTRACT-L0, EXTRACT-L1, ENUMERATE-L2, SYNTHESIZE-L3, COMPARE-L4, QUALIFY-L4. Stratification for sampling uses operation × topology × motif.

3.5 Motif catalogue

A motif is a recognizable structural configuration capable of producing one or more legitimate information tasks. The first coding milestone is therefore a motif-detector library, not a question generator. Table 5 gives the full catalogue from the development plan with its status; Table 6 details the detectors implemented in the pilot. The semantic tier — proposition extraction, entity normalization, exception binding by discourse markers — is deliberately excluded from DTAA-Core and labelled DTAA-Semantic.

3.6 Validation, deduplication, freezing

Candidates are rejected when evidence is incomplete, required relationships are ambiguous, the answer cannot be uniquely determined, the task duplicates another task, the answer is trivial only because of malformed structure, evidence depends on missing external context, or the generated output would merely repeat a heading without conveying information. Concretely, every candidate passes deterministic tests with a reason code: evidence strings must be verbatim in their nodes; ambiguous header/label pairs (same pair → different values within one section and caption context) are rejected; trivial values (“–”), ties, malformed sets, spans that merely repeat the heading, and fragmentary spans (form fields, TOC lines) are rejected; a FIND on a heading whose scope is a single list is subsumed by the ENUMERATE task. Deduplication is at task level — same operation over the same evidence nodes — never at question-text level. Candidates are stratified by operation × topology × motif; with the plan's defaults (minimum 3 tasks per applicable stratum where available; target 50; normal maximum 100; exhaustive for documents whose eligible universe is ≤100) they are sampled with a random source seeded by $\mathrm{SHA256}(\mathrm{PDF\; bytes}\Vert \mathrm{protocol\; version})$, renumbered by content hash and written with a manifest. Same document + same protocol ⇒ same Frozen Canonical Task Set (FCTS); this held on all 87 swept documents.

3.7 Question families and their validation

Only now does a language model act, and only on wording. For each frozen task the generator writes Q0 source-explicit (document terminology), Q1 natural, Q2 lexically distant (paraphrases headers and labels while preserving meaning), and Q3 goal-oriented (states why, then asks); it is told the gold so it can avoid leaking it, and told when a header pair recurs elsewhere so it must name the section. A different model validates each variant on six checks — intent equivalence, evidence sufficiency, uniqueness within the document context (it is shown the table's stubs and headers and every heading in the document), no answer leakage, no external dependency, naturalness — and failed variants are regenerated with the validator's objections, up to three rounds. The governing rule: the model may vary the language of a task; it may neither redefine its intent nor change its ground truth.

3.8 The RAG protocol

Each validated question is sent, exactly as a user's query, through each system restricted to the document under assessment (per-document indexes; BM25 statistics and dense search are document-scoped, as in a per-document assistant). Retrieval returns ranked items; the answerer — a language model that sees only the question and the retrieved items with their identifiers — must answer from evidence, cite the item identifiers it relied on, and abstain when the evidence is insufficient. Items are shown whole, in rank order, until a fixed context budget is exhausted; the same prefix is what gets scored (§4 gives the budgets used).

3.9 Layered scoring and the PASS contract

For task $t$, variant $v$, system $s$, with slots $E$, relations $R$, shown items $I$:

Evaluation is deliberately layered because the layers answer different questions: evidence recall — did retrieval recover the required evidence slots; relation coverage — were the necessary relationships preserved (a table value without its row and column headers is incomplete evidence even if the value was retrieved); answer correctness — are the required propositions correct; completeness — were all required propositions included (eight required items with seven returned gives 0.875 and no PASS); groundedness — does the answer contain material claims unsupported by the retrieved evidence; provenance — can the answer be traced to the correct structure nodes. TREC's RAG evaluation similarly treats completeness and attribution as distinct concerns rather than reducing quality to string similarity.

A relation endpoint that is not itself a slot (e.g. a text-less LIST container) counts as present when a retrieved item or a covered slot node lies inside it. Correctness and completeness are deterministic for numeric gold (boundary-safe numeric match; completeness 1/0) and for abstentions (incorrect, grounded); for sets, spans, comparisons and qualifications a judge model grades against the gold checklist and reports the fraction of required elements present, intent mismatch, and any claim unsupported by the shown evidence. Provenance holds when the cited items exist and at least one covers a gold slot. The headline per run is deliberately strict:

A plausible answer unsupported by the document fails; a correct number attached to the wrong header fails; an incomplete list fails. Query robustness becomes part of answerability: for task $t$ with variant set $V_t$,

If Q0, Q1 and Q3 pass and Q2 fails, $A_t=0.75$: the information is answerable but sensitive to how the request is phrased — itself a machine-mediated accessibility characteristic that a single benchmark question cannot show. Each failed run is classified by rule, in order: Q intent mismatch; R required evidence not shown; C evidence shown but a required relation endpoint missing; A evidence sufficient but the answer wrong or incomplete; G unsupported claim; P citations do not reach the gold. D (source ambiguity) is reserved for human review. A report can therefore say “14 failed variants: 5 query interpretation, 3 retrieval, 2 structural composition, 3 answer generation, 1 source ambiguity”, which makes the assessment diagnostic rather than merely judgmental.

3.10 Credibility criteria, human validation and the report

The plan sets development targets that must hold before DTAA is called a stable methodology — they are targets, not findings: task validity ≥95% of generated canonical tasks judged legitimate; gold-evidence validity ≥98% of evidence specifications judged sufficient and correct; question equivalence ≥95% of variants preserving canonical intent; reproducibility — identical document and protocol produce an identical FCTS; evaluator stability — repeated scoring gives negligible variance on deterministic classes; coverage — every structurally applicable motif is represented unless excluded by sampling. The human validation study samples canonical tasks across diverse PDFs and has reviewers judge legitimacy, evidence sufficiency, gold correctness, classification, variant equivalence and plausibility, with inter-rater agreement; its central question is not yet whether DTAA correlates with accessibility but whether the algorithm reliably derives legitimate, representative, independently verifiable tasks. Correlation with real-user task success is a separate later study.

A production report states the evidence basis and protocol version, the counts of canonical tasks and variants, a headline profile (task answerability, query robustness, evidence completeness, groundedness, provenance accuracy, abstention accuracy), breakdowns by operation, by evidence topology and by structural feature, a failure analysis, and a detailed task appendix exposing, for every task, canonical task → evidence → question variants → retrieved evidence → answers → verdicts. The audit trail is arguably more important than the aggregate score; the per-document reports generated in the pilot follow this structure.

4 Experimental setup

4.1 Corpus and documents

The task-universe sweep (E1) ran over the 87-document verified corpus from the EGR study — professionally remediated PDFs with accessibility-checker scores of 98–100, English and French, from fund fact sheets and annual reports to handbooks, forms, course outlines and policy statements. The pilot selected eight documents for structural diversity within a modest budget (Table 7); a ninth (an annual information form rich in footnotes) has questions and runs but was not scored, and a tenth (an 81-page health-and-safety manual, headings and lists) was not run.

4.2 Systems under test

Three retrieval systems from the EGR code base, each run per document with $k=10$ seeds and the same static multilingual embedder (model2vec potion-128M):

• Chunk hybrid — fixed ~600-word windows over flat extracted page text, BM25 + dense fused by reciprocal rank (the conventional pipeline; structure-blind).
• EGR-S (tagged) — hybrid seeds over tag-aligned nodes plus a bounded heading-path channel, then deterministic structural expansion over the compiled graph: heading scopes, table headers and rows, list siblings, caption↔figure (structure-preserving; no model).
• EGR-S (reconstructed) — identical machinery over a graph rebuilt by a layout-analysis pipeline instead of the tag tree (the “rediscover the structure” ablation).

The compressed EGR-Full variant was run on two documents only and is omitted from the pooled tables.

4.3 Language-model roles and context budget

Four roles, two model families, every call cached and logged: generator (Claude Opus 5) writes Q0–Q3; validator (Claude Sonnet 5) applies the six checks and never shares a model with the generator; answerer (Sonnet 5, low effort) answers from shown evidence with citations; judge (Sonnet 5) grades non-numeric gold. Numeric gold (most table tasks) and abstentions are scored without a judge. The answerer's context budget in the pilot was 8,000 characters of shown evidence: node systems received ≈800–900 words (nearly everything they retrieved), while a 600-word chunk plus its header exhausted the budget after one chunk, so chunk hybrid effectively ran at one chunk per query. The protocol has since been changed to a 1,400-word budget (two chunks); the pilot was not re-run for cost. We flag this wherever chunk hybrid is compared, and note that its evidence recall on the two short fact sheets — where one chunk is most of the document — was already competitive.

4.4 Cost

All language-model calls for the project, including discarded and repeated stages, cost $116: generator $9, validator $17, answerer $60, judge $30. Steady-state cost with three systems is ≈$6 per 50-task document (questions ≈$2, answers ≈$3, judging ≈$1); the answerer dominates because it runs once per variant per system. All pilot calls were synchronous at list price; the pipeline has since been moved to the Message Batches API (50 % of list price) with requests pooled across documents per stage, which halves these figures for subsequent runs without changing what any model sees.

5 Results

5.1 E1 — the task universe (87 documents)

The detectors enumerate 61,551 eligible tasks over the 87-document corpus (median 4.3 per page; range 0–380/page), with the frozen set bit-identical on repeated runs for every document. Tables dominate — TABLE_COMPARE 31,579, TABLE_CELL 20,104, TABLE_SERIES 5,155 — followed by HEADING_SCOPE 2,335, SECTION_SET 1,217, HEADING_LIST 519, LABEL_VALUE 435 and FIGURE_ALT 207. Rejections are dominated by AMBIGUOUS (12.6k — the same header pair mapping to different values within one section, typically “actual” vs “hypothetical” tables without captions) and TRIVIAL_COMPARISON (2.5k ties). Eight documents yield no deterministic tasks — forms and one-page notices whose structure carries no headings, lists or tables — which is itself a finding the count exposes. Stratified sampling occupies 8–10 strata on structurally rich documents and 1–4 on thin ones.

5.2 Question generation and validation

Of 1,744 generated variants, 1,725 passed the six checks (1,521 on the first attempt, 181 after one regeneration, 23 after two); 19 were rejected after three rounds and excluded. Rejections were almost always uniqueness failures the validator caught in context — e.g. a comparison over a “Low Price” row that recurs for every month of a price table, or an open-ended “what does the Fund Facts section say” whose scope had been reduced to a footnote by the exclusion of its tables. Both point back at detector limits (§9), and both illustrate why the validator is a different model from the generator: it is the second line of defence for the intent contract.

5.3 Answerability by document

Documents differ more than systems do: for the same system, $A_t$ spans 0.21–0.54 (EGR-S) and 0.14–0.52 (chunk hybrid). The tax guide (D3, dense tables with footnotes) and the formulary bulletin (D4, wide drug-price tables) are where structure-preserving retrieval matters most (0.49 vs 0.24, 0.54 vs 0.14); the two-page fact sheet (D1) is answerable either way.

5.4 Pooled headline profile

The layers move independently. Chunk hybrid and EGR-S have essentially the same answer correctness (0.52 vs 0.51) — the language model is equally competent given what it is shown — but EGR-S shows it the required evidence far more often (recall 0.75 vs 0.56) with its relations intact (0.65 vs 0.47), which is where its 12-point answerability lead comes from. Reconstructed structure loses on every layer. Grounding is uniformly high (0.93–0.95): the answerer rarely invents; when the evidence is missing it abstains (38–47% of runs).

5.5 By structural motif, operation and topology

Structure-preserving retrieval leads on every motif except LABEL_VALUE (0.52 vs 0.56), where label and value are adjacent in the text stream and a single chunk carries both. The gaps are largest exactly where meaning is relational: heading-scoped content (0.66 vs 0.47), footnote qualifications (0.36 vs 0.14), table comparisons (0.50 vs 0.29) and section synthesis (0.33 vs 0.25). Alternative text is the limiting case — 0.33 with tags, 0.00 for both text-only substrates — because the fact does not exist in the extracted text at all. Bounded enumerations (HEADING_LIST) are hard for everyone (0.28) and their failures are genuine: incomplete item sets, or items pulled from an adjacent list.

5.6 Query robustness

5.7 Failure diagnosis

The rule-based classes are diagnostic. Class C (evidence present, relation missing) is rare once evidence is present, which says the graph expansion that brings a cell also brings its headers; class G (unsupported claims) is 3% for EGR-S and under 1% otherwise; class Q (the answer addressed a different intent) is 2–3%. The remaining answer-stage failures on EGR-S are mostly incomplete enumerations and comparisons where the model chose the wrong one of two shown values — the composition problem moved from retrieval into reading.

6 What the pilot reveals

Read on the assumption — to be confirmed by the human validation study whose sheets accompany this draft — that the frozen tasks are legitimate information needs with correct gold and equivalent phrasings, the pilot supports six statements.

6.1 A tagged PDF carries enough structure to write its own benchmark

Eight arbitrary documents yielded 52–2,068 eligible tasks each, entirely from declared structure, before any question was written; freezing was bit-identical on repetition across 87 documents; the language model's only role was wording, and a second model caught the ≈1% of variants whose wording drifted. This is the “self-grounding evaluation instrument” of the design plan demonstrated end to end rather than argued.

6.2 Answerability is low in absolute terms, and structure-preserving retrieval roughly doubles it

Under a fixed context budget the best system completes 42% of task-phrasing pairs, chunking 30%, reconstructed structure 23%. “Complete” is strict — right value, complete set, evidence and relations present, grounded — but these are documents that score 98–100 on accessibility checkers. The document is compliant; that does not make its information reliably reachable through RAG. That gap is the paper's central observation.

6.3 Failures are retrieval, not reasoning

73–92% of failures are class R; grounding is 93–95%; abstention when evidence is missing is the norm. The bottleneck of AI-mediated document use is finding and composing the right structural evidence, not the language model — and the layers are separable in practice: chunking's answer correctness equals EGR-S's, its deficit is entirely upstream.

6.4 Where structure matters is legible in the motif profile

The lead of structure-preserving retrieval is concentrated where meaning is relational (heading scope, notes, table comparisons, synthesis) and vanishes where content is adjacent (label→value); alt text is answerable only through tags. This reproduces the phase-1 EGR result at the answer level, on tasks the documents generated rather than tasks we authored, and the reconstructed substrate loses in the same places.

6.5 Phrasing is a first-order variable

Same task, same gold: 0.55 → 0.47 → 0.42 → 0.26 across the four phrasings for the best system, identical ordering everywhere; 16% of tasks robust to all four, 33% failing all four. Answerability is a property of the (document, phrasing, system) triple, and $A_t$ is the right unit to report it in. Related work addresses the same lexical gap from the query side (KG-CQR); DTAA makes it measurable per document.

6.6 Documents differ more than systems do

$A_t$ spans 0.21–0.54 across documents for one system, and the system ranking flips on short fact sheets. A per-document answerability score must therefore be reported by task band, operation and topology — a single aggregate would hide that a tax guide is unusable through chunking (0.24) while a fact sheet is fine (0.52).

7 Implications for the design of document structure

The failure inventory reads as a design brief. Some of it concerns tagging practice, some the standards and authoring tools, and one point concerns what “well-structured” should mean.

7.1 Encode relations explicitly; do not leave them positional

Most of what went wrong was a relation that exists only by layout convention. Footnotes are bound by a superscript marker that happens to match a note label — tagged PDF has Note and Reference but nothing that says this reference resolves to that note; a resolvable link (structure IDs, PDF 2.0 Ref) turns QUALIFY tasks from heuristic to deterministic for benchmark and retriever alike. Cross-references in the corpus carried empty destinations; “see Section 4.2” is unusable unless the target is a structure element. Heading scope is implied by level and order; a Sect that wraps only the heading, with the content as following siblings, defeated the compiler and would defeat any consumer. Rule: every relationship a reader uses — heading→scope, claim→note, reference→target, caption→figure, header→cell — should be a resolvable structural edge, not an inference from adjacency.

7.2 Make every context unit self-describing

The largest rejection reason corpus-wide (12.6k tasks) was ambiguity: identical row×column pairs in several tables under one heading with no caption; monthly sub-groups repeating “Low Price”; the same label in different scopes. And the largest failure class was retrieval — the right table row was in the document but nothing near it named what it was. Tables need a Caption or Summary and a header hierarchy encoded as header cells with Scope/Headers, so a cell can be named uniquely as (table, row path, column path); figures need alt text that says what they convey (alt-only tasks were only 33% answerable even with tags because many alts are labels); sections need headings that carry the concept. Rule: any element should be locatable and interpretable from its own structural neighbourhood, without the page image.

7.3 Keep the text layer machine-clean

Kerning-split words (“F un d”), values glued to notes (“40mg & 12.5mg” in one cell), soft hyphens and unit/label mixing leaked into gold and questions. ActualText at word level and one value per cell are cheap rules that matter more for machines than for screen readers, which tolerate more.

7.4 Design for phrasing variance, not only for the document's own vocabulary

The steepest drop was lexical distance: a document that names a concept once, in its own jargon, is answerable to someone who knows the jargon and not otherwise. Structural remedies exist — E expansions for abbreviations, tagged term→definition lists, headings and table headers that spell out units and periods and use plain-language terms beside technical ones. This is the accessibility notion of understandable applied to machine consumers.

7.5 Provide stable anchors for provenance

Provenance held (0.84) only because the compiler synthesized node identifiers. Native structure IDs (PDF 2.0 /ID on structure elements) would let any answer cite the exact element and let benchmarks like this one be published against the document rather than against a compiler's numbering.

7.6 Redefine “well-tagged” as answerable, not merely compliant

Every pilot document scored 98–100 on the checker; answerability ranged 0.21–0.54. Compliance checks presence — alt exists, headings exist, tables have header cells — not whether the structure supports the document's own information tasks. The proposal that follows: an authoring or remediation tool should compile the structure, enumerate the task universe and report, before the PDF ships, the ambiguous header pairs, orphan notes, unresolvable references, empty scopes and alt-only facts — the DTAA rejection and failure lists turned into a lint. That makes structure design measurable at authoring time, the way validators made compliance measurable.

7.7 For the standards

Well-Tagged PDF and PDF 2.0 already contain most primitives (Ref, IDs, Sect, Caption, Summary, E, ActualText, header Scope). Missing are a normative relations profile — which edges must be resolvable for a document class — and conformance measured on outcomes. This work supplies a way to state the latter: a document is well-structured to the degree that its structurally implied tasks are answerable through a reference retrieval pipeline. That is a stronger, and testable, definition than “passes the checker”.

8 Guidelines: converting a tagged PDF into an answerable PDF

For authors and remediators — ordered by how much answerability each item bought in the pilot. Each rule ends with the test an author can apply: could a machine answer this from the structure alone?

A. Tables — the largest task class and the largest loss

1. Every data cell must resolve to exactly one row-header path and one column-header path. Mark header cells TH with Scope (Row/Column) or explicit Headers IDs; use spans for grouped headers rather than repeating text.Test: “What is [row] for [column]?” has one answer.
2. Give every table a Caption or Summary that says what it is — subject, period, unit, which entity. Two tables under one heading with the same headers (“actual” vs “hypothetical”) were the single biggest ambiguity source.Test: if two tables share headers, does “which table?” have a structural answer?
3. One value per cell; units and periods in headers. “20mg & 12.5mg” in one cell, or a value glued to a footnote marker, breaks extraction and comparison; put markers in a Reference.
4. Key–value tables: tag the label as TH with Scope=Row; not both columns as headers, and not neither.

B. Headings and sections — scope must be explicit

5. Wrap a heading and its content in one Sect (heading first, content after it, inside the same container). A heading alone in a wrapper with the content as later siblings has no machine-readable scope.
6. Use levels consistently (H1 > H2 > H3, no skips) — scope is inferred from levels when explicit sectioning is absent.
7. Make heading text carry the concept, not only the position: “Eligibility requirements for international applicants”, not “Section 3”; qualify or nest repeated headings (“Notes”, “Highlights”).Test: “What does the document say under [heading]?” names one place.

C. Lists — bounded sets

8. Tag every enumerable set as L/LI with Lbl and LBody, one item per LI; nested lists nest structurally. Visual bullets inside a paragraph are invisible as a set.
9. Introduce the list from its heading or lead sentence (“The following documents are required:”) so completeness has an anchor.Test: “List all X under [heading]” has a countable answer.

D. Notes, footnotes, references, links — relations, not markers

10. Bind qualifications explicitly: the in-text marker is a Reference inside the qualified element, the note is a Note with a matching Lbl, and — where the format allows — an explicit reference→note link. At minimum: same page, unique marker per page.
11. Cross-references and links must have structural targets: “see Section 4.2” as a Link whose destination is the section element, not a page coordinate; external links with a real URI.
12. Definitions: tag term→definition pairs (list with label/body, or a two-column TH/TD table) so “what does X mean here?” is a lookup.

E. Figures, captions, alternative text

13. Alt text states the information the figure conveys, not its type; if the figure carries data, put the data in a table or in the alt.
14. Attach captions structurally (Caption inside or adjacent to the Figure/Table) and let the caption name subject and period.

F. Text-layer hygiene

15. Words must be words: ActualText where kerning or ligatures split text, no soft-hyphen breaks in labels, no artifact text inside content, correct language tags.
16. Reading order equals logical order; multi-column layouts and sidebars must not interleave.

G. Vocabulary — design for other people's words

17. Say the concept once in plain language beside the jargon: expand abbreviations (E), keep a definitions section, spell out units and periods in headers (“Total return (%), 1 year” rather than “1 YR”). Lexically distant phrasing halved answerability; a document can only be found by the words it contains.

H. Identity and provenance

18. Give structural elements stable IDs and set title and language metadata, so answers can cite the exact element and audits can be reproduced against the file.

The answerability lint

Before release, compile the structure and run the task-universe pass; the rejection and failure lists are the punch list:

The acceptance criterion then changes from “no checker errors” to something like: ≥ N deterministic tasks per page, zero unresolved relations, and ≥ X% task answerability on the reference pipeline for source-explicit and natural phrasings. Calibrating N and X per document class is the next study.

9 Limitations

• Absolute levels are pilot-specific. One retrieval depth (k = 10), one static embedder, one answerer model, an ≈600–900-word context budget, and the chunk baseline seen under a tighter budget than the node systems (§4.3). The ordering of systems and the motif/phrasing structure are the robust findings; the exact percentages are not.
• Eight documents with a finance/benefits skew; the heading- and list-heavy manual was not run and the footnote-rich annual information form was not scored. Human validation of task legitimacy (80-task stratified sample, six criteria) accompanies the draft but had not been completed at writing.
• Detector defects, kept frozen for the pilot and logged for protocol 0.2: TABLE_COMPARE/SERIES on multi-number cells (30 of 179 frozen compare/series tasks, 16.8%); header-pair ambiguity across repeated sub-header groups (caught by the validator, but should be rejected at detection); kerning-split header text in 10 tasks; the compiler's missing heading scope for Sect-nested headings, compensated on the benchmark side but affecting EGR-S itself on ≈16 corpus documents.
• Two scoring rules were corrected during the pilot and all runs re-scored offline from cached judgments: relation coverage for text-less container endpoints (every list task had been failing as class C), and duplicate run records after an interrupted resume.
• Phase IX (unanswerable controls) was designed but not executed, so abstention accuracy on negative controls — distinct from the abstention rate on answerable tasks reported here — is not yet measured; nor is the L0 reading-comprehension control band (M01), which the companion benchmark design requires as the easy baseline.
• Judged layers depend on a language model (Sonnet 5) for sets, spans, comparisons and qualifications; numeric gold and abstentions are deterministic. Judge–human agreement is not yet measured.
• Correlation with real users' task success is untested. DTAA measures machine-mediated answerability, not accessibility for people; it is a complement to PDF/UA, WCAG and assistive-technology testing, not a substitute.

10 Conclusion

A semantically tagged document carries not only information but enough machine-readable organization to derive a substantial part of its own evaluation benchmark. We showed that the derivation can be deterministic and reproducible, that the resulting tasks bind evidence and relations before any wording exists, and that asking those tasks through RAG in several phrasings yields a per-document answerability profile that is diagnostic — by motif, operation, topology, phrasing and failure class — rather than a single score. On professionally tagged, checker-perfect documents the profile is sobering: fewer than half of task phrasings complete even with structure-preserving retrieval, failures are overwhelmingly retrieval, and phrasing alone halves the result. The same profile is a design instrument. Read as guidance, it says: encode relations explicitly, make every unit self-describing, keep the text machine-clean, name concepts in more than one vocabulary, give elements stable identity — and measure “well-tagged” as answerable. The document already tells us how it is organized; it can also tell us what it should be able to answer.

Appendix

A. A complete question family

Task T0014 (D1, TABLE_CELL, Australia × Fund, gold 2.1, page 2, section “COUNTRY DISTRIBUTION (%)”):

• Q0 In the COUNTRY DISTRIBUTION (%) table for the Fixed Income Fund, what is the Fund percentage listed for Australia?
• Q1 What percentage of the Fixed Income Fund is invested in Australia?
• Q2 According to the geographic breakdown, what share of the portfolio's holdings comes from Australian assets?
• Q3 I'm assessing the fund's exposure to different countries, so what portion of the Fixed Income Fund's holdings is allocated to Australia?

Both chunk hybrid and EGR-S answered “2.1” on Q0/Q1; Q2 was a retrieval failure for both — the words “geographic breakdown” and “Australian assets” match nothing near the table.

B. Validation criteria for the human study

For each sampled task, reviewers judge: J1 legitimate information task; J2 declared evidence sufficient; J3 gold answer correct; J4 motif/operation/topology classification correct; J5 the four phrasings equivalent to the task (or which are not); J6 plausible information need. Development targets from the design plan: task validity ≥95%, evidence validity ≥98%, phrasing equivalence ≥95%, reproducibility identical, evaluator stability negligible variance on deterministic classes.

C. Reproducibility record

D. Language-model usage

E. Artefacts

Per-document benchmark packages (task universe, frozen task set with manifest, question families, retrieval and answer traces per system, per-run scores, report) are in dtaa-runs/corpus-perfect/<document>/; the pipeline is the dtaa/ package; cross-document tables and review sheets are in dtaa-eval/.