In the Evidence Graph Retrieval paper I argued that a tagged document already tells you how it is organized, and measured what retrieval loses by ignoring that. It left a question open on the evaluation side, and the more I sat with it the less comfortable it got: who decides what a document is tested on? Every RAG benchmark starts with questions written by people or by language models, and both favour the parts of a document that read like prose. A table cell that means nothing without its row and column headers, a list that is only a list under its heading, a rule silently reversed by a footnote — these rarely make it into the exam. And when the same model family writes the questions and answers them, the exam and the student share a prior.
The new stage of this research, Document Task Answerability Assessment (DTAA), inverts the order. A tagged PDF declares its own organization: headings and their scope, tables with header relations, lists, figures with alternative text, notes and references. DTAA compiles that structure and enumerates every extraction, comparison, enumeration, finding and qualification task the document deterministically supports — each bound to its gold evidence before any question is written. A seeded, stratified sample is frozen as the document’s own benchmark; a language model only writes the wording, in four controlled phrasings from document-literal to lexically distant, and a different model validates it. Answers are scored in layers — evidence, relations, correctness, completeness, grounding, provenance — so every failure is attributed, not just counted.
What the pilot found, on eight documents that all score 98–100 on accessibility checkers:
- Across 87 tagged documents the method enumerates 61,551 tasks, reproducible bit-for-bit; the pilot froze 436 of them into 1,725 validated questions and 5,175 scored runs.
- The best system — structure-preserving EGR — completes only 42% of task phrasings under the strict pass contract. Chunk-based hybrid retrieval completes 30%; the same machinery over layout-reconstructed structure, 23%.
- Failures are 73–92% retrieval, not reasoning. Given the evidence, the answering model is equally competent everywhere; grounding stays at 93–95%.
- Phrasing is a first-order variable. Ask in the document’s own words and the best system passes 55%; move the wording away from the document’s vocabulary and it halves, to 26% — the same slope for every system.
- Facts that exist only as a figure’s alternative text are simply unreachable to text-only pipelines: 0% without the tags.
The conclusion I care most about is again aimed at my industry: “well-tagged” should mean answerable, not merely compliant. Every one of these documents passes the checker with a near-perfect score, yet fewer than half of the tasks their own structure supports can be completed reliably through retrieval. The paper closes with eighteen authoring guidelines and an “answerability lint” for tagging services — derived from the observed failure inventory, not from checker rules. A document’s benchmark is no longer someone’s opinion about it; it is a property of the document itself.
The full paper — method, motif catalogue, pilot results and guidelines — is here: Document Task Answerability — Li (2026), a companion to Evidence Graph Retrieval.