Daejeon, July 27–31
Digital humanities (DH) scholars increasingly rely on research infrastructures that transform heterogeneous page images (scans, microfilm, photographs) – especially historical and archival materials – into searchable, citable, interoperable data. While conversational LLMs enable exploratory use, they are ill-suited as primary extractors because of hallucination risks (including silent normalisation of historical spelling variants, fabrication of named entities and dates, and generation of fluent but source-untethered text passages) and opaque provenance - limitations that undermine reproducibility and source criticism central to DH (Ji et al., 2023, & more recent evidence to be presented at the Conference). At the same time, the field faces a structural risk of model collapse when systems are trained on recursively generated content, underscoring the need for pipelines grounded in human-sourced evidence (Shumailov et al., 2024). We, as a research-driven organisation launched after an academic, EU-funded project, present a suite of lightweight, end-to-end document-understanding models designed for non-profit, collaborative deployment within European DH infrastructures. These models move upstream - from pixels to structured, provenance-bound data - thereby supplying reliable inputs that subsequently ground retrieval-augmented and generative tools for DH analysis (Lewis et al., 2020).
DH platforms must provide (i) interoperable outputs aligned with community standards (e.g., TEI; IIIF Presentation API), (ii) evidence binding between extracted facts and source regions for scholarly audit, and (iii) sustainably governed services deployable on institutional or cooperative infrastructure (e.g., DARIAH). The AI models presented in this study explicitly adopt the FAIR principles and emit outputs compatible with IIIF manifests and TEI encodings, enabling precise citation, entity linking, and machine-actionable corpora across repositories, datasets, and tools (Wilkinson et al., 2016; IIIF, 2023; TEI Consortium, 2025; DARIAH-ERIC, 2025).
The machine-learning models offer two complementary types:
Core models. An encoder–decoder transformer for whole-document understanding. The encoder is a hierarchical vision transformer with shifted-window self-attention to capture local and global layout efficiently; the decoder autoregressively emits a sequence augmented with special structural tokens (named entities, key–value fields, layout marks). This design follows Swin Transformer, delivering favourable accuracy–efficiency trade-offs at high resolutions typical of DH page images (Liu et al., 2021).
Deep models. An OCR-free, end-to-end architecture: a fully convolutional encoder with 2D positional encodings produces spatial features; a transformer decoder with character-level attention generates text interleaved with structural tokens. This reduces error propagation endemic to cascades (line segmentation → OCR → NER) and improves robustness to faint ink, bleed-through, skew, and curved baselines common in historical materials - following the design of the Donut on OCR-free document AI (Kim et al., 2022).
The recent dominance of decoder-only multimodal LLMs (MLLMs – e.g. GPT-4V, Qwen-VL) in general AI benchmarks does not make them suitable as primary extractors in scholarly DH pipelines. In this presentation we demonstrate these limitations through concrete, worked examples drawn from our development experience: (i) Dialogue output format: MLLMs generate conversational responses by design; suppressing this via system-prompt engineering or ‘Skills’-style wrappers is fragile, opaque, and ill-suited to automated batch processing. (ii) Long-form hallucination: unlike schema-constrained encoder–decoder models, autoregressive LLMs hallucinate extended passages, invented entity names, and fabricated dates that are syntactically fluent yet untethered from source regions – directly undermining the evidence-binding requirements of DH scholarship. (iii) Normalised spelling variants: MLLMs silently modernise or regularise historical orthography (e.g., early modern spelling forms, dialect variants), destroying the philological evidence that is often the primary object of study. (iv) Schema enforcement: generating valid, schema-constrained output (TEI, JSON-LD) is structurally at odds with free-form autoregressive generation; post-hoc validation reintroduces the cascaded error propagation that end-to-end encoder–decoder models are designed to eliminate. Taken together, these failures reinforce rather than contradict the rationale for specialised encoder–decoder architectures: our technical choice is not an oversight of recent AI developments but a deliberate design decision grounded in the epistemological requirements of historical scholarship. (v) Long context window issues: For long documents (e.g. newspaper pages) we still observe errors that are not made when the document is fed to the models in smaller parts, despite the now multi-million token count that some of these models are able to process.
The presented models produce schema-aware sequences (e.g., JSON-LD/TEI elements with coordinates) and links each field to bounding regions plus calibrated confidence, supporting human-in-the-loop verification and scholarly audit. Outputs can be packaged as PAGE XML files for viewers and annotation tools and exported as TEI XML for critical editing, thus bridging digitisation, annotation, and analysis workflows typical in DH. As clean, verifiable inputs to Retrieval-Augmented Generation, model outputs measurably improve grounding and factuality of downstream query and narrative generation used by DH teams (Lewis et al., 2020; IIIF, 2023; TEI Consortium, 2025).
Sustainable DH infrastructure depends on governance models that prioritise community control, transparency, and long-term autonomy. We present READ-COOP/Transkribus – the platform through which the models presented here are deployed and made available to DH researchers worldwide – as a case study which demonstrates how a platform cooperative can host ATR/HTR/AI services as community-owned, shared infrastructure while mitigating vendor lock-in and aligning incentives with the public good (Terras et al., 2025). This cooperative framing complements existing European DH e-infrastructures such as DARIAH-ERIC (Terras et al., 2025; DARIAH-ERIC, 2025).
(i) End-to-end, schema-aware generation: he decoder directly emits validated structures (TEI/JSON-LD) with minimal post-processing. (ii) Reduced error propagation: OCR-free variants provide benefits by collapsing recognition and understanding into a single model (Kim et al., 2022). (iii) Evidence binding and calibration: every extracted field carries coordinates and calibrated confidence (ECE/Brier), enabling curator triage and reproducible scholarship. (iv) Open benchmarks for DH: we will release recipes, evaluation code, and representative printed/handwritten corpora under permissive licences, aligning with FAIR for data, tools, and workflows (Wilkinson et al., 2016).
We assess extraction quality (CER/TED) and layout grounding (mAP, IoU for regions) across printed and handwritten DH corpora. We hypothesise improved factuality and traceability due to higher-quality evidence.
Smart Extract turns fragile page images into durable, machine-actionable knowledge that is (a) interoperable with IIIF/TEI tools used in pedagogy and research; (b) computationally tractable for institutional clusters; and (c) trustworthy for citation and critique. By elevating extraction to a first-class, evidence-preserving step under cooperative governance, DH scholars gain better corpora for close & distant reading, higher-quality inputs for LLM-assisted exploration, and shareable datasets that integrate into DARIAH-style workflows, moving decisively beyond ChatGPT toward scholar-first AI (Terras et al., 2025; DARIAH-ERIC, 2025). (Terras et al., 2025; DARIAH-ERIC, 2025.)