Daejeon, July 27–31
This paper introduces the data architecture of Humanitext Antiqua, a project designed to transform static TEI/XML archives of Western Classics into "AI-Ready" data structures suitable for Retrieval-Augmented Generation (RAG). While the Digital Humanities have successfully established TEI as the standard for text preservation, a significant gap remains in utilizing these hierarchical trees within the flat, vector-based probability space of Large Language Models (LLMs). We propose a multi-layered architecture that maintains TEI as the "Single Source of Truth" while generating a derivative "inference layer." Specifically, this paper details our Python-based "Structure-Aware Chunking" algorithm, which utilizes Canonical Text Services (CTS) protocols to resolve the conflict between philological citation standards and the fixed-size context windows of LLMs.
For decades, the Text Encoding Initiative (TEI) guidelines have provided the rigorous framework necessary for digital scholarly editing. However, the paradigm shift toward Generative AI presents a fundamental challenge: the "Tree" structure of XML is incompatible with the "Vector" space required by RAG systems.
Standard RAG implementations typically rely on "naive chunking"—splitting text mechanically by fixed token counts (e.g., every 500 tokens). In the context of Classical scholarship, this approach is fundamentally flawed. Mechanical slicing frequently cuts across traditional division boundaries (e.g., cutting a chapter in half) or merges disparate sections (e.g., the end of chapter 1 and the start of chapter 2) into a single vector. This results in a loss of granular addressability, making it impossible for LLMs to accurately cite the specific provenance of the retrieved information. This paper reports on the development of Humanitext Antiqua, focusing on our "Layer 2" architecture: a methodology for converting TEI/XML into semantically preserved, CTS-compliant chunks optimized for precise vector retrieval and integration with scholarly reading environments.
The necessity of transforming TEI for computational use is well-documented (McCarty, 2005; Flanders / Jannidis, 2018). Recent initiatives like Santini (2024) and the Perseus Digital Library’s ATLAS architecture highlight the move toward "machine-actionable publications."
However, a unified standard for "TEI-to-RAG" conversion remains unestablished. While recent dynamic segmentation methods like LumberChunker (Duarte et al. 2024) leverage LLMs to identify semantic breakpoints effectively in narratives, they do not guarantee alignment with canonical citation schemes (e.g., Book/Chapter/Section). Consequently, current solutions often sacrifice citation precision for retrieval speed. When a chunk mechanically or semantically straddles two distinct div units, the semantic integrity of the citation is compromised. Our work addresses this specific deficiency by prioritizing "Citation Integrity" over semantic fluidity, ensuring that the AI’s retrieval unit aligns perfectly with the scholarly citation unit.
To reconcile preservation with inference, we propose a decoupled, three-layer data model:
Layer 1: The Archive (TEI/XML). The immutable master data, prioritizing philological rigor.
Layer 2: The Text Body (JSON/Vector). The focus of this paper. A derivative layer where chunks are strictly aligned with CTS divisions, serving as the retrieval engine.
Layer 3: The Knowledge Graph. A reasoning layer (future work) where complex intertextual relationships and the apparatus criticus are modeled as semantic graph structures.
The core contribution of this work is the rag_chunker algorithm. Instead of arbitrary token slicing, our system utilizes the CTS URN hierarchy to define immutable boundaries.
Standard chunkers often leave "orphaned" text segments—small remainders of a chapter that are mechanically appended to the next chapter to fill a context window. This confuses the LLM regarding the source of the text. Our algorithm implements Island Isolation and Equal Balancing:
A key innovation of Humanitext Antiqua is the alignment between the retrieval layer (Layer 2) and the user interface. By enforcing CTS div alignment in the RAG chunks, we establish a 1:1 mapping between the vector search results and the Humanitext Reader, a reading environment currently under development.
While Layer 2 provides a clean "Base Text" for high-accuracy vector retrieval, the Critical Apparatus (variant readings) and editorial notes are preserved in the Reader. Because the RAG chunks share the exact CTS URNs with the Reader's data structure, a user (or AI agent) can retrieve a concept from the vector database and instantly "jump" to the corresponding locus in the Reader to inspect the full philological context, including critical notes. This architecture delegates "search" to Layer 2 and "close reading/verification" to the Reader, solving the hallucination problem by grounding every AI response in a verifiable scholarly interface.
Initial tests on the Perseus Greek corpus indicate that CTS-aligned chunking significantly improves the model's ability to self-cite sources accurately. By preventing the mixing of distinct text divisions within a single vector, we eliminate the primary cause of "citation hallucination."
Future work will focus on Layer 3, where the apparatus criticus will be modeled not just for reading, but for graph-based inference, allowing the AI to reason about textual variance and secondary literature. However, the foundation remains the alignment established in Layer 2: respecting the canonical structures of the Western Classics to create data that is not only "AI-Ready" but "Scholarship-Ready."