Daejeon, July 27–31
The integration of Large Language Models (LLMs) into Digital Humanities (DH) has shifted the focus from the labor-intensive digitization of sources to the automated extraction of meaning. Generative AI offers unprecedented capabilities in structuring unstructured corpora and identifying latent associations, promising a new era of efficiency for historical network analysis. However, this technological leap introduces a fundamental epistemic friction: the conflict between the probabilistic nature of LLMs and the deterministic requirements of historical scholarship.
Historians operate under a regime of strict evidence; a claim must be traceable to a specific locus in a primary source. LLMs, acting as “stochastic parrots” (Bender et al., 2021), generate outputs based on statistical likelihood rather than factual verification. This often results in “hallucinations”—plausible but fabricated relationships—which render raw AI outputs unsuitable for serious historiography.
This paper proposes a methodological intervention. Rather than viewing GenAI as a replacement for traditional DH methods For the purposes of this study, we define 'traditional digital humanities methods' as the suite of computational techniques predominant in the field prior to the widespread adoption of LLMs. These encompass rule-based processing (e.g., regular expressions, keyword concordance), statistical analysis (e.g., frequency distributions), and explicit network modeling, etc. Unlike the probabilistic and generative nature of LLMs, most of these traditional methods are characterized by deterministic logic and reproducibility, making them suited to serve as verification mechanisms in an AI-driven workflow.
A common misconception in the age of GenAI is that unstructured text interaction (Chat-with-PDF) renders structured data obsolete. We argue the opposite: structured relational data (e.g. social networks) are more critical than ever, for both human interpretation and AI reasoning.
Historical knowledge is often scattered across vast, heterogeneous corpora. As the scale of data exceeds human cognitive limits, structured networks serve as a crucial mediator between qualitative close reading and quantitative distant reading (Moretti, 2013). A network graph is not just a visualization; it is an intermediate representation that allows historians to toggle between the macro-structure of social capital and the micro-details of individual agency (Düring et al., 2016). Without the rigid backbone of structured data, this multi-scalar analysis is impossible.
Even for AI, structure is the key to accuracy. While Retrieval-Augmented Generation (RAG) significantly reduces hallucinations by grounding LLMs in external documents (Lewis et al., 2020; Garcia & Weilbach, 2023), it suffers from “knowledge truncation” when dealing with global questions (e.g., “What are the major clan conflicts in this era?”). Recent advancements in GraphRAG (Edge et al., 2024) demonstrate that when AI is supported by a structured knowledge graph, its ability to reason about complex, inter-connected topics improves dramatically. This validates our premise: Structuring and linking data is not an outdated practice, but the prerequisite for advanced AI capabilities.
We present a workflow applied to the extraction of a historical social network, designed to ensure that every edge in the graph is philologically sound, as shown in Figure 1 The flowchart illustrates the transformation from unstructured text to verified structured knowledge. It highlights the critical "Grounding" phase, where AI-generated claims are mathematically validated against original sources before analysis.
Figure 1 The Grounded Generative Pipeline for Historical Data.
We utilize a generative AI agent (based on DeepSeek-V3) to process historical texts. To mitigate the inherent unpredictability of LLMs, we adopt the Spec-Driven AI paradigm recently proposed for scientific data analysis (Chen et al., 2025). Unlike open-ended prompting, this approach requires defining a rigorous Knowledge Specification (Spec) before generation, treating the LLM as a schema-compliant inference engine rather than a creative writer.
To manage context window limits, we employ a sliding-window chunking strategy. The AI is then constrained to map unstructured text onto our pre-defined Spec. This specification is enforced via a strict JSON schema containing {Source, Target, Relation, Original_Text}. Crucially, aligning with the principles of rigorous data modeling in digital humanities (Flanders & Jannidis, 2019), we impose a hard “Evidence Constraint”: the model is instructed that every extracted relationship must include a verbatim excerpt (Original_Text) from the source. This constraint serves as a “validity contract” (Willard & Louf, 2023), ensuring that the generated structure is not a hallucination but a verifiable index of the primary source.
Figure 2 Visual Implementation of Algorithmic Grounding in Orange 3.
This workflow, as shown in Figure 2, illustrates the core verification mechanism. To implement this sophisticated pipeline without imposing a high technical barrier on humanities scholars, we utilized Orange 3 https://orangedatamining.com/
Functionally, the workflow is inspired by the "word cloud and concordance" example https://orangedatamining.com/widget-catalog/text-mining/concordance/
The execution of this lower branch proceeds through three stages. First, in the Vectorization phase, we project both the AI-extracted evidence and the segmented corpus into a shared high-dimensional vector space using the embedding models such as Multilingual SBERT and fastText in Orange 3. Second, we perform a Similarity Calculation to compute the Cosine Similarity between the AI’s generated citation and the ground truth. Finally, in the Anchoring phase, relationships exceeding a strict similarity threshold (e.g., >0.95) are marked as "Verified." Crucially, this step automatically replaces the AI’s potentially approximated quote with the precise string from the original corpus, recovering exact metadata such as page numbers. Conversely, matches falling below the threshold are flagged as "Unanchored" for subsequent human review, ensuring rigorous quality control.
We applied this framework to Lives of the early Medici: As told in their correspondence (Janet Ross, 1910), a collection of letters central to understanding the political and artistic patronage networks of the Renaissance. The application of our Spec-Driven pipeline yielded the following key findings regarding extraction efficacy, reliability, and analytical potential.
First, regarding extraction, the LLM demonstrated a capability to identify complex, implicit relationships (e.g., subtle patronage requests) which traditional keyword-based methods often miss.
Second, our verification results highlighted the necessity of the epistemic anchor. Acknowledging the inherent stochasticity of LLMs—where extraction yields vary across different models and prompting strategies—our multiple experimental runs consistently indicated an "unanchored rate" ranging between 10% and 15%. Our vector-based grounding mechanism successfully flagged these entries. Crucially, we classify these items as "unverified" rather than definitively "hallucinated"; while they lacked sufficient vector similarity to specific text chunks, they may still represent historically plausible associations. Therefore, our pipeline preserves them as a prioritized queue for subsequent expert human verification, ensuring no potential insight is discarded without scrutiny.
Finally, the resulting clean dataset enabled a high-fidelity network analysis in Orange 3 or Gephi-like network analysis and visualization tools. The verified graph revealed distinct clusters that not only aligned with established historical scholarship but also offered new granular details regarding the intersection of financial and artistic networks.
As AI becomes a standard tool in the humanities, we must resist the temptation of relying on intuition and unverified outputs. This paper demonstrates that the future of Digital Humanities lies not in abandoning traditional methods for AI, but in fusing them. By using structured data as the scaffold and traditional algorithms as the quality control, we can harness the semantic power of GenAI while adhering to the rigorous standards of the historical discipline. In this framework, the scholar remains not just a user of tools, but the architect of valid knowledge.