Context and motivation

Large language models are increasingly used to explore digitized historical corpora. However, in historical research, it is essential that historians can trace a model’s claims back to precise sources. Retrieval-Augmented Generation (RAG) (Lewis et al. 2020) addresses this need by grounding answers in retrieved passages, thereby reducing hallucinations. Yet historical data pose specific challenges: increased hallucinations when processing very long documents (Leng et al. 2024), the necessity of cross-referencing heterogeneous sources, and substantial OCR noise. Furthermore, during training, large language models are rarely exposed to highly specific and hard-to-access materials such as historical sources (Guo et al. 2024). This limits their ability to reason over specialized historical corpora, where domain expertise and contextual understanding are essential.

This poster presents an evaluation framework that treats “source tracing” not merely as a component of answer generation, but as a measurable capability that conditions interpretability, citation granularity, and usability for historical scholarship (Pellet et al. 2025). We focus on (i) evaluating source retrieval under realistic constraints in long, noisy documents, and (ii) building a benchmark aligned with historians' practices, composed of single-hop and multi-hop questions (Yang et al. 2018; Ho et al. 2020; Schnitzler et al. 2024) spanning two types of historical documents.

Data

Our case study focuses on the French Third Republic (1870--1940). We use parliamentary debates published in the Journal Officiel de la République française and available as OCR-derived plain text online for the years 1880-1940 for both the Chamber of Deputies and the Senate.These documents are intrinsically long and structured (sessions, sections, speaker turns), but they are also affected by OCR artifacts that complicate segmentation and retrieval. To test multi-source historical reasoning, we complement the debates with two widely circulated contemporary newspapers, Le Gaulois and L'Intransigeant, which reported on and commented parliamentary activity from distinct political positions.

For question-driven evaluation, we rely on two historian-supervised datasets. The first contains 1,486 single-hop factual questions generated from debate segments, each paired with the exact source excerpt used to create the question. The second, HistoriQA-ThirdRepublic,(Pellet et al. 2026) is a French-language corpus of 1,782 questions (897 single-hop; 885 multi-hop) designed to reflect realistic patterns of historical inquiry (cross-source synthesis, temporal constraints, sparse evidence). It is built from 3,386 document chunks drawn from debates and newspapers, with historian validation and metadata integration.

We operationalize “source tracing” at the retrieval stage and extend it to end-to-end RAG evaluation. First, we compare structure-aware chunking strategies (Smith / Troynikov 2024) for long debates: segmentation by section, by speaker turns, and bounded variants that enforce a maximum chunk size while preserving document structure. Retrieval is evaluated with Recall@K and nDCG@K; crucially, we introduce token-budgeted comparisons, where retrieval is assessed under a fixed maximum number of input tokens. This reflects frugality constraints (context windows, cost, latency) and better matches historians' practical needs: citing precisely rather than pointing vaguely to “somewhere in the debate.”

Figure 1. A sample of multi-hop question generated with Retrieval-Augmented Generation (RAG) pipeline. Given a query about the 1887 military law, the system retrieves relevant documents from heterogeneous sources and generates a multi-hop answer, which can be compared against the ground truth.

Methods

Second, we generate multi-hop candidates (Zhuang et al. 2024) by pairing semantically similar passages across corpora using dense embeddings and similarity thresholds, and we guide question generation through an iterative, historian-in-the-loop prompting process. Automatic question generation has indeed shown promising results for enhancing the evaluation of RAG systems in domain-specific settings, by enabling a systematic assessment of a model’s reasoning abilities over long documents, and reate more realistic evaluation scenarios for domain specialists (Lin et al. 2024; Li et al. 2025; Hwang et al. 2024). We distinguish multi-hop types inspired by established taxonomies (follow-up, bridge-entity, comparative) and impose source-pairing strategies. Finally, we evaluate not only retrieval but also answer quality, and we test LLM-as-a-judge (Tan et al. 2025) protocols against human annotations to estimate when automatic evaluation is trustworthy.

Results

Three main findings emerge:

• Chunking is a first-order design choice for historical RAG. Section-level segmentation yields the best raw retrieval scores but produces very large chunks (around 11k tokens on average), which weakens citation granularity and increases computational cost. A bounded, structure-aware strategy (sections + speaker turns with a maximum size) offers a better cost--performance trade-off: it stays close to section-level recall while drastically reducing the token burden and enabling more precise source pointing. Under token-budgeted evaluation, smaller, more controlled chunks can outperform very large sections by reducing “semantic dilution” in embeddings.
• Multi-hop retrieval is substantially harder than single-hop retrieval. In HistoriQA, single-hop retrieval is strong (embedding Recall@3 = 67.8, improved to 70.6 with reranking). By contrast, cross-newspaper multi-hop retrieval is low (embedding Recall@3 = 14.3), while cross-domain newspaper--debate retrieval is higher (Recall@3 = 47.8) but suffers from a systematic corpus bias: in a mixed database, the retriever over-selects debate chunks, even for press-only queries. Restricting retrieval to source-specific subsets markedly improves cross-newspaper performance, showing that corpus-aware retrieval (or explicit source-type priors) is crucial for multi-source historical tasks.
• Reranking is not uniformly beneficial. While it can improve Recall@10 and MRR in some settings, it can also reduce top-K recall and accuracy for multi-hop queries (e.g., in newspaper--debate retrieval, Recall@3 drops from 47.8 to 45.5 after reranking). This suggests domain mismatch and bias amplification effects, particularly harmful when questions require balancing heterogeneous sources. On the generation side, accuracy drops sharply from single-hop to multi-hop across all evaluated models (best LLM-judged accuracy around 58.9\% for single-hop vs.\ 18.9\% for multi-hop), indicating that compositional reasoning and cross-document synthesis remain the main bottleneck even when retrieval partially succeeds.

Contribution and poster content

The poster proposes an integrated, historian-oriented evaluation framework for RAG applied to long and noisy archives: (i) token-budgeted retriever evaluation coupled with structure-aware chunking choices that directly control citation granularity, (ii) a historian-aligned multi-hop benchmark supporting cross-source analysis, and (iii) empirically grounded guidance on when reranking and LLM-based evaluation are reliable - or, conversely, misleading.

The poster will include a pipeline diagram, a cost--performance comparison of chunking strategies, examples of multi-hop question types with cross-source links, and a set of actionable recommendations: chunk design, source-aware retrieval, and evaluation protocols that reflect real historical reasoning.