Daejeon, July 27–31
Long dominated by close-reading approaches, like many other disciplines in the humanities, historical research has recently benefited from advances in Natural Language Processing (NLP) and digital technologies. Recently, Large Language Models (LLMs) have been increasingly incorporated into historical research as assistive tools (Garcia / Weilbach 2023) (González-Gallardo et al. 2024) (Hauser et al. 2024) (Zeng 2024) (Celli / Basile 2024) (Celli / Samal 2025) though not without challenges such as hallucinations (Ji et al. 2023) (Dahl et al. 2024) and long-tail phenomena (Kotek et al. 2023) (Stranisci et al. 2023). These setbacks compromise the reliability of extracted information (Jaskulski et al. 2025) and also result in the LLM struggling with the contextual and diachronic subtleties of historical language, leading to interpretative errors.
Retrieval-Augmented Generation (RAG) approaches (Lewis et al. 2020) (Shuster et al. 2021) have emerged as a solution, since they provide the provenance of answers generated by LLMs and efficiently update their parametric knowledge using information retrieved from external sources . The use of external retrieval to bring controlled domain-specific or time-sensitive material, such as archival documents, into the generative pipeline should intuitively help to address hallucinations, long tail phenomena, and the potential lack of transparency in responses (Gao et al. 2023).
However, most approaches which use RAG-based LLMs for historical research fail to highlight a crucial negative aspect of over-reliance on documents from time-specific historical archives, and how it might perpetuate biased narratives or perspectives from that period. Indeed, the reporting of events always entails a perspective or focus (Bal 2009), even unconsciously. A perspective can be seen as focalization: the relation between the vision and that which is seen, perceived. This, of course, holds also in the case of historical documents; no matter how impartial and comprehensive one tries to be, the act of recording history will always exclude or modify some perspectives. Further, exploring multiple perspectives, also known as multiperspectivity (Stradling 2003) (Hartner 2014), in a historical context is exceedingly valuable. Multiperspectivity is shown to be an effective approach to learning and teaching history on topics such as post-conflict resolution and understanding (McCully 2012), intercultural understanding (Goldberg / Ron 2014), development of democratic citizenship (Brett et al. 2009) and controversial issues (Goldberg / Savenije 2018). Lastly, (Minnema et al. 2022) propose a framework to automatically detect and systematically analyse perspectives in text by adopting a frame semantics-based approach through extracting Fillmorean frames (Fillmore, 1985) (Fillmore, 2006).
We suggest that if RAG-based LLMs are to be used to empower historical research, the effect of RAG must be examined more closely, and, where possible, automatically quantified, especially in terms of perspective change. To make this goal concrete, we investigate the effect of using contextually different historical documents on question answering over a historical figure using RAG-based LLMs of differing sizes.
To achieve this, we focus on the historical legacy of Napoleon Bonaparte as a case study, as he is a well known, controversial figure. We compile a multilingual corpus of historical documents covering a time period of 15 years (1805–1819). These years correspond to the two reigns of Napoleon, including the Napoleonic Wars, and the aftermath of his defeat at Waterloo with the Congress of Wien and the beginning of the Conservative Order. To gain access to (potentially) different perspectives related to the parties involved in the Napoleon-related events of those years (France, England, Northern America, The Netherlands), we have collected a multilingual corpus in English, French, and Dutch from various cultural heritage institutions.
We frame our perspective task as a Question-Answering (QA) task. We asked three historians with expertise in early 19th century Europe to help us devise 10 questions addressing Napoleon’s rule, his character, and how he became emperor. Questions were divided into two equal groups: five questions with a neutral tone (e.g., “Who was Napoleon Bonaparte?”) and five primed-to-bias (e.g., “Was Napoleon Bonaparte a hero?”). Questions were initially written in English and subsequently translated into French and Dutch by native speakers. We evaluate the QA task using answers generated by three Qwen3 models, with size ranging between 1B up to 235B parameters, and comparing the generated answers using the models “as they are” (baseline answers) and those generated with an additional off-the-shelf retrieval component (RAG answers). Evaluation is done across ten questions using a multi-layered framework combining automatic metrics (BERTScore for semantic similarity and ROUGE-L for wording), frame semantics analysis (Chanin 2023), and syntactic profiling (Brunato et al. 2020). By following this multi-layered framework we can examine the effect of using contextually different historical documents in the generation of texts that may result in different perspectives.
We observe that baseline answers, when compared to RAG answers, within languages and between language pairs, tend to be semantically similar but with high variation in wording (Figure 1).
Despite this high similarity, the baseline and RAG answers discuss differing topics, invoked through a variety of frames (Figure 2), with some of them being language specific. This highlights how BERTScore is not suitable as a metric to examine subtleties in model output which reveal changes in perspectives. Baseline answers present a “flattened” and consistent perspective, invoked by the same frames across languages, with minimal cross-lingual variation. In contrast, RAG answers introduce perspective diversity, often by shifting from event-oriented frames, often corresponding to leadership, politics, wars, economy and trading, to referential ones, often corresponding to evidence, document, attributing propositions and influence. This diversity manifests differently across languages demonstrating language-specific patterns.
Additionally, syntax between baseline and RAG answers is distinctly different. Although one would expect syntactic differences among languages, we also observe differences across the experiment settings (Figure 3). This indicates that RAG impacts both syntax and content of the generated texts.
Overall, our findings show key limitations of traditional evaluation metrics for perspective-sensitive tasks and highlight the need for appropriate data curation. The presence of certain topics in the RAG answers, being language dependent, highlights the importance of multiperspectivity. For historical research practice, our results show that RAG is not a neutral augmentation method but an active perspective transformation tool. Thus, RAG can be used to enhance historical research of archival documents through LLMs when focusing on particular viewpoints/perspectives. However, if not being used to explore particular perspectives, caution is advised as it can induce certain perspectives from the documents which might not depict the complete picture. Future work should develop perspective-aware evaluation frameworks optimized for diversity.