There is a danger, as there was in the 1990s when technologies such as .html, .css, and .xml were adopted in the commercial world, that humanities researchers will assume corporate capture of artificial intelligence (AI) is inevitable. It would be natural to assume that the best we can do is experiment with ‘black box’ commercial services or slightly more transparent open-source tools, rather than working at the level of code and infrastructure. If this occurs DH risks having our traditional focus on open data, open code, open methods, and open epistemology undermined. There is an urgent need for DH researchers to engage with AI at a technical level, developing methods that support technical, cultural, and epistemological transparency (Ries, van Dalen-Oskam, and Offert 2024). This needs to be augmented with design methods and documentation that aligns work to existing standards such as FAIR (including FAIR software) (Chue Hong et al. 2021) and CARE (Carroll et al. 2020), emerging standards in Indigenous data sovereignty (Abdilla et al. 2020; Munn 2024), and principles of responsible and slow AI (McLennan et al. 2020; Levy 2024). Theoretical frameworks such as Critical AI (Bode and Goodlad 2023; Raley and Rhee 2023), Critical Infrastructure Studies (CIS) (Liu, Pawlicka-Deger, and Smithies 2026; Liu 2017), and Critical Technical Practice (CTP) (Agre 1998) can guide more theoretical aspects of such work.

This paper describes a Large Language Model (LLM) Retrieval Augmented Generation (RAG) tool developed by the AI as Infrastructure (AIINFRA) project that was used in focus groups of 20–30 users at the Australian National University, Aotearoa New Zealand Department of Internal Affairs, and the UK National Archives during 2025 (Smithies et al. 2025). Rather than being a typical end user product ATLAS (Analysis and Testing of Language Models for Archival Systems) is a ‘test harness’ that enables reproducible and transparent experiments with multiple LLMs, vector stores (databases), and word embeddings. Built using open-source code, the tool uses a common RAG architecture based on FastAPI, Chroma DB, Redis, Langchain, and Vue to enable users to query documents in a vector store. This has allowed us to design ATLAS to prioritise scholarly values and explore how Indigenous values and data sovereignty can also be supported. The harness provides users with detailed technical information about the calibration of each experiment (model version, word embedding, retrieval ranking, vector store characteristics, system prompt), as well as the source documents used by the foundation model to generate its response. The goal was not to produce another AI product for non-technical users, but to understand how feasible it is to expose LLM RAG technology to transparent and holistic DH analysis and understand whether it is still possible to build tools that provide scientific transparency from code through to data collection, processing, and management. By running the same test prompts against different test configurations in ATLAS, it becomes possible to observe and assess the impacts of different models and design choices in an iterative and reflexive way and save the resulting test data in a research repository. Details of cost per prompt and observability of code functions are supported through a connected cloud platform. Code (including extensive technical documentation) and data generated from experiments with ATLAS are available on Zenodo alongside supporting process documentation (Smithies and Berman 2025a; Smithies et al. 2025; Berman and Smithies 2025).

Our primary design goal was to move (as far as possible) beyond black box assessment of AI research tools, enabling reproducible research within the constraints of probabilistic systems (Berman and Smithies 2025). This required careful development of a baseline vector store for the system that can act as the foundation of a longitudinal data collection process. The vector store can be configured for different sources depending on research interest. This paper describes a version of ATLAS that uses a corpus comprising the letters of Charles Darwin digitized by the Cambridge Digital Library (Cambridge Digital Library) (Smithies and Berman, 2025a). Every effort has been made to ensure data consistency across the corpus. A processing pipeline chunks the documents and extracts TEI-XML metadata to enable filtering in the user interface and the core RAG retrieval process. The vector store creation is self-documenting, in that the enabling code is contained in the primary ATLAS code repository on GitHub and documented in such a way that users can gain a clear understanding of the database design strategy. Historical word embeddings produced by the Living with Machines project ensure the semantic relationships in the vector store align to the historical semantics of the letters, and the tool’s architecture enables the impact of different word embedding models to be observed and evaluated (Hosseini et al. 2021). Hybrid search, using BM25 with Reciprocal Rank Fusion (RRF), supports both dense and sparse search patterns, and the open source code allows technical users granular control over the configuration of that process. Each query generates detailed telemetry that records the retrieval and reranking process, providing a high degree of transparency.

Other free and commercial tools enable some of that functionality but cannot provide the same level of transparency and reproducibility. They usually severely limit control of the document sources, do not support historical word embeddings, do not allow the document retrieval process to be configured, and do not provide detailed telemetry (which is a valuable source of information for companies eager to monetise their products). This reduces complexity for end users but does not support best practices in research software test engineering (RSTE) or enhance AI literacy. It also limits the potential for experimentation with open data and severely compromises scientific transparency and reproducibility (Winner 1993). To facilitate the broadest range of test scenarios as possible, ATLAS supports commercial language models such as OpenAI and Anthropic, but users can also use the open source Ollama product to run open-source models locally. The freemium Phoenix Arize platform is used to provide deep levels of observability, tracing queries between the user interface, vector store, and LLM.

The understanding we gain from ATLAS will be used to test other LLM RAG products and (ultimately) to develop a framework for the analysis of LLM tools in the GLAM and humanities sectors, but it primarily functions as an “epistemology engine” (Smithies 2017), “that bring[s] human knowers into intimate relations with technologies or machinic agencies through which some defined model of what is taken as knowledge is produced” (Ihde, quoted in Van Den Eede 2015, 151). The transparency provided by ATLAS takes users beyond the surface of AI systems towards the technical nuts and bolts of RAG systems, highlighting the many configurations that inform their output (Berry 2022). Our approach is supported by process documentation that guides the evaluation and testing process, including HASTRICT principles refined in consultation with the Aotearoa New Zealand Department of Internal Affairs (Smithies and Berman 2025b): H: Human; A: Accountable; S: System targets; T: Transparent; R: Reproducible; I: Iterative; C: Contextually Informed Measures & Baselines; T: Tasks, workflows & use cases. This is part of a broad socio-technical approach to testing and evaluation of LLM systems for the historical and GLAM sectors, that acknowledges the need for technical harnesses such as ATLAS and guidance that supports AI literacy across expert user groups.

Regardless of whether RAG architectures are optimal for use in DH and the GLAM sector (an open question), ATLAS demonstrates that it possible to implement systems that support best practice in RSTE and promote scientific transparency, reproducibility, and AI literacy. In keeping with this, ATLAS also enables structured feedback by domain experts. Each question and answer is followed by a series of Likert inputs that allows users to record their interpretation of the system’s performance in terms of factual accuracy, corpus fidelity, analysis quality, relevance, difficulty, and clarity. Users can record if they consider themselves an expert or non-expert user, flag issues such as hallucination and bias, and enter free text comments. This data is added to system telemetry and stored for research purposes (Smithies et al. 2025). Significant attention has been paid to user privacy, so that user logins cannot be associated with questions or feedback, and users can choose to toggle a privacy mode that does not save telemetry for research purposes.

After describing our critical methods and providing a technical overview suitable for a general audience, the paper will provide a live demonstration of ATLAS, describing its vector store (Figure 1), its user interface (Figure 2), user feedback form (Figure 3), and the Phoenix Arize telemetry platform (Figure 4). The audience will have an opportunity to request logins to the system for time-limited experimentation. We will aim to focus the presentation on the end-to-end flow of data (from code -> user input -> user feedback -> telemetry -> publicly available research data) to demonstrate the value of ATLAS as a test harness and educative tool. Code, telemetry data including expert user feedback, and presentations for the project are available at https://zenodo.org/communities/aiinfra/.

Figure 1: ATLAS Vector Store Database

Figure 2: ATLAS User Interface

Figure 3: ATLAS User Feedback Form

Figure 4: Phoenix Arize Observability Interface