Daejeon, July 27–31
Large Language Models (‘LLMs’) now dominate debates about technology and textuality across the humanities. Much of this work takes what we call a representationalist approach, which assumes that a model’s outputs reflect properties of their training data and can be investigated separately from their conditions of production and use. This paper reports on a practical inquiry into the limits of, and alternatives to, this representationalist understanding, pursued through masking experiments with BERT models finetuned on historical corpora. Building on a recent theoretical account of LLMs as demonstrating the probabilistic nature of humanistic textual practices, we offer a speculative experiment that explores the potential of embracing rather than resolving computational indeterminacy, and ask what it would mean to approach LLM outputs as situated enactments of historical probabilities.
Representationalism has been extensively questioned within the history and philosophy of science as a basis for understanding modelling. Various arguments instead maintain that scientific models do not mirror or reflect existing systems but participate in their articulation (Knuuttila 2011; de Oliveira 2021). Even accounts that retain a representationalist component emphasise that models do far more than represent their targets, including enabling interference, coordinating practices, and enacting phenomena (Morrison & Morgan 1999; Cartwright 1999; Giere 2004). Yet representationalism remains the default framework through which word embeddings and LLMs are understood, in technical and humanistic arguments alike (Mikolov et al. 2013; Rogers et al. 2020).
In machine learning research, this framework is most apparent in the ‘probing’ paradigm: the systematic attempt to determine what linguistic, semantic, or world knowledge is encoded by the ‘internal representations’ of artificial neural networks (Kantamneni et al. 2025). Researchers treat models as containers for semantic content that can be read off from activation patterns or geometric relations in vector space (Heinzerling & Inui 2024; Engels et al. 2024; Li et al. 2025). This framing underpins both celebratory accounts in which LLMs “learn” rich and “highly interpretable” representations of syntax, semantic, and world knowledge (e.g. Cunningham et al. 2023) and critical accounts in which they fail to “ground” their representations in genuine meaning (e.g. Bender & Koller 2020; c.f. Mollo & Millière 2023). Representationalism similarly organises humanistic critiques of LLMs – as repositories of cultural or linguistic bias (Bonil et al. 2025) or as distorting or losing detail (Delétang et al. 2023) or meaning (Bender, Gebru, et al. 2021) in what they copy – and descriptions of them as powerful semantic search devices capable of surfacing latent cultural affinities and textual connections (Wilson et al. 2023).
This latter claim is made by the largest digital humanities project, to date, to pursue historical research with LLMs. The Living with Machines (‘LwM’) project, conducted at the British Library’s Turing Institute from 2018-2023, based many of its arguments on a BERT-derived model fine-tuned on nineteenth-century British Library books and journals (‘BLERT’). They used this model for various masked-token querying experiments, hiding a keyword and examining the model’s “unexpected” or “unusual” completions as evidence of usages in the underlying archive, and presenting changes in completions between models trained on different historical corpora as signalling shifts in cultural understandings and usages (Wilson et al. 2023). While the LwM arguments are careful with their findings, they generally assume that the model reproduces the latent patterns in the corpus closely enough that its completions can be taken as evidence of features of that corpus.
Our experiments with the LwM approach demonstrate the practical limits of this representational framework. To explore Irishness in Australian literature, we began by fine-tuning the LwM’s BLERT model with text from To Be Continued (TBC) (Bode and Hetherington 2018-): a database of textual and bibliographical data for over 51,000 publications of novels, novellas and short stories published in historical Australian newspapers. We created an overall BLEAT (British Library-Australian BERT) model based on 100 years of TBC data (1840-1940), as well as five successive models trained on 20-year subsets of the data (1840-1859, 1860-1879, 1880-1899, 1900-1919 and 1920-1939).
For an initial masking experiment, we used 90 sentences from our corpus containing “Irish” or “Irishman” and examined the most likely completions from our different BLEAT models. The results included some correct predictions (14% overall) and, alongside some mundane responses (e.g. “man,” “men,” different nationalities), a tendency towards positive cultural associations (e.g. “musical,” “noble,” “humble,” “simple,” “sweet”). But when we turned to interpreting these individual results and what they suggested about the corpus or the language model, we found ourselves assuming causal mechanisms for what were probabilistic situations - an assumption undermined by the evident influence of the unmasked words in the query sentences on the results produced. While the LwM authors observe the influence of model opacity on their findings, in using specific results to support claims about cultural meaning and historical change they assume a much more direct and deterministic relationship between inputs and outputs of the model than we found reasonable.
A range of theoretical arguments challenge representationalism without offering alternative approaches to employing LLMs for historical and textual research. Wendy Hui Kyong Chun’s Discriminating Data (2021) argues that algorithmic systems perpetuate social categories not by reproducing what training data “contains” but through operations that structure learning operations, including homophily, clustering, and optimisation. Phan and Wark (2021) make a parallel argument for race in Large Image Models, describing it not as a preexisting category reflected (or mis/represented) by the data, but as an epiphenomenon of algorithmic processes of classification, proxy selection, and inductive inference. Such arguments resonate with earlier digital humanities critiques of the limits of representationalism, in studies that explore how computational architectures and modelling procedures participate in the distinctions and relations through which race (McPherson 2012) and gender (Mandell 2019) become conceivable. The problem, for our purposes, is that while these accounts reconceptualise what models do in non-representational terms, they offer little practical guidance on how such insights can be translated into methods for working with LLMs.
This lack of methodological guidance is highlighted by a recent article by Fabian Offert and Ranjodh Singh Dhaliwal (2024) that highlights the representationalist assumptions structuring current critical AI methodologies through three “casuistries.” The “benchmark casuistry” connects most explicitly to the approach that we initially took with our results, and that is suggested by the LwM project: individual completions invite interpretations as textual evidence - a word surfaced from the archive or library collection, citable and traceable - when they were artefacts of a probabilistic process with no such evidentiary pathway. The “black box casuistry” reinforces this error, in treating LLMs as fixed objects with inner (opaque) workings, and the “stack casuistry” describes how this logic connects to assumptions about training data, in assuming a traceable pathway from input through to algorithmic operations and output. While the authors argue that LLMs demand different kinds of methodological engagement, their “propaedeutic” essay does not offer them. It does, however, point to an insight from Johanna Drucker that we have developed into an experiment with our BLEAT models.
Proposing what it might mean to operate at the “interface” of qualitative and quantitative critique,” Offert and Dhaliwal (2024: 4) refer to Drucker’s (2018) proposal “to understand the act of interpretation as the collapse of the probability space of all possible interpretations – to always keep in mind, in other words, that even the best anecdote is a sample.” This suggested a different approach to our BLEAT models: rather than selecting individual high-probability completions as findings, we extract full probability distributions over all possible substitutes, treating the entire vocabulary as the object of analysis. For each of our five temporal models, we analyse:
(1) mean probability and variance assigned to “Irish” itself,
(2) the distribution of high-probability alternatives; and
(3) divergence from the BLERT and BLEAT baseline models.
In thereby tracking which substitute tokens show consistent rises or falls in probability across each period, we examine the entire vocabulary as a probability distribution. Computing the Jensen-Shannon divergence between each period-specific model’s predicted distribution and that of the overall baseline gives us a coarse-grained measure of the period’s drift. While this single-scalar measure is a simple one, it offers a useful first orientation, indicating which periods diverge most sharply from the aggregate and where token-level inspection is likely to be most productive. We explore the results of this experiment, not as revealing the meaning of Irishness in different periods, but as engagements with the probabilistic, contextual, and apparatus-specific nature of LLM outputs.
Even with millions of probability assignments, no accumulation of completions will resolve the underlying epistemic and evidentiary difficulty. The problem is not that the models are small or the data is limited, but that each output is a completion within an apparatus that contingently configures text, query, and model, in ways that resist the forms of citation, corroboration, and archival location through which textual evidence is ordinarily incorporated into humanities arguments. Approaching LLM outputs in this way shifts attention from what a model “contains” to what practices of querying, fine-tuning, and interpreting bring forth. We conclude by suggesting that grappling with LLMs for historical inquiry may require abandoning the separation of “cultural object” and “humanistic interpretation,” and treating LLM outputs as situated performances of historical probabilities.