Daejeon, July 27–31
Vector space alignment remains a difficulty for word embedding comparison in humanities research. Many embedding algorithms demonstrate instability--a result of their dependence on stochastic machine learning methods and the subsequent dimensionality reductions imposed on the model outputs (Wendlandt et al. 2018). This instability ramifies when we attempt to compare word embeddings, and even more so when comparing word embeddings trained on significantly different corpora. Approaches to diachronic comparison of embeddings have utilized various methods, including Procrustes alignment, but still face major challenges and tradeoffs (Wevers and Koolen 2020).
Word embedding research in the humanities also faces problems of scale: corpus-centered approaches frequently center on corpora that cannot be easily expanded. There is a cap to the number of Hugo-award-winning novels at any given moment, a permanent limit on the number of 18th- century English novels, and a finite number of lines of ancient Greek poetry available to us. Under the conditions of corpus-centered inquiry, expanding training data via the inclusion of synthetic data is unlikely to be appropriate. Word embedding algorithms and comparison methods that require large corpora, or even corpora of comparable size, may not work for humanistic objects of study.
Given the instability of embedding algorithms and corpus restrictions faced by humanistic researchers, Antoniak and Mimno argue for averaging over multiple bootstrap samples instead of relying on single models (2018), which necessarily requires excluding texts from any given sample--posing an interpretive barrier for humanists seeking to study small groups of texts in relation to the entire corpus.
The current landscape of comparative word embedding research requires tradeoffs and careful design. We offer a new method that still requires such tradeoffs and design--building on the choices humanities researchers frequently make in manipulating the text of the corpora they model under the auspices of “cleaning”--reducing complexity--and inverting them.
We propose a method, selective reconciliation, as an approach for comparing corpus-level contexts in embedding space. In this method, particular tokens are modified to reflect their corpus of origin before the set of corpora is consolidated into a single corpus in the creation of a word embedding (e.g., dog becomes dog_corpus_A, dog_corpus_B, dog_corpus_C, and so on). Unmodified tokens will collapse onto each other in the training of the embedding, but the modified tokens are able to remain distinct. If there is no meaningful difference in contexts across corpora, then the modified tokens will remain near each other in embedding space. If there is, however, a meaningful difference in context, they are free to disambiguate.
Our method replicates the functionality of contextual embedding methods (Liu et al. 2020) that create vector spaces based on tokens rather than types; however, it retains the comparable interpretability of embedding models that collapse these distinctions (Arora et al. 2020). It allows us to selectively create a vocabulary by collapsing tokens within specific context windows (in this case, a series of corpora). It also retains the comparable speed and size advantages of a traditional word embedding model while still providing the contextual information related to our object of interest.
This is a corpus-centered approach (Antoniak and Mimno 2018) that treats modeling as a critical act designed to answer a specific question. It treats embeddings as “models of texts qua texts” (Dobson 2022)--and further, as models of corpora qua corpora--rather than a general-purpose abstraction or something optimized for performance on downstream tasks. We believe that this approach creates a bridge across the dual commitments of centering the corpus-specificity of our task and centering the interactions between hermeneutics and the algorithms that facilitate those hermeneutics. Further, it supports comparison of smaller corpora than we would otherwise be able to align using current methods.
Selective reconciliation allows us to use existing meaning-making strategies for the interpretation of word embeddings--including looking for terms with high cosine similarity, using analogy tests via vector math, or looking for neighborhoods, but extends these interpretation strategies to allow for the comparison of historically or contextually variant tokens of single types.
This method has implications for several stages of embedding training and interpretation.
This method requires corpus creation and selection in a such a way that annotating tokens for their corpus of origin is meaningful--corpora are therefore assumed to be representative of something at the researcher’s creation/discretion--up to and including representative of themselves (we can think here of the canonical corpora available to researchers). This method requires researchers to consider carefully the implications of repeat texts, if the corpora overlap.
Researchers frequently take steps to reduce the “noise” in their embeddings, reducing the complexity of the model via tokenization, regularization, and other cleaning approaches such as removing page numbers, boilerplate, tables, etc. Our approach, on the other hand, adds complexity to the model, by including the corpus as a factor in the creation of the vocabulary, ultimately increasing its size. Decisions at this stage therefore need to weigh both the reductions and additions to the vocabulary carefully.
By intervening at the preprocessing stage and creating new tokens, there should be no significant impacts to most forms of embedding model training. More work needs to be done to understand the impact of changes in vocabulary size and type-token ratios on the validity of word embeddings, but researchers should be able to use the existing standards for model assessment (e.g. attributional similarity or analogy tests on the unmodified types).
This method was developed to explore a specific research question: as critics frequently claim slave and robot are synonymous, what is the measurable relationship between the two in science fiction? Are the two terms substitutable?
We turned to a GloVe model, which, because of its approach to context, captures synonymy and antonymy quite well (Pennington et al. 2014). Corpora (Table 1) were selected because they were hypothesized to provide meaningfully different examples of use for slave and robot. A select list of keyterms were modified to reflect their corpus of origin: robot, android, droid, slave, negro, the n-word, and pluralizations. The rest of the words in the model, after filtering out words that appear fewer than five times, were passed through a dictionary filter. We custom-built the dictionary, including all the words in all of the science fiction, because neologisms were of high interest to us.
We highlight here the deeply idiosyncratic nature of our “cleaning,” our embedding algorithm selection, and our term list for modification--all driven by our experience with the corpora and the scholarly literature, our particular research questions, and our hypotheses. This method is more dependent on researcher choice, not less--and we argue that “cleaning” should likewise be a matter of carefully considered researcher choice for the task at hand.
Table 1: Corpora
| Corpus Name | Corpus Description |
| Chicago | A corpus of top United States library fiction holdings, created by researchers at the University of Chicago’s Text Lab |
| Gale American Fiction | A corpus of early American Fiction |
| Hugo Nebula | Novels that have won or been nominated for the Hugo and Nebula award |
| Star Trek | A corpus of Star Trek novels (excluding graphic novels) |
| Star Wars | A corpus of Star Wars novels (excluding graphic novels) |
| SVRSF | Novels from a general science fiction corpus selected because they contained “slave” or “robot” in-text or in-title |
Figure 1: Keyterms across corpora, unscaled
A benefit to the combined corpora can be seen in Figure 1. Many of the keyterms of interest for this particular case are concentrated, or found exclusively, in single corpora--droid being the obvious example, as George Lucas literally owns the word, meaning it only shows up in Star-Wars-affiliated texts. Context connections for droid and other terms would not be easily visible in single-corpus models, and traditional Procrustes alignment would fail, as it requires a consistent vocabulary between corpora.
Figure 2: t-SNE of combined corpus embedding
Figure 3: filtered t-SNE of combined corpus embedding
Figure 2 shows a t-SNE of the full embedding of the combined corpora, and Figure 3 shows that same t-SNE of the same embedding, at the same scale, filtered on the modified keyterms. Not only does this method allow us to rule out synonymy, but it invites us to compare corpus-level usage. It, for example, facilitates close, careful reading of the neighborhood of android_startrek--the visual outlier in Figure 3.
We conclude by presenting our efforts to measure this particular method’s impacts on model stability (including comparing fixed, shuffled, and bootstrap-sampled corpora). Further validation is needed to understand the relationship of corpus composition to the modified terms. Additionally, different embedding algorithms will ultimately suggest different interpretations--here, GloVe’s context window calculations capture both proximity and emulation--another algorithm would suggest different meaning-making possibilities.
We do think this is most useful on algorithms that we have meaning-making protocols for--it will not work on anything resembling a generic “AI” or proprietary LLMs. Manipulating tokens in this way in corpora before training models affords the collapse of contexts while maintaining a particular, chosen axis of distinction; it also suggests the possibility of collapsing tokens into artificial types, much as we may train a model on a lemmatized corpus, along similar axes of distinction.