Between research and infrastructure: the semantic space of DraCor publications

In our first experiment, we attempted a broad exploration of the ‘semantic space’ of DraCor-based research. To compare papers across languages, we embedded the full texts using a multilingual transformer capable of producing language-agnostic sentence-level embeddings.

https://huggingface.co/sentence-transformers/distiluse-base-multilingual-cased-v1.

Figure 1. Semantic space of DraCor-related research papers (PCA of multilingual embeddings)

What emerges (Figure 1) is what one might call an ‘infrastructure-vs.-research’ divide along the PC1 axis (accounting for 12.2% of total variance). On the left, a group of papers – Börner 2022, Fischer 2019 (Programmable), Börner 2023 (Dockerizing), Giovannini 2023, and Börner 2023 (Onboard) – mainly deals with DraCor as a research infrastructure, while most other papers use DraCor as a data source for computational drama analysis. This seems to suggest that infrastructure-oriented papers have a somewhat more distinct semantic profile, but also underscores, in general terms, the duality of the project itself, well encapsulated by its definition as “a showcase for the concept ofhttps://doi.org/10.5281/zenodo.4284002Programmable Corpora”.

https://dracor.org/doc/what-is-dracor.

PC2 is less clear, but it appears to reflect methodological differences. Papers using machine-learning methods (for detection and classification of various elements in drama) tend to appear in the upper part of the plot, while papers using more traditional quantitative approaches are lower down. Thus, PC2 may capture a loose gradient between machine-learning-based and more classical distant-reading approaches.

Methods, formats and tool mentions

Secondly, we explored the methodological arsenal deployed across our bibliography. To do so, we extracted tools, formats and methods mentioned through manually defined multi-language word lists compiled by Schöch, Fileva, and Dudar (2022). Words were retrieved by string matching n-grams (one to four words) – case-sensitively and accounting for spelling variations (e.g. “lemmatisation”/“lemmatization”).

Figure 2. Overall count of tools.

Figure 3. Overall count of formats.

Figure 4. Overall count of methods.

To get a first overview, we summed up all mentions of a given tool, format or method across all papers.

Within the ‘methods’ category, we reconciled non-English methods names (e.g. traitement automatique des langues) to the common English term (e.g. NLP) using LLMs (with manual post-evaluation). Since most entries occur quite rarely, we filtered the lists by a count threshold of five.

Figure 5. Percentage of articles mentioning a tool at least once.

Figure 6. Percentage of articles mentioning a format at least once.

Figure 7. Percentage of articles mentioning a method at least once.

Similar insights emerge by plotting the percentage of papers in which an entry occurs at least once (Figures 5-7). In terms of format, it is unsurprising that more than 70% of the papers mention TEI, i.e. the standard in which DraCor plays are encoded. Frequent mentions of Python (40%) and API (25%) signal engagement with another key concept in the DraCor infrastructure, namely, that of API-driven ‘programmable corpora’. As for the methods, "annotation" (45%) leads over "network analysis", "classification" and "machine learning" (30%-40%), suggesting that works in the bibliography might be often concerned with further enrichment of DraCor data.

Figure 8. Diachronic development of tool mentions.

Figure 9. Diachronic development of format mentions

Figure 10. Diachronic development of method mentions. 

Plotting the diachronic evolution of the five most frequent entries per category yields further insights (Figures 8-10). For example, while early papers appear to rely only on network analysis and visualisation, later contributions feature more classification procedures and machine learning. The graphs also reflect the development of the DraCor platform, e.g. with a peak in API mentions corresponding to the official launch of the website dracor.org (2017). One could also critically explore the intersection of tools, methods and formats: for instance, visualisation is present throughout the years, but Gephi (a popular software for visualising literary networks) only appears from 2017 onwards.

Eventually, we also track which popular algorithms (e.g. SVM) and neural network architectures (e.g. LSTM) appear at which point in the chronology of the DraCor bibliography. Results (Figure 11) show how computational drama analysis has been responsive to shifting trends in DH methods. Until 2021, for example, rule-based and classical machine learning algorithms dominate the scene, while after 2021 transformers quickly overtake both traditional approaches and other neural network architectures.

Figure 11. Normalised relative frequency of selected algorithms (per paper in each year).

Citations network

Shifting our focus from the papers’ content to their metadata, we finally attempted to model scholarly references found in each publication. Figure 12 presents a bibliographic network of interconnected citations: each of the 3377 nodes represents a scientific publication and each edge links it to a further piece of research it cites. We removed from the graph Fischer et al. (2019), which gets disproportionately cited because of its role as the suggested reference to the whole DraCor project, and scaled the nodes' size to their in-degree (i.e., the number of other papers citing that publication).

The resulting network is scale-free and displays a hub-and-spoke topology: most papers have small citation clusters around them, while highly-cited works move to the centre of the picture. Examining these central nodes reveals that computational drama analysis – at least as reflected in the DraCor bibliography – has not undergone major paradigm shifts in the last decades, insofar as the most-cited works remain those rooted in structuralist-formalist traditions. This is evidenced both by early seminal works like Pfister (1977) and Marcus (1970) and by more recent contributions such as Moretti (2011) and Trilcke (2013), which were notable in introducing and popularising the methodology of literary network analysis as an effective way to study plays.

Figure 12. Citation network from the DraCor bibliography (left) and a close-up on the most cited items (right). Label size is proportional to in-degree (mentions count).