Daejeon, July 27–31
Digital humanists and researchers routinely work across languages and modalities, such as comparing a French diplomatic speech with its German press coverage, or tracing how a radio broadcast reshapes a printed debate. Yet most digital tools confine them to monolingual or single-medium corpora, forcing a disconnect between established research practices and the infrastructure meant to support them.
Multilingual and multimodal embeddings have the potential to close this gap by representing diverse source types – articles, images, advertisements, broadcasts – as vectors in a shared space where proximity encodes semantic similarity regardless of language or modality (Smits and Kestemont 2021; Smits and Wevers 2023). A growing body of work in the digital humanities has begun to explore this potential, but concrete assessments of how embeddings perform when applied to the heterogeneous, imperfect materials that characterize historical corpora remain scarce (Brottrager et al. 2021; Fedchenko et al. 2025).
This paper reports on user experiments conducted with a corpus of digitized historical newspaper collections represented as multilingual and multimodal embeddings within the Impresso project. Through six usage scenarios tested across two workshops, we examine where embeddings succeed in supporting exploratory research workflows, where they fall short, and what changes to interfaces, models, and data infrastructure would be needed to make them reliably useful for question-driven historical inquiry.
We first describe the Impresso project and its infrastructure, then detail the embedding models and data used in our experiments. We present the experimental design, report our findings organized by usage scenario, and conclude with implications and directions for future work.
The Impresso projects sit at the intersection of natural language processing, history, and design. Their shared goal is to make large-scale historical media archives more accessible and usable for research.
The first project, Impresso 1, focused on Swiss and Luxembourg newspapers. It produced the Impresso Web App, which moves beyond keyword search by integrating semantic enrichments – named entities, topic models, and text reuse detection – as entry points for content discovery (Romanello et al. 2020). The project successfully demonstrated how machine learning enhance corpus exploration and support data criticism (Düring et al. 2024).
The second project, Impresso 2, launched in 2023, aims to pioneer the joint exploration of Western European historical newspapers and radio content across temporal, linguistic, and national boundaries (see Figure 1). The project was motivated by two observations. First, the digitization of radio archives, while less studied than newspaper archives, is growing rapidly and creating new opportunities for research. Second, cultural heritage data collections remain inadequate for historical research for two reasons. They are limited by institutional and media silos that restrict access to single countries, languages, or media types. They also only allow passive exploration of static collections, whereas historical research requires iterative comparison and association across sources.
Figure 1 - Schema of the Impresso2 agenda
Impresso 2 not only provides semantic-based faceted search across structured newspaper and radio collections but also interconnects all elements in a shared multilingual vector space, enabling comprehensive connection across media archives. This approach is informed by research on historical media ecosystems and implemented through collaboration with 21 European partners, including national libraries, archives, and cultural heritage institutions https://impresso-project.ch/consortium/associated-partners/.
The experiments reported in this paper rely primarily on embeddings derived from Impresso content items. A content item is the smallest unit we distinguish on a newspaper page – an article, an image, or an advertisement – or within a radio broadcast. Sentence-level and chunk-level embeddings are also available as samples at this stage of the project.
For text, we use mGTE to generate 768-dimensional vectors at the content item, sentence, and chunk levels. Full-corpus coverage is available at the content item level, while sentence and chunk embeddings currently exist only for the sample from years 1912 and 1986. This partial coverage means that findings based on finer-grained text units should be interpreted with caution regarding their generalizability.
For images, we work with two complementary models. DINOv2, a vision transformer trained on approximately four million images from the Impresso corpus, produces 1024-dimensional visual embeddings. These support image similarity search, clustering, and downstream tasks such as classification, feature extraction, and object detection. OpenCLIP then generates 768-dimensional image-text embeddings by processing images padded to squares and resized to 224x224 pixels alongside multilingual text input with a maximum length of 77 tokens. We selected OpenCLIP for its ability to project images and texts into the same space, enabling cross-modal querying, image-text similarity, and image description generation. Table 1 offers an overview of the data.
Table 1: Overview of the embeddings used for the experiments.
Despite their promise, many questions about the practical utility of embeddings for historical research remain open. How sensitive are textual embeddings to OCR noise? What text granularity suits which tasks? How well do embeddings detect text reuse across languages? Can external documents be meaningfully linked to the Impresso corpus for question-driven research? Can embeddings support the detection and tracking of historical events? And what opportunities do they create for exploration and accessibility in the Impresso Web App and Datalab?
We began addressing these questions through two mini-hackathons held in autumn 2025. The first, a two-day internal workshop, brought together members of the Impresso team and invited external guests with backgrounds in history, computational linguistics, and digital humanities (n=21). It addressed embeddings broadly across modalities. The second, a half-day session held at the Computational Humanities Research (CHR) conference in Luxembourg (n=16), focused specifically on image embeddings. Both events were structured around concrete usage scenarios with preselected datasets, and participants ranged from historians with limited programming experience to NLP researchers, which helped surface a variety of expectations and pain points.
Table 2: Case studies to explore the utility of Impresso’s embeddings for historical research workflows.
The six scenarios can be grouped into three groups, each probing a different dimension of embedding utility. The first group concerns connecting and retrieving documents. The “Linking In” scenario tested whether external documents could be embedded through the Impresso API and meaningfully matched to newspaper coverage of the same events. The RAG scenario explored whether retrieval-augmented generation over imperfect and heterogeneous historical data could surface relevant sources, using newspaper articles on nuclear power as a test case.
The second group addresses tracing content across boundaries. The multilingual text reuse scenario used French and German medical advertisements to examine whether embeddings can capture shared meaning across languages well enough to detect content circulation. The event detection scenario tested whether embeddings, combined with metadata and semantic enrichments, could distinguish and track related but distinct events, using Olympic Games coverage as its corpus.
The third group explores cross-modal research. The multimodality scenario investigated whether images and texts embedded in the same space could support combined visual and textual analysis, specifically by geo-referencing newspaper maps using named entities found in accompanying articles. The gender scenario, designed for the CHR workshop, provided participants with a prepared notebook and query examples to experiment with different image search strategies for studying gender representation in the newspaper corpus, followed by free exploration guided by their own research interests.
This diversity in case studies reflects Impresso’s goal of serving a wide range of research practices, thematic interests, and technical skill levels, both within and beyond the project team. Table 2 summarizes the scenarios, their objectives, and their case studies.
Our experiments confirm that embeddings can reveal meaningful connections across languages and modalities. They also expose clear limitations that must be addressed before embeddings can move from exploratory tool to reliable research infrastructure.
The strongest results emerged in document retrieval and cross-modal search. Across several scenarios, embeddings surfaced relevant materials that keyword queries alone would have missed. In the “Linking In” scenario, external documents from League of Nations debates were successfully matched to newspaper coverage through the Impresso API, demonstrating that the shared vector space can accommodate external sources. The multilingual text reuse scenario showed that French and German medical advertisements with shared content were brought into proximity despite surface-level linguistic differences. Image-text embeddings also proved effective for exploratory browsing.
The central limitation that emerged across all scenarios is one of interpretability and control. Embedding-based links between documents rest on a large number of shared features, but users can currently neither predict nor inspect why two items are connected. This opacity is tolerable in exploratory contexts, where unexpected associations are welcome. It becomes a serious obstacle for question-driven research, where scholars need to understand and justify the basis for a connection. More user control over how embeddings are ranked, filtered, and explained is needed to bridge this gap.
Granularity and transparency proved to be two other persistent challenges. The observed combination of promising results and unpredictable behavior highlighted the need for clearer documentation of model training, data composition, and potential biases, so that researchers can assess how much confidence to place in embedding-based findings.
Despite these limitations, our experiments point to concrete next steps. Text-image and multilingual embeddings are well suited for integration into the Impresso Web App, where they can improve search and power document recommendations in an exploratory interface. For the Datalab, the most promising direction to emerge from the workshops was user-defined image embeddings, allowing researchers to train or fine-tune embeddings on their own curated image sets, tailored to specific research questions. This would shift embeddings from a fixed infrastructure layer to a flexible analytical tool shaped by the researcher’s own priorities.