Daejeon, July 27–31
As digital game archives expand globally, access to these collections remains largely text-centric. Current archival systems rely on metadata-based keyword search (e.g., title, release year, publisher) (Lee et al. 2015). While effective for known-item retrieval, this approach struggles to support exploratory queries about visual motifs or atmosphere—such as "1990s dark fantasy" or "isometric cityscapes." To bridge this gap, we propose a visual semantic search system.
This study positions game packages (box art) as critical "visual paratexts" (Genette 1997; Consalvo 2007; Švelch 2020; Pupáková 2025) that condense gameplay experiences into static images and function as iconic identifiers of games as products. By applying multimodal AI to these paratexts, we aim to enable "Distant Viewing" (Arnold / Tilton 2023), allowing researchers to query the archive using natural language rather than rigid metadata. This paper presents the system and evaluates its capabilities and biases (Drucker 2020).
Recent studies show that vision–language models such as CLIP are increasingly applied in GLAM contexts, including large-scale visual analysis of cultural heritage collections (Abgaz et al. 2021). However, these models reflect cultural and linguistic biases in their training data and tend to privilege Western visual conventions (Paullada et al. 2021; Offert / Impett 2023). Therefore, this study treats multimodal retrieval not only as a technical solution but also as an object of critical examination in game archives.
Our dataset comprises 14,470 images from the Ritsumeikan Center for Game Studies (RCGS) Collection, including digitized game packages and covers of related printed materials. Semantic retrieval was implemented using CLIP ViT-L/14 (Radford et al. 2021), which maps images and text into a shared embedding space. We apply this approach to video game archives, where visual materials function as paratextual representations of interactive media.
For retrieval infrastructure, we employed Qdrant with the HNSW algorithm to enable approximate nearest neighbor (ANN) search, supporting interactive exploratory discovery.
We evaluated the system using two tasks: a Relevance Task (Precision@10 for category-based queries, representing exploratory or unknown-item retrieval) and a Ranking Task (MRR and Hits@20 for specific game titles, representing known-item retrieval). For the relevance task, 32 natural-language queries describing objects, characters, and abstract concepts were constructed, while the ranking task used 10 queries describing distinctive visual features of specific games or series. Each query was tested in both Japanese and English, resulting in 64 relevance-task instances and 20 ranking-task instances. Ground truth relevance was manually annotated by the author based on whether retrieved images contained the visual or thematic elements described in the queries.
The results revealed a language bias. English queries outperformed Japanese ones, with an average Precision@10 of 6.63 (SD = 2.97) compared to 3.97 (SD = 3.68). This difference was statistically significant (Wilcoxon signed-rank test: W = 36, p < .001, Cohen's d = 0.81), indicating a large effect. In the ranking task, the mean reciprocal rank (MRR) was 0.027 for Japanese queries and 0.258 for English queries. The model showed high precision for universally recognizable objects (e.g., “dog,” “sea,” “military ship”). English queries describing abstract visual qualities—such as “dark fantasy mood” or “comical touch”—also retrieved relevant images, suggesting potential for exploratory retrieval of stylistic motifs.
However, performance declined for culturally specific terms in Japanese. For example, the query “pixel art” returned no relevant results in Japanese but six in English, and “horror” showed a similar disparity (0 vs. 9). Visual consistency also affected retrieval: The Legend of Zelda series achieved high precision due to recurring iconography, whereas the more visually diverse Yakuza (Ryū ga Gotoku) series produced lower ranking performance.
These results suggest that CLIP-based retrieval can support exploratory discovery of visual motifs in game archives while exhibiting an Anglo-centric bias reflecting the linguistic imbalance of its training data.
To understand how the AI structures the visual history of games, we projected the 768-dimensional vectors into 2D using UMAP (Figure 1). The reduction used n_neighbors=15, min_dist=0.1, metric='cosine' to preserve local stylistic clusters. The visualization revealed that the model formed clusters based on visual features rather than metadata genres.
Figure 1. UMAP projection of game images showing visual clusters.
This study demonstrates that Visual Semantic Search is a viable "new tool" for Digital Humanities, offering a complementary access layer to traditional metadata. It allows researchers to bypass the "keyword bottleneck" and engage in exploratory analysis of visual tropes. However, the study also highlights critical limitations. The inability to retrieve "pixel art" or "yokai" via Japanese queries indicates that off-the-shelf multimodal models possess a cultural blind spot. They are not neutral observers but are biased towards Anglo-centric visual cultures. This "algorithmic exclusion" poses a risk for non-Western archives, where unique visual motifs may remain unsearchable without domain-specific intervention. This phenomenon aligns with broader critiques of machine learning datasets (Paullada et al. 2021). In game archives, such biases risk reproducing existing imbalances in the global historiography of digital games.
Future work will focus on two directions: first, comparing multilingual models and evaluating language and domain adaptation using game-specific captions, followed by domain-specific fine-tuning to improve Japanese-language performance; and second, developing a hybrid ranking system that combines traditional metadata with visual semantic search. We also plan to implement multi-annotator evaluation to establish a more robust ground truth. By addressing these biases, we aim to build a more inclusive interface for the distant viewing of game history. More broadly, multimodal AI can function not only as a retrieval tool but also as a methodological instrument.