Daejeon, July 27–31
“Reading Images, Writing Metadata” is an ongoing project by the Austrian National Library which aims towards enriching metadata using various Computer Vision techniques, including AI models and Machine Learning, on a diverse collection of graphics and images, opening new collections for user engagement.
Computer Vision techniques, especially using AI models, have become increasingly more popular throughout the past decade, causing a sharp increase in the number of models programmed for different tasks (Mensink and Van Gemert 2014; Hugging Face 2025a and 2025b). While object detection models like YOLO by Ultralytics have set their focus on real-time applications and computational efficiency (Khanam and Hussain 2024), an abundance of other models has flooded the research community, focusing on specialized tasks or robustness. The model CLIP by OpenAI, for example, has marked a new path towards solving tasks it had not learned to solve before (so-called Zero-Shot Predictions), by classifying unseen images and showing strong performance on modern photographs (Radford et al. 2021; Vignoli et al. 2025). In the context of libraries and GLAM institutions, however, these models have yet to face rigorous testing by confronting them with the diversity of data available at said institutions. While some actors have already approached the topic of image processing of their data like the Rijksmuseum (Mensink and Van Gemert 2014) or the ETH Zürich (Graf 2023), many trials so far have been either technically outsourced towards AI companies and third parties – making the process less transparent and not reproducible for the end user – or concerned themselves with other types of AI applications such as using AI as part of the library services or as a tool to make the cataloguing process more efficient (e.g. Okunlaya et al. 2022). By contrast, this project proposes an open and accessible usage of AI models to enhance the data available, simultaneously aiming at making the process transparent for readers and users.
In detail, the pictures and graphics available in the online portal of the Austrian National Library (containing among other sources a vast collection of historical and modern photographs, portraits, prints, posters, watercolors, hand drawings) are to be made more accessible through automatic object detection and classification, enhancing general retrievability via search. The metadata generated hereby will be published and made available for other research applications in the future. Furthermore, images contained in the collection of digitized books (ABO, Austrian Books Online) published before 1900 as well as in the digitized journals and magazines collection of the library (ANNO, Austrian Newspapers Online) are to be identified and extracted using AI combined with layout and content classification operations, which will expand the collection of items searchable. Digitized images with enriched metadata from object detection will also enable users to find similar images. These different milestones will permit users to gain new approaches to the different collections of the library and foster serendipitous exploration as well as enhancing fundamental research on the use of digitized artworks in different contexts.
In a first and already completed step, various classification models and classification systems like ICONCLASS (van de Waal 2024), Getty Thesaurus (Getty Research Institute n.d.) and library standards (Julien et al. 2021; Kazi et al. 2021) were tested on their robustness with regard to covering a sample dataset’s diversity of art styles and iconographic content, showing that finding a fitting vocabulary for all styles is challenging. Additionally, different architectures and model sizes of the YOLO11 and YOLO12 models by Ultralytics were examined to find the model version most adaptable to the heterogeneity of images. Through this process, a combination of three different models (two from the object detection field, one from the classification field) was established, to harmonize results for different user and interest groups. These models include two versions of YOLO11, one self-trained on the LVIS-dataset (Gupta et al. 2019) and one Open Access Model from Huggingface (Shao et al. 2019), as well as CLIP. While the models being used are already pre-trained and therefore biased to a non-quantifiable degree, refining the training process as well as setting the best standards for quality control enabled the library to generate high quality metadata that has been created in a very robust setting. Additionally, the transparency of data generation increased significantly, whilst identifying the key needs from the users’ perspective and responding to those needs in a more individualized and creative way was made possible. To further validate this process, a trained librarian conducted an additional quality control review on the annotations, which revealed a significant discrepancy between the model’s built-in statistical metrics (such as mAP50-95 and accuracy scores) and the actual accuracy observed during testing. The findings suggest that standard model metrics may not be reliable indicators of performance when applied to diverse, historical data, as the two measures diverged by nearly 60%. Notably, while the model’s internal accuracy scores appeared very low, the annotations it produced were largely correct (up to 90%). This raises important questions about how much these build-in statistics can be trusted when evaluating model performance in specialized, real-world contexts. One measure that could not be tested thoroughly is the number of annotations missed by the model; the decision of combining multiple YOLO versions was based on this issue, trying to achieve the largest number of detections possible.
During the second, currently ongoing phase of the project, a YOLO26 model was trained to identify different types of images (photographs, illustrations etc.) on pages of historical journals and books. For this, roughly 600 images were manually annotated using data from the library’s resources.
Due to the formal setup and high versatility of data available at the library, the team faces an abundance of challenges. This includes questions such as the depth(s) and degree(s) of description necessary for valuable user engagement, but also the suitability of applying either a certain model to a certain type of historical sources or of applying one such model to several source types at the same time. Additionally, the models’ capabilities had to be evaluated keeping in mind not only the heterogeneity of images (e.g. the difference between hand-drawn or printed images), but also the expectations of librarians and other specialists regarding data quality, standardization, and implementation in (library) interfaces to ultimately benefit all future user groups. Moreover, it was important to be aware of the biases these models inherently reflect and reproduce. Tests so far have shown that especially manual quality control by an expert seems to be a necessary step to ensure a valuable outcome.
The goal of creating a new and enhanced form of user engagement highlights the importance of interaction between humans and technology. As such, the project’s methods and aims are inherently transferable to other domains and very suitable for broader discussion with peers during this conference. The presentation will briefly explain the institutional background of the project to then focus on the main questions and decisions made so far, before finally giving an outlook on the tasks and challenges ahead.