Daejeon, July 27–31
Increased access to GLAMS (Galleries, Libraries, Archives, Museums, Special Collections), including efforts in digital scholarship and linked data, depends on thorough and accurate description. As a result, many collections are set aside due to labor-intensive work, lack of funding, and staff. Advancements in Artificial Intelligence (AI) and Large Language Models (LLMs) could be a tool to address this issue.
Our research assesses the performance of seven LLMs in generating metadata and transcriptions for archival materials, both via Application Programming Interface (API) in an automated workflow and through their web-based interfaces. We approached the use of LLMs to foreground small data by focusing on context-rich pages requiring interpretation.
Exploration of LLMs in metadata and cataloging workflows is ongoing. Recent OCR studies show that tools like Gemini 2.0 Flash and GPT-4o match existing solutions like Transkribus and Tesseract (Grief et al. 2025). However, valid concerns in the GLAM field regarding the use of AI necessitate careful evaluation of projects to ensure that they are responsible and practical (Brador 2024). AI hallucination mandates human intervention and verification in projects within the GLAM space (Huang et al. 2024). Building on prior research, we used core principles of the PROMPT design framework, including persona creation and detailed contextualization of tasks, formats, intended purposes, audiences, and professional expectations (Hartman-Caverly 2024).
The project focused on the Southern Architect and Building News (SABN) collection (1882-1932), a monthly trade publication that documented architecture and building trades in the southern United States. Our holdings include 254 digitized issues containing illustrated covers, advertisements, editorial content, symbols, diverse fonts, photography, and drawings. Items have basic metadata but are not fully indexed.
SABN was selected for several reasons. It is fully digitized, negating the risk of exposing private or restricted materials to LLMs. The collection does not include materials from underrepresented communities or sensitive personal information. While this highlights a limitation in scope, it allows us to avoid ethical concerns that require safeguards as we explore the capabilities of LLM tools.
Our cross-departmental team included six staff members and student workers with expertise in metadata, digital collections, AI implementation, archives, architectural history, and digital scholarship. Seven LLMs were evaluated using API and web interfaces on pages from SABN. Based on local metadata practices, we tested five key tasks: OCR creation and text editing; table of contents generation; name entity recognition, and Library of Congress Subject Heading (LCSH) generation. We established an evaluation criterion: completeness, description of the full page’s content, accuracy, OCR, descriptions, term correctness, entities, and utility for users. We scored results using a scale of 1-4.
Prompts were structured according to the library repository metadata standards. The prompts asked for Library of Congress and Art & Architecture Thesaurus subject headings, full transcription, a summary of the text, named entity extraction, and to flag harmful content. For web interface testing, four pages from three SABN issues were selected: two full-page editorials and two dual-column pages. API testing used 10 pages, with visual elements and complex layouts, as well as editorial content with multiple text blocks.
Claude Sonnet 3.5 achieved the highest overall scores. OCR and content summarization were the greatest strengths of the LLMs (Thomas et al. 2024). AI-generated OCR results outperformed the original transcriptions. Entity extraction ranked lower overall, though testers felt that the metadata could improve discoverability. Subject heading creation was the weakest task across all models tested. The difficulty primarily stemmed from the complicated and variable syntax of generating exact matches of LCSH (Chow et al. 2024). This was later addressed through a purpose-built processing pipeline informed by these results, available as an open-source GitHub repository (Moutran 2025). A significant finding was that, in general, larger models produced the best results. Haiku performed best in content summarization, suggesting that a multi-model approach could be a practical and cost-efficient option.
Most LLMs struggled to handle slurs. GPT-4o was the exception, transcribing the page accurately but not unnecessarily repeating the term in the metadata. Gemini and GPT-4o-mini refused to process the page. Claude Sonnet 3.5 substituted or flagged the term. Claude 3 Sonnet, and Haiku wrote the terms in the summary and content warning, and GPT-4o-mini transcribed it directly. Importantly, web-based models pose risks of sensitive content being used in future training data.
Anthropic models, including Claude Sonnet 3.5 and Haiku, struggled with smaller text in images, likely due to lower-resolution image ingest during pre-processing compared to OpenAI and Gemini models. Budget-conscious projects can use smaller LLMs via API, though reliability, quality, and data privacy require consideration. We recommend choosing models based on task-specific strengths, collection needs, and institutional access. Batch processing via API can reduce costs. Results can be improved through prompt optimization, rigorous testing, and formalized human oversight. Due to LLM models' embedded bias, metadata workflows should follow best practices that include community stakeholders to identify ethical concerns informed by lived experience and subject expertise (University of California, Irvine Libraries 2025).
Our research demonstrates that LLMs can enhance metadata creation in archival collections, though with limitations that require strategic implementation. A significant challenge will be hosting and supporting the search for the increased volume of generated metadata, including fields describing AI usage and mechanisms that allow users to flag potential inaccuracies (Cushing et al. 2022). Other challenges include keeping up with the AI landscape, tool stability and security, and processing harmful content. This could prove to be challenging for smaller or under-resourced institutions. We conclude that carefully designed LLM workflows, in accordance with archival best practices, can increase access, enrich metadata, and help GLAMs address processing backlogs.
We employed Anthropic, OpenAI, Gemini, and Llama models for the following purposes: generating metadata for our research and generating and debugging code used in the API workflow. No custom training was conducted; all results reflect the model's behavior, guided through collection-specific prompt design. Metadata was evaluated using methods described above, and all code was tested thoroughly. The authors assume all responsibility for the content of this submission.