Cultural heritage institutions such as libraries, archives and museums face increasing ethical and regulatory pressure to describe complex visual collections such as manuscripts, photographs, artworks, and other objects for accessibility purposes. This obligation to generate succinct “alt-text” descriptions of digital images often entails remediation tasks in the tens of thousands of items, an under-funded and under-resourced mandate that is nevertheless crucial to ensure equal access to shared cultural heritage online.

Automating this work has been of interest since at least 2007 (Li / Li 2007), but the advent of vision-language models (VLMs, alternatively termed large multimodal models) has delivered robust visual comprehension at hitherto unreached levels of sophistication (Chen et al. 2024). AI approaches are appealing given their ability to automate complex tasks involving natural language, but there are plentiful reasons to look beyond simply pasting GLAM materials into ChatGPT. Importantly, multimodal architectures are available via open-weight models such as Qwen, DeepSeek and Gemma, which enables their responsible use at institutions with best practice guidelines in place regarding privacy and reproducibility, as well as the ability to retrain base models to align closely with domain-specific tasks in the cultural heritage field.

Even with the recent advances in VLMs, the limitations of taking an “image-only” approach to accessibility remediation are evident for both general archives and collections with special sensitivities around race, gender, and other forms of identity (Berry 2025). As most VLMs (both commercial and open-weight) are trained using contemporary material, they may exhibit presentist biases (describing photographs from 1990 as “vintage”) as well as cultural chauvinism (mischaracterizing non-Western clothing as “Halloween costumes”). These limitations often preclude their use as one-shot captioners in professional cultural heritage contexts (Mähr / Twente 2025). Professional labor may be used to check and, if needed, correct each generated caption, but such efforts arguably are a poor use of valuable time and expertise if the experts’ inputs do not contribute towards inflecting the model in directions that better reflect the values and priorities of the curators or institutions.

Our approach to solving this problem is to more fully leverage the multimodal power of VLMs: conditioning the production of the caption on both image pixels and textual metadata drawn from institutional archival practice (Broadwell / King March 2025). Providing item- or collection-level metadata from a museum, archive or library along with the image gives a VLM important perspectives on the origin, context, physicality, use, and interpretation of the image it is attempting to describe. This information often contains vital clues for improving the responsibility and accuracy of generated captions, and lessens the “guessing game” a VLM must play when prompted to caption an image without the benefit of context, scholarship, or critical perspective. This method aligns with new research into VLMs that has demonstrated how “each generated token gathers information from both the input image and text.” (Kaduri et al. 2024)

The approach outlined above has recently been explored through its application to collections as diverse as Stanford University Library’s collection of photography of the history of Silicon Valley (Broadwell / King June 2025), the Wylie Wong Collection of May's Studio Photographs from San Francisco’s Chinatown (Broadwell / King March 2025), and the images in Wikimedia Commons (Vershbow et al. 2025), with promising initial results. A new tool we are developing based upon this method seeks to place curators and other scholars in full control of the possibilities of dual-conditioning by exposing an array of captioning possibilities along two axes: caption length (short to long) and metadata influence (low to high).

To ameliorate the inherent stochasticism of generative models and to accommodate the rapid cadence of new open-weight model releases, we propose a fine-tuning workflow that first validates a base set of dual-conditioned captions against human-in-the-loop, gold-standard captions provided by expert archivists. This infrastructure uses a comparative model deployment evaluation framework based upon the Promptfoo open-source local VLM testing app and associated prompt optimization methods, automatically comparing the results of multiple models and prompts to desired gold standard outputs while probing for undesired behavior (so-called “red-teaming”). Then, for specific collections, the highest-rated captions selected from the human-in-the-loop dual-conditioning evaluation tool described above – those that did the best job of integrating visual description with archival context – are used as the basis for per-collection fine-tuning of well-behaved models that can be applied to describe all images belonging to that collection.

This short paper will present an overview of current limitations of vision-only models for alt-text generation and showcase our new approach, tools and workflow on diverse large-scale real-world collections. Our work has revealed that both fine-tuning of locally hosted models and “conditioning” of descriptive image captions by incorporating available metadata into the model’s instructions show promise for producing useful alternative text for images. We also find that tailoring approaches to specific collections and keeping human reviewers in the loop are keys to making the alt text as accurate as possible while gaining efficiency at scale. We intend this talk to highlight opportunities for further research and cross-institutional discussion regarding the relationship between word and image in cultural heritage contexts.