Daejeon, July 27–31
Large language models (LLMs) such as ChatGPT are increasingly used in the cultural heritage domain for tasks like metadata creation, semantic enrichment, and artwork captioning. Since these tasks depend on curated metadata, it is essential to evaluate AI-generated descriptions and understand how human metadata improves them.
We present a method for automatically generating descriptions of ceramic and porcelain artworks in Digital Humanities (DH). Ceramics suit captioning tasks: they are three-dimensional and harder to describe than flat objects such as paintings, have simple forms that reveal basic generation quality, and represent widely used historical objects whose documentation is often incomplete.
Using an 11,566-item open-access Rijksmuseum dataset, we compare an LLM (ChatGPT) with our RAG-enhanced LLM, TerraLex, which retrieves similar artworks and use their metadata. Our results show that the RAG method produces more accurate, context-rich descriptions with fewer errors, and human evaluators consistently preferred them. These findings highlight the advantage of RAG for description generation and the value of complete, high-quality human metadata. TerraLex supports humanities research by strengthening the completeness and clarity of object descriptions used in cataloguing, stylistic comparison, and interpretation. It also aids teaching and public engagement by providing accessible explanations of ceramic artifacts. The novelty of this work lies primarily in the proposed workflow and in the empirical benefits it demonstrates for metadata generation.
Automatic image captioning is well studied in computer vision, but its use in cultural heritage remains limited. Traditional systems such as Vinyals et al. (2015) and Xu et al. (2015) focus on visual features like shapes and colors, ignoring cultural context. Recent work such as Reshetnikov and Marinescu (2025) addresses heritage images—mainly paintings—using crowdsourced annotations, yet 3D artifacts like pottery remain underexplored. Zheng et al. (2023) is the only prior ceramic-captioning study but relies on complex clustering and knowledge graphs. TerraLex is simpler and offers flexible, museum-ready descriptions.
The rise of large generative models has expanded opportunities for enriching museum data. Studies such as Ahola et al. (2024) show LLMs can support metadata creation from curated catalogues, yet they remain prone to hallucinations and stylistic bias, especially for underrepresented regions or periods. Retrieval-based augmentation mitigates these issues.
RAG, introduced by Lewis et al. (2020), integrates retrieval with generation to ground outputs in factual sources. While RAG has been used in DH for text annotation and translation, such as in Miyagawa (2025), its application to visual cultural data has not been reported to the best of the authors’ knowledge, and this is where our work contributes.
From a humanistic perspective, automating descriptive language introduces a semiotic shift. Scholars such as Drucker (2018) note that DH must consider how machines produce interpretive statements, since AI language encodes cultural hierarchies and absences. Comparing ChatGPT and RAG-based captions thus becomes both an engineering task and an inquiry into how AI mediates cultural meaning.
We collected images and metadata of ceramic and porcelain objects from the Rijksmuseum API (Amsterdam). The dataset includes 11,566 items: porcelain (54.5%), ceramics (0.6%), earthenware (32.8%), and stoneware (12.1%). Each record provides an image, title, and metadata including material, year, place, and description.
The first line of Table 1 shows the proportion of missing data in this retrieved dataset. Since evaluation requires comparison with a gold standard, we randomly selected a subset of 1,000 items that contained no missing data. These were used for evaluation, while the remaining items were incorporated into our RAG system as described below.
Table 1: Percentage of Missing information across datasets and model outputs
RAG (Retrieval-Augmented Generation) improves generative accuracy by retrieving relevant external information and supplying it to the LLM before generation. We use RAG to ground descriptions in verified museum metadata, reducing hallucinations and enabling more accurate, context-rich interpretations of ceramic artworks.
Figure 1 illustrates the design of our system. The upper-left portion shows the input image and the prompt, where all images are vectorized using the standard CLIP-ViT model (Radford et al. 2021), providing reliable visual grounding for the RAG system. Our RAG is constructed from the Rijksmuseum dataset described above, retrieves the top-K(=3, for reducing tokens) most similar items. Similarity is evaluated by the CLIP vectors’ L2 distance. The multimodal LLM (GPT-4o, and GPT-5) receives both the original input image and the retrieved results, and then generates a multi-sentence description.
We compare our system, TerraLex, with ChatGPTs (OpenAI GPT-4o mini, and GPT5 mini) as baselines. Terralex and the baseline always use the same GPT kind (Terralex-GPT4o, and Terralex-GPT-5). The input is an artwork image and the output is a multi-sentence description—longer than a caption—which we refer to as a description.
Both systems use the same prompt shown in the upper-left portion of the figure. Unlike previous captioning systems, our approach is intentionally simple and broadly applicable to general DH tasks, making it important to report its effectiveness.
Figure1. Our TerraLex System
The last four rows of Table 1 compare the percentage of missing data in the outputs of the baseline system and TerraLex. While GPT-4o leaves many key fields unspecified, GPT-5 and both TerraLex variants substantially reduce missing information across attributes, indicating that TerraLex produces more complete and informative descriptions of the artworks.
Table 2 reports the automatic evaluation results using standard captioning metrics—BLEU1/4 (Papineni et al. 2002), ROUGE-L (Lin 2004), METEOR (Banerjee and Lavie 2005), CIDEr (Vedantam et al. 2015), and BERTScore, to assess lexical and semantic similarity to the references. Each item in the dataset includes a museum-provided description, which we use as the reference text.
Across all metrics, TerraLex consistently outperforms GPT. Notable gains in METEOR, ROUGE-L, and BERTScore show that TerraLex’s descriptions are not only closer to the references at the n-gram level but also better aligned semantically.
Table 2: Automatic evaluation results for image captioning models
Automatic evaluation captures only string or vector similarity, so we also conducted a human evaluation. We first assessed 220 instances using the six criteria in Table 3. TerraLex consistently outperformed GPT, even with the large improvement of GPT-5 over GPT-4o, with especially large gains for Material, Period, and Region—attributes difficult to infer from images alone.
We then ran a review session with five ceramics specialists. Each viewed 100 images with two descriptions (TerraLex vs. GPT) and selected the better one. Final choices were determined by majority vote.
Figure 2 shows that evaluators preferred TerraLex. Agreement by Fleiss’ Kappa was 0.657 for GPT-4o. GPT-4o was favored only in about one fifth of cases, typically when retrieval returned no helpful similar items or when little cultural inference was required. Such cases may be detectable for future adaptive selection between the RAG and baseline outputs. In contrast, GPT-5 performed better than GPT-4o, favored in about one third of cases, with agreement degrading to 0.234. The results still demonstrate the effectiveness of RAG enhancement using human-generated metadata.
Table 3: Percentage of correct information produced by four models
Figure 2. Human evaluation of 100 instances by 5 pottery artists
Although TerraLex improves completeness and accuracy, its use in cultural heritage raises notable diversity and ethical concerns. Reliance on metadata shaped by Western institutions can reinforce cultural and geographic biases and underrepresent marginalized or non-Western ceramic traditions, reflecting epistemic injustice (Fricker 2007) and critiques of museum authority (Kidd 2018).
Visual-similarity retrieval may obscure culturally specific meanings, and automated metrics overlook interpretive or community-centered values. Yet TerraLex can also surface overlooked objects by enriching sparse records. Future work should draw on more globally diverse collections and involve humanities experts to ensure contextual appropriateness and avoid reductive interpretations of material culture.