Daejeon, July 27–31
This project demonstrates how we can use a Knowledge Graph with expert-created domain ontologies in conjunction with multimodal LLMs to create new descriptions of visual historical sources in a structured and verifiable manner with European heraldry as a case study. We implement a Graph-RAG system to create LLM-based descriptions of coats of arms that are improved and grounded in a Knowledge Graph, leveraging existing descriptions of coats of arms, encoded in RDF and structured by a formal ontology.
One of the core challenges in cultural heritage research is scaling the description and datafication of vast amounts of complex historical source material without sacrificing depth, accuracy, or contextual richness. This becomes particularly acute when dealing with visual sources where symbolic systems, material variations, and interpretive complexity intersect.
Our case study focuses on heraldic sources from medieval and early modern Europe. Heraldry was omnipresent as a tool for visual communication across European societies, through coats of arms (Hablot 2017). These visual codes consisted of a largely fixed set of over 1,000 different symbolic primitives combined according to specific heraldic conventions. Despite their significance as historical sources, large portions of this material remain unexplored (Hiltmann 2019). We are dealing with potentially over one million coats of arms preserved across various material contexts (Pastoureau 2008, p. 42).
What makes heraldry a particularly interesting test case for scaled visual source description is its extreme longtail distribution of symbolic elements. In our existing knowledge graph containing over 80,000 representations of coats of arms in manuscripts, we observe that 96.7% of all charge types (1,102 of 1,140) appear fewer than 0.5% of the time, with 320 charges appearing only once. This represents a quintessential small data challenge. How can we generate high-quality, interpretable descriptions when most domain-specific elements appear too rarely to support pattern-based machine learning? This distribution pattern mirrors challenges found across many cultural heritage domains where rare elements carry crucial historical significance yet remain severely underrepresented in training data.What makes heraldry a particularly interesting test case for scaled visual source description is its extreme longtail distribution of symbolic elements. In our existing knowledge graph containing over 80,000 representations of coats of arms in manuscripts, we observe that 96.7% of all charge types (1,102 of 1,140) appear fewer than 0.5% of the time, with 320 charges appearing only once. This represents a quintessential small data challenge. How can we generate high-quality, interpretable descriptions when most domain-specific elements appear too rarely to support pattern-based machine learning? This distribution pattern mirrors challenges found across many cultural heritage domains where rare elements carry crucial historical significance yet remain severely underrepresented in training data.
Scholarly work with heraldic sources requires formally correct data that can be reliably used for querying and analysis. This necessitates validated descriptions. Without such grounding in expert knowledge, automated descriptions may appear superficially reasonable while containing errors that undermine their utility for scholarly use and database integration. The challenge is generating descriptions that meet these exacting standards at scale.
Existing approaches to heraldic material operate primarily on the visual level. Similarity-based machine learning methods, including embedding-based approaches, can be successfully used (Sustek et al. 2020, Webaldic). However, these methods sacrifice historical context and cannot link heraldic imagery to the persons, groups, territories, and material objects essential for scholarly interpretation.
Knowledge Graphs offer a solution to this context problem. The Digital Heraldry Knowledge Graph (Schneider 2023) demonstrates how heraldic material can be richly contextualized e.g. with manuscript metadata, image regions via IIIF annotations etc. Yet manual curation at scale remains impractical given the volume of surviving sources.
The emergence of Graph-based Retrieval-Augmented Generation offers new possibilities for leveraging structured knowledge at scale. Recent surveys by Zhang et al. (2025), Han et al. (2025), and Zhu et al. (2025) reveal two dominant paradigms. Most prevalent are approaches that extract entities from textual corpora to construct knowledge graphs for retrieval (Edge et al. 2024, p. 3). However, these approaches assume the need to build graphs from textual data rather than leverage existing expert-created ontologies. A second, less conventional strand leverages existing large-scale knowledge graphs like Freebase or Wikidata for retrieval (Zhang et al., 2025). Both methods primarily address text-to-text retrieval tasks, where both queries and retrieved content are textual.
The closest existing approach is OG-RAG (Sharma et al. 2025), which grounds retrieval in domain ontologies — across healthcare, agriculture, and news domains — by constructing hypergraphs of factual blocks extracted from unstructured documents. While philosophically aligned with our commitment to ontology-grounded retrieval, OG-RAG operates only on textual sources and constructs its hypergraph at indexing time from documents; our work extends this commitment to visual cultural heritage and to retrieval directly from a pre-existing expert-curated ontology with multilingual scholarly definitions.
Cultural heritage domains present distinct challenges that expose limitations in current Graph-RAG approaches. First, cultural heritage often involves visual objects requiring domain-specific descriptions. Second, while large-scale knowledge graphs offer broad coverage, specialized domains possess expert-curated ontologies encoding decades of scholarly knowledge that general-purpose graphs cannot capture. The heraldic longtail distribution exemplifies this problem: multimodal LLMs struggle with rare examples that lack training data, yet these elements often carry crucial historical significance. This gap between general-purpose textual Graph-RAG and the need for specialized, ontology-grounded multimodal approaches motivates our methodological contribution.
Our approach addresses this gap by adapting graph-based retrieval techniques, including hierarchical summarization, for ontology-backed knowledge graphs. Rather than constructing graphs from extracted entities, we leverage the pre-existing hierarchical structure of our expert-designed heraldry ontology. This ontology extends beyond a controlled vocabulary to provide a complex class hierarchy that captures conceptual relationships between heraldic elements.
Crucially, our knowledge graph embeds multilingual definitional texts from heraldic handbooks and scholarly literature. These definitions, drawn from German, English, and French sources, describe heraldic properties from multiple perspectives, reflecting both historical evolution of heraldic conventions and varied scholarly emphases across traditions. This multilingual, multi-perspective foundation proves essential because heraldic terminology and conventions are not uniformly structured across linguistic and temporal boundaries.
The system operates through iterative refinement. A multimodal LLM generates an initial analysis of heraldic figures present in a coat of arms image. The system then retrieves relevant definitional context from the knowledge graph based on this initial analysis, cross-references against established blazon conventions, and produces a refined description aligned with scholarly standards. Our heraldry ontology provides validation constraints beyond what fine-tuned models offer, as descriptions must conform to the formal structure encoded in the ontology itself.
While our case study focuses on heraldry, the methodological approach has broad applicability across DH and cultural heritage domains dealing with complex visual sources, described by specialized vocabularies and scholarly literature. Consider Iconclass, a widely used classification system for art-historical imagery, where concepts are similarly defined in reference works (Couprie 1983). The multilingual structure of our approach resonates with the conference theme of translinguality, demonstrating how formal knowledge structures can serve as translingual semantic bridges, enabling AI systems to operate across language boundaries without privileging a single linguistic perspective. Such systems can enable institutions and publics to collaboratively describe visual sources using AI-assisted tools grounded in scholarly expertise.