1. Introduction

As data-driven approaches increasingly shape the methodological paradigm of the humanities and social sciences(Ma 2021), an urgent challenge arises: how to systematically organize, interpret, and digitize the vast amount of archaeological catalogue resources at both breadth and depth(Bi et al. 2019). Traditional workflows relying on manual inspection and annotation are limited in efficiency and consistency, making them insufficient for large-scale knowledge base construction. Recent studies have begun to explore human–LLM collaborative approaches for metadata construction, highlighting the role of generative AI in structuring complex information (Bagchi 2025).

With advances in artificial intelligence, layout analysis has been widely explored in document processing(Zhao / Zhang 2025). However, existing models focus on block detection and lack mechanisms for semantic alignment or knowledge extraction, limiting deep understanding of archaeological catalogues. Meanwhile, multimodal models (e.g., BLIP, ChatVLM) show strong image–text reasoning, but precision and structured extraction remain insufficient(Alampara et al. 2025). Therefore, no single technology currently satisfies the combined requirements of speed, accuracy, and semantic depth needed for archaeological catalogue digitization.

The task of multimodal archaeological catalogue digitization faces several challenges:

First, catalogue layouts are visually complex with co-occurring multiple instances. A single page often contains several artifacts, identifiers, and descriptions, arranged in diverse formats. The correspondence between images and titles is typically implicit, making rule-based matching unreliable.

Second, fine-grained recognition of artifact components is essential. Humanities research requires detailed morphological information. For example, a bronze weapon is defined by a hierarchical morphological system—object, part, sub-part, and specific morphological descriptors.

Third, terminological variation and non-standard expressions are pervasive. Historical texts include variant terms, archaic vocabulary, and descriptive phrases. For instance, the same yuan morphology may be described as “willow-leaf shaped” or “elongated.”

Fourth, bridging visual–textual semantics is inherently difficult. The visual and textual modalities operate at different levels of abstraction and conceptual granularity. Constructing a unified semantic space is necessary for precise alignment. This challenge is not only technical but also conceptual, requiring the alignment of heterogeneous semantic representations across modalities (Giunchiglia et al. 2023).

2. Literature Review

Catalogue-type documents present cultural heritage information through paired images and text, including archaeological (N. Zhao 2007), art/exhibition (Dong et al. 2021), architectural (Zeng et al. 2019), and historical catalogues(Mo 2015). Archaeological catalogues serve as essential research tools, documenting artifacts in a structured manner. With the rise of digitization, catalogue formats increasingly integrate digital images and online publication. Early projects include the National Library of China’s “Digital Ancient Books Platform” and the Palace Museum’s “Digital Heritage Repository,” while video, BIM, and VR techniques offer additional documentation possibilities (Yang / Lee 2020). However, these efforts primarily focus on digitization and presentation, with limited attention to semantic structuring and knowledge organization. From a knowledge organization perspective, catalogue data should be understood as part of a broader ecosystem integrating data, standards, and interpretive frameworks (Bagchi 2021b).

Layout analysis techniques support structural analysis of complex documents. Early rule-based methods were interpretable but lacked generalizability (C. Liu et al. 2023). With the development of deep learning, accuracy in detecting elements such as figures, titles, and paragraphs has improved significantly(C. Liu et al. 2021), enabling applications in commercial and industrial domains(Pappula / Rusum 2023). Nevertheless, deep learning–based layout models remain focused on visual detection and lack semantic understanding and knowledge-level extraction. This limitation prevents their effective application to archaeological catalogues, where meaning arises from the interplay of images, text, and concepts. This limitation stems from the lack of domain-level semantic modeling. Knowledge-intensive domain methodologies emphasize that complex domains require explicit conceptual structures, such as ontologies, to support consistent interpretation and scalable knowledge construction (Bagchi 2021a).

Multimodal artificial intelligence, especially Vision-Language Models (VLMs), provides new opportunities for processing multimodal inputs(Zhang et al. 2024). Contemporary models such as GPT-4o, QWen-VL(Bai et al. 2023), and GLM-V4(Zeng et al. 2024) improve unified visual–textual reasoning for zero-/few-shot extraction. Models like GLIP(Li et al. 2022) and Grounding DINO(S. Liu et al. 2024) have also demonstrated strong performance in open-vocabulary detection, contributing to both general-purpose understanding and domain-specific tasks. However, precision remains insufficient for scholarly analysis of archaeological catalogues.

3. Methodology

Figure 1. Workflow of Ontology–Layout Analysis–Multimodal LLM Collaborative Extraction. Figure 1 shows the complete workflow from raw data preparation to structured knowledge base construction and analysis.

To address the challenges outlined above, this study proposes a collaborative workflow (Figure 1) that integrates ontology engineering, layout analysis, and multimodal large language models. It aims to transform scanned pages into a structured knowledge base. The workflow consists of the following components:

3.1 Construction of a fine-grained cultural heritage ontology

Following ISO 704/1087 and W3C standards, the ontology is developed with reference to research scope, objectives, and domain knowledge. It formalizes artifact categories, component hierarchies, attributes, and historical contexts. Competency questions guide the ontology to ensure queryability and reasoning. In this study, taking weapon catalogues as the target corpus, we construct a bronze weapon ontology and publish it at:

http://www.dhontology.com/ChineseCulture/data/bronze-weapon-ontology.owl

3.2 Layout analysis and structured extraction

First, an end-to-end layout detection model (e.g., DocLayout-YOLO) is used to segment each page into block types, including image blocks, title blocks, and text blocks. OCR is applied to obtain textual sequences with positional indices.

Second, a layout graph is constructed based on the detected layout elements. Specifically, each layout unit is abstracted as a node, whose attributes include spatial coordinates and semantic labels derived from the detection results (e.g., image, title, text). For each image node, edges are established to multiple candidate textual nodes (e.g., figure titles, paragraph titles, and text blocks), forming a set of potential image–text associations. Based on this graph structure, a graph neural network (GNN) is used to learn node representations; image–title matching is achieved through node similarity and edge-weight prediction within the layout graph.

3.3 Ontology-driven prompt engineering and multimodal extraction

Prompt templates—covering entity recognition, attribute extraction, relation judgment, and disambiguation—are automatically or semi-automatically generated based on ontology classes and properties. Each template includes examples and constraint instructions (e.g., requiring that outputs be mapped to ontology class labels).

Using these templates, the multimodal LLM conducts staged extraction:

• Visual parsing: the model identifies object parts and morphological features.

• Text parsing: NER and RE are applied to extract morphological descriptors, chronology, provenance, and related attributes.

• Semantic alignment: visual labels and text entities are mapped to ontology URIs; entity alignment and disambiguation are performed.

Finally, extracted results are converted into RDF triples and linked to OWL classes and properties to ensure semantic consistency. A graph database (e.g., Neo4j) is used to store the knowledge and provide Cypher/SPARQL-based complex querying for downstream statistical analysis, visualization, and archaeological computation. This design also facilitates data interoperability in line with FAIR principles, ensuring that the structured knowledge is findable, accessible, interoperable, and reusable across different systems and research contexts. To mitigate potential biases in multimodal large language models, ontology constraints and structured prompt templates are introduced to guide the extraction process toward domain-consistent and interpretable outputs.

3.4 Knowledge-based analysis

On top of the structured knowledge base, cross-temporal and cross-regional analyses—such as morphological statistics, evolutionary modeling, and network analysis—can be carried out in conjunction with archaeological expertise, overcoming the traditional limitations of scale and consistency in humanities research.

4. Experiment

Experimental results demonstrate that the proposed workflow achieves high-accuracy image–text separation and semantic pairing in documents with complex layouts. The standardized data generated through this workflow show good generalizability and extensibility. Using The Compendium of Ancient Chinese Weapons (《中国古兵器集成》in Chinese) as the primary source, we construct a fine-grained knowledge base of bronze weapons through the following steps:

4.1 Weapon ontology construction

Guided by the “term–characteristic–concept” methodology (Wei & Chen, 2023), we design a fine-grained ontology of bronze weapons. The ontology standardizes hierarchical weapon categories (Figure 2), component morphology terminology and attribute relations (Figure 3), and incorporates social-historical contexts and spatiotemporal dimensions.

Figure 2. Hierarchical representation of weapon categories. This figure illustrates the hierarchical classification system of ancient weapon categories.

Figure 3. Core classes and object properties of the weapon ontology. This figure presents the main core classes and relational object properties defined in the weapon ontology.

4.2 Image–text extraction and multimodal alignment

Figure 4. The results of layout analysis. This figure demonstrates the layout segmentation and structural analysis results of cultural heritage artifact documents.

Following the workflow described above, layout analysis, OCR, and block segmentation are performed (Figure 4), and a graph neural network is employed for block-level pairing. The experiments are conducted on a self-constructed archaeological catalogue dataset, where page-level layout graphs are built based on detected elements, and image–text pairing is formulated as a binary classification task over candidate node pairs. Guided by the ontology, a multimodal vision-language model (GPT-4o Vision) identifies weapon components and morphological features from images, while textual descriptions are processed to extract morphological terms, chronological information, provenance, and related attributes.

To evaluate the effectiveness of the graph-based pairing, we report the performance of the GNN on image–text matching, as shown below (Table 1):

Table 1. GNN Method Performance on image-text matching

The model is trained and evaluated on 1,380 page-level graph samples (1,100 for training and 280 for testing), where each node is represented by combined spatial and semantic features. The results indicate that the proposed layout graph and GNN-based approach can reliably capture structural relationships between layout elements, achieving high accuracy in image–text alignment even under complex multi-instance layouts.

Based on the matched image–text pairs, both visual and textual information are mapped onto ontology classes to achieve semantic alignment. For each weapon instance, RDF triples are generated in a normalized manner, covering physical dimensions, sociocultural contexts, and spatiotemporal attributes.

4.3 Quantitative Analysis of Morphological Evolution

This study further selects bronze Ge(戈) from the Shang and Zhou periods as a focused case study. A dataset comprising 329 Ge specimens encoded along 15 component-level dimensions was extracted through the ontology-based workflow. Using a multi-layer approach including temporal frequency, feature co-occurrence, and morphological clustering, the study reconstructs the evolutionary trajectory of bronze Ge from the Shang to Warring States periods.

The overall results reveal a clear and consistent evolutionary pattern. Functionally, components associated with fastening mechanisms, such as the elongation of the Hu(胡), the number and distribution of perforations, and the increasing dominance of the lower tooth, became progressively standardized. These changes reflect a structural transition from simple binding systems to more complex and mechanically stabilized composite forms.

In contrast, decorative and symbolic elements exhibit persistent diversity: from the animal-mask motifs of the Shang and Zhou, to the geometric regularization of the Western Zhou, and finally to the institutionalized inscriptions of the Warring States. This pattern demonstrates the dual role of the Ge as both a military implement and a political-symbolic artifact.

In summary, this study proposes and implements an end-to-end multimodal datafication framework for archaeological catalogues, addressing key challenges in complex layout processing, multi-instance pairing, cross-modal alignment, and semantic consistency. Second, it introduces a cultural heritage ontology together with an ontology-driven prompt engineering scheme, enhancing the interoperability of cultural heritage images and texts within a unified semantic space, while aligning with FAIR data principles to support reuse and cross-platform integration of archaeological knowledge. Finally, through a knowledge-base-driven analytical case study, the paper illustrates the potential of this approach to support cultural heritage research and preservation.

In future work, we aim to address current limitations, such as hallucination in large models and restricted ontology coverage, through ontology expansion, scalable weak-supervision strategies, and more robust cross-lingual extraction methods.

*This research is supported by the NSFC project "Research on the Construction of Cross-Language and Multimodal Knowledge Graph of Cultural Relics" (Grant No.72204011).