Introduction

This study quantitatively evaluates the accuracy of automatic knowledge graph construction using Large Language Model (LLM) and examines its applicability to historical literature texts. As a case study, we focus on “Íslendinga saga” (Saga of the Icelanders), part of the medieval Icelandic “contemporary sagas.” Written in Old Norse in the 12^th -13^th centuries, these works record events close to their actual occurrence.

We compare two knowledge graphs constructed from the first ten chapters of the saga (out of 200): (1) a gold-standard reference graph created by a domain expert and (2) a knowledge graph automatically generated by LLM. Both graphs are modeled using the CIDOC Conceptual Reference Model (CIDOC CRM) (Figure 1.). Accuracy is measured across three key stages of the knowledge graph construction pipeline: entity recognition, event identification, and relationship extraction. Through this comparison, we assess the extent to which LLM-based knowledge graph construction can support or partially substitute expert scholarly work.

Related Works

Previous research on medieval Icelandic sagas has explored network analysis and named entity extraction, including ChatGPT-based approaches (Yamada et al. 2024), but event-centered knowledge graph construction remains underexplored. Ontologies, including CIDOC CRM and Narrative Ontology (Meghini et al. 2021), enable cultural heritage modeling, with recent applications of LLMs and CIDOC CRM to historical texts (Wang / Zhang 2025). This study contributes a quantitative evaluation of LLM-generated knowledge graphs against expert CIDOC CRM annotations for medieval Icelandic narratives.

Methods

To generate the automatic knowledge graph, we used MyGPT (GPT-5.1) with structured prompts explicitly specifying CIDOC CRM classes, properties, and output format (Figure 2.). The gold-standard knowledge graph was manually created by a saga specialist who closely read the text and encoded events, persons, places, and their relationships according to CIDOC CRM. We employed a multi-layered evaluation framework based on the following perspectives.

First, entity recognition was evaluated for persons and places using precision, recall, and F1 score. To account for orthographic variation in Old Norse names, string normalization (lowercasing, separator removal, suffix standardization) and exact matching were applied after URI normalization, with substring matching used as a fallback for unmatched entities. Second, event identification was evaluated by aligning events between the two graphs using the longest common subsequence-based string similarity measure on dct:sourceText values, with a threshold of 0.8. Third, relationship extraction was evaluated for matched events, focusing on participant extraction, place assignment, and event type classification.

Results and Discussion

The results (Table 1.) show clear differences in performance across the three stages. Entity recognition achieved high accuracy. Mismatches were attributable to several factors: orthographic variation in Old Norse names (4 cases, e.g., Gizurardóttir/Gizursdóttir, Reykholt/Reykja-holt), alternative naming with epithets (1 case: Ari Þorgilsson/Ari inn sterki), and misrecognition of descriptive expressions or common nouns as proper names (3 cases, e.g., Maðr ór Hjarðarholti (the man from Hjarðarholt), Þing (assembly)). Since critical editions of the saga provide indices of personal and place names as well as genealogical tables, integrating these resources into the construction process is likely to further improve entity-level accuracy.

Event identification achieved an F1 score of 0.766 (precision 0.859, recall 0.691). Of the 156 events generated by the LLM, 134 matched corresponding events in the gold-standard data, while 60 out of 194 gold-standard events were not captured. Chapter-level analysis revealed that missing events were concentrated in Chapters 5-7, which are substantially longer than others, with coverage dropping to 58-65%. This pattern suggests limitations related to context length and difficulties in segmenting extended narrative passages into fine-grained events.

Relationship extraction proved the most challenging task (Table 2.). Participant extraction achieved an F1 score of 0.603. A major source of error was the LLM’s tendency to rely exclusively on crm:P14_carried_out_by (active agent) while ignoring crm:P11_had_participant (general participation), which appeared frequently in the gold-standard data. This distinction is semantically significant in CIDOC CRM for differentiating active agents from passive participants. Place assignment (F1 0.568) and event type classification (F1 0.480) showed even lower performance, with frequent misclassification between closely related event types. The LLM also failed to capture complex inter-event relations such as event continuation and event composition, as well as property chains representing role-based participation.

Conclusion

Results indicate that while LLM perform strongly in entity recognition and can support experts by producing initial annotation drafts, substantial challenges remain in event identification and relationship extraction. Future work will explore ontology-aware output constraints using SHACL prompt refinement with few-shot examples, and integration of indices and genealogical data. We also plan to extend the evaluation beyond the opening chapters to analyze performance across different narrative stages.