Introduction

Freytag’s pyramid (Freytag 1900) is a theoretical framework that defines the structure of a narrative according to five phases usually called exposition, raising action, climax, falling action and denouement. This article focuses on narrative segmentation of literary and historiographical texts and their annotation with Freytag’s pyramid labels using large language models (LLMs). The topic is of interest from a theoretical, technical and pedagogical point of view. Recent work in this area has showed that multi-agent generative AI systems (Arif et al. 2025) using Freytag's pyramid structure to guide narrative generation via LLMs along with multimodal outputs (text-to-speech, video, music) can produce engaging educational settings for story co-creation and enhance children’s storytelling skills. Other studies that incorporated LLMs into narrative generation and comprehension pipelines and comparison with human performances have reported both promise and challenges in tasks such as narrative topic labelling and detection of elementary components of narrative discourse (Piper / Bagga 2024; Piper / Wu 2025), as well as AI story writing (Tian et al. 2024).

In digital humanities, LLMs’ evaluation represents a key aspect that needs to be further examined due to the very nature of problem-solving in this domain, which can often imply various degrees of uncertainty and subjectivity not easily measurable by clear-cut methods. For example, how can we assess LLMs’ output and reasoning in complex tasks such as narrative segmentation and annotation? To what extent the training, mathematical and probabilistic mechanisms inherent to these technologies can reflect a certain degree of comprehension of the story and problem to be solved?

Our assumption is that to answer this question a more complex evaluation system would be required to take into account multiple perspectives on the analysed processes. To test this assumption, we propose a meta-benchmark that considers three types of tasks performed by the models: (1) annotation; (2) evaluation; (3) rebuttal. A human agent can also be involved as a meta-evaluator in all the three tasks. The rationale of such a scenario is that by asking the models to solve a problem, evaluate the solutions of other models (and itself) to that problem and defend their own way of solving it may shed light not only on the capabilities and limits of these models in assuming various roles within the problem-solving space but also on how humans can interpret this type of mirror effect sometimes reflecting pieces of their own image. The proposed framework includes a set of data and an experimental protocol as described in the following sections.

Dataset

The size of the dataset used for this project is intentionally small to ensure closer control and analysis of results and interpretation. Two types of texts were chosen for the experiments: (1) literary; (2) historiographical. The literary texts are short stories (8.000 to 14.000 characters) published in the 1910s-1920s and available via Project Gutenberg (Montgomery 1910; Daudet 1920; Chekhov 1921; Mansfield 1922). The historiographical texts are short excerpts (4.600 to 5.700 characters) from four 19-th century historical works and editions also available in the public domain (Michelet 1847; Tocqueville 1856; Burckhardt 1878; Ranke 1887).

Although the experiment language is English, with both original and translated texts, the selected works involved authors of different origin and gender to ensure a certain level of diversity. The choice of the historiographical texts was driven by White’s (1973: 7) concept of emplotment, as the way by which a “sequence of events fashioned into a story is gradually revealed to be a story of a particular kind”, and its application to the analysis of four major 19-th century historical works from which the fragments were extracted. It was therefore intended to compare the segmentation and annotation of short stories and test whether a certain plot structure, as hypothesised by White, can be detected for the historiographical selection too. While the study is not meant to discern between different types of emplotment in White’s sense (i.e., shaping historical events into narrative forms by using literary structures such as romance, comedy, tragedy and satire), a simpler annotation schema based on Freytag’s pyramid was chosen for the experiments as a way of identifying basic narrative units and trajectories of the rise and fall in narrative tension. It was assumed that story segmentation into narrative fragments and their annotation with Freytag labels by the models require a certain degree of understanding of the story arc and tension distribution across the segments, and conformity to a strict linear order of the annotated episodes.

Experiment setting and methodology

The experiments used three LLM-based chatbots (ChatGPT 5.2. Thinking, Claude Opus 4.5, Gemini Pro3 Fast) and three open Hugging Face models running locally in GGUF format (mistral-7b-instruct-v0.2.Q4_K_M, llama-pro-8b-instruct.Q4_K_M, SOLAR-10.7B-Instruct-v1.0.Q4_K_M). The choice of the models was determined by the author’s access to the first two chatbots via personal and institutional subscriptions and the free accessibility of the others. Selecting both big and small, commercial and open models was part of the plan of ensuring variety and a comparison basis for the study.

Figure 1 shows the main phases in the meta-benchmarking workflow. The models receive the same prompts (Appendix 1) and the corresponding files as input, depending on the phase (annotation, evaluation, rebuttal). Outputs are evaluated by other models and the human agent (Appendix 2), but a different architecture with additional layers and hierarchies of evaluation both by human and AI can also be envisaged.

Figure 1. Meta-benchmark workflow for Freytag annotation using LLMs

Preliminary results and discussion

The project is still ongoing and the abstract reports on preliminary results. A first round of tests has been performed so far with the AI chatbots and open LLMs for the annotation, evaluation and rebuttal related to the literary and historiographical dataset.

Table 1 shows the evaluation results for the annotation of the literary (lit) and historiographical (hist) texts, with the highest scores obtained by Claude (hist) and ChatGPT (lit) from the chatbot category, and Mistral (hist) and Llama (lit) from the open LLMs. For the historiographical annotation, Claude received the highest scores from Gemini for both criteria, conformity with prompt instructions (C1) and quality of annotation (C2), and lower scores from ChatGPT and itself. We illustrate the literary annotation, evaluation and rebuttal for Daudet’s short story. The main issue in Claude’s annotation, pointed out by ChatGPT and itself (Gemini assigned to it the maximum score, while the open LLMs either provided no numerical score or a numerical score with less specific justification), consisted of an early and diluted climax that spanned 3 segments, and the placement of the tension peak "To arms! To arms! The Prussians!“ in the denouement instead of climax. Interestingly, none of the models identified this sequence as the climax, but ChatGPT and Claude mentioned it in their evaluations (Appendix 3).

Table 1. Evaluation of the annotation task (literature, historiography) with degrees of criteria (C1, C2, C1+C2) fulfilment on a Likert scale 1 (very low) to 5 (very high) (runs 13-14.12.2025 and 19-22.03.2026)

In general, the chatbots followed more closely the instructions and produced more structured annotations, while the open LLMs sometimes struggled with the required format, story coverage, number of segments, and the strict order of Freytag’s pyramid structure. For both categories, it was observed that although some aspects were not correctly defined in their own annotations, the models were occasionally able to identify these errors and properly comment on them in their evaluations or rebuttals, an intriguing behaviour that would need further investigation.

Conclusion

The article proposes a meta-benchmark that includes three phases, annotation, evaluation and rebuttal, for human (work in progress) and AI assessment of plot annotation with Freytag’s pyramid labels by LLM-based chatbots and open LLMs. It assumes that this type of framework may shed light on the reasoning capabilities of the models and the levels of understanding implied by complex tasks such as segmentation and plot annotation in literary and historiographical texts. While the framework can be applied to other research fields, the study focuses on story comprehension rather than broader human-like understanding, which requires extended testing and larger varieties of tasks and data.

Appendix