Introduction

In Computational Literary Studies (CLS), the dominant application of machine learning follows a top-down trajectory: a phenomenon is defined a priori, and human experts annotate a dataset to train or test an algorithm's ability to detect it in a large corpus. While this supervised paradigm is effective (Bamman et al. 2024), it limits discovery to pre-defined concepts. This constraint is problematic for literary analysis of themes or plot patterns, where the material's richness clashes with a radical reduction to a few categories. Large Language Models (LLMs) offer a remedy, enabling a shift towards bottom-up research. As cooperative agents, LLMs can process corpora to identify emergent patterns and latent phenomena without pre-defined schemas. However, the integration of generative AI into the research design introduces new risks, specifically regarding reliability. Consequently, utilizing LLMs for inductive discovery requires a robust framework of checks and balances; rigorous evaluation is indispensable to distinguish between valid, data-driven insights and the stochastic noise or hallucinations inherent to the technology.

As a case study for this inductive framework, we address the challenge of computational plot analysis. Prior work (Konle et al. 2023) segmented narratives into scenes, defined as narrative units where the discours and histoire are equal, the location is fixed, a specific action is central, and the character constellation remains constant, but relied on motif repertoires (structured inventories of recurring plot devices) adapted top-down from literary theory.  While theoretically grounded, this approach remains constrained by the scope of pre-existing frameworks and faces challenges in operationalization.

In this paper, we explore a bottom-up procedure that uses LLMs to induce a motif repertoire directly from a corpus of romance novels. Additionally, we focus on developing evaluation strategies to assess the robustness of the process. This study serves as a first step in determining if generative AI can facilitate reliable data-driven discovery.

Methods

Our pipeline

Code, prompts and data: https://github.com/LeKonArD/DH26_motif_repertoire

Evaluations

Validity

To assess the quality of the induced motif repertoire, we evaluated the semantic consistency between the generated scene type labels and the narrative content. Two annotators

One of the authors and a student assistant with literary studies background

Plausibility

To assess whether the induced categories represent valid genre knowledge, we mapped the generated taxonomy against the romance plot schema defined by (Regis 2003) and operationalized for German dime novels by Konle et al. (2023). The bottom-up induction without prior prompting successfully discovered scene types which can easily be mapped to those described in theoretical works on the genre.

The model’s theme First Encounter (containing subtypes like Meet Cute) aligns with Regis’ First Contact. The induced themes of Growing Intimacy and First Sexual Encounter map to the Falling in Love stage. Most notably, the model granularly unpacked the abstract concept of Obstacle into specific narrative tropes such as Family Opposition, Romantic Rivalry, and Deception & Secrets. Finally, the narrative resolution was correctly identified, with The Big Fight mirroring the structural crisis (or Regis's "Point of Ritual Death").

Applicability

To assess the practical utility of the induced themes and scene types for yet unseen texts, we applied it to Verführung unter Palmen (Seduction under Palms), a novel excluded from the induction corpus. We compared the model's scene-by-scene annotation against a human-written summary

Complete summary and mapping: https://github.com/LeKonArD/DH26_motif_repertoire/blob/main/Applicability/Seduction_under_Palms_Mapping.md

The millionaire Max meets Janine, leading to a passionate one-night stand. To deter his ex, who is stalking him, Max pays Janine to pose as his wife; she agrees due to her own financial crisis. Despite growing attraction, they remain emotionally distant. When Janine reveals an accidental pregnancy, Max accuses her of deceit, causing her to flee. Later, Max discovers the truth, chases after her to confess his love, and proposes.

The model successfully reconstructed this sequence using the induced repertoire. It correctly identified the incitement as a First Encounter and First Sexual Encounter. The central conflict is mapped to Pretend Relationship, Forced Proximity, and Romantic Rivalry. Crucially, the model captured the dramatic climax and resolution in their correct chronological order: Unexpected Pregnancy, The Big Fight/Breakup, Grand Gesture, and finally Wedding/Happily Ever After.

Stability

Initial experiments on a corpus of 100 novels yielded a highly fluctuating taxonomy, suggesting that corpus heterogeneity was compounding model stochasticity. To isolate the model's inherent stability from this noise, we reduced the scope to a fixed subset of 10 novels and conducted 10 independent re-runs. The raw outputs are aggregated and manually mapped by two human annotators to semantic equivalents, resulting in a consolidated set of 37 Scene Subtypes and 28 Themes. The Adjusted Rand Index of the two annotations is .89 for themes and .87 for subtypes.

Figure 1: Frequency of induced Themes and Subtypes in 10 re-runs.

The results reveal a recurrent core structure surrounded by a variable periphery. As illustrated in Figure 1, the frequency distribution is heavily skewed: only 4 of the 28 Theme categories (e.g. Forced Proximity) and 5 of the 37 subtype categories (e.g., Discovery of Pregnancy) appeared in at least 8 of the 10 runs. This moderate convergence is quantified by the Mean Pairwise Jaccard Index, which yielded .36 for subtypes and .40 for themes. These metrics indicate that while the model consistently retrieves central plotpoints, the specific granular labels and peripheral concepts vary significantly between runs.

Discussion

Our experiments suggest LLMs can effectively facilitate bottom-up discovery in literary studies, provided a robust evaluation framework is used. The Validity assessment showed that although generative models hallucinate, their outputs are reliably verifiable. High agreement between human and automated judgments (F1-Score 0.91) suggests complex tasks can be scaled by using the models as evaluators via granular verification. Plausibility was confirmed as the induced categories matched established literary theory (e.g., Regis's romance schema), indicating the model captured latent genre knowledge. The Applicability evaluation further confirmed the repertoire's practical utility, demonstrating its ability to map plot structures in unseen texts. The critical limitation is stability: a core of themes and subtypes emerges consistently, but others fluctuate significantly, suggesting the taxonomy is a stable core with a fluid periphery.

Additionally, we face a fundamental recall problem. It seems like as long as the total number of latent themes and scene subtypes within a corpus is unknown, it is impossible to quantify how many valid patterns the model fails to detect. But framing it like this presupposes that this number of latent themes and scene subtypes is fixed and independent from an observer and its assumptions. Instead these concepts refer to generalizations heavily depending on genre knowledge and other factors.

It is important to acknowledge an inherent epistemological tension in data-driven induction: evaluating an LLM’s output against established literary theory risks introducing confirmation bias, as these models are pre-trained on vast amounts of human knowledge. If Plausibility were our sole metric, it would be difficult to prove whether the model discovered emergent patterns in the data or merely regurgitated existing literary theory. However, our framework mitigates this risk by triangulating Plausibility with Validity, Applicability, and Stability. Because human annotators independently verified that the LLM's induced categories genuinely reflect the narrative content of the specific scenes, and because the model successfully mapped these structures onto unseen texts, we can be confident that the categories are grounded in the textual data itself. Thus, alignment with Regis (2003) serves not as the ultimate proof of discovery, but as a demonstration that bottom-up, text-grounded induction can recover and operationalize theoretically sound genre frameworks.

To address these challenges, future work will focus on developing a multi-agent system (e.g. Wang et al. 2025) where the identification of new types and themes is decomposed into smaller operations. This approach will involve discrete steps for analyzing novels to create new candidates, verifying their validity, checking the database for similar entries, and systematically determining how to incorporate accepted candidates. It will be crucial to find the right balance between a parsimonious taxonomy and the individual demands of the texts. Ultimately, while this evaluation framework is not exhaustive, our results indicate that utilizing generative AI for bottom-up discovery is a promising avenue worth pursuing, though further rigorous evaluation is required to fully validate its capacity for genuine data-driven inquiry.