Diffusion-based image generation has begun to attract exploratory attention within digital humanities and art history. Yet its methodological implications remain poorly understood when such models are adapted to the kinds of datasets art historians frequently work with: small, fragmentary, stylistically coherent corpora that demand interpretive mediation rather than large-scale statistical regularities. One overlooked aspect is captioning. Whereas machine learning typically treats captions as functional semantic labels, art history regards description as interpretive writing - a site where stylistic judgment, contextual framing, and iconographic reasoning converge. In diffusion fine-tuning, each training image is paired with a caption; captions therefore help structure the relation between textual prompts and visual features. This paper investigates what captions do when diffusion models are fine-tuned on extremely small art-historical corpora, and how captioning interacts with prompt design, dataset structure, and pretrained visual patterns.

The study is part of a digital-humanities research project examining whether generative models might support art-historical reasoning, understood here as the formulation and testing of stylistic, iconographic, and compositional hypotheses with incomplete evidence. Our case study centres on the 17^th century Carniolan painter Almenak, whose surviving oeuvre consists of only four extant works, with two later paintings tentatively attributed as early modern copies of now-lost originals. Almenak was selected because the corpus exemplifies a common art-historical problem: how to reason from a few secure works and later images that may mediate lost compositions without treating speculation as reconstruction. Rather than reconstructing missing works, our goal is epistemic: to understand what, and how, a model learns when trained on severely limited data, how different captioning styles shape or fail to shape this learning, and what this reveals about the interpretive conditions under which generative AI might contribute to art-historical inquiry.

Recent advances in generative image modelling have been driven primarily by diffusion- based architectures trained on large-scale image–text datasets (Rombach et al. 2022). Adaptation techniques such as Low-Rank Adaptation (LoRA) enable pretrained diffusion models to be fine-tuned on small datasets by modifying limited subsets of parameters, making such models accessible beyond industrial research contexts (Hu et al. 2022). The experiment fine-tunes Stable Diffusion XL (SDXL; Podell et al. 2023), a U-Net-based latent diffusion model, through LoRA adapters. Most technical studies of fine-tuning, however, focus on stable visual domains and do not address historically specific or fragmentary material. Here, small data is not only a numerical constraint but an art-historical one: limited survival, uncertain attribution, and fragmentary evidence restrict what kinds of visual relations are available to learn.

Within digital humanities and art history, engagement with generative AI has been cautious and uneven. Existing scholarship has foregrounded ethical concerns, institutional bias, and the epistemological risks of reconstruction rather than empirically examining how models behave when trained on historical data. Nygren and Drimmer (2023) have argued that machine-learning systems risk obscuring the interpretive and institutional histories embedded in art-historical datasets, while Spratt (2021) has emphasized the tension between AI-driven image completion and the conceptual centrality of the fragment. Bishop (2018) has noted more broadly that computational methods reshape the questions art history can ask, and Manovich and Arielli (2024) situate AI-generated images within longer histories of recombination and fragmentary visual knowledge. Captioning has received little attention as a methodological problem within digital humanities. This study addresses that gap by examining captioning practices within a controlled small-data fine-tuning scenario grounded in art-historical material.

To reflect the realities of art-historical evidence, we divided the four preserved Almenak paintings into ninety-nine fragments, including facial details, hands, still-life elements, drapery, and isolated architectural or background motifs. These fragments served as the complete training set for the experiment. This approach mirrors art history’s frequent reliance on isolated stylistic clues rather than intact compositions and imposes a learning problem fundamentally different from the heterogeneous datasets on which diffusion models are originally trained. It also introduces a deliberate trade-off: fragmenting the paintings increases the number of training samples and foregrounds local stylistic features but removes many of the spatial and compositional relations through which whole pictorial scenes are organized.

Fine-tuning was conducted using LoRA (Low-Rank Adaptation), a lightweight technique that attaches trainable matrices to specific attention layers (Hu et al. 2022). LoRA is widely used for stylistic adaptation because it can learn visual tendencies from small datasets without overwriting the pretrained model’s representational structure. Our findings confirm that it adjusts surface-level features more readily than deeper compositional regularities.

To test the role of captioning, we constructed four parallel caption sets for all fragments, ranging from minimal keyword-like labels to expert art-historical descriptions and machine generated captions derived from vision–language models and CLIP. Each caption set was used to train a separate LoRA adapter under identical conditions: the same SDXL base model, image set, training resolution, training length, LoRA rank, learning-rate settings, and evaluation prompt set. We then evaluated the models through controlled prompts, including historically plausible scenes and deliberately anachronistic ones, with and without a learned stylistic conditioning token (“in style of almnk”).

The results show that LoRA fine-tuning on this micro-corpus produces a recognizably coherent stylistic shift relative to the base model. Prompts that included modern settings, such as “a woman in a 20th-century kitchen,” tended to acquire elements of a Baroque chromatic palette and painterly surface treatment across all four LoRA-trained models. While this stylistic tendency was broadly shared, the models differed substantially in how they interpreted the modern setting: some preserved contemporary interiors rendered in a historical idiom, whereas others reimagined the scene as a fully Baroque environment.

At the same time, captioning did not control the presence or intensity of surface-level stylistic features. Here, “style” is used narrowly to refer to visible surface effects such as chromatic palette, simulated texture, brushwork simulation, light, and modelling; “interpretive behavior” refers to how the model resolves subject, setting, figure relations, and prompt semantics. Whether fragments were captioned with terse labels, richly interpretive descriptions, or machine-generated sentences, these stylistic features remained broadly consistent. Differences between captioning strategies emerged instead in how the model resolved the semantic content of prompts.

Models trained on simple or vision–language-model captions tended toward literal reuse and recombination of visual fragments. In prompts such as “an elderly peddler showing trinkets to a peasant woman,” these models frequently reproduced figures closely resembling those in the training set, repeating posture and facial type. When prompted with Baroque-inflected scenes in modern interiors, they often abandoned the modern setting entirely. By contrast, models trained on expert or CLIP-based captions displayed slightly greater interpretive flexibility, more often retaining the narrative structure of the prompt or combining motifs from multiple fragments rather than replicating a single one. These outputs nevertheless demonstrated a more elastic response to prompt semantics.

A further limitation became apparent in prompts involving multi-figure tavern scenes. The base model, despite lacking exposure to Almenak’s work, produced more coherent group scenes. All LoRA-trained variants struggled with spatial organization: figures overlapped awkwardly, perspective drifted, and interactions were often incoherent. This limitation reflects a property of the dataset rather than a failure of LoRA itself. By fragmenting the paintings, we removed the compositional cues necessary for learning spatial grammar. The result demonstrates a structural constraint of fragment-based fine-tuning: while LoRA can learn painterly surface features, it cannot infer absent spatial relationships.

The broader implication is that captioning in diffusion-model fine-tuning does not function primarily as a mechanism for stylistic control. It functions instead as an interpretive framework that interacts with fragment structure and prompt semantics, shaping tendencies toward reuse, recombination or contextual retention. Captioning does not tell the model “what style to learn,” but it may influence assumptions about what a fragment represents and how it should be activated in response to textual prompts. This finding aligns with digital humanities critiques emphasizing the tension between humanistic description and machine-interpretable annotation (Bishop 2018; Nygren / Drimmer 2023; Spratt 2021), as well as with machine-learning research showing that pretrained visual representations exert a dominant influence over generation, with textual conditioning functioning primarily as a modulating rather than determining factor (Rombach et al. 2022; Hertz et al. 2022; Gal et al. 2022).

For art-historical reconstruction and visual speculation, these results highlight the need for caution and methodological clarity. LoRA fine-tuning can shift stylistic surface features even from extremely small datasets but cannot reliably learn compositional grammar or narrative structure from fragmented inputs. Captioning shapes interpretive tendencies but not stylistic fidelity, while prompt phrasing remains a disproportionately powerful determinant of generated images. The experiment therefore specifies why these outputs remain speculative: their visual plausibility arises from an interaction among corpus structure, captioning, prompts, and pretrained visual priors, rather than from a recoverable model of lost works.

By testing captioning regimes under tightly controlled conditions, this study contributes a methodological framework for understanding how diffusion models integrate and reorganize fragmentary art-historical data. It clarifies the interpretive role of captioning and its limits, offering a foundation for more historically grounded uses of generative AI.