Daejeon, July 27–31
Drawing on current debates in digital humanities on the relationship between hermeneutic and computational approaches (e.g., Tilton et al. 2024), especially on asking hermeneutic questions with computational approaches versus asking about the results obtained through computational methods from a hermeneutic perspective (Ringler 2024), this paper focuses on two aspects of engagement with small data, namely (i) interpreting small data through computational methods, which we refer to as “first-order interpretation” and (ii) interpreting computational results from the hermeneutic perspective ("second-order interpretation", see Vrangbæk et al. 2024 for a similar terminology).
As a case study, we address an underexplored phenomenon in German dialogue research: the success of interlocutors’ attempts to establish shared understanding or common ground between interlocutors (Clark & Brennan, 1991; Mohapatra et al., 2024) in counseling conversations. For example, the advice seeker's utterance, “I have lived there two years,” is followed by the pattern typical of the counselor’s social role and institutional context, namely the follow-up question, “Do you have a written rental agreement?” that advances common ground.
Prior work (Shaikh et al. 2024, 2025) has mainly focused on identifying and classifying grounding acts (e.g., acknowledgments, clarification requests) in English. However, the success of these attempts (whether an utterance advances shared understanding) requires context-sensitive interpretation as well as domain-specific and pragmatic knowledge. We investigate whether a German encoder model can approximate human judgments of grounding success and how annotation and preprocessing decisions shape the interpretation of computational modeling. Specifically, our research questions are:
- RQ1: How can we assess the success of grounding attempts through model training on small data and using only minimal context?
- RQ2: How does the interpretation of model outputs depend on hermeneutic decisions regarding the annotation process, data preprocessing, input restrictions, and dataset composition?
The dataset comprises 11,591 turns from nine institutional counseling domains. It belongs to the GRACO dataset presented in Belosevic (2026). Each turn received two judgments by two untrained linguists (Cohen’s κ = 0.17): (i) Attempt-to-advance (yes/no / NA), indicating whether the speaker tries to move grounding forward; and, if “yes,” (ii) Success (success / non-success), judged by the immediate uptake in the next turn. A third reviewer resolved disagreements by majority vote. Because success is undefined when there is no attempt, the classification dataset comprises 6,786 turns (subset with Attempt-to-advance = “yes”). The corpus includes a substantial portion of dialogues with non-native speakers (≈ 50%), reflecting linguistic diversity in institutional settings. Anonymization and normalization removed personal identifiers. Non-verbal markers and extended metadata were excluded to avoid deanonymization and structural leakage.
Methodology
We fine-tuned German BERT (deepset/gbert-base) with a small-data compatible training method QLoRA (Dettmers et al. 2023) for four epochs on the binary task (success vs. non-success) using a conversation-level split (80/10/10). To ensure predictions reflect interpretive content rather than structural shortcuts and to align with the human annotation process, inputs included only the current utterance and the immediately preceding turn. We withheld speaker roles, conversation phase markers, domain metadata, and all future turns. To address RQ1, we compared our results with human annotations and two baselines: the majority class and logistic regression, both trained on the same minimal context. Evaluation used a held-out test set (macro-F1, accuracy); 95% CIs via a 1,000-rep conversation-level bootstrap (percentile), resampling conversations with replacement (domain-stratified; seed=42).
GBERT clearly outperformed both baselines: macro-F1 = 0.86 (95% CI [0.83, 0.89]), accuracy = 0.86 ([0.83, 0.89]). Logistic regression achieved a macro-F1 score of 0.76 ([0.73, 0.79]), surpassing the majority baseline’s F1 of 0.34. Precision and recall for GBERT were both ≈ 0.86, indicating no collapse toward a single class. The confusion matrices (Figure 1) indicate a balanced performance across both classes for GBERT, suggesting a slight tendency to miss some actual successes and a somewhat less balanced pattern for the logistic regression baseline. The results confirm the strong performance of BERT-based models in classification tasks for low-resource languages, such as Marathi (Jadhav et al., 2025).
Unlike this first-order computational interpretation, the second-order qualitative perspective reveals shared error patterns between GBERT and logistic regression: 1) yes/no questions followed by acknowledgments; 2) incomplete questions or requests for confirmation followed by direct responses, and 3) next turns followed by follow-up questions are often misclassified as non-success. The error pattern 2) is also a typical case of inter-annotator disagreements. However, unlike human annotators, the models overclassify turns followed by truncated turns as success. Like human annotators, the logistic regression overrates minimal signals of attentive listening as successful acknowledgments.
These error patterns are shaped by hermeneutic decisions and extralinguistic factors (see RQ2). For example, preventing structural leakage (i.e., no speaker/phase/domain cues; no future turns) by design strengthens interpretive validity but also amplifies sparsity, as spontaneous talk encodes uptake through prosody, timing, and gesture. When such signals are scrubbed, some successes are textually under-determined. Annotators, too, may implicitly consult a slightly wider window than the single following turn when judging success. Where annotators disagree, we often find latent contextual dependencies not recoverable from two turns. In short, our second-order analysis surfaces a tension central to small data: methods that protect interpretive integrity can also induce blind spots that the model cannot overcome.
Furthermore, the dataset and training data reflect cultural patterns underlying institutional practices in German counseling dialogues. It comprises only publicly available dialogues from a limited number of domains, which may skew interactional styles. Finally, although training the model on the binary classification task did not enable the model to "see" how meaning is co-constructed by interlocutors during conversation, it allows comparability with guidelines used for human annotation.
The paper accounts for both methodological engagement with small-data practices that resist shortcut learning and foreground interpretive stakes, as well as critical engagement with language models as both an analytic instrument and an object of hermeneutic inquiry. Future work will test how presenting models with more immediate context and cross-domain generalization influences the distribution of error patterns.