Introduction

Carefully annotated datasets are essential for uncovering algorithmic bias and promoting debiasing techniques, especially in underrepresented domains that large-scale datasets tend to overlook (Yogarajan et al. 2023; Luka & Leurs 2023). Even at a small scale, structured annotated datasets allow computational analysis to “complement qualitative human readings” (Mauro 2014) and can help “pave the way for new annotations and new experiments” (Sprugnoli et al. 2023). Aligning with DH2026’s sub-theme "annotating | beyond patterns: interpretation with small data," this abstract introduces Poetry Annotator, an annotation tool developed within a broader research effort on underrepresented poetry. The tool supports building theme and emotion metadata of poems, including those with under-resourced languages (e.g., Native American and Hawaiian languages), to improve their visibility and support more equitable discovery in digital libraries. As an open-source, lightweight, and customizable tool, it can be easily adapted by other researchers for annotating their collections with different metadata schemes.

Background

We are annotating two poetry datasets to build richer metadata for underrepresented poetry and examine algorithmic biases in digital libraries. The poems were written by poets from five cultural groups, including Native American, Native Hawaiian and Pacific Islander, African American, Asian American, and Latino/a/x American, drawn from Poets.org and OCRed, copyright-protected poems accessed within the HathiTrust Research Center (HTRC) Data Capsule, a secure offline Linux environment. Dataset construction details are described in prior work (Choi & Kang 2025).

Existing annotation tools and methods were not compatible with our poetry annotation study. Simple text and spreadsheet tools flatten structure, which is problematic for poetry where lineation, spacing, and typography are essential to meaning-making (Bhyravajjula et al. 2025). They also lack administrative support for multi-annotator coordination. While annotators can read poems directly on the website (e.g., Poets.org), surrounding interface elements (e.g., navigation menus and advertisements) interrupt close reading and focused annotation. Furthermore, existing general-purpose annotation tools, such as BRAT, Doccano, TextAnnotator (Pontus et al. 2012; Hiroki et al. 2018; Abrami 2020; Colucci et al. 2024), likewise do not preserve poetic formatting or operate in secure offline environments, such as HTRC Data Capsules. We therefore developed a new annotation tool that preserves formatting, supports multiple annotators, and operates in a secure offline environment. After a two-month trial with 13 annotators and approximately 200 poems, the tool was refined and stabilized and is publicly available at: https://github.com/FAcc06/AnnotationTool_DH.

Key Features

Multidimensional annotation

Figure 1 shows the annotation interface design, where an annotator can assign theme and emotion tags to record their thematic conceptions of a poem and the emotions they feel while reading it. The interface shows 50 preset themes derived from Poets.org's top themes, offering annotators clear guidance and support for more consistent theme selection. It also includes eight preset emotions (anger, fear, anticipation, trust, surprise, sadness, joy, and disgust) from The NRC Emotion Lexicon (Mohammad & Turney 2013), a widely adopted emotion framework in DH and computational linguistics. Annotators can optionally add more theme and emotion tags. The interface also features a two-dimensional emotional coordinate based on the arousal-valence model (Russell 1980), a numeric affective representation previously tested in other cultural domains such as music (Yang et al. 2008). We expect future evaluation of its role in poetry collection building and discovery. To encourage thoughtful, grounded annotation, the interface requires annotators to provide brief explanations for their selected theme and emotion tags.

Figure 1. Poetry annotation interface (poem text blurred for copyright protection)

Workflow
Figure 2 illustrates the annotation workflows. Collections of poems are divided into small, adjustable Figure 2 illustrates the annotation workflows. Collections of poems are divided into small, adjustable worksets (e.g., twenty poems evenly sampled from the five cultural groups) to reduce bias from prior familiarity with specific groups. Worksets are automatically assigned upon completion, allowing asynchronous annotation with progress tracking. In the HTRC Data Capsule, which operates in offline mode, the workflow mirrors the online version, except that worksets are generated locally by an administrator and annotation results are stored locally. A separate administrative interface is available in the online version for user management, manual workset assignment, and export of annotation data.

Figure 2. Online and offline annotation workflows

Conclusion

We introduce Poetry Annotator, an open-source poetry annotation tool for annotating poems, including those in under-resourced languages, across online and offline environments. We hope other researchers will adapt this tool for their own annotation studies. In future work, we will use the annotated metadata to build a digital poetry library and analyze metadata and potential biases to promote equity.