Job advertisements from historical newspapers offer a rich resource for studying labor markets, but extracting structured information from them remains challenging due to their unstructured format, OCR errors, complex layouts, and archaic language—challenges that are common to Named Entity Recognition on historical texts more generally (Novotny et al., 2023; Avram et al., 2024; Schweter et al., 2022). Such challenges, as well as the research potential of these sources, have also been identified by several scholars working with job advertisements or digitized newspapers more broadly (Bunout, 2023; Håkansson et al., 2023; Oberbichler & Pfanzelter, 2023; Schulz et al., 2014; Wevers, 2022; Wevers & Smits, 2020).

Within the JobAds project, we developed a comprehensive pipeline for processing Austrian newspapers (Österreichische Nationalbibliothek, 2021) from 1850 to 1950, focusing on job advertisements. The pipeline encompasses newspaper pages segmentation, OCR, text-type classification, and training of machine learning models for NER. The NER component relies on an ELECTRA-based architecture fine-tuned on historical German newspaper data ('zeitungs-lm-v1' (Schweter, 2024)) to better capture diachronic linguistic variation typical of the period. Details of the pipeline architecture and model training are described in Adam et al. (2025) and Venglarova et al. (2024).

While these prior publications have presented our results in extracting job titles and occupational data, this poster shifts the focus to a crucial yet often underrepresented aspect of the pipeline: the manual annotation process used to train a Named Entity Recognition (NER) model to identify job titles, requirements, skills, salary information, and other job-relevant entities. The structured data produced by the pipeline enables analyses of long-term developments in job demand and supply; for a detailed study see Mölzer and Kleinert (2024).

To generate high-quality training data for the NER model, the project team, with the support of student assistants, manually annotated a representative subset of 14,985 job advertisements, using the doccano open-source annotation tool (Nakayama et al., 2018). The subset of the JobAds corpus chosen for annotation was derived through disproportionate stratified sampling with respect to newspaper title and year. The aim was twofold: to provide a small, high-quality annotated dataset and to create the training material necessary for large-scale automated extraction of job-relevant information.

Annotations covered several layers of information, starting from layout when we were annotating visual blocks on the newspaper page. Further annotations, already on the text level, included individual components of job advertisements, such as job titles, salary information, and requirements.

The process revealed the inherent challenges of translating human expression into structured categories. For instance, in the case of visual annotations of job advertisements and their classification into sub-categories (job offers, job searches, service offers, mediations and headings), syntax was sometimes ambiguous: a phrase like “Perfekte Kontoristin sucht Schuhfabrik” [ Perfect office clerk seeks shoe factory or Shoe factory seeks a perfect office clerk ] leaves it unclear whether it is a job offer or a job search, complicating classification, and the ‘correct solution’ is only available based on the context – it is more probable that a shoe factory is looking for an office clerk than vice versa. Similarly, sometimes it is unclear, whether a segment should be classified as a service offer, or if it is not a job advertisement at all and just advertising one’s business, such as in the short text “Massage” followed by an address.

Another problem of this type was a missing historical context, such as in the advertisement “Madelaine Arztesgattin” [doctor’s wife] followed by the address, as it is not clear without additional research, whether Madelaine is a name, a historical name of a position, or simply a product.

Similar problems appear when annotating entities within the text. Not all phrases clearly fit predefined labels: job titles could be literal descriptions of persons, such as “Junger Mann” [young man] , “Französinnen” [French women] , or “Mädchen” [maid] , requiring the addition of a new label for these “not-positions.”

Through iterative refinement of annotation guidelines and continuous discussion among annotators, the team developed strategies to handle such ambiguities, increasing consistency and transparency. While many cases required interpretation based on the specific historical and textual context of each advertisement, recurring patterns were incorporated into the evolving annotation guidelines. This allowed the team to balance context-sensitive judgment with consistent rule-based decisions across the dataset. These examples illustrate how seemingly small annotation decisions can have significant downstream effects on model behaviour.

The resulting dataset not only enabled the training of reliable NER models but also provided methodological insights into the complexities of annotating historical texts, emphasizing the human judgment that underpins automated analysis. Based on these rigorous and methodological guidelines the resulting training data allowed the creation of an NER model that achieved a F1 score of 0.95 on a held out test dataset with a human-corrected text, and a F1 score of 0.83 on a completely unrelated dataset of German newspapers with a messy OCR (Adam et al., 2025).