Daejeon, July 27–31
This paper addresses the technical challenges of applying handwritten text recognition (HTR) to demographic data preserved in late Ottoman census registers. Two issues are central. First, the registers are handwritten in Ottoman Turkish, an under-resourced historical language for which effective pre-trained HTR models remain scarce. Second, the data consists of handwritten entries embedded in tabular forms, many with empty cells and others containing multi-line content that challenges standard layout assumptions (fig. 1). Existing table-segmentation methods perform poorly on such material, resulting in unreliable region detection and reduced transcription accuracy.
Since the 1990s, scholars have systematically extracted demographic and socio-economic data from surviving Ottoman census registers. Foundational studies include Duben and Behar’s analysis of Muslim and Armenian populations in Istanbul (1991), Cuno and Reimer’s work on the 1848 and 1868 Egyptian censuses (1997), and Alleaume and Fargues’s examination of nineteenth-century Egyptian census practices (1998). Later research broadened both scope and method: Saleh (2013) digitized late nineteenth-century Egyptian censuses, while Buessow (2011) reconstructed social characteristics of Palestinian regions. Sarah and Johann Buessow (2020) made largely invisible population groups visible through close reading of census registers. Buessow et.al. created a database of census materials from late Ottoman Gaza (2017-2023, 2024), and Campos (2018) analyzed social structures in Jerusalem using computed census data. The UrbanOccupationsOETR project extracted and published data on urban occupation from nineteenth-century Ottoman pre-census registers Kabadayı and Erünal (2024). All these datasets were produced through manual transcription, limiting scalability and reuse.
The present paper is part of the LOOP project (https://ercloop.hypotheses.org/loop), which aims to extract the complete corpus of late Ottoman census records of Palestine (467 volumes, 80,000+ pages, Berg and Büssow 2023) into a structured database enabling analyses from the micro to the macro level. These census records constitute a key part of the documentary heritage of the Eastern Mediterranean, underscoring the importance of sustainable digital access and processing pipelines.
In the past decade, HTR has advanced considerably, with two major platforms widely used in scholarly contexts: Transkribus, powered by the PyLaia engine, and the open-source platform eScriptorium, based on Kraken. Recent years have seen growing efforts to integrate HTR into Ottoman studies. Köse (n.d.) initiated the Ottoman Text Recognition Network in 2021 focusing on the creation of recognition models for the Transkribus platform. Kirmizialtın (2023) and the Digital Ottoman Corpora project published the first public generalized text recognition model for 19th-century Ottoman Turkish periodicals on Transkribus. The 2016 founded Open Islamicate Initiative develops HTR models for Arabic script on the eScriptorium platform. As base model for our pipeline, we used Smith et.al (2023) model created in the Automatic Collation for Diversifying Corpora project.
Typical HTR workflows consist of two separately trainable steps. First, document segmentation identifies regions, lines, and masks that represent the page layout. Second, text recognition converts the handwritten content within these masks into machine-readable text. Our census materials pose difficulties at both stages.
The population registers from late Ottoman Palestine (c. 1880–1917) are handwritten within printed tabular forms. Many cells are empty, while others contain multiple lines of text, often skewed or overflowing cell boundaries. Repeated information, such as religious affiliation, is frequently indicated using ibid or ditto signs. Maintaining the tabular structure is therefore crucial for meaningful data extraction. Although Transkribus offers advanced table-detection features, these fail in our case due to the high proportion of empty cells. Existing algorithms typically infer the logical and physical table-structure by detecting text-blocks, an approach ill-suited to these registers.
HTR model availability and quality vary greatly by language. Ottoman Turkish, a predecessor of modern Turkish, is written in Arabic script and fell out of use following the language reform of 1932. The census registers employ Rika, a highly stylized variant of Arabic script. Recognition is further complicated by the multilingual nature of the data: many personal and place names derive from Arabic, Hebrew, Armenian, Greek, and other languages used in late Ottoman Palestine. At the start of our project in 2023, no public training sets existed for this combination of historical script and multilingual content. Our attempt to reproduce LLM-based table-recognition methods (Kim et. al. 2025) failed because current models lack proficiency in Ottoman Turkish.
To address these challenges, we rethought segmentation entirely. Instead of attempting table recognition, we trained the segmentation model to draw a single line spanning the full width of each table row. During text recognition, vertical column boundaries are allowed to be recognized as pipe characters. In post-processing, the table is reconstructed from the text lines with the pipe character as a separator, as if from a CSV file. This approach achieved an overall text recognition rate of 94%, with an error rate of under 2% for years of birth. A further challenge emerged in segmentation: while line placement can be trained, the associated masks that define the text recognition area cannot. Due to empty cells and multi-line entries, these masks are often positioned too low. To address this, we export the segmented files and apply a custom Python script that generates new masks aligned to the full height of each table row. The corrected files are then re-imported for text recognition.
By foregrounding reuse, modular adaptation, and user-driven intervention within existing HTR infrastructures, this paper contributes a sustainable approach to processing under-resourced, trans-lingual documentary heritage, with relevance for similar materials preserved in large-scale archival collections.