Daejeon, July 27–31
Digital collections at scale rely on Optical Character Recognition (OCR) to convert scanned texts into machine-readable data. For researchers working with digitized texts, OCR quality can be decisive. OCR noise corrupts semantic features [1], distorts named entities [2], and alters or masks stylistic features [3], limiting the reliability of downstream scholarship.
OCR quality varies, however, as technology evolves. Large repositories like HathiTrust continue to release newly OCR’d texts that promise to be of better quality. Understanding the changes in quality is essential for recognizing the limitations they impose on downstream data use, both for training data and for subsequent analyses.
We undertake this study in order to assess how often, and to what extent, HathiTrust’s collections are re-OCR’d. We pilot methods for aligning multiple OCR generations of the same volume-level sentence, quantify character- and word-level changes, and test whether there are changes in OCR quality that meaningfully affect downstream tasks.
Our study considers how HathiTrust Digital Library (HTDL) OCR quality has changed over time. Jiang et al. [4] provide a prototype for aligning gold-standard, hand-keyed Project Gutenberg (PG) books to computer-generated HTDL texts at scale, which we leverage, along with their initial study of HTDL OCR [5], for our study to quantify if HTDL OCR accuracy has changed as models have become more sophisticated.
Many studies on OCR quality in digital libraries produce aligned datasets of OCR to facilitate subsequent downstream or post-correction tasks [6, 7, 8]. As other studies, we consider how OCR quality impacts downstream tasks, situating our study with those that assess how OCR impacts out-of-the-box tools [9], downstream BERT models [5], Named Entity Recognition/Linking [2], and Information Retrieval [10]. Generally, downstream task success drops significantly when character error rate (CER) rises above 5%-7%, a finding we test further with increasing tool complexity and capability.
For this preliminary study, our dataset is limited to sentences from 346 volumes aligned between hand-keyed PG and digitized HTDL sentences. This relatively small size allows a rich examination of changes in OCR quality and focused effort on accurate sentence alignment. We hope to use this focus on a smaller subset of truly big data in order to better distinguish between changes that are due to alignment artifacts, such as the omission of a trailing semicolon, and more substantive improvements that indicate volumes have been re-OCR’d, such as where whole words are restored.
Aligning texts is itself a significant challenge that we partially alleviate by starting from a corpus of 167,079 aligned HTDL-PG sentence pairs. These sentences were computationally aligned between 19,049 HTDL volumes and their PG surrogates using a text extraction pipeline grounded in substantial manual review of the texts and their structure [4]. We use the original aligned sentences and the HTDL candidate pool for finding updated sentences using an iterative process detailed in Figure 1.
Fig 1. Complete text alignment workflow
Using original HTDL target sentences, then PG target sentences yielded 102,495 matched sentences. We then use Levenshtein distance similarity to find a further 56,925 matches, for a final aligned corpus of 159,420 sentences.
We assessed text quality through Word Error Rate (WER) and Character Error Rate (CER), evaluating current OCR text against the PG text. To ensure that our calculations are accurate, we replicate Jiang et al.’s previous textual normalization and calculate differences between our WER/CER and theirs to determine which sentences had been re-OCR’d. We also append metadata from HathiTrust MARC records to map when a HathiTrust record has been updated [12].
We set a CER/WER threshold of 1% to minimize anomalies in text normalization, and preliminary results indicate that of the 346 volumes, 96 saw the majority of sentences improve CER, and 43 had an improved WER. Of those, 81 books have an improved CER and have been updated since the original study in 2022. 36 books have an improved WER and have been updated since 2022. This indicates an increased effort on the part of Google and partner libraries to re-OCR documents (see Figure 2). However, the sentence-by-sentence median CER / WER did not significantly improve across all books, with only 15 / 346 (4.3%) showing a WER line-by-line mean delta less than -4%.
Fig 2. Book-level CER / WER changes by update year. Mean CER / WER Δ calculated by subtracting original CER / WER from current WER. Negative correlates to an improved CER / WER. Black dots represent the last updated date of a book.
To assess how OCR quality can affect analysis using digitized texts, we examine two NLP tasks using our test data: feature extraction and Named Entity Recognition (NER).
We use the HTRC Extracted Features Dataset (EF), a bag-of-words text dataset extracted from raw HTDL text using industry-standard NLP tools, to explore effects on feature extraction. We specifically examine token part-of-speech (POS) tags to explore OCR influence, comparing the EF 2.0 [13] and EF 2.5 [14], released 5 years apart. Comparing the number of hapax legomenon tokens–words only used once in the volume–and tokens with “unknown” (tagged “UNK”) as their part-of-speech tag serve as benchmarks for token-level OCR impact.
The prevalence of hapax legomenon tokens reduces from EF 2.5 to EF 2.0 (see Figure 3).
Fig 3. Shows the change in the overall fraction of hapax legomenon tokens across the 346 volumes analyzed. The left panel presents the distribution of changes, highlighting the magnitude of decreases. The right panel visualizes each volume individually; points farther to the right of the red reference line indicate larger reductions in hapax tokens
We also looked at the prevalence of the “UNK” POS tag. We find reduction in extreme prevalence (from a maximum of 4.13% of part-of-speech tags in a text to 3.34%), and find that many–though not all–of the “worst offenders” in EF 2.0 have improved (Figure 4):
Fig 4. Compares the top 15 HTIDs with the highest fraction of 'UNK' POS tokens in EF 2.0 and maps their change in EF 2.5.
For evaluating downstream effects on NER, we benchmark an established, transformer-powered method implemented with spaCy and a cutting-edge LLM-based pipeline with GPT-4o. A 1000-sentence sample yielded very different results from each NER pipeline, with no exact matching results for any passage. Single-token entities extracted with spaCy showed a higher level of accuracy than with GPT-4o, which instead excelled at multi-token entities that spaCy often missed. These early results signal that even advanced LLMs still struggle when sentence-level data is messy.
Our results show that text processing is vital to string matching and OCR evaluation. In emphasizing alignment and normalization, we measure how small changes can end up creating big differences in final data or evaluation. The effects demonstrate how OCR quality is still a challenge for downstream tasks, even when using cutting-edge methods.
If we can extrapolate our small sample findings to the larger HTDL, only about 27.7% books show an improved CER and 12.4% an improved WER. However, due to copyright limitations, we are limited to open-source HTDL and PG texts, and our data currently constitutes only 346 books of the over 19 million in the HTDL. While a useful foundation, this study is in progress and by no means comprehensive.
In future work we will consider which types of texts are being updated, by whom, and against various MARC-level metadata fields such as subjects, document type, published date, and language. Additionally, we hope to quantify why certain texts have not been re-OCR’d.