Daejeon, July 27–31
Abstract
We present an end-to-end pipeline that transforms 2,900 pages of handwritten oceanographic logbooks (1883–1912) into a searchable, citation-aware question answering system. Three OCR/HTR models are combined through context-aware LLM post-correction, reducing Word Error Rate from 49.89% to 32.98%. To support access to this noisy historical corpus, we use a hybrid RAG architecture combining BM25F and multilingual embeddings for retrieval and source-grounded answer generation. Results show improved transcription and retrieval performance, while also highlighting remaining challenges such as residual OCR noise and loss of tabular structure. Beyond this case study, the framework is intended as a reusable method for digitizing and exploring historical scientific archives.
Keywords:
OCR/HTR, LLM-based Post-Correction, Historical Archive Analysis, Hybrid Retrieval-Augmented Generation, Human-in-the-Loop Evaluation
1. Research Context and Objectives
We develop and evaluate a complete artificial intelligence pipeline for transforming fragile historical oceanographic archives into a searchable, question-answering system for domain researchers This study is based on research conducted within the ObTIC team at Sorbonne Université during a Master’s internship in spring 2025, in collaboration with the Observatoire Océanologique de Banyuls-sur-Mer.
The corpus comprises 2,900 pages, including 1,773 pages of meteorological registers and 1,127 pages of ship logbooks. The registers contain structured daily measurements such as atmospheric pressure, wind, air and water temperature, precipitation, salinity, and sea state, while the logbooks provide complementary information including sampling operations, species observations, geographic positions, and depth measurements. The corpus distinguishes between two sources: the Lacaze-Duthiers (marine biology, dredging) and the Roland 2 (navigation, logistics), reflecting distinct vocabularies and data schemas.
Table 1: Overview of the historical document corpus
The documents are digitized as high-resolution JPEG 2000 (.jp2) images, totaling approximately 45GB, with pages in varying formats (A4/A5). The corpus contains mixed layouts, combining tables and free text. It represents a rich but still largely inaccessible handwritten scientific resource. Its long-term observations of sea state, temperature, and marine life make it a strong candidate for digitization and computational exploration. This effort also aligns with global data rescue initiatives (e.g., ACRE, Old Weather) and international infrastructures such as ICOADS https://www.ncei.noaa.gov/products/international-comprehensive-ocean-atmosphere-data-set
Our work makes three main contributions: (1) a multi-model OCR pipeline with LLM-based post-correction, (2) a hybrid RAG system tailored to noisy historical scientific texts, and (3) a structured evaluation framework combining quantitative metrics and expert assessment.
2. Positioning within AI and Digital Humanities
The project is situated at the intersection of AI, digital humanities, and historical ecology (De Boer & Stork, 2024). It starts from the observation that long-term environmental questions—such as changes in marine ecosystems or coastal climate patterns—require access to observations spanning decades or centuries, which are often preserved only in archival form.
However, these archives pose several obstacles: degraded paper quality, variable and archaic handwriting styles, and domain-specific vocabulary that standard OCR and Handwritten Text Recognition (HTR) systems handle poorly. The central technical challenge is therefore to extract text that is sufficiently accurate for downstream tasks such as semantic search (Alrahabi et Desclés, 2009, Alrahabi, 2021) and question answering, while preserving the documentary context needed by experts who must trust and verify the outputs (Duranti & Rogers, 2024).
Recent advances in historical document processing combine specialized HTR systems such as Kraken with vision-language models like Qwen-VL and Mistral OCR. However, a significant gap persists between benchmark performance and real-world conditions, where average accuracy remains around 66%. Traditional post-OCR correction methods, including dictionary-based and language model approaches, are insufficient for handling contextual errors such as misinterpreted dates or proper nouns.
In parallel, Retrieval-Augmented Generation systems typically distinguish between dense (embedding-based) and sparse (keyword-based) retrieval. Despite these advances, there remains a lack of fully integrated pipelines tailored to historical scientific logs, particularly approaches that exploit temporal continuity for OCR post-correction.
3. Multi-Model OCR and Corpus Construction
To make these handwritten archives usable at scale, we introduce a multi-stage pipeline that progressively improves transcription quality. Instead of relying on a single engine, we adopt a multi-model strategy combining three complementary tools: Kraken with the McCATMuS model (Chagué, 2024), Mistral OCR, and Qwen2.5-VL-32B (Bai et al., 2025).
Each page is processed by three engines, producing complementary transcription hypotheses. To balance performance and cost, we implement an adaptive sampling strategy: commercial Mistral OCR is invoked only on complex pages where Kraken and Qwen2.5-VL-32B disagree substantially.
The system is implemented as a modular Python pipeline composed of dedicated modules for OCR processing, data preparation, retrieval, generation, evaluation, and user interaction. The entire workflow is driven by YAML configuration files, enabling reproducibility and rapid experimentation without modifying code.
Several engineering optimizations further improve efficiency, including adaptive sampling of expensive OCR models, batch processing, and caching of intermediate results. The retrieval component leverages optimized vector indexing using HNSW for efficient similarity search.
Table 2: OCR engine characteristics and adaptations
4. Context-Aware LLM-Based Post-Correction
A key methodological contribution is the design of a context-aware, LLM-based post-correction algorithm that fuses these multiple OCR outputs into a single improved transcription (Thomas et al., 2024; Kanerva et al., 2025).
We observe that many errors in historical documents are not purely local but contextual: misinterpreted dates, misaligned entries, or inconsistent naming across consecutive pages. To address this, the correction step uses a sliding window over the document sequence: for each page, the algorithm takes into account the three previous and three next pages.
For each page, a large language model is prompted to act as an expert archivist and receives not only the three OCR outputs for the current page but also the transcriptions of the pages in this window. This temporal and narrative context enables the model to exploit redundancies and continuity in the logbooks when deciding which tokens to correct and which variants to prefer. However, this context-based correction may also introduce silent orthographic normalization and coherence-driven hallucinations, as the LLM may regularize archaic spellings, abbreviations, or numerically variable entries into more plausible modern or internally consistent forms, potentially erasing historically meaningful variation.
5. Quantitative Evaluation of OCR and Post-Correction
The result of this post-correction stage is a unified, authoritative text for each page that is substantially cleaner than any single engine’s output. Quantitative evaluation is performed using a carefully constructed ground truth dataset representing approximately 5% of the corpus (around 160 pages), manually transcribed by experts at OOB.
Using this gold standard, we compute Character Error Rate (CER) and Word Error Rate (WER) for all three individual OCR engines, and then for the post-correction method (Levchenko, 2025). The results show that Qwen2.5-VL-32B significantly outperforms Kraken and Mistral OCR, and that the proposed post-correction method further reduces both CER and WER. This result suggests an advantage of vision-language models over the standalone HTR baseline on this corpus, likely because visual transformers better capture long-range layout dependencies and leverage internal linguistic priors to resolve partially illegible tokens. In contrast, Kraken’s very high error rate points to a mismatch between the McCATMuS model and the specific handwriting and tabular conventions of the OOB registers.
This improvement at the word-level is particularly important because it directly affects how well terms can be retrieved and how accurately sentences can be interpreted in subsequent retrieval and generation stages.
Table 3: OCR quality comparison (Word Error Rate & Character Error Rate)
The difficulty of the corpus is evidenced by a Word Error Rate (WER) of 96.12% for the Kraken baseline. While Qwen2.5-VL-32B lowers this to 49.89%, our post-correction pipeline achieves a final WER of 32.98%, representing a relative improvement of 34% over the best single model. These results indicate that our pipeline produces transcriptions of sufficient quality to support downstream semantic search and question answering.
6. Document Structuring and Hybrid Retrieval
To enable reliable search in this noisy historical corpus, we design a tailored RAG pipeline combining complementary retrieval strategies (Kelly et al., 2025). The documents are naturally ordered in time and structured like logbooks, which motivates a page-level chunking strategy: each page is treated as a single chunk, preserving its internal coherence and metadata such as book identifier, date, and page number. However, page-level chunking may be too coarse for dense logbooks, as it can blend multiple distinct events into a single representation, potentially diluting semantic specificity. Finer-grained strategies, such as semantic or sliding-window chunking at the line or entry level, could improve retrieval precision.
For semantic indexing, each chunk is embedded using the paraphrase-multilingual-mpnet-base-v2 model, chosen for its strong performance on multilingual and cross-lingual tasks, which is essential because the material involves French with potentially mixed terminology. The resulting embeddings are stored in a Chroma vector database, where they are associated with rich metadata that later supports filtering, contextualization, and citation.
To ensure both high recall and precision, the pipeline does not rely exclusively on dense embeddings. Instead, it implements a hybrid retrieval mechanism that combines semantic and lexical search. On the semantic side, Chroma provides nearest-neighbor search over the embedding space. On the lexical side, the system uses Whoosh with a BM25F ranking function.
We index each page in Whoosh with BM25F over the fields transcription (w = 1.0), date (w = 2.0), and metadata (book_id, vessel, location; w = 0.5), with the final score computed as a weighted sum across fields. When a user submits a query, the system retrieves candidates independently from the vector database and from the BM25F index, pools the results, and then re-ranks them by embedding similarity to produce the final top-k set of relevant pages. In this re-ranking stage, BM25F scores are not combined numerically with embedding scores: they serve only to define the candidate pool, while the final ranking score is determined solely by embedding similarity over all pooled candidates.
This hybrid strategy helps mitigate weaknesses of either approach on its own, such as missing relevant pages due to lexical variation or retrieving semantically similar but contextually irrelevant passages (Lewis et al., 2020; Wang et al., 2021). In our setting, it also serves as a partial safeguard against OCR noise, since exact lexical matching and semantic similarity may recover complementary evidence when one signal is degraded; however, this mitigation remains imperfect and does not rule out downstream misreadings.
7. Answer Generation and Prompt Design
The answer generation phase builds on the retrieved set of documents. We use targeted prompt design to align the LLM with the expectations of historical and scientific research, while keeping the overall system robust to model evolution.
At the orchestration level, the generation system is wrapped in a fallback gateway that can route requests either to an external model (e.g., via OpenRouter) or to a local LLM, ensuring robustness and adaptability when the primary model fails or reaches usage limits.
The prompt instructs the model to produce formal academic French, provide explicit source citations, and acknowledge uncertainty when evidence is missing. The prompt also enforces explicit citation and honest limitation: when data is missing, ambiguous, or not present in the corpus, the model is guided to state this explicitly rather than generate unsupported content. This design choice addresses a crucial trust issue when deploying generative models in scholarly environments (Sahoo et al., 2024).
8. User Interface and Expert Interaction
To make the system accessible to non-technical users, the project includes the development of a web application that serves as the main user interface. Through this application, researchers can pose questions in natural language, inspect the passages retrieved by the system, and view the corresponding scans or transcriptions.
The web application is implemented in Streamlit and provides the main interface for querying the corpus, inspecting retrieved passages, and accessing the corresponding scans or transcriptions See the repository: https://github.com/obtic-sorbonne/RAG_Banyuls/tree/main
The interface enables users to inspect retrieved passages and provide feedback signals for system refinement. This interface should be understood as an exploratory tool rather than a source of ground truth, as expert input remains essential to resolve ambiguities that automated methods cannot handle, such as cryptic handwritten abbreviations or highly localized references specific to the Banyuls context.
9. Evaluation of the End-to-End RAG System
The evaluation follows a multi-stage framework that assesses both intermediate outputs, namely OCR quality, and final system performance in RAG-based question answering. The end-to-end RAG system is evaluated through expert human assessment, complementing the earlier quantitative OCR metrics. This approach aligns with 'Humanities-in-the-loop' frameworks, where AI-assisted close reading ensures historical fidelity and captures the contextual complexity inherent in archival records (Zhou et al., 2025).
Domain specialists evaluate answers using a structured evaluation grid with a 0–5 scale covering accuracy, completeness, relevance, and hallucination detection. All evaluation materials, including queries, expert annotations, scores, and hallucination flags, are stored in a structured SQL database, creating a reusable evaluation framework that supports reproducibility, longitudinal comparison, and future system iteration.
We report initial pilot testing and outline a plan for more extensive analysis, focusing on how well the system supports actual research questions in historical ecology and how its performance varies across different query types (e.g., numeric questions, qualitative descriptions, temporal comparisons).
We acknowledge that the present RAG evaluation remains primarily qualitative, without incorporating standard retrieval metrics such as Recall@k or MRR over expert-annotated queries.
10. Contributions, Limitations, and Future Directions
The outcomes of the project are both practical and methodological. Practically, the pipeline delivers a corrected digital corpus of the OOB archives and a working hybrid RAG system that can be deployed as a research tool. Methodologically, it provides evidence that an LLM-based post-OCR correction strategy can meaningfully improve transcription quality on difficult historical material and that a carefully engineered RAG architecture can bridge the gap between noisy archival data and the expectations of scholarly inquiry.
Taken together, these results suggest that modern AI methods can be adapted, constrained, and evaluated in ways that respect the specificities of cultural and scientific heritage collections rather than treating them as generic text.
A major limitation remains the reliable extraction of numerical time series. Although the final WER marks substantial progress, it remains too high for scientific use of individual numerical values, where minor digit errors can compromise analyses (Zhang et al., 2024). This issue is compounded by the absence of explicit document layout analysis (DLA), which linearizes tabular data and may weaken column-wise associations between variables such as time, pressure, and temperature.
More broadly, we frame our contribution as a reproducible blueprint for other institutions confronting similar digitization challenges, stressing its broader applicability. To encourage reuse and open evaluation, both the pipeline and the user-facing application are publicly available.
In sum, the OOB pipeline provides a promising foundation but still requires greater technical maturity: explicit document layout analysis, standardized RAG evaluation metrics, and updated embedding and retrieval strategies aligned with the state of the art are necessary to ensure reliability for climate-oriented scientific use. Future work should include fine-grained error analyses of numeric values, technical terms, and named entities, given their importance for retrieval and downstream interpretation. It should also incorporate standard retrieval metrics such as Recall@k, MRR, and NDCG, alongside citation-faithfulness checks to verify that generated claims are both supported by and correctly attributed to the retrieved pages. A key next step is a robustness evaluation comparing expert judgments on answers generated from gold-transcribed pages versus OCR-derived pages, in order to assess whether residual OCR errors lead primarily to minor omissions or to substantive historical mischaracterizations. If successful, this framework could serve as a model for many other scientific archives worldwide awaiting large-scale AI-enabled data rescue.
The pipeline is designed as a modular and reproducible framework adaptable to other historical scientific archives. Future directions include integration with temporal databases, combination with structured data (e.g., text-to-SQL over digitized tables), and adaptation to other languages and scientific domains. Another promising direction is native multimodal RAG based on visual document retrieval models such as ColPali (Faysse et al., 2025), which directly address the loss of tabular structure induced by OCR linearization by indexing page images instead of transcriptions. By preserving layout, tables, and spatial cues, such approaches could be especially valuable for the OOB registers, where tabular organization is central to interpretation.
Acknowledgements:
We warmly thank the Banyuls-sur-Mer team for their support, for providing the corpus, for their expert feedback, and for their valuable assistance in the evaluation: Michel GROC, Sandrine Bodin, Nicolas Desreumaux, Rémi Gaillard, Adeline Batailler, and Mylène Lorre.