Introduction

The use of Named Entity Recognition (NER) models on historical French-language documents remains rare, particularly for notarial records, which are often inaccessible due to the limited accuracy of OCR and Handwritten Text Recognition (HTR) models. Even corpus-specific HTR models maintain a best-case character error rate of around 2 % (Romein et al., 2025), necessitating substantial manual corrections. Many projects rely on volunteer work, such as the Amsterdam City Archives Crowd leert computer lezenhttps://www.amsterdam.nl/stadsarchief/organisatie/crowdsourcing/crowd-leert-computer-lezen/, the Dutch VeleHanden projecthttps://velehanden.nl/, and Canada’s Nouvelle-France numériquehttps://nouvellefrancenumerique.info/ (Gohier, 2022). Current HTR systems struggle with irregular handwriting and document layouts, slowing initiatives on French state archival collections. For example, the citizen science project Familles parisiennes digitises selected documents without HTR, while the French National Archives Lectaurephttps://lectaurep.hypotheses.org/ project uses eScriptoriumhttps://escriptorium.rich.ru.nl/ to transcribe numerous early 20th-century notary registers (Chagué and Rostaing, 2021). Similarly, the NER4archives projecthttps://www.archives-nationales.culture.gouv.fr/innovation-et-recherche/ner4archives-reconnaissance-dentites-nommees-pour-les-services-darchives (Charbonnier et al., 2021) has advanced slowly.

NER research in historical studies remains limited and mainly focuses on digital editions (Görmar, 2024; Hansen, 2023). Historical texts pose several challenges: archaic orthography, multiple variants of the same names, disappeared or renamed institutions and locations, and entity types (e.g., noble titles) absent from modern NER training. Nonetheless, current technology allows the processing of large multilingual corpora, as demonstrated by the Odeuropea project (Menini et al., 2022). The use of a clean, typed corpus circumvents OCR/HTR issues and enables systematic evaluation of NER models. Named Entity Recognition thus represents a valuable tool for constructing a prosopography of 17th-century French life.

In this study, we evaluate spaCy small, medium, and large models for NER across two versions: 2.3.0https://github.com/explosion/spacy-models/releases/tag/fr_core_news_sm-2.3.0 and 3.8.0https://github.com/explosion/spacy-models/releases/tag/fr_core_news_sm-3.8.0. We describe the dataset [2], present the evaluation methodology and results [3], and discuss the findings alongside future research directions [4].

The Insinuations du Châtelet

The dataset is based on a detailed inventory of « insinuations », i.e., registrations of notary acts, from the Châtelet of Paris between 1539 and 1682. Following a 1539 decree by King François I, Parisians were required to record property transfers—such as wedding contracts, testaments, and donations—at the Châtelet, a practice that lasted until its abolition in 1791. These records provide rich data on social structures, networks, professions, and relations with high-status individuals. The 19th-century manuscript inventory was later digitized by the French National Archive, resulting in 89,987 records covering over 180,000 individuals. The data of the Insinuations and of our experiments is available on this GitHub : https://github.com/SimonDagenais/insinuations.

Reverse Engineering SpaCy Models

For a historical project, we had the advantage of working with clean data, as it had been manually transcribed. We wondered whether Named Entity Recognition (NER) models would perform well on this type of data, so we decided to test spaCy models. Initially, we were quite disappointed, as many names of people and places were not recognized correctly. (Koudoro-Parfait, 2025) shows that different versions of spaCy models perform very differently, and, surprisingly, older versions often outperform newer ones. We reuse the method and programs made available by (Koudoro-Parfait, 2025)https://github.com/These-SCAI2023/EXPE42_EVALUATION_ELTeC-fra_09072024/tree/main, and we believe it is essential to highlight the importance of the reproducibility and reusability of such work for the digital humanities community. Based on this experience, we annotated 36,000 tokens to create a ground truth and evaluated two versions of spaCy. Two annotators carried out the annotation process. We adopted the spaCy entity categories LOC, PER, ORG, and MISC, deciding that people’s occupations should be annotated as MISC and that Shop signs should be annotated as LOC, as they are mostly used as addresses. At the end of the annotation phase, we obtained a Cohen’s kappa score of 0.83, indicating a good level of inter-annotator agreement and ensuring a high-quality ground truth.

Table 1: Comparison of entity-type annotations across different versions of spaCy. GT = Ground Truth.

In order to evaluate the outputs of the Named Entity Recognition (NER) system, we employed two simple evaluation methods. First, we counted the number of entities present in the gold standard and in each system output. The results are reported in Table 1. We observe that, in the Person [PER] category, the spaCy 2.3 model fails to identify approximately 25 % of the entities, whereas the spaCy 3.8 model misses nearly 70 % of them. It also appears that the MISC category is the most problematic in our case with respect to the gold standard. Indeed, across all configurations, spaCy models do not annotate professions as entities, whereas they were annotated as such by human annotators. Furthermore, the spaCy 3.8 models annotate the term "Notice" as a MISC entity, which is not the case for the 2.3 versions. As a result, the latter models exhibit a particularly large number of missing entities in this category.

The second evaluation method consisted in computing the cosine distance and the Character Error Rate (CER) between the gold standard and the annotations produced by the spaCy models in versions 2.3 and 3.8. To interpret the cosine distance and CER values presented in Table 2, it should be noted that values closer to 0 indicate better performance, whereas values closer to 1 reflect poorer results. Overall, both metrics consistently show that the spaCy 2.3 models outperform those of version 3.8. Interestingly, the medium (md) model, which achieves the best cosine similarity results in spaCy version 2.3, also yields the poorest results in version 3.8. The large (lg) model of spaCy version 3.8 achieves the best results within that version, but they still remain inferior to those obtained with version 2.3.

Table 2: Results of spaCy models according to the cosine metric and the CER.

Discussion and Conclusion

It is often assumed that newer language models systematically outperform older ones. In this study, we challenge this assumption through experiments on 16th- and 17th-century French notarial archives. Our results show that the older spaCy 2.3.0 models frequently outperform the more recent 3.8.0 models, a trend also observed in prior work on 19th-century French novels. This suggests that model recency does not guarantee superior performance, particularly for historical texts where orthography, spelling variants, and domain-specific entities differ from contemporary language corpora.

The limited improvement observed in the 3.8.0 models may be linked to the inclusion of spaCy lookups data, which does not appear to enhance Named Entity Recognition (NER) for historical French sources. These findings highlight the importance of selecting NLP models based on corpus characteristics, text genre, and historical period rather than relying solely on version updates.

By carefully evaluating model performance on domain-specific corpora, researchers in digital humanities can optimize the extraction of entities from historical documents. Future work will extend this analysis to other languages, historical periods, and archival collections, providing a more comprehensive understanding of how modern NLP tools perform on premodern texts and enabling more effective applications in prosopographical research and historical social network analysis.