When transcribing medieval charters, the question of how to handle abbreviations is often quite puzzling. Expanding them can be considered more interpretative and some transcribers often prefer being more diplomatic, transcribing them abbreviated. In the context of the DiDip project (Vogeler et al.) and specifically the monasterium.netcorpus, which is heavily abbreviated, a key outcome of transcribing is the training of HTR models in order to automate the transcription process; to a great extent transcription rules are determined with this in mind. The abbreviation question has begun to gain traction in the technical literature supporting medieval studies and specifically the work by (Clérice et al. 2023) stands out. A major factor to consider is that abbreviations require special characters and can grow the character sets considerably. We argue that this work contributes a systematic, quantitative perspective on abbreviation expansion through the analysis of parallel abbreviated and expanded subcorpora of medieval charters, framing the problem in information-theoretic and machine-learning terms and enabling assessment of transcription fidelity, reversibility, model generalisation, and downstream pipeline design. The Fontenay corpus (Camps et al. 2022) which we will refer to as FTN and new unpublished yet corpus consisting of 33 charters from the archive of the former monastery of Santa Maria della Grotta in Vitulano (BN, Italy), now held at the Library of the Società Napoletana di Storia Patria, dated 1254–1380 and issued across the Kingdom of Naples (Terra di Lavoro and the Principato Ultra), in places ranging from rural villages to major centres. These are private deeds of various legal types, issued by 32 notaries with royal or apostolic authority, often involving the monastery but sometimes only private individuals, we will refer to it as SMG.

Table 1: Information Contained in the corpora

We measured the CER between the two modalities to be 21.6% for FTN and 31.1% for SMG. Quantitative comparison of information contained in the two parallel corpora, as can indicatively be seen in Table 1, provides some insights on the non-structural information contained in the corpora. Hartley information measures the theoretical maximum information per symbol given a character inventory under uniform probability assumptions, whereas Shannon entropy estimates the expected information per symbol based on observed character distributions in the corpus (Information Theory). What stands out is that expanding abbreviations in the SMG adds 14.5% more characters and 6.9% more information, while FNT adds 7% more characters but only 0.1% more information. In Figure 1, we can see the frequencies of characters in each corpus, we observe that the abbreviated texts have thicker long tails, which indicates that many of the abbreviations introduced are quite rare.

We apply a purpose-built character-level neural sequence model, which we call BandedRNN, employing Connectionist Temporal Classification (CTC; Graves et al. 2006) and perform experiments learning to predict expanded text given the abbreviated text. In the case of SMG the CER between the automatically expanded and the manually expanded abbreviations achieved is a 4.2% CER while for FTN a CER of 3.9%. The reverse transformation, predicting abbreviated text given expanded text, performs significantly worse: 10.3% CER in the case of SMG while in the case of FNT it is 16.4%. It is clear that using BandedRNNs for expanding abbreviations is quite good and qualitative inspection of the outputs indicated that many errors can be attributed to unaligned evaluation data and inconsistent letter normalizations such as "u" and "v".These results strongly suggest that abbreviation expansion is, in practice, a largely irreversible transformation under current modeling and normalization constraints. We also trained a model on the union of the subcorpora and obtained a CER of 4.3% which indicates that generic BandedRNNs tend to generalise well and that one BandedRNN model can learn to "code-switch" in order to account for different transcription practices between the two corpora. Due to space constraints, figures and tables are indicative and reporting in more detail on experiments and analysing them, goes beyond the scope of this abstract.

Figure 2: Indicative outputs of the mixed BandedRNN. Insertion errors are marked as yellow, deletions are magenta, and substitution errors are yellow.

These findings indicate that for the monasterium.net corpus, a hyperdiplomatic transcription policy—i.e. one that preserves abbreviated forms while deferring expansion to computational post-processing—would be beneficial as we could reliably infer the expanded text from the abbreviated text. This allows for two different pipelines, either have HTR produce abbreviated text and then expand it with BandedRNN, or have BandedRNN produce groundtruth for training a dedicated HTR model for directly expanding abbreviations. As several monasterium.net users mentioned, even paleographically trained users have trouble expanding some abbreviations when they are unfamiliar with a specific sub-corpus even in the same language. Discussions with them convince us that both modalities are desired by scholars yet very few scholars would be willing to transcribe twice the same document. This leads us to recommend the text-processing pipelines that can use either the abbreviated or the expanded text depending on use-case as can be seen in Figure 3.

Figure 3: Indicative proposed text processing pipelines