Abstract

1. Research Background and Problem Statement

The 18th century marked a critical turning point in the history of the Russian Empire, during which the Russian language (17th–18th century Russian) was in a dynamic transition from Old Church Slavonic influences toward modern standard Russian. Texts from this period differ significantly from contemporary Russian in orthography, lexical semantics, syntactic structure, and stylistic conventions, presenting substantial challenges for historians and philologists. Recent advances in Large Language Models (LLMs) have enabled their increasing use in translating and interpreting historical documents. However, the scholarly community still lacks systematic, evidence-based evaluation regarding LLM performance on non-standardized and pre-modern linguistic varieties. Existing studies have largely focused on translation between modern languages, thereby overlooking the contextual discontinuities and semantic shifts that are characteristic of historical language varieties.

Against this backdrop, a key methodological question arises: how should one select an appropriate type of model to effectively evaluate its capability in processing historical language? This study focuses on general-purpose LLMs rather than specialized machine translation systems. This choice is grounded in the following considerations. First, 17th–18th century Russian constitutes a low-resource historical language variety, for which digitized parallel corpora remain scarce and insufficient to support the data requirements for training or fine-tuning dedicated translation models. Second, constructing such models from scratch would entail substantial computational costs, rendering this approach impractical for most research teams in the humanities. By contrast, contemporary general-purpose LLMs—such as GPT, Claude, and Gemini—have been pre-trained on large-scale multilingual corpora and have demonstrated a certain degree of cross-temporal linguistic generalization. These models can be directly applied to downstream tasks through prompt engineering, without the need for additional training data.

Based on the above research context and methodological considerations, the primary objective of this study is to construct a comprehensive evaluation framework to determine whether LLMs can reliably translate such texts. By integrating Cross-Model Consistency and Expert Annotation, this study seeks to address two key questions: Do contemporary state-of-the-art models (e.g., GPT 5.5, Claude 4.7, Gemini 3) produce semantically consistent outputs when interpreting 17th–18th century Russian? More importantly, does such consistency correspond to actual translation accuracy?

2. Data and Methodology

This study adopts a mixed-methods approach, combining computational linguistics with traditional historical textual analysis.

Corpus selection focuses on original Russian documents from the 17th–18th centuries—from the reforms of Peter the Great to the period surrounding the Treaty of Kyakhta (1727). The corpus includes diplomatic correspondence and administrative decrees, e.g., Полное собрание законов Российской империи and Акты исторические, to ensure representativeness. A total of 20–30 texts will be selected and segmented into 80–150 semantically coherent units as the basic analytical units.

The experimental design consists of three stages:

• Model Translation and Vectorization:
  Cutting-edge models such as GPT-5.5, Claude 4.7, and Gemini 3 will generate translations under a unified few-shot prompting scheme. The resulting translations will be embedded using high-dimensional vector models.Cutting-edge models such as GPT-5.5, Claude 4.7, and Gemini 3 will generate translations under a unified few-shot prompting scheme. The resulting translations will be embedded using high-dimensional vector models.
• Consistency Measurement and Stratified Sampling:
  Cosine similarity is used to compute cross-model consistency scores. A methodological innovation of this study is the use of Cosine similarity is used to compute cross-model consistency scores. A methodological innovation of this study is the use of consistency-based stratified sampling instead of random sampling. Samples are grouped into high consistency, low consistency, and median consistency categories, from which 40–60 key segments are selected for deeper analysis. This design allows us to evaluate both “collective consensus” and “individual divergence” among models.
• Expert Blind Review and Annotation:
  Specialists in Russian history will evaluate selected segments according to multiple criteria—fidelity, fluency, terminological accuracy, and historical appropriateness—and will produce Gold Standard translations as reference outputs.Specialists in Russian history will evaluate selected segments according to multiple criteria—fidelity, fluency, terminological accuracy, and historical appropriateness—and will produce Gold Standard translations as reference outputs.

3. Core Analytical Framework: A Four-Quadrant Matrix

This research proposes an innovative Four-Quadrant Analytical Matrix that cross-tabulates Cross-Model Consistency with Expert Evaluations, revealing deeper mechanisms underlying LLM translation behavior:

• Quadrant I (High Consistency / High Accuracy):
  Indicates that models demonstrate stable interpretive capacity for certain types of text (e.g., straightforward narrative sentences).Indicates that models demonstrate stable interpretive capacity for certain types of text (e.g., straightforward narrative sentences).
• Quadrant II (High Consistency / Low Accuracy):
  Represents the most concerning region: Shared Hallucination. Models may unanimously produce confident yet incorrect translations, often due to shared biases in training data or collective misinterpretations of rare historical terms.Represents the most concerning region: Shared Hallucination. Models may unanimously produce confident yet incorrect translations, often due to shared biases in training data or collective misinterpretations of rare historical terms.
• Quadrant III (Low Consistency / High Accuracy):
  Reflects cases where a single model outperforms others, offering insights into architectural or training advantages for complex textual structures.Reflects cases where a single model outperforms others, offering insights into architectural or training advantages for complex textual structures.
• Quadrant IV (Low Consistency / Low Accuracy):
  Marks the current limits of LLM capabilities, typically associated with damaged manuscripts or highly obscure phrasing.Marks the current limits of LLM capabilities, typically associated with damaged manuscripts or highly obscure phrasing.

4. Expected Contributions

The study expects to contribute at two levels:

• Evaluating Cross-Model Semantic Consistency:
  Through the systematic computation of cross-model consistency scores, this study empirically examines the degree of semantic convergence and divergence among translations produced by leading LLMs when processing 17th–18th century Russian texts. The findings will demonstrate that while automated metrics such as cosine similarity can quantify inter-model agreement, they are insufficient as definitive indicators of translation quality, thereby establishing a more rigorous methodological foundation for evaluating machine translation of historical texts.Through the systematic computation of cross-model consistency scores, this study empirically examines the degree of semantic convergence and divergence among translations produced by leading LLMs when processing 17th–18th century Russian texts. The findings will demonstrate that while automated metrics such as cosine similarity can quantify inter-model agreement, they are insufficient as definitive indicators of translation quality, thereby establishing a more rigorous methodological foundation for evaluating machine translation of historical texts.
• Assessing the Consistency–Accuracy Relationship:
  By means of the four-quadrant analytical matrix, this study will reveal the latent risk of “shared hallucination” underlying high consistency—that is, instances in which models converge on translations that are, in fact, erroneous. The study will empirically identify the specific conditions under which archaic orthography and non-standard syntactic structures of 17th–18th century Russian induce highly consistent yet inaccurate translations, thereby delineating the gap between inter-model agreement and actual translational accuracy and providing users with critical diagnostic benchmarks.By means of the four-quadrant analytical matrix, this study will reveal the latent risk of “shared hallucination” underlying high consistency—that is, instances in which models converge on translations that are, in fact, erroneous. The study will empirically identify the specific conditions under which archaic orthography and non-standard syntactic structures of 17th–18th century Russian induce highly consistent yet inaccurate translations, thereby delineating the gap between inter-model agreement and actual translational accuracy and providing users with critical diagnostic benchmarks.

Although this study takes 17th–18th century Russian as its specific case, the proposed three-stage evaluation framework—integrating cross-model consistency computation, stratified sampling, and expert annotation—is not language-dependent. Future researchers may substitute corpora and annotation criteria to apply the same framework to the translation assessment of other pre-modern historical language varieties.