Introduction

Oracle Bone Inscriptions (OBI) constitute the earliest systematic form of Chinese writing and are indispensable primary sources for understanding Shang dynasty civilization (Qiu 2015). The task of semantic completion, predicting missing or damaged characters from contextual evidence, is fundamental to the philological study of these texts (Mo / Zhang 2023: 47-56; Wang / Cheng 2025: 121-130). While computational methods have made strides in the physical reconstruction of oracle bones through image restoration (Jiang et al. 2024) or fragment matching (Wen et al. 2025: 1-7), they largely neglect the linguistic dimension of the inscriptions. A restoration that focuses solely on physical structure while ignoring contextual linguistic information may lead to unreliable or meaningless results.

The application of modern NLP techniques to this domain is hindered by three fundamental challenges specific to OBI: (1) Extreme data scarcity, with only about two thousand deciphered characters available (Chen 2019: 156-160); (2) Absence of standardized digital encoding due to pervasive graphical variants and undeciphered signs; (3) Long-tail character frequency distribution (Chai 2010; Chen 2007). These characteristics make conventional, data-intensive pre-trained language models ineffective.

This study directly addresses these challenges with two core contributions of significant value to digital humanities and low-resorce NLP. First, we construct the Oracle Bone Inscriptions Corpus (OBIC), a machine-readable dataset specifically engineered for semantic modeling. Second, we propose the Frequency-Aware Multi-scale Transformer (FAMT), a neural architecture that integrates explicit mechanisms to handle data imbalance and capture contextual dependencies at multiple granularities. Our work demonstrates that the semantic completion solution presented in this study achieves state-of-the-art performance, offering humanities scholars a powerful auxiliary tool. This research not only advances the digital study of ancient script corpora, but also provides new methodologies for processing other low-resource, non-standard languages.

The source code and data for this project are openly available on GitHub at https://github.com/aloryori/FAMT.git.

Data

A foundational contribution of this work is the creation of OBIC, a dataset enabling rigorous semantic completion research. Constructed from the comprehensive Oracle Bone Inscriptions Multi-modal Dataset (OBIMD) (Li et al. 2024), OBIC overcomes the primary obstacle of machine readability. We implement a structured, database-inspired mapping scheme: each semantically distinct character concept is assigned a unique primary identifier, and all known graphical variants are linked to this identifier while retaining distinct codes. This two-level schema effectively decouples linguistic meaning from visual form, allowing computational models to learn the equivalence of different shapes representing the same character.

Applying this encoding and filtering out sequences too short or containing too many undeciphered characters, we yield a core corpus of 13,527 sentences, comprising 1,660 distinct characters and 81,084 tokens. Statistical analysis confirms its long-tail distribution (see Figure 1). To directly address this imbalance, we design an oversampling strategy (see Figure 2). This strategy increases the sampling weight of sequences containing low-frequency characters, thereby enriching the training signal for rare but critical lexical items without discarding common patterns. The final augmented OBIC dataset of 17,585 sequences, partitioned into standard training, validation, and test sets, represents a significant infrastructural contribution. The dataset provides an essential, high-quality foundation required for developing and benchmarking advanced semantic models in this field.

Figure 1 Cumulative frequency distribution of characters in OBIC sequences

Figure 2 Sequence Oversampling Weight

Method

Our primary methodological innovation is FAMT, a neural architecture explicitly designed to overcome the unique challenges of OBI semantic completion. The FAMT framework follows a streamlined yet powerful three-stage pipeline: adaptive input masking, frequency-aware contextual encoding, and loss-optimized prediction.

The process begins with adaptive masking, a strategic departure from uniform random masking. We assign higher masking probabilities to low-frequency characters, ensuring that the model is rigorously trained to predict these rare but informative tokens. Simultaneously, we reduce masking at sequence boundaries to preserve crucial syntactic cues that guide prediction of interior elements.

Figure 3 The FAMT Architecture

The core of our approach is the FAMT encoder (see Figure 3). It enhances the standard Transformer with two deeply integrated, synergistic innovations. The first is dynamic frequency awareness. Each token’s representation is augmented with a frequency embedding prior to the attention calculation. This allows the model’s attention mechanism to dynamically adjust its focus based on the inherent predictability and information value associated with a token’s rarity, injecting crucial prior knowledge into the learning process.

The second innovation is a multi-scale attention mechanism coupled with a Frequency-Aware Feed-Forward Network (FA-FFN). We configure different attention heads to operate over varying contextual ranges, from local character collocations to broader thematic connections across the entire inscription. This enables FAMT to simultaneously capture fine-grained syntactic patterns and high-level semantic coherence, which is vital for interpreting OBI’s concise yet dense prose. The FA-FFN further amplifies this capability by explicitly modulating the representation strength of low-frequency tokens in the network’s intermediate layers, actively compensating for their under-representation.

For the final prediction stage, we employ the Adaptive Focal Loss (AFL). This loss function automatically down-weights the contribution of easy-to-classify high-frequency characters to the gradient, while sharpening the focus on challenging predictions associated with rare characters. This optimization strategy directly targets the long-tail problem, steering the model’s learning capacity toward the most difficult and semantically valuable cases.

Analysis

We conduct comprehensive experiments on the OBIC test set to evaluate FAMT against strong and diverse baselines, including CopticBiLSTM (Levine et al. 2024: 61-70), CNN-BiLSTM (Singh / Kamboj 2025: 3900-3910), BERT (Devlin et al. 2019: 4171-4186), and RoBERTa (Duan et al. 2024: 14005-14015). The results unequivocally demonstrate FAMT’s superior performance (see Table 1 and Figure 4). FAMT achieves a state-of-the-art accuracy of 0.8051, significantly outperforming all benchmarks. The stark underperformance of RoBERTa (0.2297) highlights the inadequacy of generic, data-intensive models in this ultra-low-resource setting. Beyond accuracy, FAMT excels in recall-oriented metrics and prediction confidence, indicating its outputs are both accurate and reliable.

Table 1 The Comprehensive Performance of Semantic Completion Models

Figure 4 The Performance of Semantic Completion Models on Data with Different Frequency Distributions

Ablation studies (see Table 2) rigorously validate the contribution of each core component. Removing either frequency awareness (-FA) or multi-scale attention (-M) causes a measurable performance drop. Removing both (-FA-M) results in the most significant degradation, confirming their synergistic effect. Further stratified analysis (see Figure 5) reveals their distinct, complementary roles: the frequency module primarily stabilizes predictions for high/medium-frequency characters by correcting data augmentation bias, while the multi-scale mechanism dramatically improves disambiguation for low-frequency characters by integrating broader contextual semantics.

Table 2 The Ablation Experiment Results of FAMT

Figure 5 The Results of Ablation Experiments on Data with Different Frequency Distributions

Qualitative case studies (see Table 3 and 4) vividly illustrate that baseline models often default to predicting common but contextually loose characters, whereas FAMT successfully identifies rarer, semantically precise characters that align with the inscription’s overall meaning, demonstrating a qualitatively deeper understanding.

Table 3 A semantic completion case from the comparative experiment

Table 4 A semantic completion case from the ablation experiment

Conclusion

This research provides a systematic computational framework for the semantic completion of Oracle Bone Inscriptions. By constructing the OBIC corpus and proposing the FAMT model architecture, we address three core challenges in this domain: data scarcity, encoding inconsistency, and extreme frequency imbalance. Experimental results demonstrate that our proposed approach achieves state-of-the-art, robust, and reliable performance compared to existing methods.

The value of this work extends significantly beyond oracle bone studies. The methodologies developed, particularly for handling data imbalance through dynamic frequency integration and for modeling multi-scale context in short texts, provide a novel and transferable framework for computational research on other low-resource historical languages and non-standard corpora. This project exemplifies a principled approach to interdisciplinary research, where technical innovations in AI are deeply informed by and directly address concrete, longstanding problems in the humanities, resulting in tools that genuinely augment traditional scholarly practice.

Future work will focus on expanding OBIC with newly deciphered materials, incorporating deeper linguistic annotations, and exploring pathways for integrating such specialized, efficient models with general-purpose large language models. We believe this research direction holds strong promise for bridging the gap between advanced computational techniques and the nuanced demands of historical philology, opening new avenues for interrogating and preserving our written heritage.