Daejeon, July 27–31
The digitization of global cultural heritage has unlocked vast new opportunities for computational analysis within the Digital Humanities. A prime example is the Tibetan Buddhist literature, a corpus representing one of the largest and most complex collections of pre-modern literature in the world. Spanning philosophy, medicine, history, and ritual, this textual archive, now being digitally preserved by organizations like the (Buddhist Digital Resource Center), offers a profound insight into human intellectual history. However, its sheer scale and the linguistic complexities of Classical Tibetan mean it remains largely inaccessible to modern, large-scale computational methods.
Unlocking the potential of this archive requires semantic search—a technology that allows scholars to trace concepts, cluster passages, and detect textual reuse. Implementing such a system relies on training a sentence similarity model, a process that traditionally demands massive human-labeled datasets (Reimers and Gurevych, 2019). For Classical Tibetan, this presents a critical bottleneck. The paucity of experts capable of navigating its linguistic and doctrinal complexities makes the creation of a “gold-standard” dataset through manual annotation logistically and financially infeasible.
To overcome these barriers, we introduce Bi-Dharma and Cross-Dharma, a set of models for sentence similarity in Classical Tibetan. Our contributions are threefold:
Deep bidirectional transformers like BERT (Devlin et al., 2019) enable high-accuracy semantic similarity via Cross-Encoder architectures, which process sentence pairs simultaneously. However, the computational cost of full self-attention renders this approach intractable for large-scale retrieval tasks, as it requires quadratic inference complexity.
To resolve this bottleneck, Reimers and Gurevych (2019) introduced Sentence-BERT (SBERT). Operating as a Bi-Encoder, SBERT utilizes a Siamese network to map sentences independently into dense vector spaces. This architecture yields fixed-length embeddings that enable efficient large-scale comparison using cosine similarity. Subsequent work, such as SimCSE (Gao et al., 2022), further improved the quality of these embeddings through advanced contrastive learning objectives.
State-of-the-art retrieval systems typically employ a two-stage Retrieve-and-Rerank cascade architecture. In this framework, a Bi-Encoder first retrieves a broad set of candidates to ensure recall, followed by a Cross-Encoder to maximize precision.
In recent years, significant progress has been made in Tibetan NLP, as documented by the comprehensive surveys from Huang et al. (2025); Meelen et al. (2024). Foundational resources, most notably the monolingual TiBERT (Liu et al., 2022) which we utilize in our comparison experiments and Tibetan-BERT-wwm (Liang et al., 2024), have been developed. These models have proven effective for tasks like text classification and sentence boundary disambiguation. Furthermore, multilingual models like CINO (Yang et al., 2022) have shown strong performance for the Sino-Tibetan language family.
Despite this progress, there is a clear research gap: a lack of a dedicated, high-performance sentence transformer for Tibetan (Huang et al., 2025). While models like TiBERT provide strong general-purpose language understanding, they are not optimized for sentence-level semantic similarity tasks out-of-the-box. Applications like semantic search or clustering for modern Tibetan (Engels and Barnett, 2025) currently rely on general-purpose multilingual models.
To address the scarcity of labeled data in Classical Tibetan, we adopted a two-tiered strategy. First, we curated a high-quality “gold-standard” dataset using human experts, serving primarily as a robust evaluation benchmark. Second, to overcome the scalability limits of manual annotation, we generated a larger “Silver-Standard” dataset using an automated LLM committee. This dual approach allows us to rigorously validate model performance on human-verified data while leveraging synthetic data for efficient training.
Although Classical Tibetan uses its own script, many academic and computational resources adopt the Wylie transliteration scheme (Wylie, 1959), which represents Tibetan characters with the Latin alphabet. We use Wylie in our work: datasets and models, unless otherwise stated.
The set was compiled by seven scholars of Tibetan Studies and comprises sentence pairs drawn from major Buddhist corpora, including the bKa’ ’gyur, bsTan ’gyur, rNying ma rgyud ’bum, and autochthonous ones like the Rong zom gsung ’bum. To ensure semantic diversity, candidate pairs were pre-categorized by the annotators into four similarity levels as part of the curation process, and from this pool, we selected 1,000 pairs with a balanced class distribution to facilitate unbiased relative scoring.
For annotation, we employed Best-Worst Scaling (BWS) (LouviereandWoodworth, 1991), a comparative method superior to inconsistent discrete rating scales. Following the BWS methodology, annotators were presented with tuples of four sentence pairs for each tuple and asked to identify the “most” and “least” similar in each tuple. These ordinal judgments were aggregated into continuous scores ranging from 0 to 1 using Equation 1, yielding a metric of high granularity and reliability compared to traditional discrete scales.
We name this gold dataset tibetan-sts-gold and publicly release it.
In response to the severe shortage of annotated resources, we present a novel, scalable methodology for generating high-quality “silver-standard” semantic annotations. We leverage the advanced reasoning capabilities of modern Large Language Models (LLMs) to serve as a committee of “expert” annotators. This approach, which uses several LLMs to automate annotation, has recently shown success in complex tasks (Bagdonet al., 2024; Zhang and Feng, 2025). Following this approach, we first combine the judgments given by the LLM committee, then aggregate them using the scoring function in Equation 1 to construct a large-scale, cost-effective, and high-quality “silver-standard” dataset. To the best of our knowledge, the application of such an automated LLM-committee framework specifically for semantic similarity is a novel development.
We verified the efficacy of this approach by comparing the LLM committee’s scoring against our human gold-standard dataset. As detailed in Table 1, the ensemble approach achieves a Spearman correlation coefficient (ρ) of 0.847 with human experts, validating its use for training data generation. We name this silver-standard dataset tibetan-sts-synthetic-bws and publicly release it.
To facilitate the construction of state-of-the-art Retrieve-and-Rerank pipelines, we develop and release two optimized components designed for seamless integration into industry-standard search engines: a Bi-Encoder for efficient candidate generation and a Cross-Encoder for high-precision re-ranking.
Bi-Encoder Framework
We evaluate using the CoSent (Huang et al., 2024) contrastive loss function with weighted mean pooling.
Training Strategy
To solve the data bottleneck, we employ a hybrid training strategy. We augment the human-annotated “gold” training split with the “silver-standard” dataset generated by our BWS-LLM committee.
We experimented with both Bi-Encoders and Cross-Encoders. To ensure robust evaluation, we employed a four-fold cross-validation strategy, stratifying the “gold-standard” dataset into four distinct splits such that every unique pair appears in the test set exactly once across the folds. Table 2 lists the annotated pair sizes. We evaluated four distinct training configurations using the “gold” splits of 600 and a “silver-standard” dataset of 5,000 samples:
Additionally, we evaluated the models without any fine-tuning to serve as a baseline for comparison.
We compare our models against several state-of-the-art Tibetan language models: TiBERT (Liu et al., 2022), CINO (Yang et al., 2022), T-RoBERTa, and the multilingual LaBSE (Feng et al., 2022). Because these models are trained on native Unicode Tibetan script, we convert our data from Wylie transliteration to Unicode prior to training or evaluation. This ensures a fair comparison by allowing the baselines to fully utilize their native Tibetan tokenizers and vocabularies. Additionally, we compare our approach to the Wylie-tuned model mbert-Tibetan and the multilingual models all-mpnet-base-v2, and all-MiniLM-L12-v2.
Table 3 shows the Bi-Encoder models‘ performance, which indicates that mbert-Tibetan consistently outperformed other models. The model demonstrated the strongest zero-shot baseline (0.804), and proved most effective after fine-tuning, achieving the highest overall Bi-Encoder score of 0.855, and thus selected as our Bi-Dharma model. The best score was achieved by first training on the “Synthetic” dataset followed by the “gold”.
Cross-Encoder Performance
As shown in Table 4, Cross-Encoders offered slightly better precision, peaking at 0.864 with mbert-tibetan, which we therefore selected as our Cross-Dharma model. Unlike Bi-Encoders, which leverage pooled embeddings for zero-shot tasks, untuned Cross-Encoders showed near-zero baseline performance, highlighting the necessity for fine-tuning. Once fine-tuned, however, this architecture established the state-of-the-art. Notably, the performance gap between the best Cross-Encoder and our top Bi-Encoder is marginal (~0.01). For latency-sensitive applications, researchers can reasonably deploy the Bi-Encoder as a standalone retriever without a Cross-Encoder reranking step. Nevertheless, this efficiency trade-off must be carefully considered: because the Spearman correlation scale is non-linear, incremental gains at the upper end of the scale translate to a disproportionately substantial reduction in real-world ranking errors.
Efficacy of Synthetic Data
The LLM-generated "Silver-standard" dataset demonstrated high utility. Notably, the best Bi-Encoder result (mbert-tibetan: 0.855) and the peak Cross-Encoder performance (mbert-tibetan: 0.864) were both achieved using synthetic training data. This validates our hypothesis that the automated LLM-BWS committee effectively scales high-quality training signals for low-resource tasks, but at a quality level that does not match that of human expert annotation.
With the introduction of Bi-Dharma and Cross-Dharma, we achieve a significant improvement in measuring semantic similarity for Classical Tibetan, effectively transforming semantic search from a theoretical concept into a vital and reliable tool for digital scholarship. This advancement bridges the critical gap between the ongoing preservation of the Tibetan Buddhist literature and its semantic accessibility. By validating a novel, fully automated LLM-BWS annotation framework, we demonstrated that the "expert bottleneck", long considered an insurmountable hurdle for specialized languages, can be effectively overcome through synthetic data augmentation.
Scholars can now trace the evolution of philosophical concepts and identify textual reuse across centuries of literature. While our immediate focus is Classical Tibetan, the methodology presented here establishes a reproducible blueprint for other low-resource cultural heritage domains, proving that state-of-the-art retrieval is achievable even in the absence of massive human-labeled datasets.
Acknowledgments
This study is supported in part by the European Research Council (Intellexus, Project No. 101118558). Views and opinions expressed are, however, those of the authors only and do not necessarily reflect those of the European Union or the European Research Council Executive Agency. Neither the European Union nor the granting authorities can be held responsible for them.