Text complexity and comprehensibility are related but distinct constructs: complexity refers to measurable linguistic properties of a text, whereas comprehensibility captures a reader's perceived ease of understanding. Assessing these properties is central to language-related research, from Easy and Plain Language studies (Vecchiato 2022) to education (Friedrich / Heise 2022; Spencer et al. 2019) to automatic text simplification (Garbacea et al. 2021). However, defining text complexity and finding the exact factors that lead to a text being considered complex is a challenging task (Dmitrieva 2025: 80). Nevertheless, in education, features of text complexity should be carefully considered when text materials are matched to students (Dahl et al. 2021: 712), which is especially important in creating accessible learning content. In this work, we investigate how perceived complexity and comprehensibility relate to each other, and which linguistic factors are associated with each, using a small dataset of human judgements.

We combined two sources of Russian educational texts evaluated by human readers. The first dataset (Trushchelev 2025) comprises 12 middle-school textbook excerpts rated by 386 eighth-grade students on a 1–7 point scale; the second (Trushchelev 2023) contains 8 texts evaluated by 400 first-year undergraduates. Students evaluated the texts with several criteria, including complexity and comprehensibility.

Table 1. General dataset information.

In order to study the influence that the linguistic properties of texts might have on the perceived complexity and comprehensibility, we parsed the texts with the help of the NLTK, Mystem, and Stanza frameworks and extracted 150 features motivated by established readability research of Russian L2 texts(Laposhina et al. 2018). These included surface measures (average word and sentence length), grammatical properties (e.g., proportion of imperative verbs), syntactic complexity (e.g., average number of parataxis relationships per sentence), and vocabulary-based proxies. For the latter, we used lexical minima for different CEFR levels (see, for example, Andryshina / Kozlova 2015), which provide a standard benchmark for vocabulary difficulty in Russian as a foreign language.

Given the small sample (N=20 texts), we interpreted all statistics cautiously and limited ourselves to exploratory techniques. We began with Kendall's τ correlation between each dependent variable and the linguistic features, then used ridge regression with leave-one-out cross-validation and alpha selection via grid search to test whether the measured features may be more closely related to comprehensibility than to complexity.

Comprehensibility and complexity correlated negatively (τ=−0.69, p=0.001). Comprehensibility exhibited more than twice as many significant bivariate associations with linguistic features as complexity did.

Ridge regression with leave-one-out cross-validation yielded higher rank-order agreement for comprehensibility (Spearman ρ=0.60; Kendall τ=0.48) than for complexity (ρ=0.34; τ=0.28), with comparable root mean squared error across outcomes. These results suggest that surface linguistic features preserve ordinal information for comprehensibility more effectively than for complexity.

Several features were significant for both outcomes, including average word length, relative frequency of ordinal numerals (ANUM tag in Mystem), and some vocabulary-related features. The latter mostly represented the relative number of words from the lexical minima for the A2, B1, and B2 levels in a text. However, partial correlation testing proved that the vocabulary features’ significance was most likely caused by the fact that the words from the vocabulary lists are simply shorter, especially on the earlier levels of language acquisition. The complexity and comprehensibility scores also appear to not have a significant correlation with the text’s subject. In our data, such correlation can only be observed if the texts about Russian language are included – however, these texts were evaluated by a different, older group of participants. The Kruskal–Wallis H test did not show any significant subject-complexity or subject-comprehensibility correlation for the other subjects, where texts were evaluated by the same group of middle school students.

Despite that, some of the features that prove to have a significant negative correlation with text comprehensibility, such as the relative number of patronyms and toponyms, are mostly present only in the history textbook texts that have the lowest mean comprehensibility. Further studies are required to determine if these features can affect comprehensibility independent from the subject. A group of features that seems to have a significant negative effect on comprehensibility regardless of the subject are lexical density and type-token ratio.

In conclusion, this study provides an insight on how perceived complexity and comprehensibility function in the context of educational texts. Firstly, although these variables are correlated, our results suggest that they may be influenced by different linguistic factors, and that comprehensibility may be more directly captured by linguistic features than complexity. Secondly, the influence of the text’s subject on the scores should still be considered when analysing the features’ relationships with complexity and comprehensibility. These findings carry practical implications for matching materials to student abilities, designing accessible content for diverse audiences including foreign-language learners, and informing automatic simplification systems. Future work will expand the sample and examine how these perceived-difficulty dimensions align with established accessibility guidelines (see, for instance, Netzwerk Leichte Sprache 2022, or Ruel et al. 2018).