Daejeon, July 27–31
People regularly tell stories, for instance, in news articles, when gossiping, or in literature. If we want to investigate whom these stories are about and how the different characters relate to each other, we can try to build a social network of the characters in the story (Agarwal et al., 2012). These social networks are typically visualized using nodes to represent the characters while the edges show the relationships between the characters. These networks can then be analyzed, see, for example, Fields et al. (2022), James Lee and Jason Lee (2017), and Min and Park (2016).
To build social networks, we first need to identify the characters in the narrative. This is typically done using a technique called Named Entity Recognition (NER). NER systems identify names of people, locations, organizations, etc. Names of characters that co-occur in a small context (e.g., in a sentence or paragraph) can then be assumed to have some sort of relationship to each other (e.g., see Labatut and Bost (2019) and Ven et al. (2018)).
There are two main challenges with this approach. First, even though the task of NER is considered well understood (Nadeau and Sekine, 2007; Seow et al., 2025), NER systems are not always perfect. This is particularly valid for languages that are under-resourced in the digital space (Ramukhadi et al., 2020). Second, NER 1systems only identify named entities and, as such, do not recognize non-named characters, such as “the sniper”, or “the chrononaut”.
In this research we investigate whether large language models (LLMs) can be used to identify the characters that occur in stories. While LLMs have already been used for NER (see, e.g., Wang et al. (2025), Wei et al. (2023), Xie et al. (2023), Yan, Yu, and Chen (2024), and Ziems et al. (2024)), here we focus on identifying all (including non-named) characters that occur in a story. Additionally, this task is slightly different from NER as it does not identify all occurrences of the characters, only their existence.
Several LLMs have multi-lingual support, which means that one model can be used to identify characters in stories in different languages. This allows us to consider identifying characters in stories in the official South African languages. South Africa has eleven official languages in addition to South African Sign Language (which we will not take into account here): Afrikaans, IsiNdebele, IsiXhosa, IsiZulu, Sepedi, Sesotho, Setswana, Siswati, Tshivenḓa, and Xitsonga (ordered alphabetically). Most of these languages are highly under-resourced and the available digital language material is highly genre (the government domain) specific, see, e.g., Eiselen (2016).
To investigate the suitability of LLMs for the identification of characters in stories, we perform a pilot study. We start with a selection of short English stories. These stories are automatically translated to the South African languages. Next, an LLM is requested to identify the characters in these stories. To evaluate the outcomes, the characters are identified manually in the English stories. These are then compared against the characters identified by the LLM in the translations.
To evaluate the performance on multiple stories, we take the first five nanofictionary stories from the first season archive by Andrew Looney (https://www.wunderland.com/WTS/Andy/Archives/NanoficSeason1.html). These are very short stories (on average 59 words) and contain on average three characters per story. These very short stories are selected to ensure that the entire story can be fully captured in the context of the LLMs.
The English stories are then translated to the other languages using the web interface of Google translate. This ensures that the same story is presented in the different languages, which allows for a direct comparison of the results as the characters in the stories will be the same. Note that we currently do not have access to stories with known characters written originally in the different South African languages. This approach, however, assumes that the translation is of high enough quality.
The different stories are handed to the LLM for identification of the characters. This is done through Ollama with the Gemma3:4b model using default parameters and temperature set to 0. This is a model that contains multiple languages and still runs on a CPU.
For these pilot experiments, we experiment using a zero-short approach to ensure that the entire stories are available in the context of the model. If models with large contexts are used, further experiments with few shot learning may be considered as well, but this also assumes availability of additional annotated stories.
The prompt used is as follows: “Please identify all main characters in the following [LANG] nanofictionary story: [STORY]” with [LANG] the name of the language of the story which is given in [STORY].
To evaluate, we compare our manually identified characters (in the original English text; our gold standard) against the characters identified by the LLM. From this we can compute precision (are all characters found by the LLM characters in the story?), recall (does the LLM find all characters in the story?), and F1-score, which is the geometric mean of precision and recall. We provide both micro and macro averaged results. The micro results provide averaged results per character, whereas the macro results give equal importance to each story (as stories may have a different number of characters).
The results can be found in Table 1. We see that overall, the results are quite good, especially if we realize that the stories are very short. Characters often only occur once or twice in the story, which means that identifying the characters is a relatively challenging task.
English shows highest results, followed closely by Afrikaans (which is also a Germanic language). The other languages, however, also perform quite well. The recall is typically high (higher than precision) which indicates that mostly the characters that occur in the story are found, but sometimes additional characters are “made up” by the LLM.
Table 1: Micro and macro precision (P), recall (R), and F1-score (F) values for the identifition of characters for each of the languages.
| Micro | Macro | |||||
| Language | P | R | F | P | R | F |
| Afrikaans | 81.25 | 86.67 | 83.87 | 82.00 | 90.00 | 83.81 |
| English | 92.31 | 80.00 | 85.72 | 93.33 | 85.00 | 88.57 |
| IsiNdebele | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
| IsiXhosa | 75.00 | 64.29 | 69.23 | 73.33 | 73.33 | 69.14 |
| IsiZulu | 64.29 | 60.00 | 62.07 | 63.33 | 65.00 | 61.33 |
| Sepedi | 55.56 | 66.67 | 60.61 | 57.00 | 75.00 | 60.19 |
| Sesotho | 56.25 | 60.00 | 58.06 | 60.00 | 65.00 | 58.10 |
| Setswana | 64.29 | 60.00 | 62.07 | 68.33 | 70.00 | 66.90 |
| Siswati | 60.00 | 60.00 | 60.00 | 60.00 | 65.00 | 58.57 |
| Tshivenḓa | 70.59 | 80.00 | 75.00 | 73.67 | 85.00 | 75.48 |
| Xitsonga | 70.59 | 80.00 | 75.00 | 70.00 | 85.00 | 73.14 |
We see, however, that IsiNdebele is a major outlier. In fact, the LLM does not find any correct characters in the IsiNdebele translation of the text. When manually investigating the IsiNdebele results, we realize that the quality of the automatic translation from English into IsiNdebele is not good at all.
A proper investigation into the quality of the translation is needed to understand the results. Unfortunately, we do not speak most of the South African languages. Instead, we back translate the automatically translated texts into English (again using Google translate). We do not directly compare the original text to the back translated text (as this is not a useful approach to measure machine translation performance (Zaanen and Zwarts, 2006)), but we can manually check whether the characters can still be found in the back translated text. This is the case for almost all texts, except for all IsiNdebele texts (also looking at the IsiNdebele texts, it is obvious that the translations are not good). Furthermore, two Sepedi and two Sesotho texts miss one character and one Xitsonga text misses one character in the back translated texts. Additionally, sometimes names of characters are mangled during translation, e.g., “Commander Thompson Thompson” or “Sinuper” instead of “Sniper”. When it is clear who the character should be, we have marked these as correctly identified.
We presented a simple LLM-based approach that can identify characters in stories. The outcomes of this approach can be used as the basis for the creation of social networks. The approach was tested on short English stories that were automatically translated to ten South African languages (and English). The results were compared against manually identified characters, which showed promising results.
The approach has two major benefits. First, the approach can be used to identify characters, even if they do not have explicit names (such as “the Tabby”, “the Siamese”, or “the Time Police”), which extends what NER systems can do. Second, it can be used for the identification of characters in stories written in under-resourced languages, such as the South African languages.
Future work can go in several directions. First, new LLMs are made available regularly. These new models may lead to improved performance on this task. Second, different prompting strategies may be investigated. For example, a few-shot approach may lead to improved results. Finally, the performance and evaluation of the approach on different texts should be investigated. This not only includes texts in different genres, but also texts originally written in the different South African languages (i.e., not using automatic translation).