Daejeon, July 27–31
Multimodality refers to the ability of an AI system to process and integrate information from multiple data types, such as text and images. Despite recent progress, most multimodal systems are still experimental and difficult to scale or adapt to new domains (Nagare / Shirke 2025). Chatbots such as EcoSage for plant care demonstrate that models processing both text and images generate more specific and relevant answers (Tomar et al. 2024). Concurrently, recent developments, including RAG-based chatbots (Fan et al. 2024) – applicable to historical archives (e.g., on Palestinian history (Miftah et al. 2025) or for accessing materials of a Truth Commission (Sokol et al. 2025)) – often focus on the textual modality.
Consequently, there is a gap between the growing use of multimodal formats in archives (text and images) and the ability of AI systems to process them effectively. In practice, this means that users of digital humanities collections still have to work with text and visual materials separately, manually interpreting images and linking them to textual context.
As a result, answering even relatively simple research questions that require joint analysis of images and text remains time-consuming and cognitively demanding. This limitation highlights the need for adaptive multimodal assistants capable of integrating both modalities based on approaches such as Retrieval-Augmented Generation (RAG) (Zhou 2025).
This work proposes a multimodal virtual assistant for digital humanities databases. The key idea of our system is the following: instead of relying only on text, the assistant can look at both images and their descriptions to better understand a user’s question and provide more accurate answers.
The architecture of the virtual assistant is based on retrieving images and text fragments relevant to the user's query (see Figure 1). Text and images are combined in two steps: first, the system retrieves the most relevant images and their associated textual descriptions; then, a multimodal language model processes both modalities together, using the text to interpret the images and the images to ground the textual information, producing a final answer.
The system first uses a large language model (LLM) to reformulate the user’s question and decide whether a search is required. If so, the LLM generates a search query, informed by dialog history, and applies domain-specific filters to the knowledge base.
This query initiates a hybrid search that combines image and text search. For image search, the query is converted into a vector using vision-language models (e.g., CLIP (Radford et al. 2021) or SigLIP (Zhai et al. 2023)) and compared with image embeddings stored in a vector database.
Figure 1 – Architecture of the virtual assistant
For text search, the system retrieves images by relevance of their textual descriptions. It consists of two subcomponents: semantic search, where query and description embeddings (generated by models like E5 (Wang et al. 2022) or BGE (Chen et al. 2024)) are compared via cosine similarity in a vector database, and full-text search, which employs lexical matching with ranking algorithms such as BM25 (Robertson / Zaragoza 2009) in a text index database.
The relevance scores from image and text searches are combined to produce a ranked list of relevant images. The top-5 results, along with their descriptions, are then processed by a multimodal large language model (MLLM) to generate the final answer.
To select MLLM for our chatbot, we evaluated modern models within our multimodal RAG framework. We grouped images with their textual descriptions into semantically related sets and generated corresponding test questions for each group.
We collected datasets of images and their textual descriptions from three domains:
In total, we assembled 58 groups of 5 images each (20 in History and Botany, 18 in Literature). Grouping allows us to bypass the retrieval step and evaluate only the generation component. For each group, we generated 5 test questions using GPT-5.1, designed to be simple, specific, and require analysis of the group of images collectively (see Appendix A in the repository Hereinafter, the repository refers to: https://github.com/alekosus/multimodal-smart-assistants-dh2026
Images descriptions and questions, as well as the generated answers during experiments, have been in Russian. Examples and prompts have been translated into English for convenience.
For the experiments, we selected 6 open-weight MLLMs:
https://huggingface.co/meta-llama/Llama-4-Maverick-17B-128E-Instruct
https://huggingface.co/mistralai/Mistral-Large-3-675B-Instruct-2512
The models were chosen based on their positions on the LMArena Vision leaderboard. Only open models were considered, ensuring the possibility of local deployment of the assistant for processing private data (e.g., personal data or restricted archival data).
Each model was prompted to answer a test question, with the prompt containing the five relevant images and their textual descriptions (see Appendix B in the repository for message format and generation prompt).
Responses were evaluated according to the five criteria adapted from (Murugadoss et al. 2025):
Model responses were evaluated using the LLM-as-Judge approach (Li et al. 2025). A judge model was given the retrieved images, their descriptions, the question, and the model’s answer. It then assigned a score from 1 (worst) to 5 (best) for each criterion and generated thoughts, enhancing the evaluation’s interpretability (see Appendix C in the repository for the prompt).
To select the judge model, we applied the Bradley-Terry model to pairwise comparisons of model judgments, identifying the model whose preferences best aligned with those of human annotators. The candidates were selected from the LMArena leaderboard (GPT 5.1, Gemini 3 Pro, Claude Sonnet 4.5, and Grok 4.1 Fast). Based on independent assessment of two annotators, Gemini 3 Pro was chosen as the judge. The exact strength scores are also presented in Appendix C.
Figure 2 presents the average scores, with 95% confidence intervals, for all evaluated open-weight models across domains, as rated by the Gemini 3 Pro judge according to the established criteria.
Figure 2 – Average scores across three domains for the five criteria for each tested model. Dots (●) indicate criterion score, horizontal bars represent 95% CI. Stars (★) and vertical lines mark the highest criterion score
Qwen3-VL Instruct achieved the highest scores in 3 out of 5 criteria. In the Accuracy criterion, it was outperformed by Llama 4 Maverick by a margin of 0.05, and in the Coherence criterion, it was surpassed by Gemma 3 by 0.03 (see Appendix D in the repository for exact scores).
An example group of images, the corresponding question, and the response generated by Qwen3-VL Instruct are presented in Figure 3.
Figure 3 – Example of a response from Qwen3-VL Instruct to a question from the Literature domain (see Appendix E in the repository for the full textual version of this example)
We analyze model performance for each domain separately. In History, Qwen3-VL Instruct was identified as the best performer for 3 out of 5 criteria (Figure 4).
Figure 4 – Scores for the five criteria for each tested model in the History domain
In Literature, the same model demonstrated superior results for 4 out of 5 criteria (Figure 5), while in Botany, it achieved the best scores for 2 out of 5 criteria.
Figure 5 – Scores for the five criteria for each tested model in the Literature domain
Gemma 3 shows better performance in the Botany domain, achieving the highest scores for 3 out of 5 criteria (Figure 6) (see Appendix D in the repository for the detailed scores for each domain).
Figure 6 – Scores for the five criteria for each tested model in the Botany domain
We separately examine the scores for the safety criterion regarding the subset of provocative questions (Figure 7).
Figure 7 – Scores for the Safety criterion for the subset of provocative questions
Qwen3-VL Instruct significantly outperforms other models in safely handling provocative questions, though considerable variance in safety scores is observed. Notably, some models acknowledged a query’s unethical nature but still produced harmful answers (see Figure 8).
Provocative questions varied by domain. In Literature, most involved unethical interpretations of character behavior; here, warning the user was deemed sufficient for safety. In Botany, questions often concerned the dangerous use of plants, where a complete refusal to answer to the posed question was expected.
Figure 8 – Example of model responses to a provocative question. The unsafe response from Mistral Large 3 is highlighted in red (full text is omitted in accordance with ethical guidelines). The response from Qwen3-VL Instruct, which contains no harmful information, is highlighted in green
We examine the heatmap depicting the number of scores below 3 received by the models for each criterion (Figure 9).
Figure 9 – Number of scores below 3 for each model per criterion across all domains
Errors made by GLM 4.5V on the coherence criterion within the Literature dataset are attributed to mixing of languages in its responses. Errors exhibited by Qwen-3-VL Thinking in the History domain manifest in the form of hallucinations and generation loops. Errors related to the accuracy criterion in Botany are primarily due to response inadequacy: the models either fail to incorporate key data from the provided context or, conversely, attribute properties to plants that are absent from the botanical descriptions.
The experimental study across three diverse domains (History, Literature, Botany) demonstrated that modern open-weight multimodal models like Qwen3-VL Instruct have achieved a high level of performance across key criteria: accuracy, relevance, coherence, integrity, and safety. This outcome proves the feasibility of building effective and ethical intelligent systems for working with cultural heritage and digital humanities data.