Emblems, within Gesture Studies, are word-like gestures that have a particular form and meaning for a cultural group (e.g. a “thumbs up”, “peace sign”, or an affirmative “nod”) with dictionary-like meanings (Ekman / Friesen 1969, McNeill 1992). Much of the emblem inventory across cultures is globally un(der)documented (Gawne / Cooperrider 2024). Existing analysis methods include elicitation surveys (Johnson et al. 1975, Morris et al. 1980) and text corpus analysis (Freestone et al. 2023). Corpora have been traditionally difficult to build and to search. AI models (computer vision, NLP) are increasingly able to support the processing of audio-visual research data, such as videos which include gestures. Recent advances in transfer learning enable accurate gesture classification with limited labeled examples per class (Sarwat et al. 2024), which can reduce annotation time significantly and rapidly increase the scale and speed of emblem analysis, while maintaining high label accuracy (Trujillo et al. 2019).

In this poster, we demonstrate the process of collaborative design between researchers in Data Science and Gesture Studies to build automated tools for gesture detection and disambiguation. This work focuses on the application of gesture recognition algorithms to a set of “heart” emblems, using HAGRIDv2 (Nuzhdin et al. 2024) and Google MediaPipe hand landmark detection methods (Lugaresi et al. 2019). Heart emblem gestures represent hearts, but they do so in varied ways which are often associated with different demographics. For example, “finger hearts” (Figure 1A) were initially popularised in Korean media (Kiaer / Kim 2024). Multiple heart gestures use the whole hand, including the “hand heart” in Figure 1B. These are often associated with Millennials. There is also a heart gesture made with the index fingers at the point and the middle fingers curled at the top, typically associated with Gen Z users.

Figure 1 Two common forms of Heart emblems (images from Noto Emoji Font open license)

This work aims to address two methodological questions: (1) Can transfer learning from pre-trained gesture models (HAGRIDv2) enable accurate heart emblem classification with limited labeled data (<500 examples)? (2) How does classification accuracy vary across heart emblem variants when training data is constrained? Answering these questions can demonstrate the feasibility and limitations of applying AI methods to emblem gesture research. To do this, our workflow involves four key ML stages (Figure 2).

Figure 2 Four ML stages in this project

The process is as follows. (1) We build a unique dataset from a combination of self-recorded videos and public databases (e.g.HAGRID, Roboflow) and web-sourced images and frames extracted from videos using MediaPipe. We capture both millennial and Gen Z hand variants for this dataset. (2) Manual labeling of images across three classes (Korean finger heart, Millennial hand heart, other heart gestures) is conducted using Roboflow annotation tools. (3) We fine-tune HAGRIDv2's 1M+ image pre-trained model with a Transformer backbone (e.g. VitB16 or YOLOv10x) with robust feature representations (Kapitanov et al. 2024). This model achieves 98.1% accuracy on 18 gesture classes including two types of heart gestures, applying transfer learning to adapt its final layers to multi-class heart gesture classification; (4) We monitor performance using confusion matrices, precision/recall, F1-scores and Mean Average Precision (mAP) (Gao et al. 2025). The model flags low-confidence predictions for manual review, progressively improving its accuracy through iterative retraining. Our approach follows computer vision best practices for small datasets (Foteinos et al. 2026), including stratified train/validation/test splits (60/20/20), data augmentation (rotation and brightness adjustment), k-fold cross-validation and early stopping to prevent overfitting.

In discussing these results, we reflect on the challenges of collaborating across disciplinary boundaries. As computer vision improves to allow gestural input and gesture detection in images (Uba / Jurewicz 2024), we need to ensure these systems are based on a meaningful understanding of the variation in emblem performance and function. Preliminary results demonstrate both the promise and the challenges of transfer learning for small-data emblem gesture analysis. HAGRIDv2 achieves 96% accuracy on traditional two-handed heart gestures, benefiting from the model's pre-training on similar hand configurations. However, accuracy drops to 50-60% for Korean finger hearts and other variants, where training examples are limited and hand poses differ significantly from HAGRID's original dataset. While this proof-of-concept does not address the varied performance we found on these demographically distinct emblem gestures, it establishes a methodological baseline for future research. Pre-trained models can enable emblem detection at scale, but improved automated annotation of specialized variants will require either targeted data collection to increase the scale of training data or few-shot learning approaches (Rastgoo et al. 2024). Further work may expand this framework for large-scale corpus analysis of emblem gesture variation to test hypotheses about geographic and demographic patterns (e.g. cultures, ages, genders) currently inaccessible through labour-intensive manual transcription methods. This framework could also be extended to other domains such as gesture detection in art (Bernasconi et al. 2023) or medical settings (Huang et al. 2022).