Introduction

The visualization of poetry has long been explored across visual and digital media (Michel 1999; McVee et al. 2008; Benito-Santos et al. 2023). In the case of classical Chinese poetry, existing work has examined AI-assisted poetic generation and the reinterpretation of poetry through animation and visual art, but has focused largely on imagery, theme, and affective atmosphere rather than on the poem’s sonic organization (Zhao et al. 2024; Zhang / Eger 2024; Chen / Chen 2023). Although some projects have transformed poetry into abstract graphics or sound-responsive visual systems (Huang et al. 2012; Zhao / Lee 2022; Cui 2024), the visual representation of prosodic structures such as tonal patterning and internal rhythmic articulation remains underexplored.

Sound is integral to classical Chinese poetry. As Liu and Cai have shown, the monosyllabic structure of Chinese, the distinction between level and oblique tones, and the patterned articulation of heptasyllabic lines together produce a highly organized sonic form (Liu 2022: 21; Cai 2008: 104). This article examines how animation can make these prosodic structures perceptible through visual design and dynamic movement.

Using Li Shangyin’s “Jin Se” as a practice-based case study, combining TouchDesigner, After Effects, and manual compositional design, it explores abstract waveform animation, landscape-based spatial composition, and metaphorical animated imagery.

Visualizing Experiments

This study adopts a three-layer visual encoding method to translate the auditory structure of classical Chinese poetry into visual form, moving from physical sound signal to semantic rhythm and tonal prosody.

Layer I: Signal Mapping through Waveform Synchronization

The opening line of “Jin Se” introduces the se, an ancient string instrument, so strings were adopted as the key visual element in this sequence. Here, I tested direct synchronization between poetic recitation and visual waveform. In TouchDesigner, the recorded narration was used as audio input, and its fluctuations were translated into moving white lines, later layered and refined in After Effects.

The experiment showed first that waveform intensity did not correspond directly to the poem’s tonal pattern ( ping/ze). Under identical parameter settings, amplitude was driven mainly by the instantaneous volume of the recitation rather than by tonal category. The waveform therefore responded more directly to vocal energy than to prosodic structure.

Second, vocal energy was closely shaped by emotional emphasis. In the opening line, for instance, wuduan (“for no reason”) carries a stronger tone of questioning and lament than the surrounding nouns, and accordingly produces a noticeably larger waveform amplitude, as shown in Figure 1. The waveform thus made visible not only acoustic contour but also the distribution of affective pressure within the recitation.

Figure {1} Waveform-based white lines generated from recitation of the source poem.

It also suggested the presence of semantic rhythm, especially the characteristic 2+2+3 grouping of seven-character regulated verse. This is consistent with Cai’s observation that, in addition to the main pause at the end of a line, there is often a secondary caesura within the line, particularly in the final unit (Cai 2008: 104). What appeared in the waveform was therefore not sound intensity alone, but a pattern shaped by internal segmentation and phrasing.

Layer II: Semantic Rhythm through Animated Action

This stage examined how the internal grouping of the verse could be made perceptible through animated action. Key words and semantic units were treated as cues for movement, so that rhythm could be conveyed through transformation and camera motion.

This is clearest in the circular pattern sequence. The motif draws on three images from the poem, sea, moon, and pearl, visualized into wave-like outer patterns, a circular ring, and a pearl-like form between two birds. The bird motif derives from the cuckoo image in the previous sequence, linking this passage to the film’s broader visual system.

Figure {2} Circular motif based on the poem’s sea, moon, pearl, and bird imagery.

The sequence unfolds according to both the semantic progression of the line and the characteristic 2+2+3 grouping of seven-character verse. During canghai (“vast sea”), mist-like patterns extend inward; at yueming (“bright moon”), the outer ring appears in clockwise motion; and during zhu you lei (“pearls have tears”), the pattern fractures and disperses, as shown in Figure 2. Since the recitation proceeds at the same pace as elsewhere in the poem, rhythm becomes legible here through the alignment between semantic grouping and visual change rather than through tempo alone.

This experiment also clarified the relation between voice and music. Continuous movement aligned more stably with the music, while sharper transitions could be anchored more precisely to the recited words. For example, the droplet’s transformation into a circle coincides with a gentle plucked note, while the bird’s appearance within the white circle aligns with a short musical phrase. Recitation articulates semantic emphasis, while music sustains the broader continuity of movement.

Layer III: Visual Encoding of Tonal Prosody

The third experiment translated prosody into spatial form. In this system, each Chinese character corresponds to a mountain form. Level tones ( ping) become low, gentle slopes, while oblique tones ( ze) become higher, sharper peaks, as shown in Figure 3. Tonal alternation is thus translated into visible rise and fall. The handscroll-like landscape also draws on the poem’s image of lantian, understood as a mountain setting in which jade glimmers in sunlight and gives rise to drifting mist (Liu 1969: 51). The image therefore functions as both poetic landscape and prosodic structure.

This system was applied to the line “ lantian ri nuan yu sheng yan”. Its tonal pattern, ping-ping-ze-ze-ze-ping-ping, was translated into topographic units organized by both tone and semantic rhythm. Lantian and ri nuan are rendered as paired peaks, while yu sheng yan was treated as a 1+2 grouping, with one dominant peak followed by two lower connected forms.

Figure {3} Tonal-topographic landscape, P = ping, Z = ze, and brackets indicate semantic groupings.

This experiment showed that the auditory organization of classical Chinese poetry can be spatialized in animation. Here prosody does not simply drive motion; it generates the inner architecture of the visual world. The landscape therefore functions not only as scenery but as a visible trace of tonal rhythm.

Conclusion

Using “Jin Se” as a practice-based case, this study shows that the sound structure of classical Chinese poetry can be translated into animation through different visual strategies. Waveform, motion, and spatial composition each reveal different aspects of poetic sound, from vocal energy and semantic rhythm to tonal prosody. Animation therefore does more than visualize poetic imagery; it can also make the poem’s sonic organization perceptible in visual form.

Benito-Santos, Alejandro / Muñoz, Salvador / Therón Sánchez, Roberto / García-Peñalvo, Francisco J. (2023): “Characterizing the visualization design space of distant and close reading of poetic rhythm”, in: Frontiers in Big Data 6: 1167708. DOI: 10.3389/fdata.2023.1167708.

Cai, Zong-qi (ed.) (2008): How to Read Chinese Poetry: A Guided Anthology. New York: Columbia University Press.

Chen, Hung-Cheng / Chen, Z. (2023): “Using ChatGPT and Midjourney to Generate Chinese Landscape Painting of Tang Poem ‘The Difficult Road to Shu’”, in: International Journal of Social Sciences and Artistic Innovation 3, 2: 1–10. DOI: 10.35745/ijssai2023v03.02.0001.

Cui, Qiran (2024): Visualizing Poems. Master’s thesis, Rochester Institute of Technology.

Huang, Chih-Fang / Lu, Huei-Ping / Ren, Jian (2012): “Algorithmic Approach to Sonification of Classical Chinese Poetry”, in: Multimedia Tools and Applications 61, 2: 489–518. DOI: 10.1007/s11042-011-0856-4.

Liu, James J. Y. (1969): The Poetry of Li Shang-yin: Ninth Century Baroque Poet. Chicago: University of Chicago Press.

Liu, James J. Y. (2022): The Art of Chinese Poetry. London / New York: Routledge.

McVee, Mary B. / Bailey, Nancy M. / Shanahan, Lynn E. (2008): “Using digital media to interpret poetry: Spiderman meets Walt Whitman”, in: Research in the Teaching of English 43, 2: 112–143.

Michel, Karl F. (1999): “Turning poetry into paintings: An experiment in visualization”, in: Art Education 52, 3: 6–12.

Zhang, Ran / Eger, Steffen (2024): “LLM-based multi-agent poetry generation in non-cooperative environments” <https://arxiv.org/abs/2409.03659> [24.04.2026].

Zhao, Cheng / Wang, Bin / Wang, Zhen (2024): “Understanding Literary Texts by LLMs: A Case Study of Ancient Chinese Poetry” <https://arxiv.org/abs/2409.00060> [24.04.2026].

Zhao, Jianli / Lee, Hyo Jong (2022): “Automatic Generation and Evaluation of Chinese Classical Poetry with Attention-Based Deep Neural Network”, in: Applied Sciences 12, 13: 6497. DOI: 10.3390/app12136497.