DH 2026

Daejeon, July 27–31

Thu, July 3011:00–12:30S079107
Short Paper

The Multimodal Television Generation: Experimental Synchronization in Early U.S.and Swedish Children’s Television

Johan Malmstedt
University of Gothenburg · johan.malmstedt@umu.se

Children’s television of the 1970s remains a debated and often controversial field, at times celebrated as a site of creative experimentation and progressive pedagogy and at other times dismissed as overly didactic and moralizing (Mares and Pan 2013; Janson 2014). Yet this decade marked a turning point in how audiovisual media addressed young audiences in both Europe and the United States, merging education and entertainment in ways that continue to shape contemporary media culture. Programs such as Sesame Street (PBS, 1969) and Fem myror är fler än fyra elefanter (SVT, 1973-1975) have been recognized for both their learning outcomes and their aesthetic innovation, but scholarship has primarily focused on policy, institutional history, and measurable effects, leaving the audiovisual mechanisms of learning comparatively underexplored.

Audiovisual pedagogy was nevertheless a central concern in the 1970s. Writers such as Ray Birdwhistell argued that the emerging television generation was learning that communication is a multimodal process, structured through a range of auditory and visual cues, and that television producers bore a special responsibility for shaping the relation between sound and image (Birdwhistell 1970). Taking this insight as a point of departure, this study asks how children’s television of the 1970s arranged the relationships between sound and sight. It examines how these programs taught young viewers to interpret meaning across modes and how verbal and visual codes were combined to stabilize meaning for novice audiences. In a comparative analysis of Sweden and the United States, I explore how contrasting public-service and commercial conditions shaped these multimodal pedagogies. Building on semiotic studies of children’s television (Hodge and Tripp 1986; Fiske 1987) and theories of audiovisuality (Altman 1985; Chion 1994; Donnelly 2019), I approach these series as early laboratories for what I call cross-modal literacy.

The corpus consists of children’s television from Sweden and the United States produced during the 1970s and drawn from digitized archives in the Svensk Mediedatabas and the American Archive of Public Broadcasting. It includes long-running educational formats such as Sesame Street and The Electric Company together with Anita och Televinken, Fem myror är fler än fyra elefanter, and Vilse i pannkakan. To capture both what is shown and how it is sounded and edited, I combine transformer-based recognition with calibrated measures of audiovisual complexity (Fickers et al. 2019; Malmstedt 2023). More specifically, the analysis tracks speech, music, and other sonic events alongside visual changes, including shifts in shots and on-screen objects or scenes. By aligning visual events with spoken words, music, and other soundtrack cues, the study measures when auditory and visual information coincide, when one modality leads or lags behind the other, and when changes in overall audiovisual complexity mark segments for closer qualitative interpretation. These measures are calibrated against a small manually annotated sample in order to support close reading and assess the reliability of the automated analysis.

Preliminary findings reveal two recurring strategies. First, cross-modal redundancy, in which object content and informational complexity work in synchrony, supports viewers by aligning what is heard and what is seen. Second, sensory cueing, in which one modality briefly diverges in order to prepare anticipation for a more comprehensive change in content, trains young audiences to follow dynamic shifts in attention. Across both Swedish and U.S. programs, frequent cases appear in which sound leads the visual field, with melodic shifts, vocal stress, or rhythmic cues anticipating cuts or object reveals. This suggests that sonic structure often organizes perception rather than merely reinforcing what is seen.

These findings indicate a greater openness to modal non-synchronicity than is usually ascribed to this period of audiovisual culture. Previous scholarship has often associated more extensive experimentation with offset sound-image relations with later digital production environments and tools (Donnelly 2019). My preliminary analyses, however, suggest that children’s television of the 1970s was already experimenting with less orthodox sound-image relationships, and that the pedagogical coordination of rhythm, speech, and editing anticipated later forms of cross-modal design.

In this way, the paper contributes both to scholarship in digital film and television studies and to a critical history of cross-modality (Arnold and Tilton 2019). At a time when multimodal AI systems are increasingly promoted for education, translation, and automated content production, it is important to understand how the patterns these models learn from were historically produced. By examining what young audiences once watched, this study illuminates the historical emergence of cross-modal logics and points to alternative approaches to synchronization and meaning-making that risk being forgotten in an age of automated multimodal media.

References
  1. Altman, Rick (1985): 'The Evolution of Sound Technology', in: Elisabeth Weis / John Belton (eds.): Film Sound: Theory and Practice. New York: Columbia University Press, 44-53.
  2. Arnold, Taylor / Tilton, Lauren (2019): Distant Viewing: Computational Exploration of Digital Images. Cambridge, MA: MIT Press.
  3. Birdwhistell, Ray L. (1970): Kinesics and Context: Essays on Body Motion Communication. Philadelphia: University of Pennsylvania Press.
  4. Chion, Michel (1994): Audio-Vision: Sound on Screen. New York: Columbia University Press.
  5. Donnelly, Kevin J. (2019): The Spectre of Sound: Music in Film and Television. London: British Film Institute.
  6. Fickers, Andreas / Snickars, Pelle / Williams, Michella (2018): 'Audiovisual Data in Digital Humanities', in: VIEW Journal of European Television History and Culture 7, 14: 1-4.
  7. Fiske, John (1987): Television Culture. London: Methuen.
  8. Hodge, Robert / Tripp, David (1986): Children and Television: A Semiotic Approach. Cambridge: Polity Press.
  9. Janson, Malena (2014): När bara den bästa TV:n var god nog åt barnen: Om sjuttiotalets svenska barnprogram. Stockholm: Karneval.
  10. Malmstedt, Johan (2023): 'Scale Exercises: Listening to the Sonic Diversity in 5000 Hours of Swedish Radio with Computers and Ears', in: Zoomland 21.
  11. Mares, Marie-Louise / Pan, Zekun (2013): 'Effects of Sesame Street: A Meta-Analysis of Children’s Learning in 15 Countries', in: Journal of Applied Developmental Psychology 34, 3: 140-151.