Often lost in the discussions and implications of generative artificial intelligence (AI), is the value that analytic applications of AI offer digital humanists via powerful tools to quantitatively classify and understand cultural heritage. The difference between the two is that generative AI has been positioned as the ultimate in user-friendly interfaces. Indeed, some would say it is an interface without an inherent application behind it. Analytic AI can be thought of as the opposite: it is an application of the same machine learning techniques that lead to generative AI, but focused entirely on creating more machine-readable data rather than serving that information to humans in an interface they can easily use to make meaning.

We report on the creation of a pipeline designed around producing an additional, subsequent stage of analytic AI in which the final interface allows humanists to apply the same interpretive skills they use on traditional texts when they view the otherwise-obscure output of one specific analytic AI system: 3D human pose analysis extracted from 2D video. We will discuss the design goals of the interface, titled Deep Screens XR, as well as the technical infrastructure needed to generate pose data from historic videos and transform that data into a visual environment. The pose data is applied to animated figures presented to scholars in 3D, recreating the motion of actors in those videos outside the context of the original films so their actions can be studied both quantitatively through statistical analysis of the extracted data as well as qualitatively through interpretation and discussion of the visualization of that data. By presenting data in a form more familiar to humanistic interpretation, Deep Screens seeks to bridge the gap between AI-centric machine vision analysis and traditional film and media studies methods of reading time-based media.

Deep Screens is a project that reverse engineers its methodology from two goals. The first is to answer a specific research question: can we use 3D pose analysis software to characterize differences in acting and cinematography styles across different facets of a large video corpus drawn from U.S. commercial film and television in the 20th century? Our corpus consists of over 900 curated titles spanning U.S. cinema and television history from the 1890s to the late 1970s. This research question necessarily engages technical understandings of how pose lifting models operate, conceptual questions around what it means to delineate a performance style, operational methods related to legally extracting, cataloging, and preparing a corpus of commercial moving images, and analytic tools needed to make sense of large-scale numeric data. We have found the research infrastructure needed to study these questions largely underdeveloped and will report on our experiences building it as well as best practices others can adopt when approaching such scholarly projects.

The curation of the film and television corpus to be studied takes advantage of many contemporary advances in expanding the available subject matters of U.S. cinema and television history. DVD and Blu-Ray releases in the past 15 years have revealed dedicated archival and scholarly collaborative pursuits to address many of the imbalances and repressed histories of the U.S. canon (and sub-canons), especially in developing surprising and detailed collections with intersectional emphases: large collections of films directed by women, or films intended for the segregated audiences that characterized exhibition practices for many decades, etc. As a result, our collection emphasizes the broad and uneven qualities of both High and Low-budget production practices through the most elaborate and expensive productions; early experiments with cinema as a series of “attractions”; the slow and uneven development of performance styles that moved from rigid but expressive theatrical traditions toward more naturalistic styles; the equally slow and uneven development of narrative forms that gradually internalized story-telling within the development of new experiments in mise-en-scene; the grandeur of cinematic form on late silent cinema; the loss of that grandeur in the development of early sound cinema; and the gradual success in the development of multi-genre classical cinema, always inflected by lower budget experimentation plus the essential era of “minority cinemas” (African-American, Yiddish, LatinX, etc. Early television was developed with an entirely different visual regime in scale and technology, and was additionally a critical context for the rise of Method Acting (which soon grew in significance in an era of mature themes and wide-screen spectacle in the cinema). Our final segment of the historical overview includes both classical and newly experimental studio productions plus the emigration of B-production Hollywood film units into production for network television Other selections include representatives of the decline of the Studio system, dramatic shifts of thematic and censorable material in post-WWII cinema, and the growth of post-classical cinema in the 1960s and late 1970s extensive to the LA Rebellion movement that arose at the UCLA film school.

The second question is one of communication: traditionally, quantitative analysis of large bodies of data in digital humanities get represented by abstractions like graphs. Representing the deeply expressive content of narrative film using raw numbers or abstract visualizations like graphs is a lossy form of communication that does not allow students and scholars to closely interpret a statistical summary in the same way they would interpret the film that statistical summary describes. Visualizations built into off- the-shelf pose analysis generally show stick figure overlays atop the original videos, both flattening 3D data back into a 2D representation and tying it to the original copyrighted material, limiting its ability to be redistributed. Newer forms of visualization common to multimodal generative AI lack the ability to show semantically meaningful data, such as joints and bones connecting those joints, because the models themselves operate on mimicking blobs of pixels it has labeled as human rather than understanding the physical properties of a human body.

To address these two questions, the Deep Screens project has developed methods and tooling needed to a) extract corpus videos from physical media under the provisions of the text and data mining exemption to the US Digital Millennium Copyright Act, and store those videos securely; b) analyse large numbers of videos using open source cut detection and pose analysis packages such as mmpose and pyscenedetect in a Slurm-based high performance computing environment; c) merge descriptive film-level metadata with extracted 2D and 3D pose data; d) optionally import 3D pose data into Autodesk’s Maya to manually clean specific sequences and export the cleaned data back into the original pose format; and e) provide scholars the option to see this pose data in a 3D environment that allows scholars and artists to move around figures (visualized data avatars) in either a desktop environment or virtual reality. While data analysis related to understanding the stylistic fingerprints of specific facets of the corpus is ongoing, we will discuss the methods used in this pipeline and how it may be generalized to other digital humanities studies.

A key consideration in developing this technology stack is its dependence upon models trained on generic video data, which is quite different than the video found in a narrative film and television corpus. For legal and ethical reasons, the research team did not train any new machine vision models for this study. Using pre-trained models such as rtmpose-wholebody and videopose-lift, though, meant accepting the limitations of models not designed to understand dynamic camera movement, artistic cinematography, and frames that represent human figures in ways ranging from a single body part to huge crowds wearing period costumes. Pre-trained models introduce other limitations for further research in this direction as well: for example, testing to integrate object recognition into the pipeline showed anachronistic results exposing the disconnect between model training and application when models identified objects like iPhones on screen in films shot decades before these devices were invented.