This paper presents the ongoing development of a digital tool to analyse the changing use of the phrase ‘human rights’ in English-language historical newspapers. Combining relative frequency measures, named entity recognition, and qualitative analysis, it offers a more precise, reproducible account of temporal and spatial variation in human rights discourse. The language of human rights has been described as ‘the lingua franca of moral thought’ in the ‘West’ (Ignatieff 2001). In the field of international politics, scholars have argued that human rights have become the dominant moral framework for international norms and transnational advocacy (Keck and Sikkink 2014; Sellars 2002; Thomas 2001). A semantic analysis suggests that the phrase ‘human rights’ today has a unique ‘symbolic force’ (Neves 2007). Many historians of human rights have argued that the ubiquity of human rights discourse is a relatively recent phenomenon, one that took shape long after the adoption of the 1948 Universal Declaration of Human Rights. They disagree, however, on which decade of the twentieth century marked the moment of ‘take off’ or ‘breakthrough’ (Eckel and Moyn 2014; Hoffmann 2016; Jensen 2016; Keys 2014). Their findings have been arrived at through a close reading of historical sources, with quantitative analysis largely limited to tracking changes over time in the raw frequency of the phrase ‘human rights’ in a single corpus (Moyn 2010). Recently, scholars outside of the historical discipline have applied advanced quantitative methods to study human rights discourse in corpora of human rights treaties (Law 2018) and Google search data (Dancy and Fariss 2024), but without engaging in a temporal and spatial analysis of variations since the 1940s.

A new approach is required to mapping fluctuations in the frequency of the phrase ‘human rights’ in a nuanced, precise, and systematic fashion across a large time scale. To this end, we have created a bespoke digital tool using Clarivate’s TDM Studio (Clarivate 2024). This platform is designed to enable researchers to carry out textual and data analysis on the full text of historical newspapers, magazines and other publications, and then export the results of this analysis. However, working in the Python notebook environment provided by TDM Studio requires coding skills, and is not suitable for exploratory data analysis.

In this paper, we describe an approach to working with digital archives of this kind which separates the data analysis and cleaning phase from exploration of the resulting aggregate dataset. A dataset of 29,243,734 texts from the New York Times and the Guardian in the time period 1923-2021 was filtered to 132,814 news articles which contained the phrase ‘human rights’ using the ProQuest dataset definition interface, and a set of Python notebooks were developed which used the natural language processing library spaCy 3.8.6 (Honnibal et al. 2020) to perform named entity recognition on the full text of the articles with the model en_core_web_sm (v3.8.0) and write the results of this analysis as a compressed dataset which could be exported from TDM Studio. The compressed dataset contains key metadata for each article (title, ID, date, authors and article type) and lists of the geopolitical and organisational entities which are mentioned in the article. Named entity recognition was chosen as an analysis technique which could be run on the limited computational resources available in TDM Studio, and which produced a dataset which could be exported within the size limits imposed by the platform.

We then developed a web interface to the dataset using Observable Framework. (Observable, 2024). Observable is a toolkit for developing interactive data science visualisations which allows rapid prototyping without any requirement for a dedicated backend server. The website allows a researcher to explore the dataset by selecting data ranges, entities, groups of entities such as major transnational NGOs, and then viewing a graph showing the number of articles or percentage of total articles per year for the required period.

Because the raw text in ProQuest is based on OCR from source documents of variable quality, the named entity recognition dataset required a semi-automated cleaning stage using OpenRefine, and automated deduplication was carried out for articles which appeared multiple times in the source databases under different ids. The mappings from raw entity names to cleaned values and removed duplicates were recorded as reference CSV files. OCR error rates are estimated by checking raw entity names against dictionaries and geospatial databases.

The web interface includes a diagram showing the data flows from the original ProQuest databases, through the named entity recognition notebooks, data cleaning and pre-processing scripts and the final dataset used in the Observable data visualisation tool. This diagram includes links to the source code used in the different stages, and a machine-readable file using the RO-Crate standard for describing research data workflows, allowing the process used to arrive at the final data product to be reproduced or re-used with a different base corpus of articles or alternative text-mining techniques, such as vector embeddings.

The results are thus far promising. As our corpora comprise two newspapers, we have been able to engage in an intertextual and cross-national comparison of trends. Rather than calculate absolute frequency, we have calculated the relative frequency of ‘human rights’ in each corpus to account for changes in the total number of news items in each corpus over time. Then we have used named entity recognition (Kim and Cassidy 2015; Neudecker 2016) to compare how the frequency of the phrase ‘human rights’ differs according to the country, international organisation, or transnational advocacy group featured in a news item. Finally, we have coupled this quantitative analysis with a qualitative analysis of individual news items that further contextualises each of the trends. Emerging from this preliminary analysis is a more accurate and differentiated picture of how the incidence of the phrase ‘human rights’ has changed from the early Cold War to the present. We aim to release an open access version of our tool that will serve as a model for an analysis of the use of other phrases in corpora of newspaper text.