Daejeon, July 27–31
Over the last decade, bibliometrics and citation network analysis have emerged as key methodological pillars in the digital humanities for understanding the evolution of scholarly communities, intellectual lineages, and disciplinary boundaries. Brughmans (2013) and Colavizza (2018) have already shown how computational methods of citation analysis can be applied to historiography, and there is emerging work as well on analysis of co-citations (Rossini 2024). Yet, despite the size, interdisciplinarity, and rich textual ecosystems of Jewish Studies, no large-scale, field-wide citation network has been constructed. This paper reports on a work-in-progress project that begins to fill that gap by building the first comprehensive citation graph of Jewish Studies scholarship from 1980 to 2023, using AI-assisted citation extraction from JSTOR and a multi-stage pipeline for data cleaning, canonical author identification, and community detection. An additional unusual feature of this project is our use of the Resource Description Framework for modeling our data. While the project aims to model the intellectual structure of Jewish Studies as a whole, this abstract focuses on our extraction methods and data ontology.
The dataset consists of 15,342 research articles culled from 29 selected journals in Jewish studies and adjacent fields. The journals over the years 1980-2013 and were obtained from JSTOR (JSTOR) in JSON format. One limitation of the data is that not all of the important journals in Jewish studies were available in digital form, due both to issues of digitization and licenses. We also confined ourselves to articles written in English. As is common in humanities corpora, the files display high heterogeneity: OCR noise, inconsistent citation conventions, multilingual content, and varied formatting (Kuster 2011).
Rather than reducing citations to node-to-node links, we modeled the dataset as RDF triples in order to support richer analysis (Raji and Surendran 2016). This structure encodes dates, publication types, venues, and distinctions between primary- and secondary-source citations. While RDF is well suited to representing complex scholarly relationships, it also introduces challenges of modeling and implementation. We originally planned on using the FaBiO and CiTO ontologies, which were designed to model bibliographies (FaBiO) and how citations were being used (CiTO). The latter turned out to raise too many difficulties to be useful so we based our ontology on FaBiO alone (Peroni and Shotton 2012).
There are two parts to our pipeline. The first uses Llama -3.3-70B (Grattafiori et al. 2024) run locally, per our licensing agreement with JSTOR, to produce structured citation records. The second pipeline cleans and links author names from messy bibliography data to structured identifiers using a multi-set approach. The heart of that approach is using calls to OpenAlex (Priam et al. 2022). The two pipelines (Figures 1 and 2) are detailed below.
For our benchmark for the first pipeline (LLM extractions) we used manually produced benchmarks from 15 articles across seven distinct journals. This yielded 660 ground-truth citations (secondary sources only) which we then compared to 677 LLM-extracted citations. For the data that we passed into our second pipeline, which included only author and title, we had a precision score of 0.9461 and a recall of 0.9303 for an F1 of 0.9349.
The extractions are still in production, and so we have only preliminary estimates. The first pipeline requires, per article, about 35,000 prompt tokens and about 5,000 generated tokens. Since these run on a local machine there is no direct cost to us. The second pipeline will require about 1.5M OpenAlex and 400K API calls to Serper.dev. The costs for these services are still being negotiated.
When the extractions are completed, they will be converted to RDF triples. The data will then be imported into a platform like Neo4j or GraphDB for analysis. Using Community Detection algorithms, the network will be divided into groups and manually investigated for similarity. These groups will then be subdivided into subfields. The goal is to emerge with a quantitative description of the network of modern Jewish studies fields, scholars, and journals. Change over time and “hidden” links between scholarly clusters are especially to be explored. Measures of PageRank, betweenness centrality, and influence vs. bridging scores help illuminate these dynamics, supporting recent DH work on disciplinary cohesion and fragmentation. These results can then be used in wider studies of how Jewish studies developed and how academic knowledge more broadly is transmitted.
We hope also to expand this network to non-English scholarship and books. Both of these domains have distinct challenges but will be important to complete a robust network.