Daejeon, July 27–31
Digital humanities research heavily relies on structured data extracted from large bodies of previously released academic documents (Borgman 2015), and the spread of open science has reinforced the demand more broadly (Knöchelmann 2019). Documents can be digitized with document structuring tools such as GROBID, which can produce structured data for born-digital documents at acceptable quality. However, digital humanities research commonly targets documents that are not born-digital, causing GROBID to generate inaccurate data that contains textual errors, misplaced footnotes and incorrect recognition of figures and captions.
This paper proposes a method for producing high-quality structured data of academic documents by utilizing an AI-powered document analysis tool and a multimodal LLM. The proposed method utilizes the LLM to detect in-text references to figures and footnotes from the text extracted by the document analysis tool. Experiments on publications related to the Giza Pyramids demonstrated that the proposed method produces structured data with high fidelity even from non-born-digital documents.
Multiple document structuring tools have been developed for parsing born-digital academic PDFs, including GROBID, CERMINE (Tkaczyk et al. 2015), PdfAct (Bast and Korzen 2013), and Science Parse. GROBID, built on CRF and deep-learning models trained on scientific articles, is the most widely adopted of these and underpins large-scale born-digital scholarly corpora such as S2ORC (Lo et al. 2020) and CORD-19 (Wang et al. 2020). However, the existing tools generally struggle to analyze complex layouts and document structures; the benchmark by Meuschke et al. (2023) reports that even the best-performing tools achieve an F1 score of no more than 0.5 in extracting figure captions and 0.2 in extracting footnotes.
Beyond the works specialized in academic documents, document analysis services such as Azure Document Intelligence https://azure.microsoft.com/ja-jp/products/ai-foundry/tools/document-intelligence
Figure 1: Overview of the proposed method
This paper proposes a method for extracting structured data from academic documents by utilizing Azure Document Intelligence, an AI-powered tool for OCR and document layout analysis, along with a multimodal LLM. Figure 1 shows the overview of the proposed method.
The proposed method first uses Azure Document Intelligence to obtain the text content and structural information from the PDF of documents. Azure Document Intelligence analyzes the layout of a given document to extract the raw text, structure such as sections and paragraphs, and areas of figures and captions. The extracted areas of images are clipped from the pages and saved separately.
The proposed method converts the output of Azure Document Intelligence to TEI XML format. However, unlike GROBID, references to figures or footnotes are not extracted by Azure Document Intelligence. Therefore, the proposed method utilizes an LLM to automatically detect the references from the text of each paragraph. Tags are then attached to the paragraph text at the points where the references were detected to store the information of references.
We implemented a system that extracts the structured data of documents with the proposed method using Gemini 3.1 Flash Lite, a state-of-the-art multimodal LLM developed by Google, with the thinking level set to high and the temperature to 2.0. As part of the Digital Heritage Project https://dhss.nagoya-u.ac.jp/en/projects/digital-heritage-project/
The implemented system successfully produced the structured data from all 21 tested documents, while GROBID has been reported to fail to process up to 10% of input PDFs (Lo et al. 2020; Wang et al. 2020). Moreover, the extracted data was highly accurate. In a subset of four documents, the implemented system achieved precision of 0.86 (195/227), recall of 0.95 (195/206) and F1 score of 0.90 for detecting in-text references to figures and footnotes. These results indicate that the proposed method can produce high-quality structured data even from non-born-digital documents.
This paper presented a hybrid method for producing high-quality structured data of academic documents that utilizes a document analysis tool for OCR and layout analysis and a multimodal LLM to detect in-text references to figures and footnotes. An experiment on 21 publications about the Giza Pyramids showed the robustness of the proposed method on non-born-digital documents, successfully processing all 21 documents and achieving an F1 score of 0.90 for in-text reference detection. Future work includes extending the evaluation to larger collections of publications in multiple languages and automating the detection and correction of residual errors in the output of Azure Document Intelligence and LLM.