Summary

An open-source, scalable framework for algorithmic due diligence is presented to help libraries meet EU AI Act copyright obligations for LLM training datasets. It uses a two-stage pipeline, 64-bit SimHashand MinHash LSH approximate Jaccard search, to treat high-overlap textual reuse (near-duplicates) as an auditable compliance signal. Evaluated on 19,816 books, three-word shingles with SimHash ≤29 and Jaccard ≥0.28 achieved precision 0.988 and recall 0.700. A production run over 91.7 million documents flagged 581 items in ~64 minutes. This lexical similarity method is positioned as one layer in a compliance stack, complemented by provenance and rights reservations, emphasizing deterministic thresholds for regulatory defensibility.

Introduction

The European Union Artificial Intelligence (EU AI) Act (Regulation EU 2024/1689) entered into force in August 2024 and applies on a staggered timeline. Key obligations for providers of general-purpose AI (GPAI) models took effect in August 2025, including having policies to comply with EU copyright law and mechanisms to respect machine-readable rights reservations (European Commission 2025a). In July 2025, the European Commission published the voluntary GPAI Code of Practice, offering a framework for demonstrating compliance with Article 53 (European Commission 2025b).

As frontier LLMs exhaust much of the text available online, national libraries and archives are increasingly viewed as valuable data sources. Yet these institutions operate within strict legal and copyright regimes, often supported by legal deposit laws that allow them to hold protected works. Using such collections for AI training typically requires agreements and permissions, but it also positions libraries as influential actors because they already steward large bodies of legally acquired material. At the same time, building LLMs can turn national libraries into AI model providers under the EU AI Act. Compliance becomes even harder when library holdings are combined with internet crawls, since third-party material is outside institutional control.

In this context, national libraries need practical ways to build training datasets while performing copyright due diligence at scale. This paper presents an open-source infrastructure to support that goal through scalable, auditable detection of high-overlap textual reuse. We focus on near-duplicates and substantial copying as signals that are both technically tractable and legally salient, while recognizing the limits of purely lexical methods.

RelatedWork

Copyright detection in training data is a growing challenge at the intersection of digital humanities, computational linguistics, AI, and legal compliance. Classic text-similarity methods struggle at the scale ofbillions of documents, while overly aggressive filtering can wrongly remove legitimate public-domain material. Although near-duplicate detection has been widely used for web-scale deduplication in search (Manku et al. 2007), it has not been systematically assessed for copyright-compliance contexts, where both precision and recall have legal consequences.

Recent work has approached AI and copyright from several angles: Li et al. (2023) studied verbatim memorization via reproduction experiments; Henderson et al. (2024) tested whether models respectcopyright notices in user inputs; and De la Rosa et al. (2025) showed that copyrighted, high-quality Norwegian texts materially improve model performance on complex tasks, raising the stakes for effective, defensible filtering as regulation tightens.

The EU AI Act (European Parliament and Council 2024) introduces requirements around copyright-related risk management in training pipelines, and the General-Purpose AI Code of Practice (European Commission 2025), adds expectations such as honouring machine-readable rights reservations, excluding judicially identified piracy sites, and implementing due diligence. Yet it offers limited technical direction on how to detect copyrighted content in large corpora. Our work addresses this gap with an open approach designed to balance compliance needs with computational feasibility.

Methodology

We implement a two-stage pipeline that balances accuracy with computational efficiency. Using over 500,000 in-copyright books from the National Library of Norway (De la Rosa et al., 2024), where rights status is defined by formal agreements with rights-holder organizations, we first compute 64-bit SimHash fingerprints over word-level shingles to rapidly surface candidate near-duplicates via Hamming distance (Charikar 2002). We then apply a MinHash LSH index with 128 permutations to support sub-linear approximate Jaccard similarity search (Broder 1997), followed by exact scoring on the filtered candidates. Batch retrieval is accelerated with FAISS, cutting query time by roughly 50 times versus sequential scanning (Johnson et al. 2019; Douze et al. 2024).

We evaluate using 19,816 books: 11,852 copyrighted and 7,964 public-domain controls to benchmark specificity. The corpus is mainly Norwegian with additional European languages and English, and includes realistic noise from variable OCR quality. We sweep SimHash Hamming thresholds (0–29), Jaccard thresholds (0.00–0.99), and shingle sizes k∈{3,5,7}. Results are reported with MCC (primary) plus precision, recall, specificity, and F1, alongside ROC AUC and PR AUC using SimHash-gated Jaccard as a continuous score. Threshold selection emphasizes high specificity with competitive recall, reflecting a compliance-first preference for avoiding unlicensed copyrighted retention.

Results

Our evaluation reveals that shingle size is the dominant driver of detection performance. Table 1 presents results at our primary operating point for three-word shingles (k=3) with SimHash threshold ≤29 and Jaccard threshold ≥0.28, which provide the best recall and MCC scores for our evaluation set. This configuration achieves MCC=0.681, precision=0.988, recall=0.700, and specificity=0.987, correctly flagging 8,296 copyrighted documents while generating only 105 false positives from the public-domain controls.

Table 1. Primary operating point (books; k=3; SimHash≤29; Jaccard≥0.28).

The superiority of shorter shingles becomes evident when comparing optimal operating points across different k values (Table 2). While five-word and seven-word shingles achieve marginally higher precision (0.994 and 0.996 respectively), they do so at substantial cost to recall, detecting only 52.3% and 40.2% of copyrighted documents. The MCC values decline correspondingly, to 0.547 for k=5 and 0.458 for k=7.

Table 2. Best operating point per shingle size k (books dataset).

Figures 1 through 4 visualize these trade-offs. The ROC curve (Figure 1) and precision-recall curve (Figure 2) for k=3 demonstrate strong discriminative performance, with ROC AUC=0.847 and PR AUC=0.892. Figure 3 illustrates the explicit trade-off between recall and specificity as Jaccard threshold varies at fixed SimHash threshold, making the parameter space transparent for policy decisions. Figure 4 confirms that three-word shingles consistently achieve the highest MCC across the parameter space we explored.

Figure 1. ROC curve (books; k=3; SimHash-gated Jaccard score).

Figure 2. Precision-Recall curve (books; k=3; SimHash-gated Jaccard score).

Figure 3. Operating-point tradeoff (recall and specificity vs Jaccard threshold; k=3; SimHash=29).

Figure 4. Best MCC by shingle size k.

Computational performance remains practical even at scale. Table 3 reports wall-clock times for fingerprint generation and similarity search across the evaluation corpus on a 96-core machine with 1TB of RAM. Processing time increases modestly with shingle size but remains tractable for all configurations tested.

Table 3. Runtime (books dataset; wall time from logs).

To validate specificity estimates beyond our controlled evaluation set, we applied the k=3 operating point to ten presumed-lawful corpora where any flagged document can be interpreted as a false positive. Table 4 presents flag rates for these, including web data crawls (FineWeb2), legal documents (Lovdata), government documents (Målfrid), manuscripts (Digimanus), Sámi language crawls (Sámi Web), code (StarCoder), math (Megamath), and Wikipedia, providing complementary evidence for the behavior of the system on materials unlikely to contain copyrighted books.

Table 4. Flag rate on presumed-lawful corpora at selected operating point (k=3; SimHash≤29; Jaccard≥0.28).

Large Scale Deployment

To demonstrate production readiness, we conducted a large-scale filtering as part of an existing dataset building pipeline run across multiple heterogeneous sources, including a collection of PDF documents crawled from the Internet (FinePDFs).

https://huggingface.co/datasets/HuggingFaceFW/finepdfs

Notably, most flagged matches in this run originated from the FinePDFs source, despite the input spanning multiple corpora. This pattern illustrates both the system's selectivity and the heterogeneity of copyright exposure across different data sources. The wall-clock time of approximately 64 minutes translates to a throughput of roughly 23,965 documents per second, demonstrating that fingerprint-based filtering can operate at scales commensurate with modern pre-training pipelines.

Table 5. Large-scale filter run across multiple sources.

Discussion and Implications

At our operating point, recall is ~70%, meaning lexical fingerprints alone miss about 30% of copyrighted books, expected for methods tuned to near-duplicates and verbatim reuse rather than paraphrases, translations, or heavily edited derivatives. We therefore position similarity detection as one layer in a due-diligence stack, complemented by provenance filtering, exclusion lists (rights-holder opt-outs), machine-readable rights reservations, and human audit sampling. Copyright detection alone does not replace the need for defensive mechanisms against LLMs memorization during training.

Using three-word shingles improves robustness to OCR noise, spelling variation, and small edits, with minimal loss in specificity (false positives remain <1.3%), making the trade-off attractive for cautious compliance. We also favor deterministic fingerprints, explicit thresholds, and audit trails for regulatory defensibility: unlike embedding-based methods, they support clearer explanations of why a document was included or flagged.

Conclusion

We introduce a scalable, auditable copyright-detection pipeline for EU AI Act–aligned due diligence. On a rights-managed book corpus with public-domain controls, three-word shingles enable a transparent high-precision and high-specificity setting (0.988/0.987) while flagging ~70% of copyrighted works, and deployment across 91 million documents demonstrates production-scale throughput.

Next steps include evaluation on broader multilingual and web-scale corpora, targeted tests for partial-copy detection (chapters/excerpts), and selective semantic similarity for borderline cases. Sustained compliance will require collaboration across technical, legal, rights-holder, and cultural-heritage communities; we offer these tools as shared, open-source building blocks for responsible AI development.

Keywords: copyright detection, EU AI Act, text similarity, SimHash, MinHash, FAISS, training data compliance, digital humanities infrastructure