Hugging Face's FinePDFs: A 3-Trillion-Token Dataset Revolutionizes AI Training with PDF Power

Hugging Face, the leading platform for sharing and deploying machine learning models, has unveiled FinePDFs, a groundbreaking dataset poised to reshape the landscape of large language model (LLM) training. Comprising a staggering 3 trillion tokens derived from 475 million PDF documents across 1,733 languages, FinePDFs represents the largest publicly available corpus of its kind. This monumental undertaking opens a vast new reservoir of information for AI development, tapping into a resource previously considered too complex and costly to process effectively.

The sheer scale of FinePDFs is immediately impressive. At 3.65 terabytes, the dataset dwarfs many existing LLMs' training data. While most existing datasets rely heavily on HTML scraped from the Common Crawl, a massive web crawl project, FinePDFs leverages the unique qualities of PDF documents. PDFs often contain higher-quality, more structured information, particularly in specialized fields like law, academia, and technical documentation—areas often underserved by web-based datasets.

However, the inherent challenges in processing PDFs have historically hindered their widespread use in LLM training. Many PDFs contain embedded text, requiring sophisticated extraction techniques. Others necessitate Optical Character Recognition (OCR) to convert scanned images into text. Inconsistent formatting adds further complexity. Hugging Face's solution was a multi-pronged approach, combining the text-based extraction tool Docling with the GPU-accelerated OCR engine RolmOCR. This dual strategy, coupled with deduplication, language identification, and personally identifiable information (PII) anonymization, allowed them to process documents at scale while maintaining data quality.

The linguistic diversity of FinePDFs is equally remarkable. While English dominates with over 1.1 trillion tokens, a significant number of tokens are contributed by Spanish, German, French, Russian, and Japanese (each exceeding 100 billion tokens). Remarkably, 978 smaller languages are also represented, each with more than 1 million tokens. This inclusivity promises to improve the capabilities of multilingual LLMs, potentially bridging the gap in resource availability for languages with less digital content.

To validate the effectiveness of FinePDFs, Hugging Face trained 1.67 billion parameter models on subsets of the data. The results demonstrated that FinePDFs performs competitively with SmolLM-3 Web, a state-of-the-art model trained on a comparable HTML-based dataset. Critically, combining FinePDFs and SmolLM-3 Web yielded a noticeable performance boost across various benchmarks. This underscores the complementary nature of the knowledge contained within PDFs and web data. The specific evaluation metrics, as noted in online discussions, focus on probabilities of correct choices across different benchmarks rather than a single, easily summarized score. This nuanced approach suggests a more comprehensive and robust evaluation framework.

The release of FinePDFs has generated significant interest within the AI community. The dataset's potential for advancing long-context training is particularly exciting, as PDF documents often contain significantly longer passages of text than typical web pages. This characteristic could lead to significant improvements in LLMs' ability to handle complex, multi-paragraph contexts.

Furthermore, the transparency surrounding FinePDFs' creation is a notable achievement. Hugging Face has not only released the dataset but also meticulously documented the entire processing pipeline, from OCR and text extraction to deduplication. This level of openness fosters reproducibility and trust within the research community, setting a new standard for data transparency in AI.

FinePDFs is available under the Open Data Commons Attribution license, promoting open access and fostering further research and development. Its availability on the Hugging Face Hub, accessible through their datasets, huggingface_hub, and the Datatrove processing library, simplifies access for researchers and developers worldwide. The creation of FinePDFs marks a significant milestone in open-source AI, demonstrating the potential of large-scale datasets to propel innovation and democratize access to advanced AI technologies. The impact of this resource on future LLM development and its contribution to the broader field of artificial intelligence research will undoubtedly be significant.