Most PDF datasets brag about their size in tokens, but a new analysis shows their internal stats are measured in something else entirely: documents. That mismatch hides a lot.
A study of CC-MAIN-2021-31-PDF-UNTRUNCATED, a 7.9 million-document, 32.6-billion-token web PDF corpus, found the text is wildly concentrated: 3.02% of documents hold half the tokens, and documents over 50 pages make up just 5.00% of the corpus but 53.53% of its text. PDFs built with a TeX toolchain are only 1.66% of documents yet 4.05% of the text. The bigger issue is Common Crawl's truncation cap, which affected 23.06% of documents but 63.08% of the text -- and when researchers tried to recover the missing content, two widely used extraction libraries pulled back only 11.4% and 1.4% of it, with 72% to 97% of affected documents yielding nothing at all. Even under the larger 5 MiB cap Common Crawl adopted in March 2025, 30.19% of tokens are still cut off, and recovery only improves from 3.3% to 13.2%.
Any dataset statistic reported per document -- coverage, OCR routing, language mix -- is quietly describing a small slice of the actual text, since a handful of long documents carry most of the tokens. For anyone building or auditing an LLM's training data, that means headline corpus sizes can overstate what's usable, and truncation isn't a rounding error -- it's more than half the corpus's text, gone.
The paper's fix is unglamorous: report stats in both units, documents and tokens. Not flashy, just accurate -- which turns out to be rarer than it should be.