A new benchmark shows AI language models can often tell you a topic exists in an old Czech document, but not always where.
Researchers built CzechTopic, a benchmark drawn from historical Czech documents, where human annotators marked text spans matching topics defined by a name and description. The dataset supports checking results at both the whole-document level and the individual-word level, and instead of grading against one "correct" answer, it measures models against how much multiple human annotators agreed with each other. The team tested a range of large language models against smaller BERT-based models that had been fine-tuned on a distilled development set. The dataset and evaluation code are public on GitHub.
The results split the field. Some LLMs got close to human-level agreement on identifying that a topic was present, but the same models often failed badly at localizing the actual span of text - the harder, more useful task for anyone trying to search an archive. Meanwhile, the fine-tuned BERT models, despite being far smaller, stayed competitive with the best LLMs.
That gap matters beyond one benchmark. It's a reminder that "finding the right idea in a document" and "finding exactly where that idea lives" are different skills, and that scale doesn't automatically buy you the second one. For archives and libraries digitizing historical text in smaller languages, a cheap fine-tuned model may still beat a general-purpose LLM on precision. Given the code is on GitHub, that's now a testable claim rather than a marketing line.