A new benchmark reveals that today's best large language models fall apart on long-context reasoning once task difficulty is turned up.
Researchers introduced PredicateLongBench, a benchmark that tests models by asking them to find the longest contiguous word sequence in a long input that satisfies a given constraint — for example, words in lexicographic order. The key design choice is explicit: difficulty is scaled along multiple axes, so the researchers can watch model performance degrade rather than just measure a single average score. Two generation pipelines support the tests — one fully synthetic, one drawing from real documents while preserving their statistical properties. Crucially, the tasks require no LLM-based scoring, removing a common source of noise.
The practical finding matters because most existing long-context evaluations measure average-case performance — and many are already saturated, meaning top models score so well that the benchmarks stop distinguishing between them. PredicateLongBench is designed to avoid that ceiling by systematically probing where models break down, not just how often they get things right. That makes it a more useful diagnostic tool for understanding what frontier models still cannot do.
Benchmark saturation is a recurring embarrassment in AI evaluation: MMLU, HellaSwag, and parts of the original NIAH suite all became unreliable once models improved enough to cluster near the top. PredicateLongBench's axis-based difficulty scaling is a more principled approach — though whether labs will actually use it to guide training, rather than optimize scores on it, is the usual open question.