Long-context AI benchmarks have been measuring the wrong kind of hard.
A paper posted to arXiv on September 25, 2026 (arXiv:2609.29245) introduces a formal way to size up how brutal a task actually gets as a document collection grows. The researchers call it Corpus Task Complexity, or CTC, and split tasks into two buckets: low-CTC jobs like simple retrieval, where difficulty scales linearly with corpus size, and high-CTC jobs like flagging every contradictory claim across a literature, where difficulty scales quadratically or worse. Most existing evaluations, the paper argues, have only ever tested the easy bucket. To fix that, the authors built CTC-Bench, a 22-task suite that adds 10 new high-CTC tasks, and are releasing the code and data alongside it.
The gap shows up exactly where it counts: efficient attention tricks like block-sparse and hybrid attention, often pitched as free scaling for long-context models, matched full attention on low-CTC tasks but degraded sharply once tasks got quadratically harder. That undercuts the industry pitch that cheaper attention mechanisms are a lossless shortcut for feeding models bigger documents. It also means benchmark leaderboards built mostly on retrieval-style tasks have likely been overstating how much long-context reasoning current models can actually do.
Full attention still wins on the hard stuff, but it does not scale, which makes "more context, more capability" sound less like a spec sheet and more like a marketing line.