A new attention trick promises to make long documents cheaper to feed into large language models, without retraining anything.
The method, called RBS-Attention, targets prefill, the slow first step where a model reads an entire prompt before it can start generating a response. Researchers built two selection branches: one that scores average relevance across blocks of text, and a second rescue branch that catches high-relevance tokens hidden inside blocks that would otherwise look unimportant, a problem the paper calls mean dilution. On H100 GPUs, the method sped up standalone prefill attention by up to 20.65x and cut time-to-first-token by 5.97x on a 128K-token prompt using Qwen3-30B-A3B-Instruct-2507-FP8. On the dense Qwen3-32B model, RULER benchmark accuracy landed at 88.65 versus 89.52 for standard dense attention, with additional quality checks on LongBench-v2, InfiniteBench, and Video-MME.
Prefill cost is the reason long-context AI features feel slow and expensive, since attention math scales badly as prompts get longer. A training-free fix that slots into existing block-sparse FlashAttention pipelines matters because it does not require retraining or fine-tuning a model to get the speedup, so anyone already running these models could adopt it without extra cost.
The accuracy gap is small on paper, under one point on RULER, but it is still a gap, and this is a single arXiv preprint with no independent replication or public code release mentioned.