Calling an LLM on every document in a large corpus is expensive enough to make semantic filtering impractical at scale - a new paper proposes a faster path.
The research addresses a specific and growing problem: how to run a yes/no natural-language question over thousands of documents using a large model as the authority, without paying for an oracle call on every document. The dominant approach pairs an expensive oracle with a cheap proxy model that handles easy cases - a pattern called a cascade. The paper identifies four concrete flaws in how these cascades are currently built: each proxy family wins only on a narrow query range, the strongest proxies miss token-level evidence, training uses hard yes/no labels instead of the oracle's per-document confidence scores, and calibration applies safety margins uniformly rather than reserving them for sparse-sample cases. The proposed fix composes multiple proxy strategies adaptively, trains on soft labels derived from oracle confidence, and applies the margin only where training data is thin. On three 10,000-document corpora at a 90% accuracy target, the method runs 1.6x to 2x faster than the previous best approach and meets the target on 95% of queries.
As agentic AI pipelines grow longer and more document-hungry, filtering costs compound at every step. The researchers also derive a theoretical lower bound on oracle calls, suggesting there is still 4x to 20x of efficiency headroom ahead - which frames this result less as a ceiling and more as early progress in a problem the field has barely started to optimize.
No packaged library ships with the paper, but the techniques - soft label training, adaptive cascade composition, sparse-aware calibration - are specific enough that teams running retrieval-augmented generation or bulk data-processing pipelines could begin adapting them without waiting for someone else to do the packaging.