A new research technique slashes token usage in multi-step AI search pipelines by nearly two-thirds — without retraining anything.
Multi-step retrieval-augmented generation, where a model repeatedly searches for documents and reasons over them to answer complex questions, has a compounding cost problem. Each round piles on more text — previous documents, previous reasoning traces — so the total input fed to the model grows roughly as the square of the number of rounds. That quadratic blowup hits directly in API-billed deployments: more tokens in means higher per-request cost, more network payload, and slower responses. ConCise, a protocol proposed in a new paper, attacks the problem by replacing that raw-text accumulation with a compact, append-only chain of structured conclusions. Instead of carrying the full prior context forward, each round hands off only a distilled summary. The result is linear growth instead of quadratic, and the authors add a mechanism to emit both reasoning and conclusions in a single API call rather than two serial ones. Across twelve test configurations spanning three models and two datasets, ConCise averaged 64.63% token savings while keeping accuracy in an acceptable range.
The practical value here is in what ConCise does not require: no GPU-level cache access, no pretrained compression modules, no changes to the underlying model. That matters for the growing share of AI services built on third-party APIs, serverless runtimes, or edge deployments, where you cannot touch the model internals even if you wanted to. Token costs on hosted APIs are not trivial at production scale, and any method that cuts them by more than half without a fine-tuning bill attached deserves attention.
The accuracy caveat is real — "acceptable" is doing some work in that sentence — and the technique has only been tested on two datasets, so real-world headroom will vary. Still, as a drop-in layer over existing RAG frameworks, it competes on terms that most deployed teams actually care about.