Researchers have found a way to make long-document AI queries cheaper without retraining anything or touching a GPU.
BudgetMem is a training-free system that trims long documents before they reach a language model, using plain, interpretable rules: how many named entities sit in a chunk, how important its keywords are, where it falls in the document, and whether it contains numbers or discourse markers like "however" or "therefore." That is a deliberate departure from tools like LLMLingua, which compress at the token level with a neural model. On synthetic, template-generated documents, BudgetMem discards 70% of the content and still matches the uncompressed baseline almost exactly (F1 0.859 versus 0.855). Used as a pre-retrieval filter on the same documents, it beats LLMLingua-2 by a wide margin (0.859 versus 0.554), because chopping text apart token by token destroys the phrase structure that retrieval depends on.
A single query against a 100,000-token document can already run past a dollar in API fees, and the bill grows with every token a system keeps in memory. A filtering step this cheap, running on a $10-a-month Colab instance with no trained models required, could meaningfully lower that cost for anyone building retrieval systems over long documents.
The catch is that real documents are not templates. On actual research papers from the Qasper benchmark, BudgetMem trails the uncompressed baseline by 7.4% and even LLMLingua-2 by 5.5%, a reminder that benchmark wins on synthetic text don't always survive contact with messier writing.