A new arXiv paper describes a memory system that lets AI models hold onto long context without deleting anything that might matter.
The paper proposes Attention with Routed Memory, or ARM, a replacement for the key-value caching that large language models use to track everything they have read so far. Rather than growing that cache without bound or evicting old tokens to keep it manageable, ARM routes information into a fixed-size, hierarchical memory. A Gumbel-Softmax selects which memory slots to use, and sigmoid-gated updates blend new data with what is already stored instead of overwriting it. A separate learned policy decides how much memory to pull at inference time, using less for easy queries and more for tasks that require deeper reasoning.
Every long-context system runs into the same problem: keep the full history and the cache balloons, or prune it and risk losing something the model needed later. Most current fixes - token eviction, pruning - solve the size problem by discarding information. ARM's authors report it beats fixed-cache approaches on commonsense and long-context benchmarks, with better efficiency in both memory footprint and generation speed.
That is a benchmark result, not a production one - plenty of caching techniques look elegant in a paper before anyone ships them in a system serving real traffic.