AI models keep a running scratchpad of everything they've read, and a new study says trimming that scratchpad works far better when each layer of the model gets to pick its own rules.
Large language models store a running record of the conversation, called a KV cache, so they don't have to reprocess earlier text with every new word. When that cache gets too big, models evict entries based on a chosen rule, and until now, researchers typically picked one rule and applied it to every layer of the model. A team tested what happens when different layers use different eviction rules, building fixed routes based on simple offline profiles of each layer's behavior. Running the mixed setups on the LongBench benchmark, the researchers found this layer-by-layer mixing beat single-rule setups on most tasks at the same memory budget.
This suggests the eviction rule itself is not just a knob to tune once and forget. It is a design space with real headroom left on the table. A profile-guided placement ranked second out of 100 tested combinations and beat the best single-rule baseline across all five cache budgets tested, meaning teams running long-context models could get meaningfully better compression without new hardware or bigger caches.
That's a solid engineering result, not a new architecture, so don't expect it to show up as a marketed feature anytime soon, just quietly better memory management under the hood.