Storing memory inside an AI agent's reasoning loop, rather than querying it once per turn, is fast enough to work - if you keep the store in-process.
Researchers studying language agents found that the standard architecture - observe, reason, act, query a remote store - is the problem, not the loop itself. Networked memory stores answer in tens to hundreds of milliseconds, and in-loop retrieval at those speeds inflated end-to-end latency by up to 83x in their tests. The fix they propose: move the store in-process, where it answers in roughly 100 microseconds - three orders of magnitude faster. At that speed, agents in their experiments went from making 7.2 redundant actions out of 12 (at a 110ms cloud round-trip) to zero redundant actions. Recall improved from 0 out of 5 to between 3.6 and 4.8 out of 5. The store itself never dropped a write across 244 operations; every miss was the agent's own read policy failing, not the store.
The finding reframes a long-standing tradeoff. Prior work treated retrieval latency as a fixed cost to schedule around - hide it, ration it, batch it. This paper treats latency as a function of store placement, which is a much cheaper thing to change. The authors also identify where the real bottleneck moves once you fix the store: embedding, which at network speeds costs 200-400ms per step. A small local embedder brings the full operation down to roughly 40 microseconds.
Worth noting: an alternative baseline - simply instructing the agent to restate everything every reply - also achieved perfect recall, but at a token cost that grows with the working set. That tradeoff will matter a lot at scale, which the paper leaves as an open question.