Most LLM memory benchmarks are grading the wrong thing.
A paper published on arXiv introduces PrecisionMemBench, an 89-case benchmark built specifically to measure retrieval precision rather than answer quality. The researchers argue that existing evaluations let a system "pass" simply by dumping its entire memory store — high recall, terrible precision, no penalty. Tested across 13 providers, baseline memory configurations clustered at precision scores of 0.22 and below, and failed to complete even half of the active retrieval passes. The paper also introduces Tenure, a structured belief-store proxy that resolves what memory to consult before the model ever sees the prompt, removing model-side discretion entirely. Tenure achieved perfect retrieval passes across all active, non-session, and session test cases in the benchmark.
The finding matters because precision failures are operationally invisible under current evaluation norms — a model can surface the wrong memories, dilute its context with noise, and still return a plausible-sounding answer. The researchers also flag that multi-turn topic drift compounds retrieval noise and drives up latency and cost, two metrics that answer-quality benchmarks never capture.
The memory retrieval problem is not new — RAG systems have wrestled with semantic proximity collisions since the embedding era began — but formalizing it as a precision benchmark, separate from generative quality, is a useful step toward holding vendors accountable for a failure mode they currently have little incentive to advertise.