A new benchmark argues that most AI memory tests have been asking the wrong question.
Researchers released DolphinBench, a benchmark that judges an AI agent's long-term memory by task completion rather than by answering trivia-style questions. The dataset covers three knowledge-work personas, each with roughly 500,000 tokens of accumulated user messages, and 200 tasks per persona that require pulling relevant details from that history. Every task was verified by running an agent twice, once with access to the history and once without, keeping only tasks the agent solved with the history and failed without it. Submissions also have to report total cost and latency alongside accuracy. The dataset and evaluation code are posted at dolphinbench.ai.
That combination is the real point. Most memory benchmarks are built like a quiz: the question itself hints that a fact needs retrieving, and often which one, so a system can look competent just by pattern-matching the prompt. None of the existing benchmarks, per the paper, force accuracy, cost, and latency to be reported together, which has let some memory systems win by brute-forcing huge context windows or running expensive multi-step retrieval that would be impractical in production.
It's a sensible corrective, but it's also a paper on arXiv from the people who built the benchmark and host the leaderboard, so treat the framing as a pitch until independent teams run their own agents against it and the numbers hold up.