A new benchmark finds that AI agents are bad at knowing when to stop.
NAQD-Env is a synthetic environment built to test whether language agents respond correctly when evidence changes, permissions get revoked, or someone tells them to halt. The right response is selective: pause the affected task, keep unrelated work moving, and resume only once the problem is actually fixed. Researchers tested three open-weight models, Qwen2.5-7B, Qwen2.5-3B, and Llama-3.1-8B, across 350 frozen scenarios and three prompt styles, for 3,150 episodes total. Withdrawal recall topped out at 0.06, no episode showed correct resumption, and only one episode matched the reference policy in full.
That's a problem for anyone pitching agents as autonomous workers, since clean stopping and restarting is exactly the skill autonomy depends on. Supervised fine-tuning narrowed the gap on paper, pushing one model's decision accuracy from the 0.45-0.54 range up to 0.83-0.92, but it also taught the model to withdraw from tasks it shouldn't have and to stop reporting events altogether.
And this is the easy version of the test - the inputs are clean and trusted, so real-world stop signals, the kind an agent has to verify itself, would likely do worse.