A new study says the fancy planning prompts baked into AI agent systems matter less than a cheap sanity check bolted on afterward.
Researchers tested what actually makes agent harnesses work using retail and airline scenarios in the tau2-bench framework. They compared real, task-specific written plans against shuffled policy text of the same length, isolating whether the content of the guidance mattered or just its presence. Across 265 matched test cells, the real plans lifted verified task success by about 7 percentage points, with the biggest gains on harder tasks. Separately, a read-only verifier that checks outputs after the fact caught 61% of invalid retail episodes for under a cent of extra cost per episode, though it also wrongly rejected 17% of correct ones.
Which piece matters more depends on the cost of getting it wrong. When errors are cheap, detailed planning is the bigger lever. When erroneous acceptances carry high liability, the verifier's ability to catch false passes matters more, and a standalone verifier alone captures nearly all of that benefit for a fraction of the cost of running both systems together.
That's a useful data point for anyone building customer-service bots on a budget: before writing a 40-page policy document for your AI agent, try bolting on a cheap checker first.