AI agents trained with reinforcement learning have a focus problem — and a new paper proposes a fix.
Researchers introduced Selective Trajectory-Aware Policy Optimization, or STAPO, a training framework designed to address what they call trajectory neglect: the tendency of RL-trained agents to lose sight of their original goal and task history at intermediate steps in a long sequence. The root cause, the paper argues, is that existing step-level supervision relies on standard Shannon entropy, which blurs together two distinct signals — how genuinely complex a situation is versus how uncertain the agent itself is. STAPO replaces that with normalized entropy, which measures an agent's confidence relative to its own average behavior in a given state, making it easier to flag the specific steps where things go wrong. Those flagged steps get targeted with a joint mechanism combining a trajectory-aware reward and a trajectory-independent penalty.
The distinction matters because long-horizon agentic tasks — think an AI navigating a simulated home, browsing a shopping site, or answering questions through search — are exactly where current RL methods struggle most. Getting an agent to remain coherent across dozens of steps is a harder problem than most benchmark scores let on. STAPO tested on ALFWorld, WebShop, and Search-Augmented QA, reporting state-of-the-art results on all three.
Those are established benchmarks, not production deployments, so the usual caveats apply — real-world tasks are messier and the gap between lab results and shipped agents has a way of widening.