A new reinforcement learning method aims to fix a basic flaw in how AI agents learn from multi-step tasks.
Group-based RL methods like GRPO have become the standard way to train reasoning and agentic large language models, and they work well when a task is a single response scored all at once. The problem shows up in multi-turn agent work, where GRPO judges an entire trajectory as good or bad rather than crediting individual steps. That means a failed run's one smart decision gets punished along with its mistakes, and a lucky win can reward steps that had nothing to do with the outcome. A new paper proposes GRAFT, a graph-based framework that stitches every sampled rollout into a single trajectory graph, recovers state values with Bellman iteration, and assigns credit to each step from the value difference between connected nodes. The authors also extend generalized advantage estimation to that graph structure, calling it Graph GAE, to cut down on noisy value estimates.
This matters because step-level credit assignment is the unglamorous plumbing behind every agent that calls tools, browses the web, or writes and runs code across multiple turns. Get it wrong and training either ignores good moves buried in bad runs or rewards good luck, both of which slow down or destabilize learning. As labs push agents into longer, tool-heavy workflows, getting per-step credit right matters more than another round of scaling.
The paper reports consistent gains over GRPO and other recent agentic RL methods on multi-turn benchmarks, though the code is only promised on GitHub, not yet published, so outside verification will have to wait.