A research paper out of arXiv proposes a more precise way to teach AI agents which of their decisions actually helped — and which quietly hurt.
The prevailing approach to self-distilled agentic reinforcement learning scores a completed trajectory with a single scalar: the gap between what a privileged teacher model preferred and what the student model did. CRAFT, short for Counterfactual Credit Assignment from Free Sibling Rollouts, argues that signal is too blunt in two specific ways. First, it only looks backward at what happened, never at the counterfactual paths the model did not take. Second, it has no sign — it cannot tell when following the teacher's preference would have made things worse. The method addresses both gaps with three components: a counterfactual token importance estimator that recycles rollouts already generated by the GRPO training algorithm, an asymmetric controller that trades off distillation weight against a KL-divergence penalty, and a token-level KL penalty that flips its behavior based on whether the credit signal is positive or negative.
The value here is specificity. Most reinforcement learning credit-assignment schemes operate at the trajectory level — the whole sequence wins or loses together. Pushing that signal down to individual tokens, without requiring extra model calls, is the kind of efficiency gain that makes a technique plausible at production scale. The authors also build in independent on-off switches for each component so any performance improvement can be traced cleanly to algorithmic change rather than incidental code differences.
The paper evaluates CRAFT across three agentic environments and four model scales, which is a broader sweep than most single-method papers bother with. Whether the gains hold up when the method meets real-world task distributions — messier and less instrumented than any lab benchmark — is the question reviewers will rightly ask first.