A research paper proposes a new way to train large language models that gives credit more precisely — down to the individual token.
Reinforcement learning has become a standard tool for sharpening LLM reasoning, but it has a persistent flaw: when a model generates a long response, the reward signal typically gets spread evenly across every token in that output, regardless of which decisions actually mattered. Adaptive Credit Policy Optimization, or ACPO, tries to fix that. The framework assigns credit at the token level using what the authors call a "mode-local surrogate entropy" — a measure of how confident the model was at each step. ACPO upweights uncertain decisions in successful outputs and penalizes overconfident tokens in failed ones, nudging the model to learn from the choices that genuinely drove the outcome.
Token-level credit assignment is not a new problem, but most production-scale RL methods still punt on it, defaulting to trajectory-level rewards because the math is cleaner. ACPO's surrogate approach threads a needle: the authors show it preserves the correct gradient direction without introducing the instability that comes from optimizing true entropy directly. Tested on mathematical reasoning benchmarks including AIME 2025 and the coding benchmark HumanEvalPro, ACPO outperformed established baselines like DAPO, GTPO, and SAPO.
The results are from a preprint, so peer review hasn't weighed in yet. Still, credit assignment has been one of the quieter bottlenecks in RL-for-LLMs research — if ACPO's gains hold up, it joins a growing list of techniques chipping away at the gap between what models know and what they can actually demonstrate under pressure.