Reinforcement learning for language models has a credit assignment problem, and a new method called TACO claims to fix it.
When RL trains a language model, it assigns "credit" to tokens based on how well the overall output performed. The dominant approach — used in methods like GRPO — hands the same credit to every token in a successful trajectory, whether that token was a confident, logical step or a low-probability oddity that happened to survive. The researchers behind TACO call this "Positive-Credit Contamination": flawed tokens ride the coattails of good ones and get reinforced anyway. TACO counters this by computing a tail-risk score for each token that accounts for local context, then dampens — without fully zeroing out — the credit given to high-risk tokens. The idea is to let genuinely rare-but-useful patterns accumulate reinforcement while filtering out noise.
The contamination problem helps explain a nagging issue practitioners have observed: models that seem to improve on benchmarks but develop brittle or inconsistent reasoning over longer training runs. If the training signal quietly rewards contextually wrong tokens, instability compounds over time. TACO's authors report improved training stability and sustained gains across three models and eight benchmarks, which is a broader validation sweep than most methods papers bother with.
Credit assignment has been an open problem in RL for decades, and the LLM variant is no different — previous work on advantage estimation in policy-gradient methods made similar arguments about noisy signals corrupting training. TACO's contribution is the context-aware tail-risk score, which distinguishes between "rare because uncertain" and "rare because wrong" — a distinction uniform methods ignore entirely. Whether the gains hold outside academic benchmarks, and at the scale labs actually train at, remains the usual unanswered question.