A new arXiv preprint says teaching an AI model to make better decisions does not automatically teach it to hold more accurate beliefs.
The paper, posted to arXiv as preprint 2609.38005 on September 30, 2026, breaks LLM reasoning under uncertainty into two separate skills: forming accurate beliefs from evidence, and turning those beliefs into decisions that optimize a stated payoff. Using a synthetic benchmark with known correct answers, the authors tested frontier and open-source models on both skills. They then ran reinforcement learning that targeted belief formation, decision-making, or both, across three task domains. Training on one skill mostly improved that skill alone, and the gains often failed to carry over to different ways of asking the model to state its answer.
That distinction matters because plenty of products already lean on LLMs to weigh evidence and recommend a call, in loan approvals, triage, or forecasting. If a model can look more decisive without actually reasoning better, benchmarks that only score final decisions risk rewarding the wrong thing. The paper's fix, training both skills together with matched formats between training and evaluation, sounds simple but is a real constraint on how these systems get built and graded.
A model that seems confident and a model that is right are not the same test, and right now most evaluations only check the first.