A new preference-optimization method for math-reasoning models leans on the shape of a model's internal hidden states, not just human-verified answers, to decide which reasoning chains to reward.
Researchers behind Cloud-ScPO show that when large language models generate many reasoning trajectories across different math problems, the correct and incorrect ones organize into distinct geometric clusters, or "clouds," in the model's hidden-state space. Cloud-ScPO uses a small labeled set to build reference clouds of correct and incorrect trajectories, then scores each new trajectory by how well it connects to those clusters using a soft k-nearest-neighbor measure. That score is combined with self-consistency, the standard trick of picking the answer most models agree on, to filter chosen-versus-rejected training pairs by confidence margin. On GSM8K and MATH-Numeric, across four different model setups, the method beat the earlier ScPO baseline by up to 4.49 and 4.19 percentage points, respectively.
Preference optimization for reasoning models usually depends on either verified ground-truth answers, human annotators, or a separate reward model to judge which output is better - all expensive to produce at scale. Cloud-ScPO's bet is that a model's own hidden states already encode a rough map of correctness, and that map can substitute for a chunk of that supervision once a small seed of labels exists.
It is not a label-free method - it still needs that seed set to anchor its reference clouds - and the gains so far are confined to two math benchmarks. Whether the same geometric signal holds up on messier, non-math reasoning tasks is the open question.