Validation loss keeps falling long after a learned world model has already stopped working well in practice.
Researchers studying model-based reinforcement learning on Gymnasium's LunarLander v3 found that common offline metrics — validation loss, multi-step prediction error — continue improving even as the model's real control performance collapses. To fix this, they evaluated 40 candidate diagnostics against an oracle measure of closed-loop quality and identified a standout: the Reward Observability Fraction (ROF), which quantifies how much a model's reward predictor depends on observable state rather than hidden latent structure. They then combined ROF with three structural regularizers into a single composite score called CROF, designed to be computed entirely at validation time with no environment rollouts required.
The stakes are practical. Training world models is expensive, and practitioners routinely pick checkpoints by watching validation curves — a habit this research suggests is quietly selecting worse models. A policy trained with the CROF-selected model beat a model-free baseline by roughly 24.5 return points while requiring about 65 times fewer real environment interactions; the same checkpoint also powered a zero-shot control policy without any additional fine-tuning.
The work is narrowly scoped to a single environment with shaped rewards, so how well CROF generalizes to more complex tasks — sparse rewards, continuous control, partially observable settings — remains untested. Practitioners adopting checkpoint selection heuristics from this work should treat it as a promising starting point rather than a solved problem, and budget time to validate the metric on their own environments before trusting it to guide production training runs.