Offline-to-online reinforcement learning just got a taxonomy.
Researchers have published a framework that explains one of the field's persistent headaches: design choices for online fine-tuning that work in one setting fail completely in another. The paper argues the answer lies in what they call the stability-plasticity tradeoff — the tension between preserving a useful offline prior and staying flexible enough to learn from new experience. From that lens, they identify three distinct regimes of fine-tuning, each demanding different stability properties depending on whether the pretrained policy or the raw offline dataset is the stronger starting point. A large-scale empirical study across 63 test cases found the framework's predictions held in 45 of them, with only 3 outright contradictions.
This matters because offline-to-online RL is increasingly how serious teams train agents: collect a dataset cheaply offline, then refine with live interaction. Without a principled way to pick fine-tuning strategies, practitioners are left guessing — and the wrong guess can mean a policy that was working gets worse, not better. The framework gives teams a diagnostic question to ask first: how strong is your offline prior relative to your policy?
The 45-of-63 alignment rate is solid but not ironclad — roughly 28% of cases weren't clean wins for the framework. That's honest science, and more useful than a paper claiming universal answers. The real test will be whether practitioners find the three-regime lens actionable outside controlled benchmarks.