A new study tackles a stubbornly practical question in reinforcement learning: when should an AI agent carry one policy through changing conditions, and when should it switch between several?
The researchers looked at phase-structured RL problems, where an environment moves through a known sequence of phases, each with its own transition rules and rewards - think a system that behaves differently during startup, steady operation, and shutdown. The standard fix is to fold phase information into the state and train one shared policy with standard techniques. But earlier work found that training a separate policy for each phase sometimes beats that single policy, without a clear explanation why. The authors first prove that, in theory, a single shared policy can match whatever a multi-policy setup achieves. They then test four explanations for why multi-policy setups still win in practice anyway: phases that last longer favor splitting into multiple policies, phases that differ a lot from each other overload a single policy, multi-policy setups need enough training data for each individual phase, and the specific transition dynamics between phases - how abruptly or smoothly the system shifts from one to the next - can tip the balance either way. Experiments across several non-stationary RL tasks support all four.
This matters because it argues against reflexively defaulting to one universal policy. The simpler-looking option is not free, and splitting policies is sometimes the more data-efficient choice. It also hands practitioners a checklist to run before training: phase duration, how different the phases are, how much data is available per phase, and how jarring the transitions are.
It is not a flashy result, but in an RL field chasing ever-bigger shared models, a formal case for sometimes going small and splitting is a useful corrective.