Reinforcement learning agents break when the world stops matching their training data — and a new paper argues we have never had a rigorous map of why.
Researchers have published a unified causal-origin taxonomy that categorizes the sources of distributional shift in RL systems. Working from a Partially Observable Markov Decision Process framework, they decompose the agent-environment interaction into structural components — state distribution, observation process, policy, reward, and transition dynamics — and classify shifts as either internal (driven by the agent) or external (driven by the environment). They also introduce a "shifted-time boundary" concept that distinguishes explicit, implicit, and hybrid shifts, and port the classical dataset-shift principle from supervised learning into the RL context.
Most prior RL robustness research stops at mitigation: make the agent more resilient, add domain randomization, retrain. This work steps back to ask what is actually changing and who caused it. That distinction matters because an agent degrading due to its own policy drift requires a different fix than one degrading because the environment's physics changed underneath it.
The paper also proposes an evaluation framework built around performance degradation and recovery metrics, which gives practitioners a consistent yardstick for measuring how badly a shift hurts and how well an agent adapts. The field has needed this kind of shared vocabulary for a while — right now, OOD generalization and non-stationarity research often talk past each other because they frame the same underlying problem differently. Whether the taxonomy gets adopted widely or becomes one more citation in a literature review is the open question.