A new paper shows that a common assumption behind AI systems that learn from past data might be quietly wrong. It also offers a fast way to check it.
Researchers studied Regular Decision Processes, a model for environments where what happens next depends on the whole history of events, not just the current moment, represented as a finite automaton (think a flowchart that remembers where it has been). Earlier work assumed that data collected under a fixed behavior policy could reliably distinguish between competing models of that environment, an idea called distinguishability. The new study proves that assumption can fail even when the policy visits every state in the automaton: in those cases, no amount of data shifts the odds between two equally plausible explanations of the data. The authors verified this result formally using Lean 4, a proof-checking tool, and built an algorithm called PEC that detects the problem in time linear in the size of the automaton being tested.
Tested against four environments, the researchers found that the standard distinguishability assumption broke down in three of them. That is a quiet failure mode: a model trained offline can look fine on paper while being structurally unable to tell two different explanations of the world apart. PEC does not just flag the problem. It also identifies an experiment that restores distinguishability in each failing case.
It is a reminder that a lot of offline reinforcement learning research rests on assumptions nobody actually checks, until someone builds the tool that does.