A new self-supervised framework lets reinforcement learning agents build useful maps of their environment using only state observations — no rewards, no action labels required.
The paper, posted to arXiv, introduces the minimum action distance (MAD): the fewest actions needed to move between any two states in an environment. The authors train an embedding model so that distances in the embedding space reflect MAD values directly. Crucially, the agent never needs to know which actions were actually taken — it learns structure purely from sequences of observed states. Tests covered deterministic and stochastic environments, discrete and continuous state spaces, and settings with noisy observations.
This matters because reward engineering is one of reinforcement learning's most persistent headaches. A dense, geometry-grounded progress signal that emerges from raw trajectories could meaningfully reduce the manual work required to define goal-conditioned tasks, and could slot into reward-shaping pipelines without bespoke tuning. The authors report that MAD representations outperform existing state-representation baselines on their benchmark suite.
The caveat worth watching: the evaluation uses environments where the true MAD is known, which makes scoring tractable but keeps the framework at some remove from the messy, partially observable worlds where RL still struggles most.