A fragmented research field just got a shared ruler.
Researchers released RoAd-RL, an open-source Python package designed to unify how the field evaluates adversarial robustness in deep reinforcement learning. The library provides common abstractions for policies, attacks, defenses, and metrics, and plugs directly into Stable-Baselines3 and Gymnasium — two widely used RL toolkits. The team tested three agents (DQN, PPO, and SAC) across two environments (LunarLander and Highway-v0) under 192 attack-defense configurations. It is available now on PyPI.
The reproducibility problem in adversarial ML is well-documented, and RL has it worse than most: every lab rolls its own attack setup, making published robustness numbers almost impossible to compare. A standardized benchmark does not fix bad research, but it does make bad research easier to catch — and good research easier to build on. The finding that some popular defenses hurt performance more than the attacks they are meant to stop is the kind of result that only surfaces when you hold the evaluation conditions constant.
Temporal smoothing came out as the most consistently effective defense across configurations — a useful data point, though two environments and three agents is a narrow slice of the problem space. Treat the headline numbers as a starting point, not a verdict.