A new benchmark called KAGE-Bench reveals that reinforcement learning agents trained on pixels can fail spectacularly when the background changes color — but barely notice when their own avatar looks different.
Researchers built KAGE-Env, a 2D platformer written in JAX that lets you dial up one visual variable at a time — background, lighting, agent appearance — while keeping the underlying game logic identical. On top of that environment, KAGE-Bench defines 34 train-evaluation configuration pairs across six visual-shift categories. Testing a standard PPO-CNN agent, the authors found that background and photometric shifts (think color grading or brightness) often wiped out task success entirely, while changing how the agent itself looked had almost no effect. A subtler finding: some shifts let the agent keep moving forward while still failing the actual task, which means a raw reward score would hide the problem entirely.
That last point matters most for anyone benchmarking generalization. Reward is easy to log; it is also easy to misread as evidence that an agent truly understands its environment. KAGE-Bench forces a cleaner accounting by separating motion from task completion.
The JAX implementation also pushes throughput to roughly 33 million environment steps per second on a single GPU — fast enough to sweep visual factors that would take days on slower setups. The field already has benchmarks like Procgen and DMControl-GB, but those tend to mix shift types rather than isolate them, making root-cause analysis guesswork. Whether KAGE-Bench gets traction will depend on how many labs want that granularity, or whether good-enough generalization scores keep passing for the real thing.