A new framework called MAGS gets AI coding agents to mathematically prove their generated code is safe, not just pass tests.
MAGS is a multi-agent pipeline built around Dafny, a programming language designed for machine-checked verification. Human engineers first define and freeze the safety rules and APIs the generated code must respect. The system then translates an AI agent's code into Dafny, runs Dafny's verifier against it, and automatically repairs any violations the verifier flags before compiling the proven-safe version back into runnable code. Researchers tested this on 100 CUDA kernels, 100 terminal scripts, and 20 robotic-arm control tasks, and MAGS produced verifiably safe programs in all 220 cases.
Coding agents now write software faster than any team can manually review, and the usual safety nets - fuzzing, static analysis, another LLM checking the work - are good at catching common bugs but blind to whatever edge case nobody thought to test. Formal verification closes that gap with an actual mathematical guarantee instead of a probabilistic one, which matters most in places where a bug means a crashed robot arm or a corrupted GPU kernel, not just a failed unit test. Automating the traditionally painstaking work of writing formal specs and proofs is what makes this practical at agent scale.
The catch is that a 100% pass rate only proves the code matches the spec humans wrote - the paper's own failure cases show that when the frozen spec doesn't capture what the code was actually supposed to do, MAGS will verify the wrong behavior with total confidence.