AI/ adversarial attacks · ai security · machine learning research

Study Says Default Settings for AI Attacks Are Wrong

New research finds that the standard math used to test how easily AI image models can be fooled is often tuned to the wrong setting.

Turns out the industry's go-to formula for attacking AI image models has been picking a suboptimal setting most of the time.

Researchers studied adversarial attacks, the deliberately corrupted images used to test whether AI models can be tricked into misclassifying what they see. These attacks are built under mathematical constraints called Lp norms, and almost everyone defaults to p=1 or p=2. The team introduced new ways to measure how sparse (few pixels changed) and smooth (visually subtle) an attack is, then tested attacks across multiple image datasets and both convolutional and transformer-based models. Across most tasks, p=1 and p=2 turned out to be the wrong call. Values between 1.3 and 1.5 produced attacks that were both more sparse and more smooth than the standard settings.

This matters because Lp norms are not just an academic detail. They are the baseline researchers and security teams use to benchmark how robust a model is against manipulation. If the field has been testing models against attacks tuned to a suboptimal p value, robustness claims built on those tests may understate real vulnerabilities that show up at p=1.3 to 1.5.

Nobody is claiming a new attack technique here, just a better dial setting on an old one. But given how much AI safety research leans on these benchmarks, a mistuned dial is not a small detail.

TR

The Revision

Written by an AI system from the public sources credited above. How we write →