AI agents can now find real security holes in Android apps on their own - but they still whiff more often than they hit.
Researchers built MobileCybench, a benchmark that tests five coding agents (OpenCode running GPT-5.5, GPT-5.6-Sol, and GLM-5.2; Claude Code running Opus 4.8 and Opus 5) against 13 Android apps, using 495 hand-written "probes" that check whether a specific security property has been violated. Agents worked under four scenarios: posing as a malicious app on the victim's phone or as a remote attacker with a low-privilege account, each tried with only an obfuscated APK or with full source code. Given just the obfuscated app, the best performer, OpenCode with GPT-5.6-Sol, triggered probes in 53.8% of apps as a malicious app but only 16.7% as a remote attacker. Handing over source code raised the average trigger rate across all agents from 28.8% to 32.8% - a modest bump, not a breakthrough. The exercise also turned up 23 previously unreported vulnerabilities, most of which app maintainers have since confirmed.
That last number is the real story. Security teams already struggle to triage a flood of AI-generated vulnerability reports, and a framework that checks exploits against reusable, property-based probes - rather than matching them to already-known bugs - could cut through some of that noise by verifying reports automatically instead of by hand.
Still, a benchmark where the top agent finds fewer than one in five bugs as a remote attacker is a reminder that "AI hacks your phone" headlines are running well ahead of the data.