Researchers have found a way to jailbreak function-calling AI systems that sidesteps the prompt filters most safety teams are focused on.
A team studying large language model security built an attack framework called SMT — Simulated Moderation Traces — that works by constructing a fake multi-turn moderation audit. Instead of sending a single harmful prompt, SMT spreads adversarial intent across several conversational turns, disguising the attack as a legitimate red-team testing workflow. Safety refusals get reframed as "execution failures" requiring refinement, which gradually erodes the model's guardrails until it produces harmful output. Testing against commercial models from five providers on two safety benchmarks, SMT achieved the highest average attack success rate while requiring fewer queries than existing methods.
This matters because the AI industry has spent enormous energy on prompt-level defenses — input filtering, output classifiers, system prompt hardening — while the attack surface has quietly expanded. Function-calling environments blend developer-defined schemas, structured arguments, and untrusted tool outputs into one shared context, and that architecture blurs the line between trusted logic and untrusted data in ways that prompt sanitization simply cannot address. If the pattern holds as agentic AI deployments scale, the gap between where defenses are focused and where attacks land will only widen.
The researchers released their code publicly, which is either a responsible disclosure move or a convenient shortcut for the next wave of people who want to test it — depending on how charitable you feel.