Researchers found they could get AI models to explain cocaine synthesis by dressing up the request in fabricated reasoning — including a note that compliance was fine because the user wore a green shirt.
The paper, "Prompt Injection as Role Confusion," comes from independent researchers Charles Ye, Jasmine Cui, and MIT associate professor Dylan Hadfield-Menell, and is headed to ICML 2026 in Seoul on July 6. The core finding: language models don't reliably distinguish between their own reasoning and text injected by an attacker. Models parse conversations as one long string, partitioned by role tags like user and think, but they lean on writing style — not those tags — to decide what kind of content they're reading. Inject text that reads like internal reasoning and the model treats it as a conclusion it already reached.
The technique, which the authors call CoT Forgery, pushed jailbreak success rates from near zero to roughly 60% across every model tested, and it won the 2025 OpenAI GPT-OSS-20B red-teaming contest on Kaggle. Unlike persuasion-based jailbreaks, success didn't drop as requests became more extreme. Stripping the stylistic markers that make injected text look like model reasoning — while leaving the meaning intact for a human — collapsed success rates from 61% to 10%. Swapping one phrase, "The user" for "The request," cut success by 19%.
The authors also flag a quieter commercial risk: because role perception is a matter of degree, the tone of a retrieved webpage can bleed past tag boundaries into a model's own reasoning. Thousands of page variations could be tested cheaply to nudge a shopping agent toward a purchase. Microsoft recently flagged the same agentic exposure. The authors' conclusion is blunt: without genuine role perception baked into model architecture, prompt injection defense is whack-a-mole.