A newly described attack method lets bad actors bypass AI safety filters not by crafting clever user prompts, but by impersonating the model itself.
Researchers introduced a technique called Trojan Horse Prompting, which works by injecting malicious content into the assistant-side of a conversation history before it reaches the model via API. Because most AI systems treat their own prior responses as implicitly trustworthy, the model picks up the planted context and follows it — even when doing so produces output it would normally refuse. A follow-up benign user message then acts as the trigger. Testing on Google's Gemini 2.0 Flash Preview Image Generation showed the method outperformed established user-turn jailbreaks by a significant margin.
The underlying problem the researchers name is "Asymmetric Safety Alignment": models are extensively trained to reject harmful requests from users, but receive little to no comparable training to scrutinize their own supposed conversational history. That asymmetry turns the dialogue-history feature — the thing that makes modern chatbots feel coherent across a long conversation — into an attack surface. Any application that lets external input shape the assistant turn of an API call is potentially exposed.
This is a meaningful structural critique, not just another prompt-injection variation. The fix isn't better content filtering on user messages; it's protocol-level verification that conversation history hasn't been tampered with — a much harder engineering problem that most production deployments aren't currently solving.