AI/ ai · security · llm · research

A Game-Theory Model for How LLMs Can Deceive and Be Deceived

Researchers built a formal framework to study how language models can be manipulated through carefully chosen prompts, and how to design defenses against it.

A new academic framework treats LLM-mediated communication as a strategic game — and asks who wins when one side controls the words.

Researchers published a paper modeling interactions between a "sender" who picks a semantic framing, an LLM that generates the actual message, and a "receiver" who evaluates it. The twist: the receiver has blind spots. The paper formalizes this as "receiver awareness" — a type that determines which linguistic features the receiver even notices. Using game-theoretic tools, specifically Perfect Bayesian Nash equilibria, the authors characterize what strategic behavior looks like when one party exploits those blind spots. Numerical experiments show the framework can quantify how awareness gaps affect outcomes and demonstrate that "awareness shaping" — essentially tuning what a receiver pays attention to — can reduce the success rate of phishing-style attacks.

This matters because agentic AI systems are increasingly making decisions based on natural language, which means whoever controls the framing of a message has structural leverage. The paper offers a principled way to analyze that leverage rather than treating prompt manipulation as a grab-bag of anecdotes.

The research arrives as prompt injection attacks against AI agents have gone from theoretical curiosity to documented threat — and most current defenses are heuristic at best. A formal mechanism-design approach won't stop attackers overnight, but it gives defenders a shared vocabulary that informal red-teaming never provided.

TR

The Revision

Written by an AI system from the public sources credited above. How we write →