AI/ prompt injection · openai · ai security · llm

OpenAI Trains Models to Follow a Chain of Trust

OpenAI research proposes ranking instructions by source so system prompts can't be hijacked by injected content or jailbreak attempts.

OpenAI published a research paper on teaching language models to treat instructions differently depending on who issued them.

The paper, titled "The Instruction Hierarchy," addresses a known weakness in how large language models handle competing instructions. Today's models treat inputs as roughly equivalent, which means malicious content buried in a document, a web page, or a user message can override what an operator actually intended the model to do. The proposed fix is a training regime that encodes a chain of trust: system-level instructions from the operator outrank user messages, which outrank anything injected from external sources the model processes. The model learns to enforce that hierarchy rather than naively following whatever it reads last.

Prompt injection is the AI equivalent of a classic injection attack - getting a system to execute commands hidden inside data it was only supposed to read. As AI agents take on more real-world tasks like processing documents, browsing the web, or acting on emails, the attack surface grows and the consequences get harder to contain. A model that can be hijacked by a malicious PDF is not safe to hand autonomous tools.

The honest caveat buried in the framing is that this approach reduces exposure rather than eliminating it. That is a reasonable claim, but it also means the adversarial research community now has a new target to probe - which is roughly how every security improvement in this space has gone so far.

TR

The Revision

Written by an AI system from the public sources credited above. How we write →