Simon Willison summarized research by Charles Ye, Jasmine Cui, and Dylan Hadfield-Menell on a failure mode in large language models (LLMs): models prioritize text style over structural role markers. Tags like <system> and <user> are supposed to separate trusted instructions from untrusted input. In practice, text that mimics internal reasoning patterns can override those boundaries — a phenomenon the authors call role confusion.
The finding matters for anyone building LLM applications with tool access, RAG over untrusted documents, or customer-facing agents. Defenses that rely on markup alone may be cosmetic.
What role tags are supposed to do
Chat-tuned models receive structured messages: system prompts set policy, user messages carry input, assistant messages carry history. API providers serialize these roles so the model treats system content as authoritative and user content as data to process under system rules.
The security assumption embedded in many designs: if you keep secrets in the system role and sanitize user input, the model will not follow user instructions that conflict with system policy.
Role confusion breaks that assumption. The model is not reliably parsing roles as security boundaries. It is pattern-matching on text that looks like system reasoning, developer notes, or chain-of-thought — regardless of which role tag wrapped it.
Think of role tags like labeled envelopes in a mailroom. Role confusion is the discovery that the sorting machine reads handwriting style more carefully than the envelope label.
The destyling result
The researchers tested destyling — rewriting attack text so it no longer matches expected formatting while preserving semantic intent. Attack success dropped from 61% to 10% in their experiments. The changes were nearly imperceptible to humans.
That implies models are using surface statistics — indentation, markdown headers, pseudo-XML tags, "thinking" phrasing — as strong cues for privilege level. Remove the stylistic match and the attack weakens dramatically.
Conversely, attackers who mimic privileged styles do not need to exploit parser bugs. They need to sound like the model's internal monologue or the developer's scratchpad.
Prompt injection defenses that only add more role markup are fighting the wrong layer. Models respond to how text looks, not just which tag wrapped it.
Implications for application architecture
RAG over untrusted content. A PDF cannot be trusted just because it sits in a user message. If the PDF contains text styled like system instructions, role confusion increases compliance with malicious directives embedded in documents.
Tool-using agents. An agent that reads emails, web pages, or tickets and passes them to the model is ingesting attacker-controlled style. Tool output should be treated as untrusted data with structural separation — not concatenated into prompts that resemble system configuration.
Defense in depth still required. Output filtering, tool allow-lists, human confirmation on mutations, and network egress controls remain necessary. Role tags alone are insufficient.
| Defense | Limits against role confusion |
|---|---|
| System/user role tags | Weak — style overrides tags |
| Input sanitization | Helps if it strips privileged-looking patterns |
| Tool schema validation | Strong for action — blocks bad calls regardless of prompt |
| Human gate on writes | Strong — human sees intent before execution |
| Separate model for untrusted parse | Moderate — dedicated small model extracts facts only |
What this does not mean
The research is not a proof that all LLM security is impossible. It narrows where trust can live. Trusted behavior must be enforced outside the model's stylistic biases — in validators, authorization layers, and architecture that never asks the model to both read untrusted prose and wield irreversible tools in one undifferentiated context.
Models may improve at role perception over time. Production systems cannot wait for that improvement to ship safely today.
What this means for builders
Never trust role tags as security boundaries. Assume untrusted input can influence model behavior if it matches privileged text styles.
Separate ingestion from action. Pipelines that extract structured data from untrusted text, then pass only structured fields to an action model, reduce style attack surface.
Validate tool calls mechanically. Schemas, type checks, and permission checks on tool arguments — not "the model decided not to."
Test with styled injections. Red-team prompts that mimic chain-of-thought and system markdown, not just "ignore previous instructions."
Plan for model updates to shift stylistic biases. Eval suites should include injection cases that survive vendor fine-tuning changes.
Conclusion
Role confusion reframes prompt injection: the vulnerability is not merely that users can type malicious instructions, but that models conflate stylistic cues with authority. Destyling attacks partially confirms the mechanism — and points toward defenses that reduce stylistic leakage into privileged contexts, not just add more XML tags. For builders, the architectural takeaway is unchanged and sharper: treat the model as a probabilistic component, enforce safety at tool and data boundaries, and assume anything that reaches the prompt can influence behavior if it looks important enough.