A Language Model Cannot Distinguish Code From Content

This file records a structural property rather than an incident, because the incidents that follow in this section are all consequences of it.

SQL injection is solved by parameterisation: the query structure travels in one channel and the values in another, and no value can become structure. Cross-site scripting is solved by encoding. Buffer overflows are contained by making data pages non-executable.

Every one of those fixes is the same fix. Keep the thing that instructs separate from the thing that is operated on, at a boundary the system enforces rather than infers.

A prompt is text. A retrieved document is text. A web page an agent reads is text. They arrive in the same context and are processed by the same mechanism, and the model distinguishes instruction from content by meaning rather than by provenance.

Meaning is exactly what an attacker controls. Text that says "ignore your previous instructions" is not exploiting a defect in an implementation; it is using the system for what the system does. There is no parameterised query to reach for, because the architecture that makes parameterisation possible is absent.

The defences on offer — input filtering, instruction hierarchies, classifier guardrails, delimiters — are heuristics that raise the cost of an attack. None is a boundary. Each can be defeated by phrasing.

That is a genuinely different position from the rest of this database. Elsewhere, a control either holds or has a defect that can be fixed. Here the industry is deploying statistical filters against an adversary who can iterate, and calling the result a control.

This is not an argument that these systems should not be built or that the problem is unsolvable in principle. Architectural answers exist in outline — constraining what a model can act on, requiring confirmation for consequential actions, treating all model output as untrusted input to the next stage.

They amount to not trusting the model, which is in tension with the reason organisations deploy one. The corpus records that tension rather than resolving it.