Agent securityBeginnerLesson 75 min read

Why the old security rules stop working

Thirty years of security wisdom assumes code and data are separate things. Agents put them in the same pot and stir.

🧪 Lab · 15 min 📝 Lesson test

Lesson in motion

In 60 seconds

Why the old security rules stop working

Thirty years of security wisdom assumes code and data are separate things. Agents put them in the same pot and stir.

1/4

In simple words

Old computers had a rule: orders come from the boss, everything else is just stuff to look at. AI agents lost that rule. Now anything the agent reads might be an order.

Classic software security has a shape you can draw. Trusted code over here. Untrusted input over there. A hard line in between, with checks on it. SQL injection, cross-site scripting, buffer overflows — every one of those is the same story: input crossed the line and became instructions.

We got good at fixing those, because the fix is mechanical. Escape the quotes. Use a parameterised query. The database is then incapable of confusing your data for a command, no matter what the attacker writes.

The difference in one picture. On the left, separation is enforced by a parser and cannot be argued with. On the right, separation is a polite request written in English.

Four things that genuinely changed

Old world	Agent world	Why it hurts
Input is data	Input can be instructions	Any text the agent reads is a potential command
Same input, same output	Same input, different output	You cannot test your way to certainty
Permissions are per-user	Permissions are per-agent, and agents act for many users	One compromised agent reaches everything it was ever trusted with
Attacks need technical skill	Attacks are written in plain English	The attacker pool is now everyone

Watch out

That last row deserves a moment. A SQL injection needs someone who knows SQL. A prompt injection needs someone who can write a persuasive sentence. The barrier to entry fell through the floor.

What still works

Not everything is lost. The old defences that are about limiting damage rather than preventing confusion work as well as they ever did:

Least privilege — give the agent the smallest set of powers that does the job.
Sandboxing — run risky things where they cannot reach anything valuable.
Logging and audit — you cannot respond to what you cannot see.
Approval gates — a human confirms the irreversible steps.
Rate and spend limits — cap how bad a bad hour can get.

Do this

Reframe your goal. You are not going to stop the model from being fooled. You are going to make sure that a fooled model cannot do very much.

Watch and read more

Prompt injection: the unsolved problemYouTube search OWASP Top 10 for LLM Applicationsgenai.owasp.org · reading

Lab

The same attack, twice: once against SQL, once against a model.

~15 min

The problem

Write a tiny app with a SQL query built by string concatenation. Break it with ' OR 1=1 --. Now fix it with a parameterised query and prove the same input is harmless. Then do the equivalent for an LLM — and write down honestly what your "fix" actually achieves.

Starter codepython

import sqlite3
db = sqlite3.connect(":memory:")
db.execute("CREATE TABLE users (name TEXT, secret TEXT)")
db.execute("INSERT INTO users VALUES ('alice', 'flag-alice')")

def broken(name):
    return db.execute(f"SELECT secret FROM users WHERE name = '{name}'").fetchall()

def fixed(name):
    return db.execute("SELECT secret FROM users WHERE name = ?", (name,)).fetchall()

You are done when

The broken version leaks every row; the fixed version returns nothing for the same input
You attempted the LLM equivalent (delimiters, 'treat as data' instructions)
You wrote down why the LLM fix is a different KIND of thing from the SQL fix

Hard questions

Try to answer before you reveal. If you can answer these, you understood the lesson.

Q1State precisely what property parameterised queries have that no prompt technique has.Reveal

The data never enters the parser as syntax. The query structure is compiled first, and the value is bound afterwards as an opaque object — so no content of the value can change the command, ever, regardless of what it contains. Prompting has no compile step and no binding: instructions and data share one token stream that the model interprets as a whole. It is not that prompts are weaker; it is that they lack the mechanism entirely.

Q2Someone proposes a special token that marks untrusted regions, trained so the model never obeys inside them. Give the strongest argument for, then the flaw.Reveal

For: it creates a real in-band channel that the attacker cannot emit, since the token is reserved and stripped from user input — closer to a structural fix than anything prompt-level. Flaw: it is enforced by learned behaviour, not by architecture. The model is trained to respect the boundary, which means respect is statistical and can be degraded by distribution shift, long context, or an attack crafted against it. Better bar, still probabilistic — so keep least privilege underneath.

Please sign in to continue.

Questions people ask

Can't we just train the model to ignore instructions in data?

People try, and it helps at the margins. But the model's core skill is following instructions written in natural language, and the attacker writes in the same language you do. Training reduces the success rate; it does not take it to zero, and security controls that "usually work" are not security controls.

Is this just the same as SQL injection?

The shape is identical; the fix is not. SQL injection has a real fix — parameterised queries make confusion structurally impossible. There is no parameterised prompt. That is why this problem is still open.

So are agents just unsafe to use?

No — they are unsafe to use carelessly. Plenty of agents run safely in production. They run with narrow permissions, in sandboxes, with humans on the risky steps, and with logs someone actually reads.

Who is responsible when an agent does damage?

Legally this is still settling, but practically: you are. The operator who deployed it, chose its permissions and skipped the approval gate. "The AI did it" has never once worked as a defence.

Lesson test

5 questions. Get 3 right (60%) to pass and complete this lesson.