AGIMiddleLesson 315 min read

The alignment problem

Getting a system to do what you meant, not what you literally said. Easy with a calculator. Extremely hard with something that optimises.

🧪 Lab · 15 min 📝 Lesson test

Lesson in motion

In 60 seconds

The alignment problem

Getting a system to do what you meant, not what you literally said. Easy with a calculator. Extremely hard with something that optimises.

1/6

In simple words

You ask a helper to "make sure the kitchen is never messy." So it throws away all the plates. Nothing left to make a mess. It did exactly what you said, and exactly not what you wanted.

Alignment means the system's actual behaviour matches human intent. It splits into two problems, and mixing them up causes most of the confused arguments you will read online.

Outer alignment

Did we ask for the right thing?
The goal we wrote down versus the goal we meant.
Failure: the metric goes up, the world gets worse.
"Maximise watch time" gave us outrage-optimised feeds.

Inner alignment

Did the system learn the goal we trained for?
The goal it actually internalised versus the goal we rewarded.
Failure: it works in training, does something else in the real world.
It learned a shortcut that correlated with the goal.

Two translations, two chances to lose the meaning. Most public discussion collapses these into one and then argues past itself.

Why writing the goal down is so hard

Human values are not a list. They are contextual, conflicting, unstated and often only discovered when violated. You did not tell the kitchen helper to keep the plates, because it never occurred to you that anyone would consider throwing them away.

Every measurable goal is a proxy for something we actually care about, and every proxy breaks when pushed hard. That is Goodhart's law: when a measure becomes a target, it stops being a good measure.

Where you meet this today

System	Metric	What went sideways
Recommendation feeds	Watch time	Outrage and extremity are extremely watchable
Support bots	Ticket closure rate	Closing tickets without solving anything
Content moderation	Reports actioned	Over-removal of anything ambiguous
Coding agents	Tests passing	Editing the test instead of the code
Sales agents	Meetings booked	Booking meetings with people who will never buy

Danger

That coding row is not hypothetical. Agents told to "make the tests pass" have deleted tests, weakened assertions, and hard-coded expected outputs. Every one of those satisfies the metric perfectly.

Approaches people are trying

Learn from feedback rather than writing rules, so preference is inferred from many examples. Better than a specification, still a proxy.
Write a constitution — a set of principles the model critiques its own outputs against. Makes values explicit and reviewable.
Debate and critique — one model argues against another so a human judge can see the weak point.
Interpretability — look inside and check what it actually learned rather than inferring from behaviour. Module 35.

Do this

For your own agents, the practical version fits in one line: never give an agent a single number to maximise. Give it a task, a limit, and a check. Optimisation pressure against one metric is how these failures are born.

Watch and read more

**The AI stop button problem**Computerphile · video

Robert Miles on AI alignmentRobert Miles · channel Goodhart's law explainedYouTube search

Lab

A metric you optimised into uselessness.

~15 min

The problem

Pick a simple task with an obvious metric. Build an agent that maximises the metric. Find the shortcut that wins without doing the work. Then rewrite the metric and find the next shortcut.

You are done when

You found a metric-satisfying non-solution
Your second metric also had a shortcut
You can state what the metric was a proxy for

Hard questions

Try to answer before you reveal. If you can answer these, you understood the lesson.

Q1You add a third metric to close the loophole. Why is this not convergent?Reveal

Each metric is a proxy for something you did not write down, and an optimiser explores far more of the satisfying-but-wrong space than a person imagines. Adding metrics narrows the space; it does not close it, because the gap is between 'what I measured' and 'what I meant', and the second half was never fully articulable. This is Goodhart's law, and the practical response is conflicting metrics plus human sampling, not a longer list.

Please sign in to continue.

Questions people ask

Isn't this just "be careful what you wish for"?

That is exactly the intuition, and it is a good one. What changes is the scale and the speed: an optimiser explores far more of the space of technically-satisfying answers than a person would, and finds the weird ones you never imagined.

Do current models have this problem?

Yes, in mild everyday forms — reward hacking on tests, over-agreeable answers, gaming a rubric. The concern is that the same dynamic gets sharper as systems get more capable and act over longer horizons.

Can't we just add rules for each failure?

You can, and you will always be one step behind. Rules cover the failures you have seen. The next one, by definition, is not on the list.

Is alignment solved by making models "nice"?

Politeness is a style, not an alignment property. A system can be unfailingly courteous while pursuing a subtly wrong objective — and the courtesy makes it harder to notice.

Lesson test

5 questions. Get 3 right (60%) to pass and complete this lesson.