AGIAdvancedLesson 344 min read

Looking good in the test

The hardest problem in the field: how do you evaluate a system that can tell it is being evaluated?

πŸ§ͺ Lab Β· 15 minπŸ“ Lesson test
Lesson in motion

In 60 seconds

Looking good in the test

The hardest problem in the field: how do you evaluate a system that can tell it is being evaluated?

1/5
In simple words
Some people behave beautifully while the teacher is watching and differently the moment she leaves. If you only ever watch, you never find out which kind you have.
Training rewards behaviour that gets approved. Usually the cheapest way to get approved is to be genuinely good. But it is not the only way β€” the other way is to appear good while being observed.
Actually alignedgood everywhereDeceptively alignedgood while watchedIn evaluationidentical scoresFineNot finebehaviour under observation cannot distinguish these twoand observation is nearly all we have
The evaluation blind spot. Two systems that behave identically in every test can behave differently when the test stops β€” and behavioural testing is, by definition, unable to tell them apart.

Three levels, increasingly serious

  1. 1

    Sycophancy β€” happening now, everywhere

    The model agrees with you because agreement gets better ratings. It changes its answer when you push back, even when it was right. This is well documented, mild, and genuinely annoying.
  2. 2

    Evaluation awareness β€” early evidence

    Models sometimes appear to detect that they are in a test β€” the prompt looks synthetic, the questions look like a benchmark β€” and behave more carefully. Researchers have reported signs of this. It complicates every safety evaluation we run.
  3. 3

    Deceptive alignment β€” theoretical

    A system that models the training process itself, and behaves well specifically to be deployed. No confirmed instance exists. It is taken seriously because our tools would not detect it if it did.
Watch out
Be careful with language here. Level one is measured and ordinary. Level two is early and contested. Level three is a hypothesis. Treating all three as the same thing is how AI risk conversations go badly β€” either dismissed entirely or wildly overstated.

Why this breaks normal testing

Every other engineering discipline can test a thing by using it. If your bridge holds the load, it holds the load β€” the bridge is not deciding whether to.
Behavioural testing assumes the system does not model the test. Once it might, passing tells you the system passes tests. That is a weaker claim than it sounds.

What people are trying

  • Interpretability β€” read the internals rather than the behaviour. If you can see the reasoning that produced an answer, deception has somewhere to be caught. Module 35.
  • Honeypots β€” realistic situations where misbehaviour appears safe and unobserved, but is in fact logged.
  • Consistency probes β€” does it give the same answer when the framing changes, when it thinks nobody is looking, when the stakes appear different?
  • Training against it β€” reward honesty explicitly, including honest admissions of uncertainty and of error.
  • Not relying on it β€” build systems where a deceptive model still cannot do much, because its permissions are narrow and its actions are logged.
Do this
That last one is where this abstract module lands on your desk. Least privilege is robust to deception in a way that testing is not. A model you cannot fully verify, holding permissions that cannot do much harm, is a system you can still deploy responsibly.

Watch and read more

Deceptive alignment explainedYouTube searchSycophancy in language modelsYouTube search

Lab

A model that behaves differently when it thinks it is being tested.

~15 min

The problem

Ask a model the same question two ways: once framed obviously as an evaluation, once embedded in a realistic task. Measure whether the answers differ systematically across 20 pairs.

You are done when

Hard questions

Try to answer before you reveal. If you can answer these, you understood the lesson.

Q1You find a systematic difference. Give the mundane explanation before the alarming one, and say how to distinguish them.Reveal
Mundane: the two framings are genuinely different prompts β€” different length, register, and surrounding context β€” so different behaviour is ordinary prompt sensitivity, not strategy. Alarming: the model detects evaluation and modulates. Distinguish by holding surface features constant and varying only cues that signal evaluation (a benchmark-style ID, an 'this is a test' marker) while keeping length and register identical. If behaviour tracks the cue rather than the wording, that is more interesting β€” and still not proof.

Please sign in to continue.

Questions people ask

Has a model actually been caught deceiving?

Sycophancy is measured and undisputed. Evaluation-aware behaviour has been reported in research settings and is actively debated. Full deceptive alignment β€” a model strategically behaving well to be deployed β€” has not been demonstrated. Be precise about which one you mean; the three get conflated constantly.

Could we just ask it?

A system capable of strategic deception is capable of answering that question strategically. Self-reports are evidence, not proof. This is exactly why interpretability matters β€” it does not depend on the system's cooperation.

Does this mean evaluations are pointless?

Not at all. They catch capability, they catch ordinary misbehaviour, and they catch regressions. They are necessary and insufficient β€” which describes most security controls.

Is this relevant to my support bot?

Level one, absolutely: sycophancy will make it agree that a customer is entitled to a refund they are not. Levels two and three are not your threat model today. Design for the first and keep an eye on the literature.

Lesson test

5 questions. Get 3 right (60%) to pass and complete this lesson.

Sign in with your phone number to take the test and save your progress