How would we even measure it?
Every benchmark that gets famous gets beaten, and then people say it never really measured intelligence. Here is why that keeps happening.
In 60 seconds
How would we even measure it?
Every benchmark that gets famous gets beaten, and then people say it never really measured intelligence. Here is why that keeps happening.
The benchmark treadmill
- 1
A hard benchmark appears
Experts agree it needs real understanding. Models score badly. - 2
Models improve, scores climb
Sometimes from genuine capability, sometimes because the test leaked into training data. - 3
The benchmark is saturated
Everyone scores near the top. It no longer distinguishes anything. - 4
People say it was never real
"That was just pattern matching." And a new benchmark appears.
Ways a benchmark lies to you
| Problem | What it means | Everyday version |
|---|---|---|
| Contamination | The test was in the training data | Seeing the exam paper beforehand |
| Teaching to the test | The model was optimised for this benchmark | Cramming past papers without learning the subject |
| Narrow measurement | Tests one skill, gets read as general ability | Judging a doctor by their handwriting |
| No error weighting | A wrong answer counts the same everywhere | A typo and a wrong dosage both cost one mark |
| Single-turn only | Real work is long and messy | A driving test with no traffic |
What better evaluation looks like
- Held-out and rotating β fresh questions the model cannot have seen.
- Long-horizon β multi-step real tasks, measured by whether the outcome was achieved, not whether the answer looked right.
- Adversarial β someone actively trying to make it fail, not a static set.
- Cost-aware β how many attempts, how much compute, how much human help was needed.
- Failure-shaped β not just the score, but which things it gets wrong and whether that pattern is changing.
Safety evaluations are a different discipline
| Eval type | Question it answers |
|---|---|
| Capability | Can it do the dangerous thing at all? |
| Propensity | Does it choose to, unprompted? |
| Robustness | Does it hold up when someone attacks it? |
| Oversight | Can a human reliably catch it when it goes wrong? |
| Dangerous capability | Can it help with weapons, large-scale cyber attacks, or manipulation? |
Watch and read more
Lab
A benchmark you contaminated on purpose.
The problem
# Contamination check you can run on any dataset
import hashlib
def canary(example): return hashlib.sha256(example.encode()).hexdigest()[:16]
# Publish canaries with your eval set. If a model reproduces one, it saw the set.You are done when
Hard questions
Try to answer before you reveal. If you can answer these, you understood the lesson.
Q1Design an evaluation that stays valid for two years. What property must it have?Reveal
Questions people ask
Is there one good AGI test?
No, and there probably cannot be, because "general" means "not covered by any fixed list of tasks". The best available approach is a broad portfolio of evolving tests plus honest reporting of failures.
What about the Turing test?
Largely retired as a capability measure. It tests whether something can imitate human conversation, which turned out to be much easier than general competence β and it rewards deception, which is a strange thing to optimise for.
Why do labs publish their own evals?
Partly transparency, partly marketing. Read them, and weight independent third-party evaluations more heavily β the incentive structure is different.
Can I write my own evals?
You should. Twenty real tasks from your actual domain, with a clear right answer, beats any public leaderboard for deciding whether a model works for you.
Lesson test
5 questions. Get 3 right (60%) to pass and complete this lesson.
Sign in with your phone number to take the test and save your progress