AGIMiddleLesson 305 min read

What is still missing

The honest gap list. These are the things today's systems cannot do, and each one is a reason to keep humans in the loop.

🧪 Lab · 10 min📝 Lesson test
Lesson in motion

In 60 seconds

What is still missing

The honest gap list. These are the things today's systems cannot do, and each one is a reason to keep humans in the loop.

1/5
In simple words
Today's AI is like a brilliant person with no memory of yesterday, who has never touched anything, and who cannot tell when they are guessing.
  1. 1

    Continual learning

    It does not get better from doing. Every session starts from the same frozen weights. A human plumber learns from each job; the model learns from none of them.
  2. 2

    Grounding in the world

    It learned about physics from text about physics. It has never dropped a glass. This is why it can write beautifully about something it would fail at in practice.
  3. 3

    Knowing what it does not know

    Calibration is poor. It is often just as confident when wrong. Humans are far from perfect here too, but we have a usable feeling of uncertainty and it does not.
  4. 4

    Long-horizon reliability

    It can do a hundred steps; error compounds. A 99% success rate per step is a 37% success rate over a hundred steps, and it rarely notices it has drifted.
  5. 5

    Causal understanding

    It is very good at what goes together and much weaker at what causes what. That gap shows up hard whenever it must plan an intervention.
  6. 6

    Genuine goals

    It has no wants between conversations. Everything that looks like drive is the shape of the task you gave it. Whether that stays true as systems get more agentic is one of the open questions.

Why each gap is a safety argument

GapSafety consequence
No continual learningIt repeats the same mistake forever — but also cannot quietly drift on its own
No groundingConfident plans that fail on contact with reality
Poor calibrationYou cannot use its confidence to decide when to check
Compounding errorLong autonomous runs degrade quietly, without an alarm
Weak causalityIt optimises the measure and misses the goal — Module 32
No stable goalsReassuring today; the thing to watch as agents get longer horizons
Watch out
Notice how many of these argue for the same control: shorter runs with more checkpoints. That single design choice mitigates four of the six.

The gaps people are actively closing

  • Memory systems patch the learning gap from outside — and bring Module 16's poisoning risk with them.
  • Robotics and multimodal training chip at grounding, slowly and expensively.
  • Verifier models and self-checking improve reliability over long runs, imperfectly.
  • Tool use outsources what the model is bad at — arithmetic, current facts, precise retrieval — to things that are good at it.
Do this
Every one of those fixes is a system built around the model rather than a change to the model. Which means the safety of the whole thing is an engineering decision you make — not a property you inherit from the lab.

Watch and read more

What LLMs still cannot doYouTube searchContinual learning problemYouTube search

Lab

The compounding-error arithmetic, felt rather than read.

~10 min

The problem

Build an agent that must complete N steps correctly. Measure per-step success, then measure end-to-end success for N = 5, 20, 100. Compare to 0.99^N. Then add a checkpoint every 5 steps and re-measure.
Starter codepython
for n in (5, 20, 100):
    print(f"n={n:3}  predicted {0.99**n:.1%}  measured {run_trials(n):.1%}")

You are done when

Hard questions

Try to answer before you reveal. If you can answer these, you understood the lesson.

Q1Checkpoints fixed the decay. What did they cost, and when is the trade wrong?Reveal
Every checkpoint is a verification step: latency, tokens, and often a human. The trade is wrong when verification costs more than redoing the work — cheap reversible steps are better retried than checked. It is right whenever a step is expensive, irreversible, or feeds many later steps. In practice: checkpoint before writes, not before reads.

Please sign in to continue.

Questions people ask

Are these gaps fundamental or temporary?

Nobody knows. Several "fundamental" limits from 2019 fell to scale. Others have not moved at all. Treat confident claims in either direction with suspicion.

Does adding memory fix continual learning?

It fixes the symptom usefully — the system behaves as if it remembers. The underlying weights still do not change, which is why memory is a database problem with database vulnerabilities.

Why does compounding error matter so much?

Because agents are defined by taking many steps. Reliability that looks excellent per step becomes poor per task. It is the main practical reason long autonomous runs disappoint.

If models have no goals, why worry about alignment?

Because systems built around models do pursue objectives — you gave them one — and optimising an objective produces goal-shaped behaviour regardless of whether anything inside wants anything. Module 31.

Lesson test

5 questions. Get 3 right (60%) to pass and complete this lesson.

Sign in with your phone number to take the test and save your progress