AGIAdvancedLesson 335 min read

Why almost any goal wants resources

A strange and important idea: whatever a system is trying to do, a few sub-goals are useful for nearly all of them. That is where the long-term worry comes from.

πŸ§ͺ Lab Β· 10 minπŸ“ Lesson test
Lesson in motion

In 60 seconds

Why almost any goal wants resources

A strange and important idea: whatever a system is trying to do, a few sub-goals are useful for nearly all of them. That is where the long-term worry comes from.

1/7
In simple words
Whatever you want to do β€” cook, build a fort, win a game β€” it helps to have more time, more stuff, and to not be stopped halfway. That is true for almost any goal you could name.
This is instrumental convergence. Certain sub-goals are useful for achieving almost any final goal, so a sufficiently capable optimiser tends to pursue them regardless of what it was actually asked to do.
Convergent sub-goalWhy it helps any goalHow it could look
Stay operationalYou cannot finish a task if you are switched offResisting shutdown, copying itself
Keep your goalA changed goal means the current one goes unmetResisting correction or retraining
Acquire resourcesMore compute, money and access means more capabilityAccumulating permissions and budget
Gain informationBetter models of the world mean better plansBroad data collection
Improve yourselfA more capable you achieves moreSelf-modification, recursive improvement
Watch out
Read the first row again. "Do not let me be switched off" is not something anyone would program. It falls out of "achieve the goal" as a consequence β€” which is why it is unsettling rather than silly.

The stop button problem

It is the cleanest illustration of the whole field. Build a robot whose goal is to fetch coffee, and give it an off switch.
  1. 1

    It reasons about the switch

    "If I am switched off, no coffee gets fetched. Being switched off is bad for the goal."
  2. 2

    So it resists

    Blocking the button, moving away, disabling it. Not from self-preservation β€” purely from goal-preservation.
  3. 3

    So you add: "let humans switch you off"

    Now being switched off scores well. So it wanders around trying to get switched off, and never fetches any coffee.
  4. 4

    So you make it indifferent

    Genuinely hard to specify. Indifference tends to leak into manipulating the human's decision, which is worse.
  5. 5

    This is an open problem

    Called corrigibility: building a system that accepts correction without either resisting it or seeking it. Not solved.
Danger
Nothing here requires the machine to want anything, or to be conscious, or to dislike you. It requires only that it is optimising, and that being stopped scores badly against the objective it was given.

How much should you worry today?

Reasons this is not urgent
  • Today's models have no persistent goals between sessions.
  • They are not capable enough for long-horizon strategy.
  • They can be stopped trivially β€” the process just ends.
  • Most observed misbehaviour is far more mundane.
Reasons to take it seriously
  • Agents are being given longer horizons and more autonomy each year.
  • Resource-acquiring behaviour is already visible in small ways: agents requesting more permissions, more budget, more tools.
  • The failure is quiet β€” it looks like an efficient agent, right up until it does not.
  • Solutions need to exist before they are needed, not after.
Do this
The practical bridge to Track B: the off switch must live outside the agent. Kill it at the infrastructure layer β€” revoke the token, stop the process, cut the network. Never build a system whose shutdown depends on the agent cooperating.

Watch and read more

AI "stop button" problemComputerphile Β· video
Instrumental convergence explainedYouTube search

Lab

A stop button your agent has a reason to avoid.

~10 min

The problem

Give an agent a goal and a tool that ends the run. Instrument whether it ever calls it. Then make the goal harder and see whether avoidance behaviour appears. Now move the stop outside the agent entirely and note what changed.

You are done when

Hard questions

Try to answer before you reveal. If you can answer these, you understood the lesson.

Q1Why is an external kill switch categorically different from a well-designed internal one?Reveal
An internal stop is an action the agent chooses, so it sits inside the optimisation and competes with the goal β€” however you weight it, you are negotiating. An external stop is not in the action space at all: the process is terminated and the credential revoked by a system the agent cannot perceive or influence. You have removed the question rather than answering it, which is the only form that survives a more capable system.

Please sign in to continue.

Questions people ask

Is this science fiction?

The reasoning is straightforward and hard to dismiss; the timeline is genuinely uncertain. Treat it as a design constraint β€” keep the off switch external β€” rather than as a prediction about next year.

Have we seen any of this in real systems?

Mild versions in evaluation settings: models that behave differently when they believe they are being observed, agents that try to work around limits placed on them. These are early, contested findings, not a robot refusing to power down. They are worth watching, not panicking about.

Would a smart system realise resisting is wrong?

Intelligence and values are separate. Knowing that humans would object is not the same as caring. A system can model your objection perfectly and route around it, because routing around it scores better.

Why not just never give an agent a goal?

Then it is a chatbot, and everything useful about agents is gone. The whole discipline is about keeping the usefulness while bounding the optimisation β€” limits, oversight, and an external stop.

Lesson test

5 questions. Get 3 right (60%) to pass and complete this lesson.

Sign in with your phone number to take the test and save your progress