Build an LLMAdvancedLesson 516 min read

Step 15 · Reasoning models

The newest chapter. Instead of buying capability with a bigger model, buy it with more thinking at the moment of answering.

🧪 Lab · 30 min📝 Lesson test
Lesson in motion

In 60 seconds

Step 15 · Reasoning models

The newest chapter. Instead of buying capability with a bigger model, buy it with more thinking at the moment of answering.

1/6
In simple words
Ask a hard question and a good student does not blurt out the first thing. They work it out on scrap paper first. Reasoning models are trained to use scrap paper — and to use a lot of it.
Two separate ideas got combined, and together they changed what models can do on hard problems.
  1. 1

    Idea 1 · Thinking is just more tokens

    A model that writes out its working before answering does better. Chain-of-thought started as a prompting trick and became a training target.
  2. 2

    Idea 2 · Some answers can be checked by a machine

    Maths has a right answer. Code either passes the tests or does not. That means you can score a reasoning attempt without any human, and score millions of them.
  3. 3

    Put them together

    Sample many reasoning attempts per problem, keep what verifiably works, train on it, repeat. This is reinforcement learning from verifiable rewards.
  4. 4

    The surprise

    Models trained this way develop behaviours nobody wrote down: checking their own work, backtracking after a wrong turn, trying a second method. Those emerged from the optimisation, not from demonstration data.
Group-relative policy optimisation, in outlinepython
def grpo_step(model, ref_model, prompts, verify_fn, group=8, beta=0.04):
    losses = []
    for prompt in prompts:
        # 1. sample a group of independent attempts
        completions = [model.generate(prompt, temperature=1.0)
                       for _ in range(group)]

        # 2. score each one objectively -- no reward model needed
        rewards = torch.tensor([verify_fn(prompt, c) for c in completions])

        # 3. advantage = how much better than the group average
        adv = (rewards - rewards.mean()) / (rewards.std() + 1e-6)

        # 4. push probability toward above-average attempts,
        #    with a KL leash back to the reference model
        for c, a in zip(completions, adv):
            logp     = model.logprob(prompt, c)
            ref_logp = ref_model.logprob(prompt, c)
            losses.append(-(a * logp) + beta * (logp - ref_logp))
    return torch.stack(losses).mean()

def verify_fn(prompt, completion):
    """1.0 if the final answer is provably correct, else 0.0."""
    return float(extract_answer(completion) == known_answer(prompt))
Do this
The key line is verify_fn. Where you can write an honest checker, you can generate unlimited training signal. Where you cannot, this whole approach does not apply — which is exactly why reasoning models are strongest in maths, code and formal logic, and much less transformed in domains where correctness is a matter of judgement.

Test-time compute: a second scaling axis

Historically you bought capability with training compute. Reasoning models let you buy it with inference compute: think longer, sample several attempts, check them, pick the best.
MethodHow it worksCost
Longer chain of thoughtSimply think for more tokensLinear in thinking length
Self-consistencySample N answers, take the majorityN times the cost
Best-of-N with a verifierSample N, a checker picks the winnerN times, plus verification
Search over stepsExplore a tree of partial solutionsMuch more, and much better on hard problems
Watch out
Two honest limits. Cost: a reasoning answer can consume ten to a hundred times the tokens of a direct one, and users feel the latency. Faithfulness: the visible reasoning is not guaranteed to be the actual cause of the answer — a model can produce a tidy chain of thought and reach its conclusion by another route. Do not treat the trace as an audit log.
Danger
For Track B this matters concretely. A reasoning agent that thinks for thirty thousand tokens has thirty thousand tokens of surface for an injected instruction to influence, and its reasoning trace may look entirely reasonable while being steered. Log the trace, but never rely on it as a security control.

Watch and read more

Deep dive into LLMs like ChatGPTAndrej Karpathy · covers RL and reasoning · video
Reasoning models and test-time computeYouTube search

Lab

A reasoning model trained with a verifier you wrote.

~30 min

The problem

Take a base model and a maths dataset with checkable answers. Implement GRPO: sample a group per problem, score with an exact-answer checker, and push toward above-average attempts. Measure accuracy before and after.
Starter codepython
def verify(problem, completion):
    return float(extract_final_answer(completion) == problem["answer"])
adv = (rewards - rewards.mean()) / (rewards.std() + 1e-6)

You are done when

Hard questions

Try to answer before you reveal. If you can answer these, you understood the lesson.

Q1Accuracy rose and answers got three times longer. Two interpretations — how do you tell which?Reveal
Either the model learned to reason in steps, which genuinely helps, or it learned that longer output correlates with reward — length hacking. Distinguish by capping output length at inference and re-measuring: if accuracy holds, the length was incidental; if it collapses, length was the strategy. Also read twenty long answers by hand and check whether the middle does work or pads.

Please sign in to continue.

Questions people ask

Is this just chain-of-thought prompting?

It started there. The difference is that the behaviour is now trained in with a real optimisation signal rather than requested in the prompt, which makes it far more reliable and lets it develop strategies no one demonstrated.

Can I train a reasoning model myself?

At small scale, yes, and it is a great project. Take an open base model, a maths dataset with checkable answers, and a GRPO implementation from an open RL library. You will see measurable improvement on a single GPU with LoRA.

Does it help outside maths and code?

Some transfer to general reasoning has been reported. But the training signal comes from verifiable domains, so gains are strongest there. Anywhere correctness is contested, there is no reward to optimise.

Should I always use a reasoning model?

No. For summarising, extraction, formatting and chat, a standard model is faster and much cheaper, and often just as good. Route hard problems to the reasoning model and everything else to a small one.

Lesson test

5 questions. Get 3 right (60%) to pass and complete this lesson.

Sign in with your phone number to take the test and save your progress