Build an LLMMiddleLesson 496 min read

Step 13 · Teaching it to follow instructions

A base model completes text. It does not answer questions. Supervised fine-tuning is the step that turns one into the other.

🧪 Lab · 25 min 📝 Lesson test

Lesson in motion

In 60 seconds

Step 13 · Teaching it to follow instructions

A base model completes text. It does not answer questions. Supervised fine-tuning is the step that turns one into the other.

1/6

In simple words

A base model has read everything but has never been told what its job is. Ask it a question and it might write ten more questions, because that is what a list of questions looks like. SFT is teaching it "when someone asks, you answer".

Pretraining gives you raw capability. Supervised fine-tuning (SFT), also called instruction tuning, shapes it into an assistant by training on examples of good conversations.

The data format

One SFT examplepython

example = {
  "messages": [
    {"role": "system", "content": "You are a careful assistant."},
    {"role": "user", "content": "Why is the sky blue?"},
    {"role": "assistant", "content": "Sunlight contains all colours. Air "
     "scatters short wavelengths more than long ones, so blue light bounces "
     "around the sky and reaches your eyes from every direction."}
  ]
}

# Rendered with the model's chat template, then tokenised. Each role turn
# is wrapped in special tokens, and the newlines matter:
#
#   <|im_start|>system      + newline + You are a careful assistant. + <|im_end|>
#   <|im_start|>user        + newline + Why is the sky blue?        + <|im_end|>
#   <|im_start|>assistant   + newline + Sunlight contains...        + <|im_end|>
#
# Use tokenizer.apply_chat_template() rather than building this by hand --
# every model family differs, and a mismatch degrades quality silently.

Danger

Two mistakes here account for most failed fine-tunes. One: using the wrong chat template — every model family has its own, and a mismatch degrades quality badly while still "working". Two: computing loss on the whole sequence instead of only the assistant tokens.

Mask the loss to assistant tokens onlypython

def build_labels(input_ids, assistant_spans, ignore=-100):
    """Only the assistant's own tokens contribute to the loss."""
    labels = torch.full_like(input_ids, ignore)
    for start, end in assistant_spans:
        labels[start:end] = input_ids[start:end]
    return labels

# cross_entropy ignores -100 by default, so the model is never
# trained to generate the user's questions back at you.

Quality beats quantity, dramatically

Dataset	Size	Result
Scraped chat logs	500,000	Mediocre — inconsistent style, factual errors learned as targets
Carefully curated	1,000–10,000	Often better on every axis
Curated + verified synthetic	20,000–100,000	The current mainstream recipe
Your own domain examples	200–2,000	Excellent for narrow, format-heavy tasks

Do this

This surprises people every time: a thousand excellent examples usually beat a hundred thousand mediocre ones. The model is not learning facts here — it already has those. It is learning format, tone, refusal behaviour and how to be helpful. Those transfer from very few examples.

Hyperparameters that differ from pretraining

Learning rate 10–100x lower — typically 1e-5 to 2e-5 for full fine-tuning, 1e-4 to 2e-4 for LoRA.
1–3 epochs only. More and you get memorisation and a sharp drop in general ability.
Small batches are fine — 32 to 128 sequences is normal.
Watch for catastrophic forgetting. Evaluate on general benchmarks, not only your task. A model that got great at your format and forgot how to reason is a bad trade.

What SFT can and cannot fix

SFT is the right tool for

Output format and structure.
Tone, persona and length.
Domain-specific phrasing and jargon.
Reliable tool-call formatting.
Refusal behaviour on a defined policy.

SFT is the wrong tool for

Adding facts — use retrieval instead.
Fixing reasoning — that comes from pretraining scale and Module 51.
Keeping up with changing information.
Anything you could achieve with a better prompt, which you should try first.

Watch out

A serious caution for Track B: fine-tuning can strip safety behaviour. Published work has shown that a small number of benign-looking examples can substantially degrade a model's refusal training. If you fine-tune, re-run your safety evaluations afterwards — do not assume the base model's properties survived.

Watch and read more

**State of GPT: pretraining to RLHF**Andrej Karpathy · video

Fine-tuning explainedYouTube search

Lab

An SFT run where the loss mask is the whole lesson.

~25 min

The problem

Fine-tune a small model on 200 conversations. Run it twice: once with loss over the entire sequence, once masked to assistant tokens only. Compare the outputs.

Starter codepython

labels = input_ids.clone()
labels[~assistant_mask] = -100        # cross_entropy ignores -100
# Run once without this line. Read what the model generates. That is the lesson.

You are done when

The unmasked model generates user turns and system prompts
The masked model answers
You used the model's own chat template

Hard questions

Try to answer before you reveal. If you can answer these, you understood the lesson.

Q1Your fine-tune is excellent on your task and much worse at general reasoning. Diagnose and fix.Reveal

Catastrophic forgetting: full fine-tuning on a narrow distribution moves weights away from general capability. Fixes in order of cost: use LoRA so base weights are frozen; lower the learning rate and train fewer epochs; mix 10-30% general instruction data into your set; and always evaluate on a general benchmark alongside your task, or you will not notice until users do.

Please sign in to continue.

Questions people ask

How many examples do I need?

For a narrow format task, 200–1,000 well-made examples often get you most of the way. For a general assistant, tens of thousands. Start small, evaluate, and only add data where evaluation shows a gap.

Full fine-tune or LoRA?

LoRA for nearly everyone — Module 52. Full fine-tuning is worth it when you have a lot of data, need a large behaviour shift, and have the memory for it.

How do I know it worked?

A held-out set of your real task, scored by a rubric or a human, plus a general benchmark to detect forgetting. Training loss going down tells you almost nothing about whether the model got better at the job.

Can I fine-tune on my company documents?

You can, and it usually disappoints. The model picks up style but recalls facts unreliably and cannot be updated when documents change. Retrieval is the right tool for knowledge; fine-tuning is the right tool for behaviour.

Lesson test

5 questions. Get 3 right (60%) to pass and complete this lesson.