Build an LLMMiddleLesson 436 min read

Step 7 · Data is the product

Architecture is nearly free — you can copy it. Data is where models actually differ, and where almost all the work is.

🧪 Lab · 25 min📝 Lesson test
Lesson in motion

In 60 seconds

Step 7 · Data is the product

Architecture is nearly free — you can copy it. Data is where models actually differ, and where almost all the work is.

1/7
In simple words
A student who reads good books becomes clever. A student who reads only spam becomes confused. The books are what matter, not the shape of the desk.
Two teams with identical code and identical compute will produce very different models if their data differs. Every frontier lab treats architecture as roughly public and data as the crown jewels.

The pipeline, in order

  1. 1

    1 · Acquire

    Web crawls, books, code, papers, curated collections. Check the licence. "It was on the internet" is not a licence, and this is now litigated regularly.
  2. 2

    2 · Extract

    HTML to text is harder than it looks. Boilerplate, navigation, cookie banners and comment spam all end up in naive extractions and all poison the model.
  3. 3

    3 · Filter for quality

    Language identification, a classifier trained on known-good text, perplexity filters, heuristics on punctuation and line length. Most crawled text is discarded — often 90% or more.
  4. 4

    4 · Deduplicate

    Exact and near-duplicate removal, usually with MinHash. This is the highest-value single step: duplicated text wastes compute and causes memorisation.
  5. 5

    5 · Decontaminate

    Remove anything matching your evaluation sets. Skip this and your benchmark numbers are fiction — see Module 28.
  6. 6

    6 · Mix

    Choose proportions per source and how many times each is repeated. This is a design decision with large effects on the final model.
  7. 7

    7 · Tokenize and shard

    Pre-tokenise into flat binary shards. Reading tokenised uint16 arrays is dramatically faster than tokenising on the fly.
Near-duplicate detection with MinHashpython
from datasketch import MinHash, MinHashLSH

def shingles(text, k=5):
    words = text.split()
    return {" ".join(words[i:i+k]) for i in range(len(words) - k + 1)}

lsh = MinHashLSH(threshold=0.8, num_perm=128)
kept = []
for i, doc in enumerate(documents):
    m = MinHash(num_perm=128)
    for sh in shingles(doc):
        m.update(sh.encode("utf-8"))
    if not lsh.query(m):          # nothing similar seen yet
        lsh.insert(str(i), m)
        kept.append(doc)

A realistic mixture

SourceShareWhy
Filtered web text50–65%Breadth, and the only source available at scale
Code10–20%Improves reasoning and structure even for non-code tasks
Books and long-form5–15%Long-range coherence, which web text rarely has
Academic and reference5–10%Factual density
Curated / synthetic5–15%Targeted skills, instruction shapes, underrepresented languages
MultilingualvariesDeliberate, or your model will be an English model that stumbles elsewhere
Do this
The code proportion surprises people. Training on code measurably improves performance on tasks that have nothing to do with programming. The best current explanation is that code is unusually rich in explicit long-range structure and step-by-step causality.

Synthetic data: the current frontier

High-quality human text is running out. So labs increasingly generate training data with existing models: textbook-style explanations, question-answer pairs, verified reasoning traces, and rewrites of messy web text into clean prose.
Where synthetic data works
  • Rewriting scrappy text into clear explanations.
  • Reasoning traces that can be verified — maths with checkable answers, code with tests.
  • Instruction and format diversity.
  • Filling gaps: rare languages, rare domains.
Where it goes wrong
  • Unverified facts — the generator's errors become training targets.
  • Style collapse: everything starts sounding the same.
  • Narrowing diversity, generation after generation.
  • Quietly training on your own benchmark answers.
Danger
Data poisoning is a genuine supply-chain risk at this layer. A small number of crafted documents in a crawl can install a backdoor: a trigger phrase that produces attacker-chosen behaviour. This is OWASP LLM04, and it is why provenance on training data matters as much as on dependencies.

How much data?

See Module 44 for the arithmetic. Short version: a compute-optimal model wants roughly 20 tokens per parameter, and models intended for heavy inference use are deliberately trained far past that — often 100–1000 tokens per parameter — because a smaller, longer-trained model is cheaper to serve forever.

Watch and read more

FineWeb: decanting the web for LLM dataHugging Face · readingLLM training data pipelinesYouTube search

Lab

A data pipeline that throws most of its input away, correctly.

~25 min

The problem

Take 10k raw web documents. Build the pipeline: extract, filter, deduplicate with MinHash, decontaminate against an eval set. Report what fraction survived each stage and read 20 rejected documents to check you are throwing away the right things.
Starter codepython
stages = ["raw", "extracted", "quality", "deduped", "decontaminated"]
for a, b in zip(stages, stages[1:]):
    print(f"{a:16} -> {b:16} {len(sets[b])/len(sets[a]):.1%} survived")

You are done when

Hard questions

Try to answer before you reveal. If you can answer these, you understood the lesson.

Q1Your quality filter keeps 8% of the crawl. Argue that this is too aggressive, then argue it is not.Reveal
Too aggressive: quality classifiers are trained on a narrow notion of 'good' — often Wikipedia-like prose — and systematically discard dialect, code-switching, informal registers and non-Western sources, so you are narrowing whose language the model learns. Not aggressive enough: most of a raw crawl is navigation chrome, spam and near-duplicates, and published pipelines routinely keep single-digit percentages while improving downstream loss. Resolve it by reading rejects by category rather than trusting the rate — the number tells you nothing without knowing what it removed.

Please sign in to continue.

Questions people ask

Can I just use a public dataset?

Yes, and you should to learn. FineWeb, The Pile, RedPajama, Dolma and similar collections are already filtered and deduplicated. Read their datasheets — the filtering choices they made are now your choices.

How much does deduplication matter?

A great deal. Published results show meaningful improvements in final loss and large reductions in verbatim memorisation. It is the highest return-per-hour step in the entire pipeline.

Is training on copyrighted text legal?

Actively contested, differs by jurisdiction, and being decided in court right now. If you are shipping commercially, get advice specific to your market rather than following what the big labs appear to do.

What about personal data in the crawl?

It is there, and models can memorise and reproduce it. Filter for personal information, support deletion requests, and understand that "the model memorised it" is not a defence under most privacy regimes.

Lesson test

5 questions. Get 3 right (60%) to pass and complete this lesson.

Sign in with your phone number to take the test and save your progress