Build an LLMAdvancedLesson 446 min read

Step 8 · Scaling laws and the compute budget

How to decide model size and dataset size before spending money. Move the sliders and watch the bill.

🧪 Lab · 15 min📝 Lesson test
Lesson in motion

In 60 seconds

Step 8 · Scaling laws and the compute budget

How to decide model size and dataset size before spending money. Move the sliders and watch the bill.

1/5
In simple words
If you have a fixed amount of money, you can buy a bigger brain or more books. There is a right balance, and people worked out roughly what it is.
Given a fixed compute budget, you must split it between parameters and training tokens. Too big a model on too little data wastes capacity; too small a model on too much data plateaus.

The three numbers you need

QuantityFormulaNote
Training computeC ≈ 6 · N · DN parameters, D tokens. Forward is 2ND, backward is 4ND
Compute-optimal ratioD ≈ 20 · NThe Chinchilla result — roughly 20 tokens per parameter
Inference compute≈ 2 · N per tokenWhich is why serving cost favours smaller models

Interactive · compute budget

7.1B
141.3B
$2.00
Training compute6.0e21
GPU-hours10k
Rough cost$21k
Tokens / parameter20x
Near compute-optimal.Around the Chinchilla ratio of ~20 tokens per parameter — best loss for this training budget. Right if training cost is what you are minimising, which it usually is not.Inference costs about 14.2 GFLOPs per generated token.

Why nobody actually trains compute-optimal any more

Chinchilla answers "what is the best model for a fixed training budget?" That is the wrong question if you are going to serve the model to millions of people, because then inference dominates lifetime cost.
  1. 1

    Compute-optimal

    A 70B model on 1.4T tokens. Best loss for the training budget. Expensive to serve forever.
  2. 2

    Inference-optimised

    An 8B model on 15T tokens. Worse loss per training dollar, far past "optimal" — and dramatically cheaper to run, small enough to fit on one GPU, and fast.
  3. 3

    The industry chose the second

    Nearly every widely deployed open model today is deliberately over-trained relative to Chinchilla. The training cost is paid once; the inference cost is paid forever.
Do this
Rule of thumb for anyone actually shipping: pick the smallest model that clears your quality bar, then train it much longer than the compute-optimal point. Your users experience latency and your finance team experiences inference cost; neither experiences your training FLOPs.

Estimating memory before you rent a GPU

Will it fit?python
def memory_gb(n_params, bytes_per_param=2, optimizer="adamw", train=True):
    weights = n_params * bytes_per_param            # bf16 weights
    if not train:
        return weights / 1e9 * 1.2                  # + KV cache and activations
    grads = n_params * bytes_per_param
    # AdamW keeps two fp32 moments, plus an fp32 master copy of the weights
    opt_state = n_params * 12 if optimizer == "adamw" else 0
    total = weights + grads + opt_state
    return total / 1e9 * 1.25                       # + activations, fragmentation

print(memory_gb(7e9))              # ~ 100 GB  -> will NOT fit on one 80GB card
print(memory_gb(7e9, train=False)) # ~ 17 GB   -> inference fits easily
print(memory_gb(1.5e9))            # ~ 22 GB   -> trainable on one 24GB card
Watch out
The gap between those two numbers is the thing beginners miss. Full training needs roughly 6–8 times the memory of inference, almost entirely because of optimiser state. This is exactly the gap LoRA exists to close — Module 52.

Emergence, and why the curve is not the whole story

Loss falls smoothly. Specific capabilities sometimes do not — arithmetic, multi-step reasoning and instruction-following have appeared fairly sharply at particular scales. How much of that is real and how much is an artefact of all-or-nothing metrics is genuinely debated.
The practical consequence: a small model can be excellent at style, format and narrow domains, while some abilities simply are not available below a certain scale, no matter how long you train.

Watch and read more

Scaling laws and ChinchillaYouTube searchTraining Compute-Optimal LLMsthe Chinchilla paper · reading

Lab

A compute budget defended against a real alternative.

~15 min

The problem

For a target quality, cost three plans: compute-optimal, inference-optimised, and using an existing API. Include training, one year of inference at your volume, and engineering time. Pick one and write the memo.

You are done when

Hard questions

Try to answer before you reveal. If you can answer these, you understood the lesson.

Q1Your inference-optimised plan trains a 3B model on 300B tokens — 100 tokens per parameter, far past Chinchilla. Justify it in cost terms.Reveal
Chinchilla minimises loss for a fixed training budget, a one-time cost. You are minimising total cost of ownership, where inference is paid per request forever. A 3B model serves at roughly a quarter the cost of a 12B one and fits on cheaper hardware, so at a few hundred million tokens a month the extra training compute pays back in weeks. The compute-optimal point is the right answer to a question you are not asking.

Please sign in to continue.

Questions people ask

Does the 6ND formula include attention cost?

It ignores the quadratic attention term, which is a good approximation while sequence length is small relative to model width. At very long context it understates the true cost substantially.

Are scaling laws still holding?

For loss versus compute, broadly yes, within the ranges that have been published. What has changed is that the binding constraints have moved to data availability, energy and inference economics rather than the curve itself.

Can I predict my final loss before training?

Approximately, by fitting a scaling law on a series of small runs at your own data mixture, then extrapolating. Labs do exactly this before committing to a large run, and it is a good habit at any scale.

What does one training run actually cost?

Rough orders of magnitude at 2026 cloud prices: a 100M model, tens of dollars. A 1B model, hundreds to low thousands. A 7B model trained properly, hundreds of thousands. Frontier scale, hundreds of millions.

Lesson test

5 questions. Get 3 right (60%) to pass and complete this lesson.

Sign in with your phone number to take the test and save your progress