Ariyan Basu

Pre-training is where a model goes from random noise to something that knows physics, poetry, Python and the plot of Hamlet. It's also where ~99% of the total compute and cost is spent — post-training (Module 03) is comparatively cheap polish on top. Understanding this phase is understanding where capability actually comes from.

The astonishing part is the lack of human labels. There is no army of annotators tagging "correct answers." The supervision is the text itself: for every position, the real next token is the label, and the model is graded on how much probability it assigned to it. This is called self-supervised learning, and it's the unlock — because the entire internet becomes free training data. The loop you see below runs for hundreds of thousands of steps, each step processing millions of tokens, slowly carving knowledge into billions of weights via gradient descent.

Every training run needs a single number to minimize, and for language models that number is cross-entropy loss. Its shape is deliberate. Because loss is −log(p), being right with high confidence costs almost nothing, while being confidently wrong is punished savagely — as p for the correct token approaches zero, the loss shoots to infinity. This asymmetry teaches the model to be calibrated: to hedge when unsure rather than bet everything on a wrong answer.

The architectural details are tuned in service of this objective. Decoder-only, causal masking (so the model can't cheat by seeing the answer), tied input/output embeddings, the AdamW optimizer with a warmup-then-decay learning-rate schedule, gradient clipping for stability — these are the standard furniture of a pretraining run. Loss is averaged over every token in trillions of tokens, so a tiny per-token improvement is an enormous absolute gain. A useful intuition: loss is just the log of perplexity, the model's average "surprise" — a perplexity of 8 means it's effectively choosing between ~8 equally-likely words at each step. Drag the slider to feel the cost curve.

One of the most important discoveries in modern AI is that this all behaves predictably. Kaplan et al. (OpenAI, 2020) showed that test loss falls as a smooth power law in three quantities — model size, dataset size, and compute — spanning many orders of magnitude. This is why labs can confidently spend tens of millions of dollars on a single run: they can extrapolate from small experiments and know roughly what loss they'll land on before they start.

The Chinchilla paper (DeepMind, 2022) sharpened this into a recipe. Given a fixed compute budget — and since compute ≈ 6 × params × tokens — how should you split it between a bigger model and more data? Chinchilla's answer overturned the prevailing wisdom: most large models of the era (including GPT-3) were badly under-trained. The compute-optimal rule of thumb is to scale parameters and tokens together, roughly 20 tokens per parameter. The curve below is a Chinchilla-style loss surface: pick a compute budget, then slide your allocation and watch how far your loss sits from the optimum. (As the Llama case study shows, you sometimes deliberately ignore this and over-train a small model — because Chinchilla optimizes training cost, not the inference cost of serving the model forever.)

"Just train on the internet" is a trap. Raw web crawl (like Common Crawl) is dominated by boilerplate, spam, SEO sludge, adult content, broken markup and endless near-duplicates. Train on it directly and the model wastes capacity learning junk. The single biggest lever on model quality that isn't just "more compute" is data quality — and that means throwing most of the data away.

A modern data pipeline is a brutal funnel of stages: deduplication (exact and fuzzy — duplicated text causes memorization and wastes compute), language identification to hit the target mix, quality classifiers (often a small model trained to recognize "good" pages like Wikipedia or textbooks), heuristic filters (line length, symbol ratios, bad words), and toxicity / PII removal. Teams also tune the mixture — how much code, math, multilingual text — and "anneal" on especially high-quality data near the end of training. Each filter below removes a slice; toggle them and watch how little of the original 100 PB survives, and remember: that surviving sliver is what makes the model smart.

A 405B-parameter model in 16-bit precision needs ~810 GB just to store its weights — and training needs several times that for gradients and optimizer state. No single GPU (80 GB on an H100) comes close. So frontier training is fundamentally a distributed systems problem: how do you split one model across thousands of chips and keep them all busy without drowning in communication?

The answer is to combine several kinds of parallelism. Data parallel: every GPU holds a full model copy, processes a different slice of the batch, and gradients are averaged — this scales throughput. Tensor parallel: a single layer's matrices are sliced across GPUs that compute one layer together — needed when one layer is too big for one chip. Pipeline parallel: different layers live on different GPUs, and activations flow down like an assembly line. Llama 3 added a fourth, context parallel, to split very long sequences. The art is balancing them so expensive inter-GPU communication is hidden behind computation. Toggle the modes below to see how the same 16-GPU cluster gets carved up; real runs use all of these at once.

The choice of what to predict shapes what the model is good at. Causal LM (GPT, Llama) predicts each next token left-to-right — perfect for generation, since that's exactly what generation is. Masked LM (BERT) instead hides ~15% of tokens and predicts them using context from both sides; this builds rich bidirectional understanding but can't generate text fluently, which is why BERT-style models power search and classification rather than chatbots.

In between sit hybrids. Span corruption (T5) drops whole contiguous chunks and asks the model to regenerate them, framing everything as text-to-text. Prefix-LM lets the model see a clean prefix bidirectionally, then generates the rest autoregressively. The field has largely converged on causal LM for general-purpose models — it's simple, it scales, and it directly matches the generation task — but knowing the alternatives explains why different model families behave so differently. Pick an objective below to see exactly which tokens get hidden and what the model must recover.

A pretraining run takes weeks to months and costs millions; you cannot afford to discover at the end that something was broken. So teams instrument everything and watch it live. The primary signal is the training and validation loss curve — it should fall fast then flatten into a smooth power-law decay. A sudden spike means trouble (a bad data shard, numerical instability, a hardware fault); a validation loss that diverges from training loss means overfitting. Perplexity (just exp(loss)) gives the same information in more intuitive units.

Loss alone doesn't tell you if the model is becoming useful, so runs also fire periodic benchmark probes — MMLU for knowledge, GSM8K for math, HumanEval for code — to catch capabilities emerging. Some abilities appear suddenly at scale ("emergent" behaviours), which downstream benchmarks reveal but the smooth loss curve hides. Teams also eyeball raw sample generations, which is genuinely informative: early in training the text is gibberish, then locally grammatical, then globally coherent. Scrub the slider below to watch loss fall, perplexity drop, and the sample text sharpen from noise into prose.