Ariyan Basu

The base model from Module 02 is a paradox: it has read most of the internet and knows an extraordinary amount, yet it's almost unusable as an assistant. It doesn't answer questions, it continues them. It has no sense of when to stop, no preference for being helpful over being plausible, no notion of which requests it should decline. All that knowledge is locked behind the wrong behaviour.

Post-training (or "alignment") is the relatively short, cheap phase that unlocks it — typically a tiny fraction of pretraining's compute, yet it's the difference between GPT-2 and ChatGPT. It doesn't teach the model new facts so much as teach it how to use what it already knows: follow instructions, prefer good answers over bad, call tools when it can't know something, and refuse genuine harm. The stages below run in sequence, each building on the last. Click through them for the map, then dig into each one.

Supervised fine-tuning is the first and most impactful step. The recipe is simple: collect a dataset of high-quality (instruction, ideal response) pairs — written by experts, or increasingly generated and filtered by other strong models — wrap each in a chat template (the <|user|> / <|assistant|> special tokens that mark turns), and continue training with the exact same next-token objective from pretraining. The model is just imitating good answers, but that imitation is enough to flip it from "text completer" into "instruction follower."

Quality beats quantity dramatically here. The influential LIMA result showed that as few as ~1,000 carefully-curated examples can produce a capable assistant, because SFT mostly surfaces abilities the base model already has rather than teaching new ones. The one technical subtlety is loss masking: the loss is computed only over the response tokens, not the prompt or template. The model reads the prompt as context but is graded solely on producing the answer — so it learns to respond, never to parrot the question back. Toggle the mask below to see exactly which tokens are trained.

SFT has a ceiling. For most prompts there's no single "ideal" answer to imitate — there's a spectrum from great to terrible, and the differences (tone, helpfulness, honesty, not being preachy) are easier for a human to compare than to write from scratch. So the next stage learns from preferences: show people two model answers, ask which is better, and train the model to produce more of the preferred kind.

The classic method is RLHF: train a separate reward model to predict human preference, then use reinforcement learning (PPO) to optimize the LLM against that reward — while a KL penalty keeps it from drifting too far from the SFT model and collapsing. It works, but it's a fiddly, unstable, multi-model dance. DPO (Direct Preference Optimization) is the elegant shortcut that has largely taken over: a clever derivation shows you can skip the separate reward model and RL loop entirely, and instead directly fine-tune on (chosen, rejected) pairs with a simple classification-style loss. It's what Meta used for Llama 3 — simpler, cheaper, more stable at scale. The β term controls how aggressively each update widens the gap between chosen and rejected. Run a few steps below and watch the preference margin grow.

A language model alone is a closed box: it can't do exact arithmetic reliably, can't know today's weather, can't run your code. Tool use (or "function calling") fixes this. During post-training the model is taught a format for emitting a structured call — calculator(...), search(...), an API — which the surrounding system executes; the result is fed back in, and the model writes its final answer using real data instead of a hallucinated guess. The crucial learned skill is judgment: knowing when a tool is needed (live or exact information) versus when it can just answer. This is the foundation of "agents."

Safety tuning teaches the boundary. Using the same SFT and preference machinery, the model learns to refuse genuinely harmful requests — weapons, malware, abuse — while still helping with the vast majority of benign ones, including superficially scary-sounding ones ("how do I kill a Python process?"). The hard part is calibration: over-refusing is its own failure mode that makes a model useless and patronizing. Good safety tuning refuses narrowly and, where possible, offers a safe alternative. Click the prompts below to watch the assistant decide: answer directly, reach for a tool, or decline.

Tülu 3's pipeline is the whole module made concrete, in order: careful SFT data curation across many skills, then DPO on large-scale preference data, then a novel final stage Allen AI calls RLVR — Reinforcement Learning with Verifiable Rewards. RLVR's insight is that for tasks with a checkable answer (math problems, code that must pass tests, instructions with format constraints), you don't need a noisy learned reward model at all — you can reward the model directly when its answer is verifiably correct. That clean signal pushes the hardest reasoning skills higher than preference data alone can. It's the same family of idea behind the "reasoning model" training that followed in 2025.

Step through the stages below and watch the average benchmark score climb — and notice the shape of the gains: SFT delivers the biggest single jump (teaching behaviour), DPO refines helpfulness and tone, and RLVR adds a final lift concentrated in verifiable domains like math.