Ariyan Basu

That one-sentence definition hides something remarkable. "Predict the next token" sounds almost trivial, yet to do it well across all of human text, a model is forced to absorb grammar, facts, reasoning patterns, translation, code, and a rough model of the world — because all of those are needed to guess what comes next. Capability is a side-effect of compression: the better you compress text, the more you must have understood it.

The architecture that does this is the Transformer (Vaswani et al., 2017). Modern LLMs use the decoder-only variant: text flows left-to-right, each token can only see the tokens before it, and the same stack of operations repeats at every position in parallel. The genius of the design is that it's almost entirely matrix multiplications — which is exactly what GPUs do fastest — so it scales to thousands of chips and trillions of words. As you walk through the seven stages below, keep one thing in mind: there is no separate "knowledge database" inside. Everything the model knows lives in the numbers (the weights) that these operations multiply against.

Why not just feed the model words, or letters? Words fail because language has an unbounded vocabulary — every typo, name, hashtag and foreign word would be an unknown. Single characters work but waste the model's effort spelling out common words letter by letter. Subword tokenization is the compromise that won: keep frequent words whole, and break rare ones into reusable fragments, so a fixed vocabulary of ~50k–130k tokens can express any string.

The dominant method is Byte-Pair Encoding (BPE). It starts from individual bytes and repeatedly merges the most frequent adjacent pair into a new token, learning a vocabulary bottom-up from the training corpus. The result is that "the" is one token, but "tokenization" might be token + ization. This has real consequences you can feel as a user: tokens are why models charge per-token, why they're weak at character-level tasks like counting letters or rhyming, and why a leading space changes a token's identity. Try a rare or made-up word below and watch it shatter into pieces.

A token id like 50143 is just an arbitrary index — it carries no meaning on its own. The embedding layer is a giant lookup table (in GPT-2, 50,257 × 768 numbers) that turns each id into a dense vector of, say, 768 real numbers. These vectors are learned during training: nothing tells the model that "king" and "queen" are related, but because they appear in similar contexts, gradient descent nudges their vectors close together. Meaning is never stored as a definition — it's stored as a position in space.

This is what makes the famous "word arithmetic" possible: in a well-trained space, king − man + woman ≈ queen, because directions in the space encode consistent relationships (gender, tense, plurality). We measure closeness with cosine similarity — the angle between two vectors — rather than raw distance, because what matters is direction, not magnitude. The scatter below is a flattened 2-D shadow of a high-dimensional space; pick any two words and watch their similarity score, and notice how words from the same category cluster.

Here's a subtle problem. The attention mechanism (next topic) treats its input as a set, not a sequence — it has no built-in notion of first, second, third. Shuffle the words and the raw math gives the same answer. Yet order is everything in language. So we have to add position information explicitly.

The original Transformer used sinusoidal encodings: each position gets a unique fingerprint made of sine and cosine waves at many frequencies, added to its embedding. Low-frequency dimensions change slowly across the sequence (coarse position), high-frequency ones change fast (fine position) — together they give every position a distinct, smoothly varying signature the model can do arithmetic on, even for sequences longer than it saw in training. Modern models like Llama use a refinement called RoPE (rotary position embeddings) that rotates the query and key vectors by an angle proportional to position, which encodes relative distance directly inside attention and extrapolates better to long contexts. The heatmap below shows the classic sinusoidal pattern — each row is a position, each column a dimension.

Attention is the idea that made the Transformer — the "T" in GPT — and it's worth slowing down on. Think of it as a soft, learned database lookup happening at every position simultaneously. Each token projects its vector three ways: a query (what am I looking for?), a key (what do I offer?), and a value (what will I contribute if chosen?). To update a token, the model takes the dot product of its query with every key, turns those scores into weights with softmax, and blends the corresponding values. A pronoun like "it" can thus reach back and pull in the noun it refers to; a verb can gather its subject and object.

Two details matter. First, this is causal (masked) attention: a token may only attend to tokens at or before its own position — you can't peek at the future you're trying to predict. Second, models run many attention "heads" in parallel, each free to specialize — one might track syntax, another long-range coreference, another simple position. The grid below is one head's weights; click any query token to see exactly what it chose to look at, and watch how it can only see backwards.

Attention alone only moves information between tokens; it doesn't do much computation on each one. That's the job of the second half of every block: a feed-forward network (FFN), a little two-layer MLP applied independently to each position, usually expanding to 4× the width and back. If attention is "gather what's relevant," the FFN is "now think about it." Much of a model's factual knowledge is believed to live in these FFN weights.

Two unglamorous pieces make deep stacks trainable. Residual connections add each sub-layer's output back to its input, giving gradients a clean highway down through dozens of layers so they don't vanish. Normalization (LayerNorm, or RMSNorm in newer models) keeps the numbers in a stable range at every step. Stack this block 12, 80, even 126 times and a rough hierarchy emerges — early layers handle surface form, middle layers handle syntax and reference, late layers handle meaning, task structure and reasoning. Drag the slider and watch the kinds of features that tend to appear as depth grows.

After the last block, each position holds a richly-processed vector. To predict, the model multiplies that vector by an output matrix (often the embedding table reused, "tied weights") to produce one logit — a raw score — for every token in the vocabulary. Softmax exponentiates and normalizes those scores into probabilities that sum to 1. That distribution is the model's answer; everything else is how we pick from it.

Temperature reshapes the distribution before sampling: divide the logits by a small number (<1) and the peaks sharpen — the model plays it safe and nearly always takes the top token; divide by a larger number (>1) and the distribution flattens — lower-probability, more surprising tokens get a real chance. This is the single knob behind "make it more creative" vs. "make it more focused." Real systems add tricks like top-k and top-p (nucleus) sampling to clip the long tail of nonsense. Generation is autoregressive: sample one token, append it, feed the whole thing back in, repeat. Adjust the temperature below, then hit sample a few times to feel the randomness.

Here's a counterintuitive fact: the base model that comes out of pretraining is not the helpful assistant you talk to. It's a pure text-continuation engine. Ask it "What is the capital of France?" and it might continue with "What is the capital of Germany? What is the capital of Spain?" — because that's a plausible continuation of a quiz, not because it's unhelpful. The knowledge is all there; the behaviour isn't.

Two steps fix this. Instruction tuning (supervised fine-tuning) shows the model thousands of example (instruction → good answer) pairs so it learns the assistant format and to actually respond. Then RLHF aligns it with human preference: the model generates several answers, humans (or a reward model trained on human rankings) say which is better, and the model is updated to produce more of what people prefer — more helpful, honest and harmless. This is a quick preview; Module 03 is entirely about this post-training stage. Toggle base vs. aligned below to see the difference one short fine-tuning phase makes.