Transformers & Large Language Models

From RNNs to attention: why Transformers exist

Language is a sequence, and the meaning of a word depends on words that may be far away. In "The animal didn't cross the street because it was too tired", the word it refers to animal, six words earlier. Any model of language needs a way to connect distant positions. The history of NLP architectures is really a history of better ways to make those connections.

The recurrent approach and its problems

A recurrent neural network (RNN) reads one token at a time and keeps a running summary, the hidden state: ht = tanh(Whht-1 + Wxxt + b). LSTMs and GRUs add gates that decide what to keep and what to forget, which helps gradients survive longer. But the design has five structural weaknesses:

1. No parallelism Step t cannot start until step t-1 finishes. Training on a 1,000-token sequence needs 1,000 sequential steps, so GPUs sit mostly idle.
2. Long information path For token 1 to influence token 500, its signal must pass through 499 hidden-state updates. Each update dilutes it. The path length is O(n).
3. Vanishing and exploding gradients Backpropagation through time multiplies many Jacobians together. If their size is below 1 the gradient shrinks exponentially; above 1 it blows up. LSTMs soften this but do not remove it.
4. Fixed-size memory The whole history is squeezed into one vector of fixed size, no matter how long the input is.
5. Uniform treatment The model has no explicit mechanism to decide which earlier words matter most for the current one.

Seq2seq and the bottleneck

For translation, the classic design used two RNNs: an encoder reads the source sentence and its final hidden state becomes the context vector; a decoder generates the target sentence from that single vector. For a 50-word sentence, everything the decoder needs is forced through one vector. Quality collapsed as sentences got longer. This is the fixed-length context bottleneck.

Attention fixes the bottleneck

The fix (introduced for translation in 2014-2015, often called additive attention) was simple: keep all encoder hidden states, and at every decoder step compute a fresh weighted average of them. When generating "Je", the decoder puts most weight on "I"; when generating "t'aime", it shifts weight to "love". The context is recomputed per output step, so nothing has to be crammed into one vector.

But RNN+attention still had an RNN inside: the encoder was still sequential, tokens inside the source could not attend to each other directly, and training was still slow. The 2017 paper "Attention Is All You Need" asked: what if we delete the recurrence entirely and build the whole model out of attention?

The representation story that led here

In parallel, the way words are represented evolved. It is worth knowing because interviewers often ask "why did we need contextual embeddings?"

If you need a refresher on RNNs, LSTMs, backpropagation or word embeddings, see Deep Learning and NLP Fundamentals.

Tokens, embeddings and token economics

A Transformer never sees characters or words. Text is first split into tokens by a tokenizer, each token is mapped to an integer ID, and each ID is mapped to a learned vector (its embedding). Everything downstream, including context limits and API prices, is measured in tokens.

Tokenization recap

Worked example: three BPE merges

Suppose the training corpus has these words with counts: play x6, played x3, playing x4, say x5. Start with single characters.

1. Count adjacent pairs "a y" appears in play (6), played (3), playing (4) and say (5) = 18 times. "p l" and "l a" each appear 13 times. Everything else is lower.
2. Merge 1: a + y → ay Words become p l ay, p l ay e d, p l ay i n g, s ay.
3. Merge 2: p + l → pl (13 occurrences; ties are broken by a fixed rule). Words become pl ay, ...
4. Merge 3: pl + ay → play (13). Now play is one token, and played = play e d. Further merges would create ed, in, ing.
5. Result Frequent words become single tokens; an unseen word like "replaying" is still encodable as pieces such as re + play + ing.

# Inspect tokenization with two popular libraries
import tiktoken
enc = tiktoken.get_encoding("cl100k_base")            # a GPT-4-era BPE vocabulary
ids = enc.encode("Tokenization isn't free!")
print(ids, len(ids))
print([enc.decode([i]) for i in ids])                   # see each piece, note leading spaces

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("bert-base-uncased")  # WordPiece
print(tok.tokenize("Tokenization isn't free!"))
# e.g. ['token', '##ization', 'isn', "'", 't', 'free', '!']

Tokenization quirks that matter in practice

• Leading spaces are part of tokens. " hello" and "hello" are usually different IDs. This is why prompt formatting and trailing spaces can change outputs.
• Numbers split oddly. "12345" might become "123" + "45". This is one reason LLMs are weak at digit-level arithmetic. Some tokenizers (Llama 3, many newer ones) split numbers into single digits or groups of three to help.
• Letters are invisible. The model sees "strawberry" as maybe two or three tokens, not ten letters, so "how many r's in strawberry?" is genuinely hard without reasoning or tools.
• Languages are not equal. A vocabulary trained mostly on English packs English efficiently (about 4 characters or 0.75 words per token) but may need 2-4x more tokens for Hindi, Tamil, Thai or Korean. Same content, higher cost, less effective context.
• Special tokens. [CLS], [SEP], [MASK] (BERT), <s>, </s>, end-of-sequence (EOS), padding, and chat-template markers that separate system, user and assistant turns.
• Vocabulary size trade-off. Typical sizes are 32K-256K. Bigger vocabularies mean fewer tokens per text (cheaper, longer effective context) but a larger embedding matrix and output softmax, and rarer tokens get less training signal.

Embeddings

The embedding layer is a lookup table E of shape vocab_size × d_model. Token ID 4031 simply selects row 4031. The rows start random and are learned by backpropagation like any other weight. In most decoder-only LLMs the final output projection (the "LM head") that turns the last hidden state back into vocabulary logits has the same shape, and many models tie the two matrices to save parameters.

Important distinction: the input embedding is static (one vector per token ID). The vectors that come out of each Transformer layer are contextual: the "bank" vector after layer 12 differs between "river bank" and "bank loan". Those contextual hidden states are what embedding models (for search and RAG) pool into a single sentence vector. See RAG for how they are used in retrieval.

Token economics

Hosted LLMs bill per token, and context windows are counted in tokens, so you should be able to do quick token arithmetic in an interview.

Worked example. A support bot sends a 1,500-token system prompt, 2,000 tokens of retrieved documents and a 100-token question, and gets a 400-token answer. At a hypothetical price of 1 unit per million input tokens and 4 units per million output tokens: input = 3,600 × 1/10⁶ = 0.0036 units; output = 400 × 4/10⁶ = 0.0016 units; total ≈ 0.0052 units per request. At 1 million requests per day, that is about 5,200 units per day, and 69% of it comes from input. The cheapest optimisations are therefore: shorten the system prompt, retrieve fewer but better chunks, and use prompt caching (many providers discount repeated prompt prefixes).

Positional encoding: giving attention a sense of order

Self-attention on its own is permutation-equivariant: shuffle the input tokens and each token's output is the same, just shuffled. The attention score between "dog" and "man" does not care which one came first, so "dog bites man" and "man bites dog" would look identical. Position information must be injected explicitly.

1. Sinusoidal (original Transformer)

Tiny example (d = 4). Frequencies: pair 0 uses 1/10000⁰ = 1; pair 1 uses 1/10000^2/4 = 0.01.

The fast pair changes a lot between neighbours (fine position), the slow pair changes gradually (coarse position). Properties: deterministic (no parameters), unique per position, and because sin/cos of (pos + k) is a fixed linear function (a rotation) of sin/cos of pos, the model can learn relative offsets. Using both sin and cos also guarantees that when one is zero the other is not.

import math, torch
def sinusoidal_pe(max_len, d):
    pe = torch.zeros(max_len, d)
    pos = torch.arange(max_len).unsqueeze(1).float()               # (max_len, 1)
    div = torch.exp(torch.arange(0, d, 2).float() * (-math.log(10000.0) / d))
    pe[:, 0::2] = torch.sin(pos * div)                             # even dims
    pe[:, 1::2] = torch.cos(pos * div)                             # odd dims
    return pe                                                      # add to token embeddings

2. Learned absolute positions (BERT, GPT-2)

A second lookup table P of shape max_len × d, trained like the token embeddings. Simple and effective, but the model has literally no vector for position 513 if it was trained with 512, so it cannot extrapolate.

3. RoPE: rotary position embedding (Llama, Mistral, Qwen, Gemma, DeepSeek...)

Instead of adding a vector to the input, RoPE rotates the query and key vectors inside every attention layer. Split each q and k into pairs of dimensions; rotate pair i of the token at position m by the angle m·θ_i.

Why it became the default: it encodes relative position (what attention actually needs), adds no parameters, works with the KV cache (a cached key never needs updating), and can be stretched to longer contexts with tricks such as position interpolation (squeeze new positions into the trained range), NTK-aware scaling (raise the base so low frequencies stretch more than high ones) and YaRN (a refined per-frequency interpolation plus attention temperature fix), usually followed by a short fine-tune on long documents.

4. ALiBi: attention with linear biases (BLOOM, MPT)

No position vectors at all. Each head adds a penalty proportional to distance directly to the attention score before softmax: score(i, j) = qi·kj/√d − mh·(i − j), with a different fixed slope m_h per head (a geometric sequence like 1/2, 1/4, 1/8 ...). Near tokens are favoured, far tokens fade. It extrapolates to longer sequences than seen in training well, but biases the model towards locality.

5. Relative position biases (T5) and NoPE

T5 learns a scalar bias per head for bucketed relative distances and adds it to attention scores. Some research shows decoder-only models can even work with no positional encoding (NoPE), because the causal mask itself leaks order information (token i can see exactly i tokens), but this is not mainstream.

Self-attention step by step

Self-attention is the operation that lets each token build a new representation of itself by gathering information from the other tokens in the same sequence. It is best understood as a soft, differentiable lookup table.

The formula

The five steps

1. Project Each token vector is multiplied by three learned matrices to produce its query ("what am I looking for?"), key ("what do I offer?") and value ("what information do I pass on if chosen?").
2. Score Dot product of every query with every key: sij = qi·kj. Large when the two vectors point the same way.
3. Scale Divide by √d_k so scores do not grow with dimension (explained below).
4. Mask and softmax Set disallowed positions (future tokens, padding) to −∞, then softmax each row into weights α_ij that are positive and sum to 1.
5. Mix Output for token i is Σj αij vj, a weighted average of value vectors.

Worked numeric example (3 tokens, d_k = 2)

Tokens: "the", "cat", "sat". Assume the projections have already produced these vectors (in a real model they come from X times learned matrices, and d_k is 64-128):

Step 1-2: raw scores QK^T (row = query token, column = key token):

Step 3: scale by √2 ≈ 1.414: the = [0.707, 0, 0.707]; cat = [0, 1.414, 0.707]; sat = [0.707, 1.414, 1.414].

Step 4a: bidirectional (encoder) softmax for the row "the": e^0.707 = 2.028, e⁰ = 1, e^0.707 = 2.028, sum 5.056 → weights [0.401, 0.198, 0.401]. Output = 0.401·[2,0] + 0.198·[0,2] + 0.401·[1,1] = [1.203, 0.797]. "the" now carries information from both later tokens.

Step 4b: causal (decoder) softmax. Apply the mask: future positions become −∞.

Read the result: with the causal mask, "the" can only see itself, so its output is just its own value. "cat" leans heavily on itself because its key [0,2] matches its query strongly. "sat" (query [1,1]) matches "cat" and itself equally and "the" less. Every output is a convex combination of value vectors, which is why attention alone is a smoothing, mixing operation and the network also needs the non-linear feed-forward layer.

Why divide by √d_k?

If the components of q and k are independent with mean 0 and variance 1, their dot product is a sum of d_k terms each with variance 1, so it has variance d_k and standard deviation √d_k. With d_k = 64, raw scores are typically around ±8. Softmax of numbers that spread out is almost one-hot: one weight near 1, the rest near 0, and the gradient of softmax in that saturated region is almost zero, so learning stalls. Dividing by √d_k brings the variance back to about 1, keeping softmax in its sensitive range. It is the same maths as temperature: √d_k acts as a fixed temperature for attention.

Masks

Causal mask for "I love NLP <eos>"   (✓ = attend, ✗ = score set to -inf)

              keys →   I     love   NLP   <eos>
  queries ↓  I         ✓     ✗      ✗     ✗
             love      ✓     ✓      ✗     ✗
             NLP       ✓     ✓      ✓     ✗
             <eos>     ✓     ✓      ✓     ✓

Shapes you should be able to recite

For a 10-token sentence in a BERT-base-sized model (d = 768, 12 heads, d_k = 64): X is 10 × 768. Per head, Q, K, V are 10 × 64. Scores QK^T are 10 × 10. Weights times V give 10 × 64. Concatenating 12 heads gives 10 × 768, then W_O (768 × 768) keeps it 10 × 768. The FFN expands to 10 × 3072 and back to 10 × 768. Sequence length never changes inside a block; only the feature dimension expands and contracts. Time is O(n²d) for scores and mixing plus O(nd²) for the Q, K, V and output projections; the n² term dominates once sequence length exceeds model width. Memory for the score matrix is O(n²) per head unless FlashAttention tiles it down to O(n).

Code: attention from scratch in PyTorch

import math, torch, torch.nn.functional as F

def attention(q, k, v, causal=False, pad_mask=None):
    # q, k, v: (batch, heads, seq, d_k)
    d_k = q.size(-1)
    scores = q @ k.transpose(-2, -1) / math.sqrt(d_k)          # (B, H, T, T)
    T = scores.size(-1)
    if causal:
        future = torch.triu(torch.ones(T, T, dtype=torch.bool, device=q.device), 1)
        scores = scores.masked_fill(future, float("-inf"))
    if pad_mask is not None:                                   # (B, 1, 1, T), True = padding
        scores = scores.masked_fill(pad_mask, float("-inf"))
    weights = F.softmax(scores, dim=-1)                        # each row sums to 1
    return weights @ v, weights

# Production code uses the fused kernel (FlashAttention under the hood when available):
# out = F.scaled_dot_product_attention(q, k, v, is_causal=True)

What is learned, and what is computed

The attention weights are not parameters. They are computed fresh for every input. The learned parameters are W_Q, W_K, W_V (and W_O). Training adjusts those projections so that useful relationships (pronoun to noun, verb to subject) produce high dot products. That is how the same frozen model attends from "it" to "animal" in one sentence and to "street" in another: the weights are fixed, but the input, and therefore Q and K, changes. Nobody tells the model that "animal" is a noun; such structure emerges because it helps predict the next token.

Multi-head attention and cross-attention

One attention operation produces one set of weights per token, so it can express only one "kind" of relationship at a time. Language has many simultaneously: the subject of a verb, what a pronoun refers to, the adjacent word, the matching bracket, the topic of the paragraph. Multi-head attention runs h smaller attentions in parallel, each with its own projections, and combines them.

Worked example: two heads, one sentence

Two tokens, "animal" = [1, 0] and "it" = [0, 1], model dimension 2, two heads of dimension 1. To keep the numbers readable we skip the √d_k scaling (d_k = 1 anyway).

softmax([1, 0]) = [e/(e+1), 1/(e+1)] = [2.718/3.718, 1/3.718] = [0.73, 0.27]. After concatenation, "animal" = [1.46, 1.50] and "it" = [1.46, 2.19]. Head 1 made "it" carry the same animal-flavoured signal as "animal" (coreference); head 2 kept "it" distinguishable. W_O then mixes these two views into the final 2-dimensional output. Real models have 12-128 heads, and probing studies find heads specialised for the previous token, syntactic dependencies, coreference and copying (so-called induction heads that find an earlier occurrence of the current token and copy what followed it, a key mechanism behind in-context learning).

import torch, torch.nn as nn, torch.nn.functional as F

class MultiHeadAttention(nn.Module):
    def __init__(self, d_model=768, n_heads=12):
        super().__init__()
        assert d_model % n_heads == 0
        self.h, self.d_k = n_heads, d_model // n_heads
        self.qkv = nn.Linear(d_model, 3 * d_model, bias=False)   # W_Q, W_K, W_V fused
        self.out = nn.Linear(d_model, d_model, bias=False)       # W_O

    def forward(self, x, causal=True):
        B, T, D = x.shape
        q, k, v = self.qkv(x).split(D, dim=-1)
        # (B, T, D) -> (B, h, T, d_k): each head gets its own slice
        q, k, v = (t.view(B, T, self.h, self.d_k).transpose(1, 2) for t in (q, k, v))
        y = F.scaled_dot_product_attention(q, k, v, is_causal=causal)
        y = y.transpose(1, 2).contiguous().view(B, T, D)          # concatenate heads
        return self.out(y)

Why not one big head?

A single head computes one softmax per query, so it must choose one distribution over positions: it cannot put 90% on the subject and 90% on the previous token at the same time. Several heads can. With the same total width, multi-head costs the same FLOPs, so it is a free increase in expressiveness. More heads is not automatically better: with a fixed d_model, more heads means smaller d_k per head, and research on head pruning shows many heads are redundant and can be removed after training with little loss. Typical choice: d_k = 64 or 128, so h = d_model/64 or /128.

Cross-attention

Cross-attention uses exactly the same formula but takes queries from one sequence and keys and values from another. In an encoder-decoder model, the decoder's queries attend over the encoder's final-layer outputs, so each target token can "look back" at the source.

Example: translating "The cat sat" into French, when the decoder is producing "chat", its query matches the key of "cat" strongly (say weight 0.70), "sat" weakly (0.20) and the rest barely, so its context is mostly the encoder's representation of "cat". Because the encoder output does not change during generation, its K and V are computed once and reused at every decoding step.

The Transformer block and the full architecture

A Transformer is a stack of identical blocks. Each block has two sub-layers: multi-head attention (tokens talk to each other) and a position-wise feed-forward network (each token thinks on its own). Each sub-layer is wrapped with a residual connection and a normalization layer. Stack 12 blocks and you have GPT-2 small; stack 96 and you have GPT-3.

The full diagram (original encoder-decoder Transformer)

                                                        Output probabilities
                                                                 ▲
                                                          ┌──────┴──────┐
                                                          │   Softmax   │
                                                          ├─────────────┤
                                                          │   Linear    │  (d_model → vocab)
                                                          └──────▲──────┘
                                                                 │
            ENCODER  (× N)                          DECODER  (× N)
   ┌───────────────────────────────┐       ┌───────────────────────────────────┐
   │   ┌───────────────────────┐   │       │   ┌───────────────────────────┐   │
   │   │     Add & Norm        │◀─┐│       │   │        Add & Norm         │◀─┐│
   │   └──────────▲────────────┘  ││       │   └────────────▲──────────────┘  ││
   │   ┌──────────┴────────────┐  ││       │   ┌────────────┴──────────────┐  ││
   │   │  Feed-Forward Network │  ││       │   │   Feed-Forward Network    │  ││
   │   │ d → 4d → ReLU/GELU → d│  ││       │   └────────────▲──────────────┘  ││
   │   └──────────▲────────────┘  ││       │                ├─────────────────┘│
   │              ├───────────────┘│       │   ┌────────────┴──────────────┐   │
   │   ┌──────────┴────────────┐   │       │   │        Add & Norm         │◀─┐│
   │   │     Add & Norm        │◀─┐│       │   └────────────▲──────────────┘  ││
   │   └──────────▲────────────┘  ││       │   ┌────────────┴──────────────┐  ││
   │   ┌──────────┴────────────┐  ││  K,V  │   │  Multi-Head CROSS-Attn    │  ││
   │   │  Multi-Head SELF-Attn │  ││ ─────▶│   │  Q: decoder  K,V: encoder │  ││
   │   │  (bidirectional)      │  ││       │   └────────────▲──────────────┘  ││
   │   └──────────▲────────────┘  ││       │                ├─────────────────┘│
   │              ├───────────────┘│       │   ┌────────────┴──────────────┐   │
   │              │                │       │   │        Add & Norm         │◀─┐│
   └──────────────┼────────────────┘       │   └────────────▲──────────────┘  ││
                  │                        │   ┌────────────┴──────────────┐  ││
                  │                        │   │MASKED Multi-Head SELF-Attn│  ││
                  │                        │   │  (causal: past only)      │  ││
                  │                        │   └────────────▲──────────────┘  ││
                  │                        │                ├─────────────────┘│
                  │                        └────────────────┼──────────────────┘
          ┌───────┴────────┐                        ┌───────┴────────┐
          │ + Positional   │                        │ + Positional   │
          │   encoding     │                        │   encoding     │
          ├────────────────┤                        ├────────────────┤
          │ Input embedding│                        │Output embedding│
          └───────▲────────┘                        └───────▲────────┘
                  │                                         │
        Source tokens ("I love you")        Target tokens shifted right ("<bos> Je t'")

   Original sizes: N = 6, d_model = 512, heads = 8, d_ff = 2048, post-LN, ReLU, sinusoidal PE.

The modern decoder-only block (Llama-style)

   token ids ──▶ Embedding ──▶ x   (residual stream, shape T × d)
                                │
          ┌─────────────────────┼──────────────────────────┐
          │                     ├──────────────┐           │
          │               RMSNorm              │           │
          │                     │              │ residual  │
          │     Masked multi-head attention    │           │   repeated
          │     (RoPE on Q,K; GQA; KV cache)   │           │   L times
          │                     │              │           │   (32, 80,
          │                     ⊕ ◀────────────┘           │    126 ...)
          │                     ├──────────────┐           │
          │               RMSNorm              │           │
          │                     │              │ residual  │
          │        SwiGLU feed-forward         │           │
          │     W2( SiLU(W1 x) ⊙ W3 x )        │           │
          │                     │              │           │
          │                     ⊕ ◀────────────┘           │
          └─────────────────────┼──────────────────────────┘
                          final RMSNorm
                                │
                     LM head (d → vocab) ──▶ logits ──▶ sampling ──▶ next token

Residual connections

A useful mental model is the residual stream: a d-dimensional vector per token that flows from the embedding to the output, and every attention and FFN sub-layer reads from it and writes an additive update back into it. Removing residuals makes deep Transformers essentially untrainable.

Layer normalization, Pre-LN vs Post-LN, and RMSNorm

Why LayerNorm rather than BatchNorm: sequences have variable length and padding, batch statistics are noisy for small or variable batches, and at inference you often have batch size 1. LayerNorm normalises each token independently, so it behaves identically in training and inference.

The feed-forward network (FFN / MLP)

What the FFN does: attention is (apart from the softmax) a weighted average, which is a linear mixing of value vectors. Without a non-linearity between layers, stacking them would collapse to one linear map. The FFN applies the same small MLP to every token independently, adding non-linear feature transformations. Expanding to 4d temporarily gives the network more room to form combinations of features before compressing back. Interpretability work suggests FFN layers act like key-value memories that store much of the model's factual knowledge, and they hold about two-thirds of the parameters.

Parameter counting

Check with GPT-2 small: d = 768, L = 12: 12 × 768² × 12 ≈ 84.9M in blocks, plus 50,257 × 768 ≈ 38.6M token embeddings (weight tying: the LM head reuses E instead of a second |V|·d matrix), plus 1,024 × 768 ≈ 0.8M position embeddings ≈ 124M. Untied, GPT-2 small would be about 163M. GPT-3: d = 12,288, L = 96: 12 × 12,288² × 96 ≈ 174B, plus about 0.6B of embeddings ≈ 175B. Memory to hold weights = parameters × bytes per parameter: a 7B model needs about 14 GB in 16-bit, 7 GB in 8-bit, 3.5 GB in 4-bit.

class Block(nn.Module):
    """Pre-norm decoder block with RMSNorm and SwiGLU (Llama-style, simplified)."""
    def __init__(self, d=4096, n_heads=32, hidden=11008):
        super().__init__()
        self.norm1, self.norm2 = nn.RMSNorm(d), nn.RMSNorm(d)   # PyTorch 2.4+
        self.attn = MultiHeadAttention(d, n_heads)               # RoPE omitted for brevity
        self.w1 = nn.Linear(d, hidden, bias=False)
        self.w3 = nn.Linear(d, hidden, bias=False)
        self.w2 = nn.Linear(hidden, d, bias=False)

    def forward(self, x):
        x = x + self.attn(self.norm1(x), causal=True)            # tokens talk
        h = self.norm2(x)
        x = x + self.w2(F.silu(self.w1(h)) * self.w3(h))         # each token thinks
        return x

Other details worth knowing

• Dropout on attention weights and sub-layer outputs was standard in the original model and BERT; very large LLMs trained for about one epoch often use little or none.
• Initialisation is small (for example normal with standard deviation 0.02), and some schemes scale down the output projections of residual branches by 1/√(2L) so the stream does not grow with depth.
• Biases are often removed from linear layers in modern LLMs; it slightly improves stability and costs nothing.
• Why N = 6? The original paper simply found it a good trade-off for its data and compute. Depth is a hyperparameter; deeper models are more expressive but slower and harder to train, and width, depth and data are scaled together (see scaling laws).

Encoder-only, decoder-only and encoder-decoder models

The original Transformer had both an encoder and a decoder. Later models kept only the part they needed. Which part you keep decides what the model sees (both directions or only the past), how it is pretrained, and what it is good at.

Why decoder-only won for LLMs

• One objective, every token is a training signal. Next-token prediction on raw text gives a loss at every position, whereas MLM only trains on the ~15% masked tokens.
• Generation is native. The training task (predict the next token) is exactly the inference task.
• Everything is a prompt. Instructions, examples, documents and conversation history are all just the beginning of one sequence; no separate input and output streams. This made in-context learning and chat natural.
• Simple and scalable. One stack, one KV cache, well-understood scaling behaviour; the engineering ecosystem (kernels, serving systems) converged on it.

Encoder-only models remain the workhorse for embeddings, retrieval, reranking and cheap classification: a 100-300M-parameter encoder is far cheaper than an LLM for tagging millions of documents. Encoder-decoders remain strong where input and output are clearly different (speech-to-text, translation).

Using each type with Hugging Face

from transformers import pipeline, AutoTokenizer, AutoModelForCausalLM

# Encoder-only: fill in a masked word using both sides of context
fill = pipeline("fill-mask", model="bert-base-uncased")
print(fill("Paris is the [MASK] of France.")[0]["token_str"])     # 'capital'

# Encoder-decoder: text-to-text
t5 = pipeline("text2text-generation", model="google/flan-t5-base")
print(t5("Translate English to German: The house is small.")[0]["generated_text"])

# Decoder-only: continue a prompt
name = "gpt2"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(name)
ids = tok("The Transformer architecture", return_tensors="pt")
out = model.generate(**ids, max_new_tokens=30, do_sample=True, temperature=0.8, top_p=0.9)
print(tok.decode(out[0], skip_special_tokens=True))

How a decoder-only model handles "input"

A common confusion: "If GPT has no encoder, where do K and V come from, and how does it read my question?" Answer: the prompt tokens are simply the first part of the sequence. Q, K and V are all computed from that same sequence. During prefill, all prompt tokens are processed in parallel with the causal mask (each prompt token sees earlier prompt tokens); then generated tokens are appended one at a time. Chat "roles" are just special tokens in a chat template that flatten system, user and assistant messages into one token stream.

Pretraining objectives and data

Pretraining teaches a model general language ability from unlabelled text using self-supervised objectives: the labels are hidden pieces of the text itself, so no human annotation is needed and the data can be trillions of tokens.

Causal language modelling (CLM): GPT, Llama and all chat LLMs

Example: for "The cat sat on the mat" the training pairs are (The → cat), (The cat → sat), (The cat sat → on), and so on, all computed in one pass. If the model assigns probability 0.25 to the true token on average, the loss per token is −ln 0.25 = 1.386 and perplexity is 4.

Masked language modelling (MLM): BERT

Pick 15% of tokens. Of those, replace 80% with [MASK], 10% with a random token and leave 10% unchanged (so the model cannot learn that only [MASK] positions matter, since [MASK] never appears at fine-tuning time). Predict the original tokens using context from both sides. BERT also used Next Sentence Prediction (is sentence B the real continuation of A?), which later work (RoBERTa) found unhelpful and dropped, while gaining from more data, longer training, larger batches and dynamic masking. ELECTRA replaced MLM with "replaced token detection" (a small generator swaps some tokens, the main model classifies every token as original or replaced), which learns from all tokens and is more sample-efficient.

Span corruption and denoising: T5, BART

T5 masks contiguous spans (average length 3, about 15% of tokens), replaces each span with a sentinel token, and the decoder must output the missing spans:

Original : Thank you for inviting me to your party last week.
Input    : Thank you <X> me to your party <Y> week.
Target   : <X> for inviting <Y> last <Z>

T5 also cast every task as text-to-text ("translate English to German: ...", "summarize: ...", "cola sentence: ..."), so one model, one loss and one decoding procedure handle classification, translation and summarisation. BART corrupts text with several noise types (masking, deletion, span infilling, sentence shuffling, rotation) and trains the decoder to reconstruct the original. Related: prefix LM (bidirectional attention over a prefix, causal over the rest) and fill-in-the-middle (FIM), where code models are trained on documents rearranged as prefix + suffix → middle so they can complete code inside a file.

Pretraining data

Modern LLMs train on roughly 2-36 trillion tokens mixed from filtered web crawls, code repositories, books, scientific papers, encyclopedias, maths, multilingual text and increasingly synthetic data generated by other models. The data pipeline often matters more than architecture tweaks:

1. Extraction Pull clean text out of HTML and PDFs; drop boilerplate.
2. Language identification and filtering Heuristics (length, symbol ratios, repetition) and quality classifiers trained to recognise "reference-like" text.
3. Deduplication Exact and near-duplicate removal (MinHash). Duplicates waste compute and increase memorisation and regurgitation.
4. Safety and privacy Remove personal information, toxic content and known benchmark test sets (decontamination).
5. Mixing Choose proportions of web, code, maths and languages; often upweight high-quality sources in a final "annealing" phase.

Training mechanics: AdamW optimizer, learning-rate warm-up then cosine or linear decay, bf16 mixed precision, batches of millions of tokens, and parallelism across thousands of GPUs (data parallel with sharded optimizer states such as ZeRO/FSDP, tensor parallel inside layers, pipeline parallel across layers). A frequent later stage, mid-training or continued pretraining, extends context length and adds more code, maths and reasoning data.

Scaling laws, Chinchilla and emergent abilities

The single most important empirical discovery about LLMs is that their loss improves predictably as you scale parameters, data and compute. That predictability is why organisations were willing to spend enormous sums on training runs: they could forecast the result from small experiments.

Power laws

Early scaling studies (2020) found that test loss falls as a power law in each of parameters N, data D and compute C when the other factors are not bottlenecks, over many orders of magnitude. Doubling compute gives a roughly constant fractional improvement in loss. A later, widely used form models both together:

Chinchilla: most early LLMs were undertrained

The 2022 Chinchilla study trained over 400 models and concluded that, for compute-optimal training, parameters and tokens should grow in equal proportion: roughly 20 tokens per parameter. GPT-3 (175B parameters, about 300B tokens, under 2 tokens per parameter) and a 280B-parameter model trained on 300B tokens were far from that. A 70B model trained on 1.4T tokens with the same compute outperformed the 280B model across benchmarks, while being 4x cheaper to serve.

Beyond Chinchilla: overtraining for inference

Chinchilla optimises training compute only. But a model that serves billions of requests pays inference cost (about 2N FLOPs per token) forever. So modern open models deliberately train small models far past the Chinchilla point: Llama 3 8B saw about 15T tokens, close to 1,900 tokens per parameter. Loss keeps improving slowly, and the result is a small, cheap model of much higher quality. The practical rule became "train the smallest model that reaches the quality you need, on as much good data as you can get". The emerging constraint is running out of high-quality human text, which drives interest in synthetic data, multi-epoch training and multimodal data.

Emergent abilities and in-context learning

Some capabilities appear to switch on abruptly with scale: multi-digit arithmetic, following few-shot examples, chain-of-thought reasoning, translating low-resource languages. GPT-3 (2020) showed in-context learning: with a handful of examples in the prompt and no weight updates, performance rose sharply (for example on closed-book trivia QA from about 64% zero-shot to 71% with 64 examples), and smaller versions benefited far less. Two caveats every strong candidate mentions:

• Metric artefacts. Some "sudden" jumps come from all-or-nothing metrics like exact match. Measured with smooth metrics (per-token log-likelihood, partial credit), improvement is often gradual and predictable.
• Loss is predictable, abilities less so. Scaling laws forecast next-token loss well, but which downstream task improves at which scale is much harder to predict.

The LLM training pipeline: pretraining, SFT and preference alignment

A raw pretrained model is a brilliant autocomplete, not an assistant. Turning it into a helpful, honest and safe chatbot takes several further stages. This section gives the conceptual picture; the hands-on techniques (LoRA, QLoRA, dataset formats, training scripts) are covered in Fine-Tuning.

  ┌────────────────────┐   ┌───────────────────────┐   ┌──────────────────────────┐   ┌──────────────────────┐
  │ 1. PRETRAINING     │   │ 2. SUPERVISED FINE-   │   │ 3. PREFERENCE ALIGNMENT  │   │ 4. REASONING RL      │
  │ trillions of tokens│──▶│    TUNING (SFT)       │──▶│  RLHF: reward model+PPO  │──▶│  (optional)          │
  │ next-token loss    │   │ 10K-1M+ instruction / │   │  or DPO / other direct   │   │  RL on verifiable    │
  │ → BASE model       │   │ chat demonstrations   │   │  preference methods      │   │  rewards (maths,code)│
  │ weeks-months,      │   │ → follows instructions│   │  → helpful, harmless,    │   │  → long chain-of-    │
  │ thousands of GPUs  │   │   in chat format      │   │    preferred style       │   │    thought reasoning │
  └────────────────────┘   └───────────────────────┘   └──────────────────────────┘   └──────────────────────┘
        ~99% of compute           small                          small                    growing share

Stage 2: supervised fine-tuning (instruction tuning)

Train the base model on curated (prompt, ideal response) pairs covering questions, instructions, coding, maths, refusals and multi-turn chats, formatted with the model's chat template. The loss is the same next-token cross-entropy, but usually computed only on the response tokens (prompt tokens are masked out of the loss). Quality beats quantity: a few thousand excellent, diverse examples can dramatically change behaviour. Instruction tuning on many tasks (the FLAN line of work) also improves zero-shot generalisation to unseen tasks.

Stage 3a: RLHF (reinforcement learning from human feedback)

1. Collect comparisons For each prompt, sample two or more responses from the SFT model; human labelers rank them.
2. Train a reward model Usually the SFT model with a scalar head, trained so the preferred response scores higher: L = −log σ(r(x, yw) − r(x, yl)) (the Bradley-Terry model).
3. Optimise the policy with RL Generate responses, score them with the reward model, and update the LLM with PPO to increase reward, while a KL penalty keeps it close to the SFT model.

The landmark result: labelers preferred outputs of a 1.3B instruction-following model aligned this way over the raw 175B base model, showing that alignment can matter more than size for usefulness. This recipe is what turned GPT-3 into the first widely used chat assistant in late 2022.

Stage 3b: DPO and other direct preference methods

RLHF with PPO is complex: four models in memory (policy, reference, reward, value), unstable training, many hyperparameters. Direct Preference Optimization (DPO) shows that the optimal policy for the KL-regularised objective can be written in closed form, so you can skip the reward model and RL loop and train directly on preference pairs with a classification-style loss:

Related methods: RLAIF and Constitutional AI use an AI model, guided by a written set of principles, to produce the preference labels instead of (or in addition to) humans; IPO, KTO, ORPO, SimPO are DPO variants (KTO needs only thumbs-up/thumbs-down labels, ORPO merges SFT and preference steps); rejection sampling fine-tuning samples many responses, keeps the best by a reward model and fine-tunes on them.

Stage 4: reinforcement learning for reasoning

For maths, code and logic, correctness can be checked automatically (unit tests pass, final answer matches). Training with RL against these verifiable rewards teaches models to produce long chains of thought, check themselves and backtrack. GRPO (group relative policy optimization) is a popular PPO variant for this: it samples a group of answers per prompt and uses each answer's reward relative to the group average as its advantage, removing the need for a separate value model. See the reasoning-models section below.

Decoding strategies with worked examples

At each step the model outputs a vector of logits, one raw score per vocabulary token. A decoding strategy turns those logits into the next token; the token is appended and the loop repeats until an end-of-sequence token, a stop sequence or the token limit. The model's weights define a probability distribution; decoding decides how to use it. Temperature and friends are not part of the model and have nothing to do with Q, K or V.

The pipeline, in order

1. Logits Forward pass returns logits z for the last position.
2. Logit processors Apply repetition, frequency and presence penalties, logit bias, and any grammar or JSON constraint masks (disallowed tokens set to −∞).
3. Temperature Divide logits by T.
4. Truncation Keep only candidates allowed by top-k, top-p or min-p; renormalise.
5. Select Take the argmax (greedy) or sample from the remaining distribution.
6. Stop check Stop on EOS, a stop sequence or max_tokens; otherwise append and repeat.

Greedy and beam search (deterministic)

Greedy picks argmax at every step. Fast and deterministic, but locally optimal choices can lead to a worse overall sequence, and it tends to fall into repetitive loops in open-ended text.

Beam search keeps the B highest-probability partial sequences (the beam) at each step, expands each by every token, and keeps the best B by total log-probability. Worked example with a tiny tree:

Greedy picks A (0.5), then x, ending with probability 0.20. Beam search with B = 2 keeps {A, B} after step 1, expands both, and keeps {B z = 0.36, A x = 0.20}: it finds the better sequence B z that greedy missed. Beam search still is not guaranteed to find the global optimum. It is good for tasks with one right answer (translation, summarisation, speech recognition) but in open-ended generation it produces bland, generic and repetitive text, because the highest-probability text is often not the most human-like. Long outputs get lower total probability, so a length penalty (dividing the log-probability by length^α) is used to avoid favouring short outputs.

Temperature

Worked example with logits [2, 1, 0] for tokens [Paris, Lyon, cheese]:

At T = 2 the absurd option "cheese" is sampled almost one time in five. Typical settings: 0-0.3 for extraction, classification, code and maths; about 0.7 for general assistant chat; 0.9-1.2 for creative writing and brainstorming. Note that even T = 0 is not always perfectly reproducible on hosted APIs because of batching and floating-point non-determinism on GPUs.

Top-k, top-p (nucleus) and min-p

Pure sampling at T = 1 occasionally picks a very unlikely token from the long tail of a 100K-token vocabulary, and one bad token can derail the rest of the text. Truncation methods cut the tail. Example distribution after "I drank a cup of":

• Top-k (k = 3): keep tea, coffee, water (total 0.82) and renormalise: tea 0.40/0.82 = 0.488, coffee 0.366, water 0.146. Weakness: k is fixed. When the model is very sure (one token at 0.97), k = 3 still lets in two junk tokens; when it is unsure (50 reasonable options), k = 3 cuts valid choices.
• Top-p (p = 0.9): keep the smallest set whose cumulative probability reaches 0.9: tea, coffee, water, milk (0.90). Renormalised: 0.444, 0.333, 0.133, 0.089. The set size adapts: confident distributions give a tiny nucleus, flat distributions a large one. Typical p = 0.9-0.95.
• Min-p (min_p = 0.1): keep tokens whose probability is at least min_p times the top token's probability: threshold 0.1 × 0.40 = 0.04, so tea, coffee, water, milk, juice survive. Min-p scales with the model's confidence and stays coherent at higher temperatures, which makes it popular in open-source inference engines.
• Others: typical sampling (keep tokens whose information content is close to the expected value), and contrastive search (penalise candidates too similar to earlier context).

Repetition, frequency and presence penalties

Penalties that are too strong cause strange word choices, because the model is prevented from reusing necessary words like "the", variable names in code or a person's name.

Length, stop sequences and constrained decoding

• max_tokens / max_new_tokens is a hard cap on generated tokens; it controls cost and latency but truncates mid-sentence if hit (check the finish reason).
• Stop sequences end generation when a string appears, for example "\nUser:" in a transcript format or a closing code fence. The stop text is usually not included in the output.
• Constrained decoding masks logits of tokens that would violate a grammar or JSON schema, so the output is guaranteed to parse. This is how structured-output modes work (details in LLM APIs).

Code: controlling generation

from transformers import AutoTokenizer, AutoModelForCausalLM, GenerationConfig
tok = AutoTokenizer.from_pretrained("gpt2")
model = AutoModelForCausalLM.from_pretrained("gpt2").eval()
ids = tok("Once upon a time", return_tensors="pt")

greedy = model.generate(**ids, max_new_tokens=40, do_sample=False)
beam   = model.generate(**ids, max_new_tokens=40, num_beams=4, length_penalty=1.0,
                        no_repeat_ngram_size=3, early_stopping=True)
sample = model.generate(**ids, max_new_tokens=40, do_sample=True,
                        temperature=0.8, top_p=0.9, top_k=0, repetition_penalty=1.15)

# A minimal top-p sampler, to show the mechanics
import torch
def sample_top_p(logits, p=0.9, temperature=0.8):
    probs = torch.softmax(logits / temperature, dim=-1)
    sorted_p, idx = torch.sort(probs, descending=True)
    cum = sorted_p.cumsum(-1)
    sorted_p[cum - sorted_p > p] = 0.0           # drop tokens after the nucleus is complete
    sorted_p /= sorted_p.sum()
    return idx[torch.multinomial(sorted_p, 1)]

Inference, the KV cache and long context

Serving an LLM has two very different phases. Prefill processes the whole prompt in one parallel forward pass; it is compute-bound and determines time-to-first-token. Decode generates one token per forward pass; each step must read all the model weights from GPU memory to do a small amount of maths, so it is memory-bandwidth-bound and determines tokens per second.

The KV cache

In a decoder, the keys and values of past tokens never change (the causal mask means later tokens cannot affect earlier ones). So after computing them once, store them. At each decode step, compute Q, K, V for the new token only, append its K and V to the cache, and attend with a single query row over all cached keys. Per-step attention work drops from recomputing an n × n matrix to an n-length vector, turning generation from roughly quadratic to linear total work in the attention part.

This formula explains most of LLM serving engineering: long contexts and big batches are limited by cache memory, not by weights. Techniques that attack it:

Other serving speed-ups

• Continuous (in-flight) batching: add new requests to the batch as soon as others finish, instead of waiting for the whole batch; raises GPU utilisation dramatically.
• Speculative decoding: a small draft model (or extra prediction heads) proposes several tokens; the big model verifies them all in one parallel pass and accepts the longest correct prefix. Output distribution is unchanged; speed-ups of 2-3x are common when the draft agrees often.
• Weight quantization: 8-bit or 4-bit weights (GPTQ, AWQ, GGUF formats) cut memory and, because decode is bandwidth-bound, raise tokens per second almost proportionally. Details in Edge AI.
• Rough throughput rule for decode: tokens/s per sequence ≈ memory bandwidth / bytes of weights read per token. A 7B model in fp16 (14 GB) on a GPU with 2 TB/s bandwidth tops out around 140 tokens/s at batch size 1.

# Hugging Face uses the KV cache automatically (use_cache=True by default).
# Manual loop to make the mechanics visible:
past, ids = None, tok("The KV cache", return_tensors="pt").input_ids
for _ in range(20):
    out = model(input_ids=ids if past is None else ids[:, -1:], past_key_values=past, use_cache=True)
    past = out.past_key_values                         # grows by one position per step
    next_id = out.logits[:, -1].argmax(-1, keepdim=True)
    ids = torch.cat([ids, next_id], dim=-1)

Context windows

The context window is the maximum number of tokens (prompt plus output) the model can attend to in one pass. It grew from 512 (BERT) and 2,048 (GPT-3) to 128K-200K for many current models and 1M or more for some. The model has no memory between calls: chat "memory" is just the application re-sending the conversation (or a summary of it) every time.

Why long context is hard

1. Quadratic attention Score computation is O(n²) in time; naive memory is O(n²) too (FlashAttention fixes the memory part, not the time).
2. KV cache Grows linearly with n and often dominates GPU memory.
3. Position generalisation Models trained on 4K positions do not automatically work at 32K; positions they never saw behave unpredictably.
4. Effective use Even when a model accepts 128K tokens, retrieval of facts from the middle is weaker than from the start or end ("lost in the middle"), and multi-fact reasoning across long inputs degrades well before the nominal limit.
5. Training data Truly long, coherent documents are rare, so long-context skills need targeted data (books, code repositories, synthetic long tasks).

Long-context techniques

Efficient attention: FlashAttention, MQA/GQA, sliding window and sparse patterns

Standard attention has two costs: compute that grows as n², and memory traffic from writing and re-reading the n × n score matrix. It also produces a large KV cache at inference. Different techniques target different costs, and interviewers like candidates who can say exactly which one each technique attacks.

FlashAttention (exact, I/O-aware)

FlashAttention computes exactly the same result as standard attention, but tiles Q, K and V into blocks that fit in on-chip SRAM, computes the scores and softmax for a block, multiplies by V, and accumulates the output using an online softmax (keeping a running maximum and running sum so blocks can be combined correctly). The full n × n matrix is never written to slow memory. In the backward pass it recomputes the scores rather than storing them. Result: memory O(n) instead of O(n²), 2-4x wall-clock speed-ups, and much longer feasible sequences. FLOPs are still O(n²d): it is faster because attention was memory-bound, not because it does less maths. Versions 2 and 3 improved parallelism and use newer GPU features. In PyTorch you get it through F.scaled_dot_product_attention.

Multi-query and grouped-query attention (smaller KV cache)

  Multi-head (MHA)            Grouped-query (GQA, 2 groups)     Multi-query (MQA)
  Q: q1 q2 q3 q4              Q: q1 q2 q3 q4                    Q: q1 q2 q3 q4
     |  |  |  |                  \ /     \ /                        \  \ /  /
  K: k1 k2 k3 k4              K:  k1      k2                    K:     k1
  V: v1 v2 v3 v4              V:  v1      v2                    V:     v1
  KV cache: 4 heads           KV cache: 2 heads                 KV cache: 1 head

• MQA: all query heads share a single K and V head. KV cache shrinks by a factor of h (for example 32x). Some quality loss and training instability.
• GQA: query heads are split into g groups, each group shares one K/V head (for example 32 query heads, 8 KV heads). Near-MHA quality with 4x smaller cache. Default in Llama 2 70B, Llama 3, Mistral, Qwen and many more. Existing MHA checkpoints can be converted by averaging K/V heads and briefly continuing training ("uptraining").
• MLA (multi-head latent attention): used by DeepSeek-V2/V3. Keys and values are compressed into a small shared latent vector per token that is cached; per-head K and V are reconstructed from it. Cache is far smaller than MHA with comparable or better quality.

Sliding-window (local) attention

Each token attends only to the previous w tokens (for example w = 4,096 in Mistral 7B). Cost becomes O(n·w) and the KV cache can be a fixed-size rolling buffer. Information still travels further than w because it hops through layers: after L layers the theoretical receptive field is about L × w. Many models interleave local layers with global (full) layers, for example several sliding-window layers per full-attention layer, to keep long-range ability.

Sparse attention patterns

Sparse transformers restrict which query-key pairs are computed: strided and fixed patterns; Longformer (local window plus a few global tokens such as [CLS] that attend to and are attended by everything); BigBird (local + global + random connections, which keeps theoretical expressiveness). These reach O(n) or O(n log n) cost but are harder to implement efficiently on GPUs, and dense attention with FlashAttention often wins in practice for moderate lengths.

Linear attention and state-space models

Linear-attention methods (kernel approximations like Performer, low-rank like Linformer) rewrite softmax attention so cost grows linearly, usually with some quality loss on retrieval-heavy tasks. State-space and recurrent-style models (Mamba, RWKV, RetNet) process sequences with a fixed-size state, giving O(n) training and O(1) memory per generated token; hybrids that combine them with some attention layers are an active area.

Mixture of Experts (MoE)

A Mixture-of-Experts layer replaces the single feed-forward network in a block with many parallel FFNs ("experts") plus a small router that sends each token to only a few of them. The model can hold far more parameters (knowledge capacity) while each token uses only a fraction of them (compute cost).

Worked numbers

• Mixtral 8x7B: 8 experts per layer, top-2 routing. About 47B total parameters (not 56B, because attention and embeddings are shared) but only about 13B active per token. Quality comparable to much larger dense models of its time at the inference FLOPs of a 13B model.
• DeepSeek-V3: about 671B total, about 37B active per token, using many small fine-grained experts plus always-on shared experts.
• Other open MoE families include Qwen MoE variants and Llama 4; several frontier closed models are widely reported to be MoE as well.

Engineering challenges

Multimodal models: ViT, CLIP and vision-language models

Transformers operate on sequences of vectors, and nothing requires those vectors to come from text. If you can turn an image, an audio clip or a video into a sequence of embeddings, the same architecture applies. This section covers the three ideas behind most multimodal systems.

Vision Transformer (ViT)

1. Patchify Split a 224 × 224 image into 16 × 16 patches: (224/16)² = 196 patches.
2. Embed Flatten each patch (16 × 16 × 3 = 768 values) and project it linearly to d dimensions. This is equivalent to a convolution with kernel 16 and stride 16.
3. Add position embeddings and a [CLS] token Learned 2-D-aware position embeddings; the [CLS] output is used for classification.
4. Run a standard Transformer encoder Bidirectional self-attention over 197 tokens.

ViTs have weaker built-in assumptions (inductive biases) than CNNs: no locality or translation equivariance baked in. So they need more data or strong augmentation, but they scale better and now dominate large vision models. Higher resolution means more patches and quadratic attention cost, so VLMs often tile large images or pool patch tokens.

CLIP: contrastive image-text pretraining

CLIP trains an image encoder and a text encoder together on hundreds of millions of (image, caption) pairs from the web. In a batch of N pairs, it computes the N × N matrix of cosine similarities between every image embedding and every text embedding and applies a symmetric cross-entropy loss so the N true pairs (the diagonal) score highest.

Payoffs: zero-shot classification (embed "a photo of a {class}" for each class and pick the closest to the image embedding, no task-specific training), text-to-image search, and CLIP-style encoders serve as the "eyes" of many VLMs and as text encoders in diffusion image generators.

Vision-language models (VLMs)

Training usually proceeds in stages: align the projector on image-caption pairs with encoder and LLM frozen, then instruction-tune on visual question-answering and conversation data with more of the model unfrozen. Other modalities follow the same pattern: Whisper is an encoder-decoder Transformer from audio spectrograms to text; video models treat frames or spatio-temporal patches as tokens; diffusion transformers (DiT) replace the U-Net in image and video generators with a Transformer over latent patches.

# Zero-shot image classification with CLIP
from transformers import CLIPModel, CLIPProcessor
from PIL import Image
model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")
proc = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")
labels = ["a photo of a cat", "a photo of a dog", "a photo of a car"]
inputs = proc(text=labels, images=Image.open("pet.jpg"), return_tensors="pt", padding=True)
probs = model(**inputs).logits_per_image.softmax(dim=-1)   # similarity / temperature, then softmax
print(dict(zip(labels, probs[0].tolist())))

Major model families: a neutral map

Almost every major LLM is a decoder-only Transformer with the ingredients on this page (RoPE or similar, pre-norm, SwiGLU-style FFN, GQA, often MoE). Differences in behaviour come mostly from training data, scale, post-training (alignment choices and style), specialisation (code, reasoning, multimodality), context length and inference infrastructure. Model versions change every few months, so learn the families and their trade-offs rather than memorising version numbers.

Milestones that shaped the field

Families at a glance

Open weights vs closed

Reasoning models and test-time compute

Classic LLM scaling spent compute at training time. Reasoning models add a second axis: spending more compute at inference time by generating a long internal chain of thought (often hidden or summarised) before the final answer. On maths, coding, science and planning problems, accuracy keeps rising as the model is allowed to "think" longer.

Why generating more tokens helps

A Transformer does a fixed amount of computation per token (one pass through L layers). A hard problem may need more serial computation than one forward pass can provide. Writing intermediate steps turns the output sequence into working memory: each new token is computed with access to all previous reasoning, so the model effectively gets more layers of computation for harder problems. This is why chain-of-thought prompting ("think step by step") improved large models on multi-step problems even before dedicated reasoning models.

Ways to spend test-time compute

How reasoning models are trained

1. Start from a strong base or instruct model Reasoning ability rides on pretrained knowledge.
2. Optional cold-start SFT A small set of high-quality long chain-of-thought examples to establish readable format.
3. RL with verifiable rewards Sample many solutions per problem; reward correct final answers (maths answer checks, code unit tests) and correct format. Algorithms such as PPO or GRPO. Behaviours like re-checking, backtracking ("wait, that is wrong") and trying alternatives emerge because they raise reward.
4. Distillation Long reasoning traces from a big reasoning model are used to fine-tune smaller models, which gain much of the ability cheaply.
5. General alignment Further preference training so the model stays helpful and safe on ordinary tasks.

Outcome vs process rewards: outcome reward models score only the final answer (simple, scalable, but can reward lucky wrong reasoning); process reward models score each step (denser signal, but step labels are expensive and can be gamed).

Trade-offs

• Latency and cost: the major disadvantage. Hidden reasoning tokens are billed and can take many seconds or minutes.
• Overthinking: simple questions may get needlessly long deliberation; route easy queries to fast models or use a low thinking budget.
• Faithfulness: the visible reasoning is not guaranteed to be the real cause of the answer; explanations can be post-hoc rationalisations.
• Not a knowledge fix: reasoning helps with logic and multi-step problems, not with facts the model never learned; reasoning models can still hallucinate, sometimes elaborately.
• Prompting differs: they usually need less "think step by step" prompting and fewer examples; sampling parameters may be fixed by the provider.

Hallucination and the limitations of LLMs

A hallucination is output that is fluent and confident but not supported by facts or by the provided input: invented citations, wrong dates, non-existent API functions, misquoted documents. Understanding why it happens is what separates a good interview answer from "the model makes things up".

Why LLMs hallucinate

1. The objective is plausibility, not truth Pretraining rewards predicting likely text. A fluent false sentence and a fluent true one look similar to the loss.
2. Knowledge gaps and cutoffs Facts after the training cutoff, private company data and rare long-tail facts are missing or weakly stored, but the model still produces an answer-shaped continuation.
3. Compression Trillions of tokens are compressed into billions of weights; details blur, similar entities merge (the wrong author for a real book, a real journal with a fake article).
4. Incentives in post-training and evaluation Benchmarks and human raters often reward a confident answer over "I'm not sure"; models learn to guess.
5. Sycophancy Preference training can teach models to agree with the user's framing, including false premises ("Why did X win the prize in 2019?" when X never did).
6. Decoding Sampling can pick a low-probability wrong token, and autoregressive generation then commits to it and builds a consistent story around it (snowballing).
7. Context problems Relevant information missing from the prompt, contradictory sources, or facts buried in a long context that the model fails to use.
8. Exposure bias Trained on gold prefixes, the model has little practice recovering from its own earlier errors.

Note what is not a cause: residual connections, attention weights or Q/K/V matrices do not "cause" hallucination in themselves; it is a property of what the model learned and how it is used.

Mitigation toolkit

Other limitations to discuss

Quick revision

• RNNs are sequential (no parallel training), have O(n) paths between distant tokens and suffer vanishing/exploding gradients; seq2seq RNNs also squeeze the whole input into one context vector.
• Attention removed the bottleneck by letting the decoder take a fresh weighted average of all encoder states at every step; the Transformer (2017) removed recurrence entirely.
• Self-attention gives O(1) path length and full parallelism at the cost of O(n²d + nd²) time and O(n²) score-matrix memory (O(n) with FlashAttention).
• Text becomes subword tokens (BPE, WordPiece, Unigram/SentencePiece); English averages about 4 characters or 0.75 words per token; other languages often need more.
• APIs bill per input and output token; output tokens cost more and are generated sequentially; context window = input + output.
• The embedding table is vocab × d; input embeddings are static, hidden states after each layer are contextual.
• Attention is permutation-equivariant, so position must be injected: sinusoidal (added, fixed), learned absolute (added, trainable), RoPE (rotates Q and K, relative), ALiBi (distance penalty on scores).
• RoPE makes the q·k score depend only on relative distance, adds no parameters and can be extended with interpolation, NTK scaling or YaRN plus fine-tuning.
• Attention(Q,K,V) = softmax(QK^T/√d_k + mask)V; Q, K, V are linear projections of the same input in self-attention.
• Scaling by √d_k keeps dot-product variance near 1 so softmax does not saturate and gradients survive.
• Masked positions get −∞ (not 0) before softmax; causal masks block the future, padding masks block filler tokens.
• Attention weights are computed per input; only W_Q, W_K, W_V, W_O are learned.
• Multi-head attention splits d into h heads of size d/h, each learning a different relationship; concatenation plus W_O merges them at the same parameter cost as one head.
• Cross-attention takes Q from the decoder and K, V from the final encoder layer; decoder-only models do not use it.
• A block is attention + FFN, each wrapped with a residual connection and normalization; modern LLMs use pre-norm with RMSNorm.
• Residuals give gradients an identity path and let each layer learn an additive update to the residual stream.
• Pre-LN is x + F(Norm(x)) (clean residual path, modern default); Post-LN is Norm(x + F(x)) (needs warmup, unstable when very deep).
• LayerNorm normalises each token over features, so it is independent of batch size and sequence length, unlike BatchNorm.
• The FFN expands d → 4d (or about 8d/3 with SwiGLU), applies a non-linearity and projects back; it holds about two-thirds of parameters and acts per token.
• One block has about 12d² parameters; GPT-2 small = 124M, GPT-3 = 175B can be derived from this. Tying the embedding table to the LM head saves a second |V|·d matrix.
• Encoder-only (BERT) = bidirectional understanding; decoder-only (GPT, Llama) = causal generation; encoder-decoder (T5, BART, Whisper) = sequence transformation.
• CLM predicts the next token at every position (loss on all tokens); MLM predicts 15% masked tokens with the 80/10/10 rule; T5 uses span corruption with sentinel tokens.
• Perplexity = exp(average cross-entropy per token); lower is better.
• Training compute ≈ 6ND FLOPs; inference ≈ 2N FLOPs per generated token.
• Chinchilla: compute-optimal training uses about 20 tokens per parameter; modern small models are deliberately overtrained for cheaper inference.
• Emergent abilities such as in-context learning appear with scale, but some apparent jumps are artefacts of all-or-nothing metrics.
• The pipeline is pretraining → supervised fine-tuning on instructions → preference alignment (RLHF or DPO) → optional RL for reasoning.
• RLHF trains a reward model on human comparisons and optimises the policy with PPO plus a KL penalty to the SFT model to prevent reward hacking.
• DPO optimises the same objective directly on preference pairs with a classification-style loss, without a reward model or RL loop.
• Greedy and beam search are deterministic; beam search suits translation and summarisation but produces bland open-ended text.
• Temperature divides logits before softmax: below 1 sharpens, above 1 flattens, and the ranking never changes.
• Top-k keeps a fixed number of tokens, top-p keeps the smallest set reaching probability p, min-p keeps tokens above a fraction of the top probability.
• Repetition penalty scales logits of seen tokens; frequency penalty subtracts per occurrence; presence penalty subtracts once.
• Prefill is compute-bound (time to first token); decode is memory-bandwidth-bound (tokens per second).
• The KV cache stores past keys and values; size = 2 × layers × KV heads × head dim × tokens × batch × bytes.
• GQA shares K/V heads among groups of query heads (MQA uses one), shrinking the KV cache several-fold with little quality loss; MLA compresses K/V into a latent.
• FlashAttention is exact attention that tiles computation in on-chip memory: O(n) memory and faster, but the same O(n²d) FLOPs.
• Speculative decoding: a draft model proposes several tokens; the target verifies them in one parallel pass (rejection sampling leaves the output distribution unchanged).
• Sliding-window attention limits each token to recent tokens; stacked layers still spread information further.
• Long context is limited by quadratic attention, KV cache memory, position generalisation and "lost in the middle" effects.
• MoE replaces the FFN with many experts and a router choosing top-k per token: huge total parameters, small active parameters, but all experts must fit in memory.
• ViT turns an image into patch tokens; CLIP aligns image and text embeddings with a contrastive loss; VLMs feed projected visual tokens into an LLM.
• Open weights give control, privacy and fine-tuning; closed APIs give frontier capability with no infrastructure; behaviour differences come mainly from data and post-training.
• Reasoning models spend test-time compute on long chains of thought trained with RL on verifiable rewards; the cost is latency and tokens.
• Hallucination comes from a plausibility objective, knowledge gaps, incentives to guess, sycophancy and decoding; grounding and verification reduce it, temperature 0 does not eliminate it.
• Known limitations: tokenization quirks, stale knowledge, finite context, no persistent memory, bias, prompt injection and non-determinism.

Glossary

ALiBi

Attention with Linear Biases: adds a head-specific penalty proportional to token distance to attention scores instead of using position embeddings.

Alignment

Post-training that makes a model's behaviour match human intent and values (helpful, honest, harmless), for example via RLHF or DPO.

Attention

A mechanism that computes a weighted average of value vectors, with weights from query-key similarity.

Attention sink

Initial tokens that receive disproportionate attention; keeping them in the cache stabilises streaming generation.

Autoregressive

Generating one token at a time, each conditioned on all previously generated tokens.

Base model

A model after pretraining only; it continues text rather than following instructions.

Beam search

Deterministic decoding that keeps the B most probable partial sequences at every step.

BPE (Byte-Pair Encoding)

Tokenizer training method that repeatedly merges the most frequent adjacent symbol pair into a new token.

Causal language modelling (CLM)

Pretraining objective of predicting each next token from the tokens before it.

Causal mask

Lower-triangular attention mask that prevents a position from attending to future positions.

Chain-of-thought

Intermediate reasoning steps written before the final answer, prompted or trained.

Chinchilla scaling

Finding that compute-optimal training uses roughly 20 tokens per parameter, scaling model and data together.

CLIP

A model that learns a shared image-text embedding space with a contrastive loss, enabling zero-shot image classification.

Context window

Maximum number of tokens (input plus output) a model can attend to in one pass.

Continuous batching

Serving technique that adds and removes requests from a running batch at every step to keep the GPU busy.

Cross-attention

Attention with queries from one sequence (decoder) and keys/values from another (encoder output).

Cross-entropy loss

Negative log-probability assigned to the correct token, averaged over positions.

Decoder-only

Transformer with only causally masked self-attention blocks, used by GPT-style LLMs.

DPO (Direct Preference Optimization)

Alignment method that trains directly on preferred/rejected response pairs without a separate reward model or RL loop.

Embedding

A learned dense vector representing a token (input embedding) or a piece of content (sentence embedding).

Emergent ability

A capability that appears only in models above a certain scale, such as in-context learning.

Encoder-decoder

Transformer with a bidirectional encoder for the input and a causal decoder with cross-attention for the output (T5, BART).

Encoder-only

Transformer with bidirectional self-attention only, used for understanding tasks and embeddings (BERT).

Exposure bias

Mismatch between training on gold prefixes (teacher forcing) and generating from the model's own possibly wrong outputs.

Feed-forward network (FFN)

The per-token two-layer MLP in each block that expands, applies a non-linearity and projects back.

FlashAttention

Exact, I/O-aware attention kernel that tiles work in on-chip memory to avoid materialising the n × n matrix.

Frequency penalty

Decoding control that lowers a token's logit in proportion to how often it has already appeared.

GQA (Grouped-Query Attention)

Attention variant where groups of query heads share one key/value head to shrink the KV cache.

Greedy decoding

Always choosing the highest-probability next token.

GRPO

Group Relative Policy Optimization: RL method that scores each sampled answer relative to its group's average reward, used for reasoning training.

Hallucination

Fluent output that is not supported by facts or by the provided context.

In-context learning

Performing a new task from instructions or examples in the prompt without updating weights.

Induction head

An attention-head circuit that finds a previous occurrence of the current token and copies what followed it.

Instruction tuning

Supervised fine-tuning on instruction-response pairs so a model follows requests.

KL penalty

Term in RLHF that penalises the policy for drifting too far from the reference model.

KV cache

Stored keys and values of previous tokens, reused at each decoding step to avoid recomputation.

Layer normalization

Normalising each token's features to zero mean and unit variance, then applying learned scale and shift.

Logits

Raw, unnormalised scores for every vocabulary token before softmax.

Lost in the middle

Tendency of models to use information at the start and end of a long context more reliably than information in the middle.

Masked language modelling (MLM)

Pretraining objective of predicting randomly masked tokens using context on both sides.

Min-p sampling

Truncation that keeps tokens whose probability is at least a fraction of the most likely token's probability.

Mixture of Experts (MoE)

Architecture where a router sends each token to a few of many expert FFNs, decoupling total and active parameters.

MLA (Multi-head Latent Attention)

Attention variant that caches a compressed latent per token from which keys and values are reconstructed.

MQA (Multi-Query Attention)

Attention variant in which all query heads share a single key and value head.

Multi-head attention

Several attention operations in parallel with separate projections, concatenated and projected by W_O.

Open weights

Model parameters released for download and self-hosting, possibly under licence restrictions and without training data.

PagedAttention

Memory manager that stores the KV cache in fixed-size blocks to avoid fragmentation and share prefixes.

Perplexity

Exponential of the average per-token cross-entropy; the effective number of choices the model is unsure between.

Positional encoding

Any method of giving a Transformer information about token order.

Pre-LN / Post-LN

Placement of normalization before the sub-layer inside the residual branch (Pre-LN) or after the residual addition (Post-LN).

Prefill

The parallel forward pass over the prompt that fills the KV cache and produces the first token.

Presence penalty

Decoding control that lowers a token's logit by a fixed amount once it has appeared at all.

Prompt caching

Reusing the computed KV cache of a repeated prompt prefix across requests.

QK-norm

Layer- or RMS-normalizing queries and keys before the dot product to stop attention logits from growing with depth.

Repetition penalty

Multiplicative decoding control that makes already-seen tokens less likely.

Residual connection

Adding a sub-layer's input to its output (x + f(x)), giving gradients an identity path.

Residual stream

The per-token vector flowing through the network that every sub-layer reads from and adds to.

Reward hacking

A policy exploiting flaws in the reward model to get high scores without genuinely better outputs.

Reward model

Model trained on human comparisons to score responses, used as the objective in RLHF.

RLHF

Reinforcement Learning from Human Feedback: reward model from human preferences plus RL (typically PPO) to optimise the LLM.

RMSNorm

Simplified normalization that rescales by the root-mean-square of features without subtracting the mean.

RoPE

Rotary Position Embedding: rotates query and key dimension pairs by position-dependent angles so scores depend on relative distance.

Scaling laws

Empirical power-law relationships between loss and model size, data and compute.

Self-attention

Attention in which queries, keys and values all come from the same sequence.

Self-consistency

Sampling several reasoning paths and choosing the most common final answer.

Sliding-window attention

Local attention in which each token attends only to the previous w tokens.

Span corruption

T5's objective of replacing spans with sentinel tokens and generating the missing spans.

Speculative decoding

A draft model proposes several tokens that the large model verifies in parallel, speeding generation without changing outputs.

Supervised fine-tuning (SFT)

Training on curated input-output demonstrations with the standard next-token loss.

SwiGLU

Gated FFN variant: W₂(SiLU(W₁x) ⊙ W₃x), used in most modern LLMs.

Teacher forcing

Training a sequence model by always feeding the true previous tokens rather than its own predictions.

Temperature

Divisor applied to logits before softmax that controls how peaked or flat the sampling distribution is.

Test-time compute

Extra computation spent at inference (longer reasoning, sampling, search) to improve answers.

Token

The unit of text a model reads and writes, typically a subword piece mapped to an integer ID.

Top-k sampling

Sampling only from the k most probable next tokens after renormalising.

Top-p (nucleus) sampling

Sampling from the smallest set of tokens whose cumulative probability reaches p.

Vision Transformer (ViT)

Transformer encoder applied to sequences of image patches.

Weight tying

Sharing the token-embedding matrix with the output LM head (W_out = E), saving |V|·d parameters.

WordPiece

BERT's subword tokenizer, which chooses merges by likelihood gain and marks continuations with ##.

Interview questions