Fine-Tuning & Alignment

The big picture: where fine-tuning fits

A modern chat model is built in stages. Pretraining teaches a network general language and world knowledge by predicting the next token over trillions of tokens of text. The result, a base model, is a powerful autocomplete engine: ask it a question and it may continue with more questions, because that is what the internet looks like. Post-training then turns the base model into an assistant: supervised fine-tuning on demonstrations teaches it the question-answer format, and preference optimization (RLHF, DPO and relatives) teaches it which of several plausible answers humans actually prefer. When you "fine-tune" as a practitioner, you are adding one more post-training step on top of an existing base or instruct model, aimed at your own use case.

Mapping back: pretraining is the costly general education, SFT is the residency that teaches a specific way of working, and preference tuning is the senior-doctor feedback that sharpens which answers are preferred. Just like a resident, a fine-tuned model can also forget things from outside the specialty if it trains too narrowly.

                 THE LLM LIFECYCLE
 ┌──────────────┐   ┌──────────────┐   ┌──────────────────┐   ┌──────────────┐
 │ Pretraining  │──▶│ SFT /        │──▶│ Preference       │──▶│ Your domain  │
 │ next-token   │   │ instruction  │   │ optimization     │   │ fine-tune    │
 │ on ~10T tok  │   │ tuning       │   │ RLHF / DPO / RL  │   │ (LoRA/QLoRA) │
 └──────────────┘   └──────────────┘   └──────────────────┘   └──────────────┘
   base model         instruct model      chat/aligned model     specialised model
   (knowledge)        (follows format)    (preferred answers)    (your behaviour)
   months, $M+        hours-days           days                   minutes-hours

What fine-tuning actually changes

Fine-tuning runs ordinary gradient descent on the model's weights (all of them, or a small added set) using a loss computed on your examples. After training, the change lives in the weights: no extra prompt is needed at inference time. This is the key difference from prompting and in-context learning, where the weights are frozen and the model adapts only through the text you give it, and from retrieval-augmented generation (RAG), where facts are fetched at query time and pasted into the context.

Prompt vs RAG vs fine-tune: making the decision

Fine-tuning is the most expensive lever in terms of data, engineering and maintenance, so it should be pulled deliberately. The standard progression is: start with a strong prompt (zero-shot, then few-shot), add tools or structured output when you need determinism or actions, add RAG when the model is missing knowledge, and fine-tune when the desired behaviour is still unreliable or too expensive to achieve through prompts. The rule of thumb is short: new knowledge → RAG; new behaviour → fine-tune; both → both.

Mapping back: instructions correspond to prompts (cheap, instant, but must be repeated every time), the handbook corresponds to a retrieval index (fresh and citable), and the training course corresponds to weight updates (durable habits, but frozen at training time).

A decision procedure

1. Define success and build an eval set first. Without 50–200 labelled test cases and a metric, you cannot tell whether any lever helped.
2. Baseline with prompting. Zero-shot, then few-shot, then a structured system prompt. Record scores, cost and latency. See Prompt Engineering.
3. Diagnose the failures. Are they missing facts (retrieval problem) or wrong behaviour despite having the facts (format, reasoning, tone, refusal)?
4. Missing facts → RAG. Build retrieval and re-evaluate. See Retrieval-Augmented Generation.
5. Behaviour still wrong, or too costly → fine-tune. Start with LoRA/QLoRA SFT on a few thousand high-quality examples; consider preference tuning if the issue is "which answer is better" rather than "what format".
6. Combine. Many production systems fine-tune for style, format and tool use, and use RAG for facts. A retrieval-aware fine-tune (training on examples that include retrieved context) teaches the model to use context faithfully.

Types of fine-tuning

"Fine-tuning" is an umbrella term. The variants differ in what data is used, what loss is optimized and which parameters are updated. The first two axes decide what the model learns; the third (full vs parameter-efficient) decides cost.

Mapping back: raw domain text maps to continued pretraining, prompt–response pairs map to SFT, a narrow labelled task maps to task-specific tuning, and chosen-vs-rejected pairs map to preference optimization; the chef's retraining scope maps to updating all weights versus small adapters.

Full fine-tuning vs parameter-efficient fine-tuning

Instruction tuning in detail

Instruction tuning trains on examples that look like the requests a user would make. A single record typically has three fields, instruction, an optional input (extra context) and output (the target answer). Diversity of instructions matters as much as volume: models tuned on many different task types generalize to unseen instructions, which is why instruction tuning produced the jump from "autocomplete" to "assistant". Instruction tuning is different from prompt engineering: it changes weights during training, while prompt engineering changes only the input at inference.

{"instruction": "Classify the sentiment of the review.",
 "input": "The battery died after two days.",
 "output": "negative"}

{"instruction": "Explain what an ACE inhibitor does in one sentence.",
 "input": "",
 "output": "An ACE inhibitor lowers blood pressure by blocking the enzyme that makes angiotensin II, a hormone that narrows blood vessels."}

Data preparation: the part that decides the outcome

In fine-tuning, the dataset is the specification. The model will imitate whatever it sees, including typos, inconsistent formats, wrong answers and a verbose tone, so most of the work in a successful project is data work. This section covers formats, chat templates, cleaning, deduplication, synthetic data, and how to split the data for honest evaluation.

Mapping back: the finished pieces are your training examples (quality matters more than count), the inconsistent measurements are noisy labels and mixed formats, and the unfamiliar notation is a prompt that does not match the model's chat template.

Common dataset formats

{"messages": [
  {"role": "system", "content": "You are a concise support assistant for a telecom company."},
  {"role": "user", "content": "My bill is higher this month. Why?"},
  {"role": "assistant", "content": "Bills usually rise for three reasons: a promotion ended, you used extra data, or a one-off fee was added. Check the 'Charges' tab for the exact line item."}
]}

Chat templates: format must match the model

Every instruct model was trained with a specific chat template: special tokens that mark where each role's turn begins and ends. For example, one popular family wraps turns as <|im_start|>user ... <|im_end|>, another uses [INST] ... [/INST], and another uses header tokens such as <|start_header_id|>assistant<|end_header_id|>. If you fine-tune an instruct model with a different layout, you waste capacity re-teaching the format and often break its existing chat skills. In Hugging Face Transformers the tokenizer carries its template, and tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) renders it.

<|im_start|>system
You are a helpful medical assistant. Answer accurately and safely.<|im_end|>
<|im_start|>user
What is the first-line treatment for mild hypertension?<|im_end|>
<|im_start|>assistant
Lifestyle changes come first: reduce salt, ...<|im_end|>

• add_generation_prompt=True appends the opening of the assistant turn, so the model knows it should answer next. Use it when rendering prompts for training-with-masking and for inference.
• End-of-turn token. The target must end with the template's end-of-turn or EOS token (for example <|im_end|>). If you omit it, the model never learns when to stop and rambles until max_new_tokens.
• No double special tokens. A rendered template often already contains BOS; tokenizing it again with add_special_tokens=True can prepend a second BOS.
• Base models have no template. You may choose one (a common choice is to adopt the template of the matching instruct model), but then add any new special tokens to the tokenizer, resize embeddings and train the embedding rows too.

Loss masking: train on answers, not prompts

In SFT we usually compute the loss only on the assistant tokens (completion-only or "train on responses"). Prompt tokens still pass through the model as context, but their labels are set to -100 (the ignore index in PyTorch cross-entropy), or multiplied by a zero mask. Otherwise the model spends capacity learning to reproduce user prompts and system text, which is not the behaviour you want, and long prompts dominate the gradient. For multi-turn data, mask every user and system turn and supervise every assistant turn.

tokens : [sys][sys][user][user][user][asst-hdr] A1  A2  A3  <|im_end|> [pad][pad]
labels : -100 -100  -100  -100  -100    -100    A1  A2  A3  <|im_end|> -100 -100
                 context only (no loss)          loss here (incl. stop token)   ignored

Quality over quantity

A consistent finding across instruction-tuning work is that a small set of carefully curated examples (around a thousand in some well-known experiments) can produce a strong assistant from a good base model, while larger noisy sets add little. The base model already has the knowledge; SFT mostly teaches format and style, which is sometimes called the superficial alignment hypothesis. Rough sizes that work in practice:

Cleaning checklist

1. Normalize the schema. One format, consistent roles, placeholder tokens like a literal <noinput> converted to an empty string, whitespace stripped.
2. Filter bad examples. Empty or truncated answers, wrong language, broken markup, refusals where you want answers (or the reverse), answers that contradict the instruction.
3. Deduplicate. Exact duplicates via hashing; near-duplicates via MinHash/LSH or embedding similarity. Duplicates overweight some behaviours and leak between train and test.
4. Remove sensitive data. Scrub personal information, secrets and credentials; models can memorize and regurgitate rare strings.
5. Check lengths. Compute token-length percentiles and set max_seq_length near the 95th percentile; drop or split outliers instead of silently truncating answers.
6. Balance. Watch class balance and task mix; mix in some general-purpose data to reduce forgetting.
7. Decontaminate. Remove any example that overlaps with your evaluation set or with public benchmarks you plan to report.
8. Human spot-check. Read at least 100 random examples. Most dataset bugs are obvious to a human in ten minutes.

Synthetic data

Much modern fine-tuning data is generated by a stronger model: the teacher writes instructions (self-instruct style), answers them, rewrites human answers into a better format, or produces chosen/rejected pairs. Synthetic data is cheap and controllable but needs guardrails: seed it with diverse real prompts, filter with rules and an LLM judge, verify facts or run code/tests where possible, deduplicate aggressively (generators repeat themselves), and check the teacher's licence and terms of use, which often forbid training competing models. Beware model collapse: repeated training on model-generated text narrows diversity, so keep a share of real human data.

Train, validation and test splits

• Shuffle once with a fixed seed, then slice train/validation/test (for example 80/10/10, or at small scale a few hundred / a few dozen / a few dozen as in the worked example later).
• Split by group, not by row when examples are related (same customer, same document, same conversation), otherwise near-duplicates leak across splits.
• Validation drives early stopping and hyperparameter choices; test is touched once at the end.
• Keep a separate regression set of general tasks (math, coding, instruction following, safety prompts) to measure forgetting.

Packing and padding

Batches must be rectangular, so shorter sequences are padded. Padding wastes compute when lengths vary a lot. Packing concatenates several examples into one sequence of the maximum length, separated by EOS; with modern kernels, position IDs are reset per example and attention is blocked across boundaries so examples do not attend to each other. Packing can speed training several-fold on short-example datasets. Without packing, group similar lengths into the same batch to reduce padding.

How supervised fine-tuning works under the hood

SFT uses exactly the same objective as pretraining: the model predicts each next token and pays a cross-entropy penalty for putting low probability on the true one. The only differences are the data (curated examples instead of raw web text) and the mask (loss on answer tokens only). Understanding the shift-by-one and the masking removes most confusion about "how the model learns to answer".

Mapping back: the question is the masked prompt, each guess is a next-token prediction, the grade is the cross-entropy loss on answer tokens, and the correction is the gradient update. Because the true previous word is always revealed before the next guess, this is teacher forcing.

The shift-by-one

A causal language model produces logits at every position; the logits at position t are a prediction for token t+1. So we compare logits[:, :-1] with input_ids[:, 1:] and shift the mask the same way. Hugging Face models do this internally when you pass labels; in a hand-written loop you do it yourself:

def batch_loss(model, input_ids, attn, comp_mask):
    """Mean next-token cross-entropy over answer tokens only."""
    logits = model(input_ids=input_ids, attention_mask=attn).logits
    shift_logits = logits[:, :-1, :].contiguous()          # predictions for t+1
    shift_labels = input_ids[:, 1:].contiguous()           # the true next tokens
    shift_mask   = comp_mask[:, 1:].contiguous().float()   # 1 on answer tokens

    per_tok = F.cross_entropy(shift_logits.view(-1, shift_logits.size(-1)),
                              shift_labels.view(-1), reduction="none")
    return (per_tok * shift_mask.view(-1)).sum() / shift_mask.sum().clamp(min=1.0)

Token-level vs sequence-level averaging

Averaging over all answer tokens in a batch weights long answers more heavily than short ones; averaging per sequence first weights every example equally. With gradient accumulation, the naive approach of averaging each micro-batch and then averaging micro-batches is subtly wrong when micro-batches contain different token counts; libraries now normalize by the total number of supervised tokens across the accumulated batch. For most SFT it is a minor effect, but it matters when lengths vary widely.

The training loop in one picture

for each epoch:
  for each micro-batch:
      forward  ──▶ masked loss / grad_accum ──▶ backward (grads accumulate)
      every grad_accum steps:
          clip_grad_norm(1.0) ──▶ optimizer.step() ──▶ scheduler.step() ──▶ zero_grad()
  evaluate on validation ──▶ keep best checkpoint ──▶ early-stop if val loss rises

Mixed precision and numerics

• bf16 has the same exponent range as fp32, so it rarely overflows; it is the default on GPUs that support it (Ampere and newer).
• fp16 has a narrow range and needs loss scaling; full fine-tuning in pure fp16 can overflow to inf/NaN. Older GPUs such as the T4 lack fast bf16, so a common pattern there is fp32 master weights for full fine-tuning, and fp16 compute for LoRA/QLoRA (where only small adapters are trained).
• Master weights in fp32 keep tiny updates from being rounded away; mixed-precision training keeps an fp32 copy for the optimizer while computing in 16-bit.

Memory math for fine-tuning

Most practical fine-tuning questions reduce to "will it fit on my GPU?" GPU memory during training holds four things: the weights, the gradients, the optimizer states, and the activations saved for the backward pass (plus some framework overhead and temporary buffers). The first three scale with the number of trainable parameters; activations scale with batch size, sequence length, hidden size and number of layers.

Mapping back: per-book notes and histories exist only for books you are allowed to edit, which is exactly why freezing the base model removes most of the gradient and optimizer memory, while compression shrinks the weight storage itself.

Bytes per parameter

Worked example: a 7B model

Take a 7B-parameter decoder with 32 layers, hidden size 4096, 32 attention heads and an MLP width of 11008 (a common 7B shape). Parameter count is about 6.7B; use 7B for round numbers.

Where the 40M LoRA parameters come from: with rank r, an adapter on a din × dout linear layer adds r·(din + dout) parameters. Per layer: four attention projections of 4096×4096 give 4 × 16 × 8192 = 524,288; three MLP projections between 4096 and 11008 (gate, up, down) give 3 × 16 × 15104 = 724,992. That is ≈1.25M per layer, ×32 layers ≈ 40M parameters, or about 0.6% of the model. In bf16 the adapter file is about 80 MB. This 7B shape is multi-head attention; grouped-query models have thinner k_proj and v_proj, so the attention term is smaller. Embeddings and the LM head are not in this 40M unless you list them in modules_to_save.

Where the 3.9 GB NF4 figure comes from: 4 bits per weight is 0.5 bytes, so 7B weights take 3.5 GB. Blockwise quantization stores one scale per 64 weights; in fp32 that is 32/64 = 0.5 extra bits per weight, and double quantization reduces it to about 0.127 bits. Embeddings and the output head are often kept in 16-bit, which adds a few hundred MB.

Activations: the part people forget

For each transformer layer the backward pass needs intermediate tensors from the forward pass. A widely used estimate for 16-bit activations per layer, without recomputation, is roughly s·b·h·(34 + 5·a·s/h) bytes, where s is sequence length, b micro-batch size, h hidden size and a number of heads. The second term is the attention score matrix, which memory-efficient attention kernels (FlashAttention and SDPA) avoid storing.

These are estimates: exact numbers depend on MLP width, dropout masks and the framework. The important scaling rules are that activations grow linearly with batch size and layers, linearly with sequence length once fused attention is used (quadratically without it), and shrink dramatically with gradient checkpointing at a cost of roughly 20–35% more compute (one extra forward pass).

Levers to reduce memory, in the order people usually pull them

1. Smaller micro-batch, more gradient accumulation. Same effective batch, far fewer activations.
2. Gradient checkpointing. Recompute activations during backward; large savings for about a third more compute.
3. Fused/flash attention. Removes the quadratic attention-matrix term.
4. Shorter max_seq_length based on the data's length percentiles.
5. LoRA instead of full fine-tuning: removes gradients and optimizer states for the frozen weights.
6. QLoRA: 4-bit base weights, about a quarter of the bf16 footprint.
7. 8-bit or paged optimizers to shrink optimizer states and survive spikes.
8. Sharding and offload: ZeRO-2/3 or FSDP across GPUs, CPU/NVMe offload of optimizer states or parameters.
9. Chunked loss / fused cross-entropy: the logits tensor (batch × seq × vocab) is huge for 150k-token vocabularies; computing the loss in chunks avoids materializing it all.

The PEFT family: training a tiny fraction of the model

Parameter-efficient fine-tuning (PEFT) freezes the pretrained weights and trains a small number of new or selected parameters. It works because the change needed to adapt a large model to a task appears to live in a low-dimensional space: research on "intrinsic dimension" showed that many tasks can be learned by optimizing only a few hundred or thousand directions in parameter space. PEFT methods differ in where the trainable parameters are placed.

Mapping back: the house is the frozen pretrained network, each renovation style is a PEFT method that inserts trainable parameters at a different place, and "removing the changes" is detaching the adapter to recover the original model.

LoRA: low-rank adaptation in depth

LoRA (Low-Rank Adaptation) is based on a hypothesis: the update a pretrained weight matrix needs during fine-tuning has low intrinsic rank. So instead of learning a full dout × din update ΔW, LoRA learns two thin matrices whose product has rank at most r, typically 8 to 64, while the original weight stays frozen.

Mapping back: the orchestra tracks are the frozen weight matrix W, the small set of sliders is the low-rank pair B and A (the rank r is the number of sliders), each slider's broad effect is a rank-one direction, and the master knob is the scaling factor α/r.

The math

• Parameter count per adapted matrix: r·(din + dout) instead of din·dout. For a 4096×4096 projection with r = 16: 131,072 versus 16.8M, a 128× reduction.
• Initialization: A is random (Kaiming/Gaussian), B is zero. So at step 0, BA = 0 and the adapted model is exactly the pretrained model; training starts from a known-good function and is stable.
• Why not both zero? If both were zero, the gradient with respect to each would be zero (each gradient depends on the other matrix), so nothing would ever learn.
• Scaling α/r: keeps the update size roughly constant when you change r, so you do not need to retune the learning rate for every rank. A common convention is α = r or α = 2r. rsLoRA argues the right scaling is α/√r, which keeps learning stable at high ranks.
• Merging: after training, compute W' = W0 + (α/r)BA once. The merged model has exactly the original architecture and zero added latency.
• Dropout: LoRA dropout (for example 0.05) is applied to the input of the adapter path only, as light regularization.

            x (d_in)
            │
   ┌────────┴─────────┐
   │                  │
┌──┴───┐          ┌───┴───┐
│  W0  │ frozen   │   A   │ r × d_in   (random init)
│d_out │          └───┬───┘
│× d_in│              │  r-dim bottleneck
└──┬───┘          ┌───┴───┐
   │              │   B   │ d_out × r  (zero init)
   │              └───┬───┘
   │                  │ × (α / r)
   └──────── + ───────┘
             │
             h (d_out)

Why it works so well

Empirically, fine-tuning updates of large pretrained models have rapidly decaying singular values: most of the change is captured by a few directions. LoRA exploits this, and it also acts as a regularizer: a low-rank update cannot rewrite the whole matrix, which is part of why LoRA forgets less than full fine-tuning (a finding often summarized as "LoRA learns less and forgets less"). The flip side is that for large distribution shifts, such as teaching a new language or continued pretraining on code, full fine-tuning or high-rank LoRA learns more.

Choosing rank, alpha and target modules

LoRA in code with PEFT

from peft import LoraConfig, get_peft_model, TaskType

lora_cfg = LoraConfig(
    task_type=TaskType.CAUSAL_LM,
    r=16,
    lora_alpha=32,               # scale = alpha / r = 2
    lora_dropout=0.05,
    bias="none",
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],
)
model = get_peft_model(base_model, lora_cfg)
model.print_trainable_parameters()
# e.g. for a 0.5B model with all-linear r=16:
# trainable params: ~8.8M || all params: ~503M || trainable%: ~1.75

A from-scratch LoRA layer

import math, torch, torch.nn as nn

class LoRALinear(nn.Module):
    def __init__(self, base: nn.Linear, r=16, alpha=32, dropout=0.05):
        super().__init__()
        self.base = base
        for p in self.base.parameters():
            p.requires_grad = False                     # freeze W0 (and bias)
        self.A = nn.Parameter(torch.empty(r, base.in_features))
        self.B = nn.Parameter(torch.zeros(base.out_features, r))   # zero init
        nn.init.kaiming_uniform_(self.A, a=math.sqrt(5))
        self.scale = alpha / r
        self.drop = nn.Dropout(dropout)

    def forward(self, x):
        return self.base(x) + self.scale * (self.drop(x) @ self.A.T @ self.B.T)

    @torch.no_grad()
    def merge(self):
        self.base.weight += self.scale * (self.B @ self.A)   # W' = W0 + (alpha/r) BA
        return self.base

Variants worth knowing

QLoRA: fine-tuning on a 4-bit base

LoRA removed gradients and optimizer states for frozen weights, but the frozen weights themselves still occupy 2 bytes per parameter in bf16. QLoRA stores the frozen base model in 4-bit and trains ordinary 16-bit LoRA adapters on top. During each forward and backward pass, weights are dequantized block by block to the compute dtype, used in the matrix multiply and discarded. Gradients flow through the dequantized weights to the adapters, but the 4-bit weights never change. QLoRA introduced three ideas that make this work without losing quality.

Mapping back: the pocket edition is the NF4-quantized base, expanding an entry is on-the-fly dequantization to bf16/fp16, the full-quality notebook is the 16-bit LoRA adapters that are actually trained, and moving pages to the shelf is the paged optimizer spilling optimizer state to CPU memory during spikes.

1. 4-bit NormalFloat (NF4)

Pretrained weights are roughly normally distributed around zero. A uniform 4-bit grid (INT4) wastes levels in the sparse tails and has too few near zero, where most weights sit. NF4 places its 16 levels at the quantiles of a standard normal distribution, rescaled to [−1, 1], so each level is used about equally often: an information-theoretically near-optimal code for normally distributed data. It includes an exact zero.

Quantization is blockwise: weights are split into blocks of 64, each block is scaled by its absolute maximum (absmax) so values fall into [−1, 1], and each value is mapped to the nearest NF4 level. Blockwise scaling limits the damage an outlier can do to one block rather than a whole tensor.

2. Double quantization

Each block of 64 weights needs its own fp32 scale: 32 bits / 64 weights = 0.5 extra bits per parameter, which is 0.44 GB for a 7B model. Double quantization quantizes those scales too: the fp32 constants are grouped into blocks of 256 and quantized to 8-bit with one fp32 second-level constant per group. The overhead falls to about 8/64 + 32/(64·256) ≈ 0.127 bits per parameter, saving roughly 0.37 bits per parameter (about 0.3 GB on 7B, several GB on 65B).

3. Paged optimizers

Memory use during training is spiky, particularly with long sequences and gradient checkpointing. Paged optimizers allocate optimizer states in NVIDIA unified memory, which automatically pages them to CPU RAM when the GPU is about to run out and pages them back when needed, turning a crash into a small slowdown. In practice you select optim="paged_adamw_8bit" or "paged_adamw_32bit".

Compute dtype and the full recipe

• Storage dtype is NF4; compute dtype is bf16 (or fp16 on GPUs without bf16 support). Matmuls always happen in 16-bit.
• prepare_model_for_kbit_training casts layer norms (and usually the output head) to fp32 for stability, enables gradient checkpointing, and makes input embeddings require gradients so gradients can flow through a frozen, checkpointed network into the adapters.
• Adapters on all linear layers were key to QLoRA matching 16-bit full fine-tuning in the original experiments; the headline result was fine-tuning a 65B model on a single 48 GB GPU.
• Speed: QLoRA is typically 20–40% slower per step than LoRA because of dequantization, a trade of time for memory.

import torch
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from peft import prepare_model_for_kbit_training, get_peft_model, LoraConfig

bnb_cfg = BitsAndBytesConfig(
    load_in_4bit=True,                      # store frozen weights in 4-bit
    bnb_4bit_quant_type="nf4",              # NormalFloat4 levels (vs "fp4")
    bnb_4bit_use_double_quant=True,         # quantize the per-block scales too
    bnb_4bit_compute_dtype=torch.bfloat16,  # float16 on T4-class GPUs
)
base = AutoModelForCausalLM.from_pretrained(
    "your-org/your-7b-model", quantization_config=bnb_cfg, device_map="auto")
base = prepare_model_for_kbit_training(base, use_gradient_checkpointing=True)
model = get_peft_model(base, LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05,
                                        target_modules="all-linear",
                                        task_type="CAUSAL_LM"))

For the broader quantization picture (PTQ, GPTQ, AWQ, INT8/INT4 inference, quantization-aware training and on-device formats) see Edge AI. Quantization-aware training combined with LoRA is also used to recover accuracy for aggressively quantized on-device models.

Hyperparameters that matter

Fine-tuning has fewer knobs than pretraining, but the defaults differ by method, and a wrong learning rate is the most common reason a run either does nothing or destroys the model. Treat the table below as starting points, then tune on validation loss and, more importantly, on task metrics.

Mapping back: too much accelerator (learning rate) and the car spins off (divergence or forgetting), no warmup and the tires lose grip at the start (unstable early updates while Adam's statistics are still poor), and too many sessions produce a driver who only knows one track (overfitting).

The rules behind the numbers

• Learning rate is the most important knob. Full fine-tuning moves pretrained weights directly, so it needs tiny steps to avoid wrecking learned features. LoRA starts from a zero update on a frozen base and can take larger steps.
• Effective batch size = micro-batch × gradient-accumulation steps × number of GPUs. When you increase it substantially, raise the learning rate a bit (square-root scaling is a safe heuristic for Adam).
• Warmup protects the first steps, when Adam's second-moment estimates are poor and a few large updates can damage the model.
• Epochs: SFT data is often seen only 1–3 times. Many passes over a small set drive the training loss towards zero while validation loss rises, the classic overfitting sign, and the model starts repeating training answers verbatim.
• Sequence length: set from data percentiles; longer costs memory linearly (with fused attention) and wastes compute on padding if most examples are short.
• Steps, not epochs, are what matter for compute planning: steps per epoch = ⌈N / effective batch⌉.
• Seed and reproducibility: fix seeds for Python, NumPy and PyTorch; small-data fine-tuning can vary noticeably between seeds, so compare methods over several seeds if differences are small.

Reading the loss curves

The tooling landscape and distributed training

The open-source stack for fine-tuning is layered. At the bottom are PyTorch and GPU kernels; above that are model libraries; then training libraries that implement SFT and preference losses; then distributed-training backends; and finally configuration-driven wrappers. Knowing which layer each tool occupies makes them easy to compare.

Mapping back: PyTorch and kernels are the raw materials, Transformers and PEFT are the prefabricated components, TRL's trainers are the specialist trades, DeepSpeed and FSDP are the heavy machinery for multi-GPU jobs, and YAML-driven tools like Axolotl are the general contractor working from a plan.

Distributed strategies

With ZeRO-3 on 8 GPUs, the 112 GB of weights, gradients and optimizer states for full 7B fine-tuning becomes about 14 GB per GPU, leaving room for activations on 40–80 GB cards.

Minimal SFT with TRL

from datasets import load_dataset
from trl import SFTTrainer, SFTConfig
from peft import LoraConfig

ds = load_dataset("json", data_files={"train": "train.jsonl", "eval": "val.jsonl"})

cfg = SFTConfig(
    output_dir="out/sft-lora",
    num_train_epochs=2,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,       # effective batch 16 per GPU
    learning_rate=2e-4,
    lr_scheduler_type="cosine",
    warmup_ratio=0.05,
    max_grad_norm=1.0,
    bf16=True,
    gradient_checkpointing=True,
    max_length=1024,                      # named max_seq_length in older versions
    packing=False,
    assistant_only_loss=True,             # loss on assistant turns only (recent versions)
    eval_strategy="epoch",
    save_strategy="epoch",
    load_best_model_at_end=True,
    logging_steps=10,
)
trainer = SFTTrainer(
    model="your-org/your-8b-instruct",
    args=cfg,
    train_dataset=ds["train"],            # records with a "messages" field
    eval_dataset=ds["eval"],
    peft_config=LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05,
                           target_modules="all-linear", task_type="CAUSAL_LM"),
)
trainer.train()
trainer.save_model("out/sft-lora/adapter")

Alignment with RLHF

After SFT, a model imitates demonstrations, but imitation has limits: there are many acceptable answers, demonstrators vary in quality, and it is much easier for people to compare two answers than to write a perfect one. Reinforcement Learning from Human Feedback (RLHF) uses those comparisons. It was the method behind the first widely used chat assistants, where annotators preferred a 1.3B aligned model over a 175B base model, showing that alignment can matter more than size for perceived helpfulness.

Mapping back: the recipe book is the SFT model, the critic is human labellers, the assistant is the learned reward model, experimenting with scores is PPO optimization, the house-style rule is the KL penalty to the reference model, and fooling the assistant with salt is reward hacking.

The three-stage pipeline

1. Supervised fine-tuning. Train the base model on human-written demonstrations of good responses. This teaches format and produces the reference policy π_ref.
2. Reward model training. Sample several responses per prompt from the SFT model; humans rank them. Train a reward model (usually the SFT model with a scalar head replacing the vocabulary head) to give preferred responses higher scores.
3. RL optimization (PPO). Generate responses with the policy, score them with the reward model, and update the policy to increase reward while a KL penalty keeps it close to π_ref.

  prompts ──▶ [SFT model] ──▶ y1, y2, ... ──▶ humans rank ──▶ (x, y_w, y_l) pairs
                                                                 │
                                                                 ▼
                                                       [Reward model r_φ(x, y)]
                                                                 │ score
  prompts ──▶ [Policy π_θ] ──▶ response y ──────────────────────▶│
                  ▲                                               ▼
                  │   PPO update  ◀──  reward = r_φ(x,y) − β·KL(π_θ ‖ π_ref)
                  └──────────────────  (value model estimates advantages)

Reward model: Bradley–Terry loss

The RL objective

PPO optimizes this with a clipped surrogate objective. For each generated token it computes the probability ratio ρ = π_θ(a|s) / π_old(a|s) and an advantage estimate A (from a learned value model using generalized advantage estimation), then maximizes:

Classic PPO-based RLHF juggles four models in memory: the policy, the frozen reference, the reward model and the value (critic) model. That is expensive and notoriously finicky (sensitive to KL coefficient, reward normalization, generation length and many implementation details), which motivated the simpler methods in the next section.

RLAIF and Constitutional AI

Human labels are slow and costly (large programmes take weeks to months). RLAIF (RL from AI Feedback) replaces human preference labels with judgements from a capable LLM prompted with guidelines. Constitutional AI is a structured version: a written list of principles (the "constitution") is used first in a supervised phase where the model critiques and revises its own harmful responses, and then in an RL phase where an AI judge chooses between responses according to the principles. It scales harmlessness training with far less human labelling of harmful content, and makes the target values explicit and auditable.

Known failure modes of RLHF

• Reward hacking: the policy finds outputs the reward model overrates (overly long answers, flattering language, certain phrases). Mitigations: KL penalty, reward-model ensembles, length penalties, periodic reward-model retraining.
• Sycophancy: agreeing with the user because labellers tend to prefer agreeable answers.
• Verbosity bias: longer answers win comparisons, so the policy learns to pad.
• Mode collapse / reduced diversity: aligned models become less creative and less calibrated than base models.
• Alignment tax: small drops on some capability benchmarks after alignment; mixing pretraining-style loss into RL (as some pipelines do) reduces it.

Beyond PPO: DPO, ORPO, KTO, GRPO and friends

Since 2023 a wave of methods has simplified preference learning. They fall into two groups: offline, RL-free methods that learn directly from a fixed preference dataset with a classification-like loss (DPO, IPO, ORPO, KTO, SimPO), and online RL methods that still sample from the current policy but drop expensive components (GRPO, RLOO). Online methods with verifiable rewards (unit tests, exact answers) power modern reasoning models.

Mapping back: the tutor is the reward and value models, the marked pairs are an offline preference dataset used by DPO, and comparing drafts against their own average is GRPO's group-relative advantage, which replaces the learned critic.

DPO: Direct Preference Optimization

DPO's insight is that the KL-regularized RLHF objective has a closed-form optimal policy: π*(y|x) ∝ π_ref(y|x)·exp(r(x,y)/β). Rearranging, the reward can be expressed through the policy itself: r(x,y) = β·log(π(y|x)/π_ref(y|x)) + const. Substituting this "implicit reward" into the Bradley–Terry loss eliminates the reward model and the RL loop.

• Needs: a preference dataset and two models (policy + frozen reference). With LoRA, the reference is just the base with adapters disabled, so only one copy is held in memory.
• Pros: simple, stable, cheap, no sampling during training.
• Cons: offline, so it cannot explore beyond the dataset; can overfit to preferences (driving both chosen and rejected likelihoods down); sensitive to how far the data is from the policy's own outputs; can increase verbosity.
• Metrics to watch: reward accuracy (fraction of pairs where implicit reward of chosen > rejected), reward margins, and the log-probabilities of chosen responses (if they fall steadily, the model may be degrading).

from trl import DPOTrainer, DPOConfig
# dataset rows: {"prompt": [...messages...], "chosen": [...], "rejected": [...]}
cfg = DPOConfig(output_dir="out/dpo", beta=0.1, learning_rate=5e-6,
                per_device_train_batch_size=2, gradient_accumulation_steps=8,
                num_train_epochs=1, warmup_ratio=0.1, bf16=True,
                max_length=1024)
trainer = DPOTrainer(model=sft_model, ref_model=None,   # None + PEFT: base acts as reference
                     args=cfg, train_dataset=pref_ds,
                     processing_class=tokenizer, peft_config=lora_cfg)
trainer.train()

The rest of the family

GRPO and RL with verifiable rewards

Group Relative Policy Optimization keeps PPO's clipped objective and KL control but removes the value model. For each prompt, sample G responses (for example 8–16), score each with a reward function, and set each response's advantage to its reward standardized within the group. Responses better than their siblings are reinforced; worse ones are discouraged. This halves memory relative to PPO and works especially well when rewards are verifiable: a math answer matches the reference, code passes unit tests, output parses as valid JSON. Training on such rewards (often called RLVR) is how models learn long chain-of-thought reasoning, with behaviours like self-checking emerging from reward alone.

from trl import GRPOTrainer, GRPOConfig

def correctness_reward(completions, answer, **kwargs):
    # 1.0 if the final line contains the reference answer, else 0.0
    return [1.0 if a in c[-1]["content"].splitlines()[-1] else 0.0
            for c, a in zip(completions, answer)]

def format_reward(completions, **kwargs):
    return [0.2 if "<answer>" in c[-1]["content"] else 0.0 for c in completions]

cfg = GRPOConfig(output_dir="out/grpo", num_generations=8, learning_rate=1e-6,
                 max_completion_length=512, beta=0.04, bf16=True)
trainer = GRPOTrainer(model=sft_model, reward_funcs=[correctness_reward, format_reward],
                      args=cfg, train_dataset=math_prompts)   # rows: prompt, answer
trainer.train()

GRPO versus PPO

This comparison is a favourite interview follow-up. Both maximize a clipped probability-ratio objective; they differ in how the advantage is estimated and what sits in GPU memory.

RLOO is the close cousin: it also drops the critic, but uses a leave-one-out mean of the other samples as the baseline instead of a group z-score. Use PPO when you need a learned reward for fuzzy preferences; use GRPO/RLOO when a checker can score each sample.

Choosing a method

Evaluating a fine-tuned model

A lower loss is not the goal; better behaviour on the target task without breaking everything else is. Evaluation for fine-tuning therefore has two halves: did it learn the task? (held-out task metrics, human or judge ratings) and did it keep what it had? (general benchmarks, safety, regression tests). Always evaluate the base model with its best prompt on the same sets, so improvements are measured against a fair baseline. The broader methodology is covered in LLM Evaluation & Safety.

Mapping back: unseen cases are the held-out test set, the practical exam is task metrics and judge ratings, and the routine competency checks are general benchmarks, safety prompts and regression tests that detect catastrophic forgetting.

Layers of evaluation

Lexical vs semantic metrics

Reporting both gives a fuller picture: if ROUGE rises but BERTScore does not, the model may just be copying phrasing; if BERTScore rises with flat ROUGE, it paraphrases correctly. When references are themselves machine-generated or noisy, weight relative differences between methods more than absolute values.

LLM-as-judge done right

• Use a pairwise format (model A vs model B) with a clear rubric; pairwise judgements are more reliable than absolute 1–10 scores.
• Swap positions and average to cancel position bias; control for verbosity bias (judges favour longer answers) and self-preference (judges favour their own family's style).
• Validate the judge against a few hundred human labels and report agreement.
• Use a judge that is stronger than, and different from, the model being evaluated.

Evaluation hygiene

1. Same prompts, same decoding for every model (for example greedy, same max_new_tokens), and decode only the newly generated tokens.
2. Use the chat template at evaluation exactly as in training.
3. Check contamination: n-gram or embedding overlap between training data and test sets.
4. Error analysis: read failures, bucket them (too short, too long, hallucinated fact, missing caveat, wrong format), and compute simple statistics such as the prediction/reference length ratio and counts of empty or truncated outputs.
5. Report cost alongside quality: trainable parameters, peak memory, training time, inference latency.
6. Several seeds when differences are small, with confidence intervals (bootstrap over test examples).

Risks and failure modes

Fine-tuning is powerful precisely because it changes the model, and every change can break something. The main risks are losing general capabilities, memorizing instead of generalizing, eroding safety behaviour, leaking data, and violating licences. Each has known mitigations.

Mapping back: stumbling on old pieces is catastrophic forgetting, over-applied phrasing is overfitting, forgotten etiquette is safety regression, and the pirated score is a licensing problem with training data or base-model terms.

A mitigation recipe for forgetting

1. Measure first. Build a general regression suite and score the base model.
2. Prefer PEFT. LoRA with moderate rank; the base weights stay intact and the adapter can be scaled down (for example multiply the LoRA scale by 0.5) if it is too strong.
3. Mix data. Add general instruction and chat examples (replay) alongside the domain data.
4. Train gently. Lower LR, 1–2 epochs, early stopping.
5. Consider model merging. Interpolating fine-tuned and base weights often recovers general ability with a small loss of task gains.

Merging adapters and merging models

"Merging" means two different things. Adapter merging folds a LoRA update into its base weights so the result is a normal model. Model merging combines several fine-tuned models (or adapters) that share a base into one model with the skills of each, using arithmetic on weights and no further training. Both are cheap and widely used.

Mapping back: the base map is the shared pretrained model, each overlay is a weight delta (a task vector or LoRA update), printing is merge_and_unload, and resolving disagreements is what TIES and DARE do with conflicting parameter updates.

Adapter merging

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained("your-org/base-model",
                                            torch_dtype=torch.bfloat16)   # 16-bit, not 4-bit
model = PeftModel.from_pretrained(base, "out/sft-lora/adapter")
merged = model.merge_and_unload()           # W' = W0 + (alpha/r) * B @ A, adapters removed
merged.save_pretrained("out/merged", safe_serialization=True)
AutoTokenizer.from_pretrained("out/sft-lora/adapter").save_pretrained("out/merged")

• Always merge into a 16-bit copy of the exact base used in training (same revision).
• Save the tokenizer and chat template with the merged model.
• PEFT can also combine several LoRAs into one adapter (add_weighted_adapter with methods such as linear, SVD, TIES or DARE), or keep many adapters loaded and switch with set_adapter.

Model merging methods

Configuration-driven tools such as mergekit implement these methods. Merged models must be evaluated like any other: merges can silently lose instruction following or produce format mixtures.

Knowledge distillation

Distillation trains a small student model to mimic a large teacher. For LLMs it is the main way to obtain small models that behave far above their size: the student learns from the teacher's outputs (and ideally its full probability distribution), which carry much more information per example than one-hot labels.

Mapping back: the coach is the teacher model, the percentages are its soft probability distribution over next tokens ("dark knowledge"), and learning from the full distribution rather than just the top choice is logit distillation, versus sequence-level distillation that trains on the teacher's final text.

Forms of distillation

import torch.nn.functional as F

def kd_loss(student_logits, teacher_logits, labels, mask, T=2.0, lam=0.5):
    # logits: [B, S, V]; labels: [B, S]; mask: [B, S] (1 on answer tokens)
    ce = F.cross_entropy(student_logits.transpose(1, 2), labels, reduction="none")
    kl = F.kl_div(F.log_softmax(student_logits / T, -1),
                  F.log_softmax(teacher_logits / T, -1),
                  log_target=True, reduction="none").sum(-1) * (T * T)
    per_tok = (1 - lam) * ce + lam * kl
    return (per_tok * mask).sum() / mask.sum().clamp(min=1)

Serving fine-tuned models

After training you have a base model plus an adapter (or a full checkpoint). Deployment choices trade off latency, memory, the number of variants you need, and target hardware. The main decisions are: merge or keep adapters separate, how to serve many variants, and how to quantize for cost or edge deployment.

Mapping back: the glued lens is a merged model (no overhead, one task per copy), the camera body is the shared base model in GPU memory, the lenses are LoRA adapters of a few megabytes each, and the swap time is the small per-request cost of applying a different adapter.

Multi-LoRA serving

Modern inference engines batch requests for different adapters together. The base matmul is shared across the batch, and specialized kernels (segmented or gathered batched matmuls) apply each request's own A and B. Adapters live in a CPU/GPU cache and are paged in on demand. Engines such as vLLM (with LoRA enabled at startup), dedicated multi-LoRA servers, and research systems like S-LoRA and Punica follow this design; managed inference products expose the same idea. Constraints to know: a maximum rank per server, a maximum number of adapters resident on GPU at once, and all adapters must share the same base.

# serve a base model with adapters that clients select by name (vLLM CLI)
vllm serve your-org/base-8b-instruct \
    --enable-lora \
    --lora-modules support=./adapters/support sql=./adapters/sql \
    --max-lora-rank 64 --max-loras 8

# request the "sql" adapter through the OpenAI-compatible endpoint
curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" \
  -d '{"model": "sql", "messages": [{"role": "user", "content": "Top 5 customers by revenue?"}]}'

Quantizing after fine-tuning

Use a calibration set that resembles your fine-tuning domain (not generic web text) when quantizing a specialised model, and re-run your regression suite: quantization errors hit rare, domain-specific behaviours first.

GGUF export for local and edge deployment

1. Merge the adapter into a 16-bit base and save with the tokenizer.
2. Convert to a 16-bit GGUF file with the llama.cpp conversion script.
3. Quantize to the target type (Q4_K_M is a common balance of size and quality; Q8_0 for near-lossless).
4. Run and evaluate with the same chat template; check that the template embedded in the GGUF metadata is correct.

# 1) merged model already saved in out/merged (see the merging section)
# 2) convert Hugging Face checkpoint to GGUF (16-bit)
python llama.cpp/convert_hf_to_gguf.py out/merged --outfile model-f16.gguf --outtype f16
# 3) quantize to 4-bit k-quant
./llama.cpp/build/bin/llama-quantize model-f16.gguf model-Q4_K_M.gguf Q4_K_M
# 4) quick interactive test
./llama.cpp/build/bin/llama-cli -m model-Q4_K_M.gguf -cnv
# alternatively keep the adapter separate: convert it with convert_lora_to_gguf.py
# and load it at runtime next to the base GGUF (--lora adapter.gguf)

On-device stacks take this further: a shared base model ships once with the operating system or app, and small task-specific LoRA adapters (tens of MB) are hot-swapped per feature, while quantization-aware training plus LoRA recovers accuracy lost to aggressive 4-bit quantization. See Edge AI for runtimes, NPU constraints and memory budgets.

End-to-end walkthrough: full fine-tuning vs LoRA vs QLoRA

This worked example puts everything together on a realistic small-scale study: adapt a 0.5B-parameter instruct model to answer medical instructions, training the same model on the same data three ways (full instruction fine-tuning, LoRA and QLoRA, the last two at ranks 8 and 16) on a single 16 GB T4-class GPU, then compare quality against cost. It uses a hand-written PyTorch loop rather than a high-level trainer so every mechanism is visible. Swap in a 7B–8B model and more data and the same code is a production-shaped QLoRA pipeline.

Mapping back: the ingredients are the fixed train/validation/test split, the variable being changed is the trainable-parameter scope (full, LoRA) and then the backbone precision (16/32-bit vs 4-bit), and "taste vs cost" is ROUGE-L/BERTScore against trainable %, peak memory and training time.

 data (instruction, input, output)
   │ clean ("<noinput>" → ""), shuffle(seed), split 300 / 30 / 30
   ▼
 chat template ──▶ tokenize once with offsets ──▶ completion mask (answer = 1)
   ▼
 ┌──────────────┬──────────────────────┬─────────────────────────────┐
 │ Full IFT     │ LoRA r=8, r=16       │ QLoRA r=8, r=16             │
 │ fp32, lr 1e-5│ fp32 base, lr 2e-4   │ NF4 base, fp16 compute,     │
 │ 100% trained │ adapters on 7 linears│ same adapters, lr 2e-4      │
 └──────┬───────┴──────────┬───────────┴──────────────┬──────────────┘
        └────────── shared loop: masked loss, accumulation, warmup+cosine,
                    clipping, per-epoch validation, keep best checkpoint
                                   ▼
          greedy generation on test ──▶ ROUGE-L + BERTScore ──▶ error analysis
                                   ▼
            cost table: % trainable, peak GPU memory, training time

Step 1: environment, seed and configuration

# pip install -q transformers datasets peft "bitsandbytes>=0.43" evaluate rouge_score bert_score
import gc, math, random, time
import numpy as np, pandas as pd, torch
import torch.nn.functional as F
from torch.utils.data import Dataset, DataLoader
from datasets import load_dataset
from transformers import (AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig,
                          get_cosine_schedule_with_warmup, set_seed)
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training, TaskType

SEED = 42
def seed_everything(seed=SEED):
    random.seed(seed); np.random.seed(seed)
    torch.manual_seed(seed); torch.cuda.manual_seed_all(seed); set_seed(seed)
seed_everything()
DEVICE = "cuda" if torch.cuda.is_available() else "cpu"

MODEL_ID   = "Qwen/Qwen2.5-0.5B-Instruct"     # any small instruct model works
N_TRAIN, N_VAL, N_TEST = 300, 30, 30          # tiny on purpose: fits one session
MAX_LEN    = 256                              # chosen from the token-length p95
BATCH_SIZE, GRAD_ACCUM, EPOCHS = 4, 1, 3      # effective batch = 4
WARMUP_RATIO, MAX_GRAD_NORM = 0.05, 1.0
LR_FULL, LR_LORA = 1e-5, 2e-4                 # full FT needs a much smaller LR
LORA_DROPOUT = 0.05
TARGETS = ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"]

Step 2: load, clean, split and inspect the data

The dataset has three fields: instruction, an optional input with extra context (stored as a literal <noinput> placeholder when absent) and output, a reference answer that was itself generated by an older LLM. That last point matters for evaluation: the references are a noisy proxy for truth, so relative comparisons between methods are more meaningful than absolute scores.

df = pd.DataFrame(load_dataset("your-org/medical-instructions")["train"])
df["input"] = df["input"].replace("<noinput>", "").fillna("")
df = df.sample(frac=1, random_state=SEED).reset_index(drop=True)   # shuffle once
train_df = df.iloc[:N_TRAIN]
val_df   = df.iloc[N_TRAIN:N_TRAIN + N_VAL]
test_df  = df.iloc[N_TRAIN + N_VAL:N_TRAIN + N_VAL + N_TEST]

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

lens = train_df.apply(lambda r: len(tokenizer(r["instruction"] + " " + r["input"] + " " + r["output"],
                                              add_special_tokens=False)["input_ids"]), axis=1)
print("rows with input:", (train_df["input"] != "").mean().round(2))
print(lens.describe(percentiles=[0.5, 0.95]))   # choose MAX_LEN near the p95

Step 3: chat-template prompts and an offset-based completion mask

SYSTEM_MSG = "You are a helpful medical assistant. Answer accurately and safely."

def build_prompt(instruction, input_text):
    user = f"{instruction}\n{input_text}" if input_text else instruction
    msgs = [{"role": "system", "content": SYSTEM_MSG},
            {"role": "user", "content": user}]
    # add_generation_prompt=True appends the assistant header so the model answers next
    return tokenizer.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)

class SFTDataset(Dataset):
    def __init__(self, frame):
        self.rows = [{"prompt": build_prompt(r["instruction"], r["input"]),
                      "answer": r["output"].strip()} for _, r in frame.iterrows()]
    def __len__(self): return len(self.rows)
    def __getitem__(self, i): return self.rows[i]

def tokenize_example(prompt, answer):
    """Tokenize prompt+answer ONCE; mark answer tokens using character offsets."""
    full = prompt + answer + tokenizer.eos_token          # end-of-turn so the model learns to stop
    enc = tokenizer(full, return_offsets_mapping=True, add_special_tokens=False,
                    truncation=True, max_length=MAX_LEN)
    start = len(prompt)                                    # first answer character
    comp_mask = [1 if s >= start else 0 for s, e in enc["offset_mapping"]]
    return enc["input_ids"], comp_mask

def collate(batch):
    ids, masks = zip(*(tokenize_example(b["prompt"], b["answer"]) for b in batch))
    L = max(len(x) for x in ids)
    input_ids = torch.full((len(ids), L), tokenizer.pad_token_id, dtype=torch.long)
    attn      = torch.zeros((len(ids), L), dtype=torch.long)
    comp_mask = torch.zeros((len(ids), L), dtype=torch.long)
    for i, (x, m) in enumerate(zip(ids, masks)):
        input_ids[i, :len(x)] = torch.tensor(x)
        attn[i, :len(x)] = 1                               # from length, NOT from pad id
        comp_mask[i, :len(m)] = torch.tensor(m)
    return input_ids, attn, comp_mask

train_loader = DataLoader(SFTDataset(train_df), batch_size=BATCH_SIZE, shuffle=True,  collate_fn=collate)
val_loader   = DataLoader(SFTDataset(val_df),   batch_size=BATCH_SIZE, shuffle=False, collate_fn=collate)

x, a, m = next(iter(train_loader))
print(x.shape, "supervised tokens:", int(m.sum()), "masked:", int((a - m).sum()))

Three design decisions are worth defending in an interview. The chat template reproduces the special tokens the instruct model was trained with. The completion mask restricts the loss to answer tokens. Tokenizing once with offsets avoids boundary drift: tokenizing the prompt separately and using its length can mis-align by a token when the tokenizer would merge characters across the boundary. Padding fills input_ids with the pad token but the masks with 0, so padded positions are neither attended to nor scored; and because the pad token equals EOS here, the attention mask is built from lengths so the real EOS is still attended and supervised.

Step 4: the shared training loop

def batch_loss(model, input_ids, attn, comp_mask):
    logits = model(input_ids=input_ids, attention_mask=attn).logits
    logits, labels = logits[:, :-1].float(), input_ids[:, 1:]
    mask = comp_mask[:, 1:].float()
    tok = F.cross_entropy(logits.reshape(-1, logits.size(-1)), labels.reshape(-1), reduction="none")
    return (tok * mask.reshape(-1)).sum() / mask.sum().clamp(min=1.0)

@torch.no_grad()
def eval_loss(model, loader):
    model.eval(); tot, n = 0.0, 0.0
    for x, a, m in loader:
        x, a, m = x.to(DEVICE), a.to(DEVICE), m.to(DEVICE)
        k = m[:, 1:].sum().item()
        tot += batch_loss(model, x, a, m).item() * k; n += k
    model.train()
    return tot / max(n, 1)

def gpu_mem_gb():
    return torch.cuda.max_memory_allocated() / 1e9 if torch.cuda.is_available() else 0.0

def free_memory():
    gc.collect(); torch.cuda.empty_cache(); torch.cuda.reset_peak_memory_stats()

def train_model(model, lr, tag, patience=1):
    seed_everything(); model.to(DEVICE)
    params = [p for p in model.parameters() if p.requires_grad]
    opt = torch.optim.AdamW(params, lr=lr)
    total = math.ceil(len(train_loader) / GRAD_ACCUM) * EPOCHS
    sched = get_cosine_schedule_with_warmup(opt, int(WARMUP_RATIO * total), total)
    hist, best_val, best_state, bad = {"train": [], "val": []}, float("inf"), None, 0
    t0 = time.time(); model.train(); opt.zero_grad()
    for ep in range(1, EPOCHS + 1):
        run = 0.0
        for i, (x, a, m) in enumerate(train_loader):
            x, a, m = x.to(DEVICE), a.to(DEVICE), m.to(DEVICE)
            loss = batch_loss(model, x, a, m) / GRAD_ACCUM
            loss.backward()
            if (i + 1) % GRAD_ACCUM == 0 or i + 1 == len(train_loader):
                torch.nn.utils.clip_grad_norm_(params, MAX_GRAD_NORM)
                opt.step(); sched.step(); opt.zero_grad()
            run += loss.item() * GRAD_ACCUM
        tr, va = run / len(train_loader), eval_loss(model, val_loader)
        hist["train"].append(tr); hist["val"].append(va)
        print(f"[{tag}] epoch {ep}: train {tr:.4f}  val {va:.4f}")
        if va < best_val:
            best_val, bad = va, 0
            best_state = {n: p.detach().cpu().clone()          # trainable tensors only
                          for n, p in model.named_parameters() if p.requires_grad}
        else:
            bad += 1
            if bad > patience: break                             # early stopping
    model.load_state_dict(best_state, strict=False)              # restore best checkpoint
    return model, hist, time.time() - t0, gpu_mem_gb()

def count_trainable(model):
    t = sum(p.numel() for p in model.parameters() if p.requires_grad)
    n = sum(p.numel() for p in model.parameters())
    return t, n, 100 * t / n

Every standard mechanism appears here: gradient accumulation (divide loss, step every GRAD_ACCUM micro-batches), linear warmup then cosine decay, gradient-norm clipping at 1.0, per-epoch validation, keeping and restoring the best checkpoint, and early stopping. Storing only trainable tensors keeps checkpoints tiny for LoRA; for full fine-tuning it is the whole model.

Step 5: run 1, full instruction fine-tuning

free_memory()
full = AutoModelForCausalLM.from_pretrained(MODEL_ID, torch_dtype=torch.float32)
print("trainable %:", round(count_trainable(full)[2], 2))       # 100.0
full, full_hist, full_time, full_mem = train_model(full, LR_FULL, "full-IFT")

The model is loaded in fp32 because pure fp16 full fine-tuning overflows on this model family and the T4 has no fast bf16; the tiny learning rate (1e-5) prevents the update from overwriting pretrained features. Memory: 0.49B parameters × 16 bytes ≈ 7.9 GB for weights, gradients and Adam states, plus activations, which makes full fine-tuning the most memory-hungry run even at this small scale.

Step 6: runs 2–3, LoRA at r = 8 and r = 16

def lora_cfg(r):
    return LoraConfig(task_type=TaskType.CAUSAL_LM, r=r, lora_alpha=2 * r,   # alpha/r fixed at 2
                      lora_dropout=LORA_DROPOUT, bias="none", target_modules=TARGETS)

lora_runs = {}
for r in (8, 16):
    free_memory(); seed_everything()
    base = AutoModelForCausalLM.from_pretrained(MODEL_ID, torch_dtype=torch.float32)
    model = get_peft_model(base, lora_cfg(r))
    pct = count_trainable(model)[2]                  # about 0.9% at r=8, 1.75% at r=16
    model, hist, t, mem = train_model(model, LR_LORA, f"LoRA r={r}")
    lora_runs[r] = dict(model=model, hist=hist, time=t, mem=mem, pct=pct)

Keeping α/r fixed isolates the effect of rank. The parameter percentage is higher than the "well under 1%" often quoted for 7B models because small models have narrow layers: for this architecture (hidden size 896, MLP width 4864, 24 layers, grouped-query attention with narrow key/value projections) rank 16 on all seven linear layers adds about 8.8M parameters to about 494M.

Step 7: runs 4–5, QLoRA at r = 8 and r = 16

bnb_cfg = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
                             bnb_4bit_use_double_quant=True,
                             bnb_4bit_compute_dtype=torch.float16)   # T4: fp16 compute

qlora_runs = {}
for r in (8, 16):
    free_memory(); seed_everything()
    base = AutoModelForCausalLM.from_pretrained(MODEL_ID, quantization_config=bnb_cfg,
                                                torch_dtype=torch.float16)
    base = prepare_model_for_kbit_training(base)     # fp32 norms, checkpointing, input grads
    model = get_peft_model(base, lora_cfg(r))        # identical adapters to the LoRA runs
    pct = count_trainable(model)[2]
    model, hist, t, mem = train_model(model, LR_LORA, f"QLoRA r={r}")
    qlora_runs[r] = dict(model=model, hist=hist, time=t, mem=mem, pct=pct)

The only change from the LoRA runs is the backbone's precision, which makes this a clean controlled comparison. Trainable parameters match LoRA at the same rank exactly. Two small-model subtleties: first, the embedding matrix (vocabulary of about 152k × 896 ≈ 136M parameters) is not quantized, so on a 0.5B model the 4-bit saving is proportionally smaller than on 7B; second, gradient checkpointing (enabled by prepare_model_for_kbit_training) cuts activation memory but adds recomputation time, so the QLoRA runs are typically the slowest per epoch while using the least memory.

Step 8: generation-based evaluation

import evaluate
rouge, bertscore = evaluate.load("rouge"), evaluate.load("bertscore")

@torch.no_grad()
def generate_answers(model, frame, max_new_tokens=200):
    model.eval(); preds, refs = [], []
    for _, row in frame.iterrows():
        enc = tokenizer(build_prompt(row["instruction"], row["input"]), return_tensors="pt",
                        add_special_tokens=False).to(DEVICE)
        out = model.generate(**enc, max_new_tokens=max_new_tokens, do_sample=False,
                             pad_token_id=tokenizer.pad_token_id)
        preds.append(tokenizer.decode(out[0, enc["input_ids"].shape[1]:],     # new tokens only
                                      skip_special_tokens=True).strip())
        refs.append(row["output"].strip())
    model.train()
    return preds, refs

def score(preds, refs):
    rl = rouge.compute(predictions=preds, references=refs, use_stemmer=True)["rougeL"]
    bs = np.mean(bertscore.compute(predictions=preds, references=refs, lang="en")["f1"])
    return {"rougeL": round(float(rl), 4), "bertscore_f1": round(float(bs), 4)}

results = {}
runs = {"full_ift": dict(model=full, time=full_time, mem=full_mem, pct=100.0),
        **{f"lora_r{r}": v for r, v in lora_runs.items()},
        **{f"qlora_r{r}": v for r, v in qlora_runs.items()}}
for name, run in runs.items():
    p, refs = generate_answers(run["model"], test_df)
    results[name] = {**score(p, refs), "pct_trainable": round(run["pct"], 2),
                     "peak_mem_gb": round(run["mem"], 2), "train_time_s": round(run["time"], 1)}
print(pd.DataFrame(results).T)

Step 9: error analysis

best = max(results, key=lambda k: results[k]["rougeL"])
preds, refs = generate_answers(runs[best]["model"], test_df)
pl = np.array([len(tokenizer(p, add_special_tokens=False)["input_ids"]) for p in preds])
rl = np.array([len(tokenizer(r, add_special_tokens=False)["input_ids"]) for r in refs])
ratio = pl / np.maximum(rl, 1)
print("median length ratio:", np.median(ratio).round(2),
      "| empty:", int((pl == 0).sum()),
      "| much shorter (<0.5):", int((ratio < 0.5).sum()),
      "| much longer (>2):", int((ratio > 2).sum()))
for p, r in list(zip(preds, refs))[:3]:
    print("REF:", r[:300], "\nGEN:", p[:300], "\n" + "-" * 60)

What results to expect and how to read them

With only 300 examples and 3 epochs, differences between methods are usually small and can flip with the random seed, which is itself the key lesson: on a free GPU, LoRA or QLoRA gets essentially the same answer quality as full fine-tuning for a fraction of memory. Other typical observations: validation loss bottoms out after one or two epochs for full fine-tuning; lower training loss does not reliably predict better generated answers; the dominant failure modes are answers that are shorter or less specific than the references, missing safety caveats, and truncation when max_new_tokens is small. Reasonable next experiments are more data, a longer MAX_LEN, a larger model with QLoRA, or a light repetition penalty at decoding.

Step 10: save, merge and export

best_run = runs["qlora_r16"]["model"]
best_run.save_pretrained("out/qlora-r16-adapter")          # adapter only: a few tens of MB
tokenizer.save_pretrained("out/qlora-r16-adapter")

# merge into a 16-bit base (never into the 4-bit weights), then export or quantize
from peft import PeftModel
base16 = AutoModelForCausalLM.from_pretrained(MODEL_ID, torch_dtype=torch.float16)
merged = PeftModel.from_pretrained(base16, "out/qlora-r16-adapter").merge_and_unload()
merged.save_pretrained("out/merged-fp16"); tokenizer.save_pretrained("out/merged-fp16")
# next: convert to GGUF and quantize for local/edge use (see the serving section)

Quick revision

• Fine-tuning continues gradient training of a pretrained model on your data; inference and prompting never change weights.
• Rule of thumb: new knowledge calls for RAG, new behaviour calls for fine-tuning, and many systems use both.
• Always baseline with the best prompt you can write and build an evaluation set before fine-tuning.
• Continued pretraining uses raw domain text; SFT uses prompt–response pairs; preference tuning uses chosen/rejected pairs or rewards.
• SFT is next-token cross-entropy with teacher forcing, usually on response tokens only (prompt labels set to -100).
• Logits at position t predict token t+1, so shift logits and labels (and the mask) by one.
• Use the model's own chat template via apply_chat_template; mismatched templates degrade quality.
• Append the end-of-turn/EOS token to every target, or the model never learns to stop.
• Tokenize prompt + answer once and use offsets to locate the answer boundary.
• Quality beats quantity: a thousand clean, diverse examples can beat a hundred thousand noisy ones.
• Deduplicate, scrub personal data, decontaminate against test sets, and split by group to avoid leakage.
• Set maximum sequence length near the 95th percentile of tokenized length plus template overhead.
• Packing concatenates short examples to cut padding waste; block cross-example attention when possible.
• Full fine-tuning with mixed-precision AdamW needs about 16 bytes per parameter before activations: 112 GB for 7B.
• Activations scale with batch × sequence × hidden × layers; gradient checkpointing trades roughly a third more compute for large savings.
• Large vocabularies make the logits tensor a hidden memory hog; chunked or fused cross-entropy helps.
• PEFT freezes the base and trains a small set of parameters: adapters, prefix/prompt tuning, IA3, BitFit, LoRA.
• LoRA: h = W0x + (α/r)BAx, with A random and B zero so training starts from the pretrained model.
• LoRA adds r·(d_in + d_out) parameters per adapted matrix; rank 16 on all linear layers of a 7B model is about 40M parameters.
• Targeting all linear layers usually matters more than increasing rank.
• Keep α/r fixed when sweeping rank; rsLoRA uses α/√r for stability at high rank.
• Merged LoRA adds zero inference latency; unmerged adapters add a small overhead but can be swapped.
• DoRA separates magnitude and direction, training LoRA on the direction, and often helps at low rank.
• QLoRA = NF4 4-bit frozen base + double quantization + paged optimizers + 16-bit LoRA adapters; compute is in bf16/fp16.
• NF4 places 16 levels at normal-distribution quantiles with blockwise (64) absmax scaling.
• Double quantization cuts scale-constant overhead from 0.5 to about 0.127 bits per parameter.
• prepare_model_for_kbit_training upcasts norms, enables checkpointing and input gradients for k-bit training.
• Typical LRs: full FT about 1e-5, LoRA about 2e-4, DPO about 5e-7 to 5e-6 (full) with β about 0.1.
• 1–3 epochs, warmup 3–10%, cosine or linear decay, clipping at 1.0, and keep the best validation checkpoint.
• RLHF = SFT, then a Bradley–Terry reward model from human comparisons, then PPO maximizing reward minus β·KL to the reference.
• PPO-based RLHF keeps four models in memory: policy, reference, reward and value. The clip applies to ρ then multiplies by A; the objective is maximized.
• DPO loss: −log σ(β [log(π_θ(y_w)/π_ref(y_w)) − log(π_θ(y_l)/π_ref(y_l))]); no reward model or sampling; smaller β allows more deviation from the reference.
• ORPO and SimPO are reference-free; KTO uses unpaired thumbs-up/down data; IPO regularizes DPO's margins.
• GRPO keeps PPO's clip but replaces the critic with group-standardized rewards over G samples; two models in memory instead of four; stalls when a group has zero reward variance.
• RLAIF and Constitutional AI replace human preference labels with AI judgements guided by written principles.
• Reward hacking, sycophancy and verbosity are classic preference-tuning failure modes; the KL penalty limits them.
• Evaluate on held-out task data, with a validated LLM judge, on general benchmarks, safety suites and a regression suite, all against the base model.
• Catastrophic forgetting is mitigated by PEFT, data replay, lower LR, fewer epochs, regularization and model merging.
• Even benign fine-tuning can weaken safety refusals, so re-run safety evaluations on every release.
• Merge adapters into a 16-bit base, never directly into 4-bit weights, then re-quantize if needed.
• Model merging (linear, SLERP, task arithmetic, TIES, DARE) only works for models sharing a base.
• Distillation trains a smaller student on teacher outputs or soft logits with temperature; check the teacher's terms of use.
• Serve one merged model for high-traffic single tasks, or multi-LoRA over a shared base for many variants.
• For local/edge deployment: merge, convert to GGUF, quantize (for example Q4_K_M), and re-evaluate with the right template.

Glossary

Adapter

A small trainable module inserted into a frozen network; in the narrow sense, a bottleneck MLP added after attention or feed-forward blocks.

AdamW

Adam optimizer with decoupled weight decay; stores two moment estimates per trainable parameter.

Alignment

Post-training that shapes a model's behaviour towards human intentions and values, such as helpfulness, honesty and harmlessness.

Alpha (LoRA)

Scaling hyperparameter; the LoRA update is multiplied by α/r.

Base model

A pretrained model before instruction tuning or alignment; it continues text rather than following instructions.

bitsandbytes

Library of 8-bit and 4-bit quantized layers and memory-efficient optimizers used by QLoRA.

Bradley–Terry model

Probability model for pairwise preferences: P(A preferred over B) = σ(r_A − r_B).

Catastrophic forgetting

Loss of previously learned abilities when a network is trained on new data.

Chat template

The exact special-token layout a chat model uses to mark system, user, assistant and tool turns.

Completion-only loss

Computing the training loss only on response tokens, masking prompt tokens.

Constitutional AI

Alignment approach in which a model critiques and revises outputs, and an AI judge ranks them, according to a written set of principles.

Continued pretraining

Further next-token training on large unlabelled domain text; also called domain-adaptive pretraining.

DARE

Model-merging pre-processing that randomly drops most delta parameters and rescales the remainder.

DeepSpeed ZeRO

Memory optimization that shards optimizer states (stage 1), gradients (stage 2) and parameters (stage 3) across GPUs.

Distillation

Training a smaller student model to imitate a larger teacher's outputs or probability distributions.

DoRA

Weight-decomposed LoRA: learns a magnitude vector and applies LoRA to the normalized direction of each weight.

Double quantization

Quantizing the per-block quantization constants themselves to save memory.

DPO

Direct Preference Optimization: aligns a model with preference pairs using a closed-form loss on policy-to-reference log-probability ratios.

Early stopping

Halting training when validation performance stops improving, keeping the best checkpoint.

Effective batch size

Micro-batch size × gradient-accumulation steps × number of data-parallel GPUs.

FSDP

Fully Sharded Data Parallel: PyTorch's native sharding of parameters, gradients and optimizer states.

GGUF

Single-file model format with metadata and many quantization types, used by llama.cpp-family runtimes.

Gradient accumulation

Summing gradients over several micro-batches before one optimizer step to simulate a larger batch.

Gradient checkpointing

Saving only some activations and recomputing the rest during backward to save memory.

GRPO

Group Relative Policy Optimization: PPO-style RL without a value model, using rewards standardized within a group of samples per prompt.

IA3

PEFT method that learns vectors rescaling keys, values and feed-forward activations.

Instruction tuning

Supervised fine-tuning on diverse instruction–response pairs so a model follows natural-language commands.

KL penalty

Term penalizing divergence of the trained policy from a reference model during preference optimization.

KTO

Preference method learning from unpaired examples labelled desirable or undesirable.

LLM-as-judge

Using a strong language model with a rubric to rate or compare outputs.

LoRA

Low-Rank Adaptation: trains a low-rank update BA added to frozen weight matrices.

Loss masking

Excluding tokens from the loss, typically by setting their labels to -100.

Merge and unload

Folding a LoRA update into the base weights and removing the adapter modules.

Mixed precision

Computing in 16-bit while keeping an fp32 master copy of weights for the optimizer.

Model merging

Combining weights of several fine-tuned models that share a base, without further training.

Multi-LoRA serving

Serving many LoRA adapters over one shared base model, batching requests for different adapters together.

NF4

4-bit NormalFloat: a 16-level code at normal-distribution quantiles, used for storing QLoRA base weights.

ORPO

Odds Ratio Preference Optimization: single-stage SFT plus odds-ratio preference loss, with no reference model.

Packing

Concatenating multiple short examples into one training sequence to reduce padding.

Paged optimizer

Optimizer whose states live in unified memory and page to CPU RAM during GPU memory spikes.

PEFT

Parameter-Efficient Fine-Tuning: methods that train a small fraction of parameters; also the name of the Hugging Face library.

Perplexity

Exponential of the average per-token cross-entropy; lower means the model finds the text less surprising.

PPO

Proximal Policy Optimization: RL algorithm using a clipped probability-ratio objective and a value model.

Prefix tuning

PEFT method that learns key/value vectors prepended at every attention layer.

Prompt tuning

PEFT method that learns a few soft embedding vectors prepended to the input.

QLoRA

LoRA fine-tuning on a base model stored in 4-bit NF4, with double quantization and paged optimizers.

Rank (LoRA)

Inner dimension r of the LoRA matrices; bounds the rank of the update and sets adapter size.

Reference model

Frozen copy of the starting policy used for the KL penalty or DPO log-ratios.

Replay

Mixing general or earlier-task data into fine-tuning to reduce forgetting.

Reward hacking

A policy exploiting flaws in a reward signal to score highly without truly improving.

Reward model

Model trained on human (or AI) comparisons to output a scalar score for a prompt–response pair.

RLAIF

Reinforcement Learning from AI Feedback: preference labels come from an AI judge instead of humans.

RLHF

Reinforcement Learning from Human Feedback: SFT, reward model from human preferences, then RL against that reward.

RLVR

RL with verifiable rewards: rewards from automatic checks such as correct final answers or passing tests.

rsLoRA

Rank-stabilized LoRA using scale α/√r.

SFT

Supervised Fine-Tuning: training on input–target demonstrations with next-token cross-entropy.

SFTTrainer

TRL's trainer for supervised fine-tuning, with chat formatting, packing and PEFT integration.

SimPO

Reference-free preference method using length-normalized log-probability as the implicit reward plus a margin.

Sycophancy

Tendency of a model to agree with or flatter the user rather than be accurate.

Synthetic data

Training examples generated or rewritten by a model rather than written by people.

Target modules

The layers (for example q_proj, v_proj, gate_proj) that receive LoRA adapters.

Task vector

Difference between fine-tuned and base weights, used for task arithmetic.

Teacher forcing

Training by feeding the true previous tokens rather than the model's own predictions.

TIES merging

Merging method that trims small deltas, elects a sign per parameter and averages agreeing deltas.

TRL

Hugging Face library of trainers for SFT, reward modelling, DPO, PPO, GRPO and related methods.

Warmup

Gradually increasing the learning rate from near zero over the first steps of training.

Interview questions