Regularization

Two students prepare for an exam. One learns the ideas. The other memorises last year's paper, answer by answer, and scores 100% on it in practice. On the real exam, with new questions, the first student does fine and the second falls apart. A model that memorises its training examples, noise and all, instead of learning the pattern behind them is overfitting. It looks brilliant on the data it has seen and fails on data it hasn't. Regularization is everything we do to push a model toward the first student.

Worked example: four training points (0, 0), (1, 1), (2, 0), (3, 1), a zig-zag that is really "about 0.5, plus noise".

The cubic is perfect on the training data and absurd one step beyond it. You can check the 8 by hand with a difference table: the values 0, 1, 0, 1 have differences 1, −1, 1, then −2, 2, then 4; a cubic keeps that last difference constant, so extending the table gives 6, then 7, then 8.

flowchart LR
  D["Training data =<br/>pattern + noise"] --> M{Model capacity}
  M -->|too little| U["Underfits:<br/>misses the pattern"]
  M -->|about right| G["Generalizes:<br/>learns the pattern"]
  M -->|too much, unchecked| O["Overfits:<br/>memorises the noise"]
  G --> N[Good on new data]
  U & O --> B[Bad on new data]

The two numbers that tell these apart are errors on two different sets of data:

# the true values at x = 0, 1, 2, 3
y = [0, 1, 0, 1]
# the flat line predicts 0.5 everywhere
y_hat = [0.5, 0.5, 0.5, 0.5]
n = len(y)
# (1/n) Σ (y_i - ŷ_i)²
sum((y_i - y_hat_i) ** 2 for y_i, y_hat_i in zip(y, y_hat)) / n  # → 0.25

In code: zigzag_fit_error and zigzag_predict fit a polynomial of any degree to the four zig-zag points and report its training error and its prediction; true_function and make_curve_data draw the noisy sine samples in the figure.

Why it matters training error alone always rewards memorisation. The gap between training and validation error is the single most useful diagnostic in machine learning.

Goldilocks tries three bowls of porridge: too cold, too hot, just right. Model capacity works the same way, and you find "just right" by measuring, not guessing: sweep the capacity and watch both errors.

Worked example: 12 noisy points from a sine wave, 300 fresh points for validation, polynomials of increasing degree.

In code: polynomial_errors fits by least squares (choosing the coefficients that minimise training MSE) and reports both errors.

Why it matters "both errors high" and "training low, validation high" need opposite fixes. Underfitting wants more capacity, more features or more training; overfitting wants more data, regularization or early stopping. Diagnosing which one you have is the point.

A baker checks the cake with a toothpick every few minutes and takes it out when it comes out clean. Leave it longer and it burns, however good it was a minute ago. Training a flexible model is similar: validation error falls while the model learns the pattern, then rises once it starts memorising. Early stopping watches validation error during training, stops once it has failed to improve for a few checks (the patience), and keeps the weights from the best check.

Worked example: validation losses per epoch 1.0, 0.8, 0.6, 0.55, 0.58, 0.65, 0.7 with patience 2. Epoch 3 is the best. Epochs 4 and 5 are both worse, so training stops at epoch 5 and the weights from epoch 3 are kept.

flowchart LR
  E[End of epoch] --> V[Measure validation loss]
  V --> Q{Better than best?}
  Q -->|yes| S[Save weights,<br/>reset counter]
  Q -->|no| C[Counter + 1]
  C --> P{Counter = patience?}
  P -->|no| E2[Next epoch]
  P -->|yes| R[Stop; restore<br/>best weights]
  S --> E2

# validation loss after epoch t = 0, 1, 2, ...
L_val = [1.0, 0.8, 0.6, 0.55, 0.58, 0.65, 0.7]
patience = 2
t_star = 0
for t, loss in enumerate(L_val):
    if loss < L_val[t_star]:
        # a new best epoch: remember it
        t_star = t
    if t - t_star >= patience:
        # patience used up: stop here
        break
t_star, t  # → (3, 5)

In code: early_stopping replays a run's validation losses and returns the best epoch and the epoch where training stops; train_flexible_model trains the over-sized network in the figure and records both losses.

Why it matters it's the cheapest regularizer there is: no change to the model, just a rule for when to stop. It's used almost everywhere a model is trained for multiple epochs, including fine-tuning language models.

Picture two archers. The first groups every arrow tightly, but a hand's width to the left of the bullseye: consistently wrong. That's bias. The second's arrows are centred on the bullseye on average but scattered all over the target: inconsistent. That's variance. A model trained on a different sample of data is like another volley of arrows. Simple models behave like the first archer; very flexible ones like the second.

Worked example: fit polynomials to 300 different random training sets of 30 points each, and measure (averaged over the input range):

sigma = 0.3
# σ²: the error no model can remove
noise = sigma ** 2
for degree, bias_sq, variance in [(1, 0.174, 0.022), (3, 0.0035, 0.015), (9, 0.0038, 4.5)]:
    # bias² + variance + σ²
    total = bias_sq + variance + noise
    # two significant figures
    print(degree, f"{total:.2g}")  # → 1 0.29 3 0.11 9 4.6

In code: bias_variance fits one polynomial to each of many random training sets and splits their error into bias², variance and noise.

Why it matters it names the two failure modes and explains why more data helps flexible models (averaging tames variance) but not rigid ones (more data doesn't fix bias). Very large neural networks complicate the picture ("double descent": past a point, even bigger models generalize better again), but the vocabulary is universal.

A coach who randomly sends half the team home before each practice forces every player to learn every position; no single star can carry the team. Dropout does this to neurons: during training, each activation is set to zero with probability p on every step, so the network can't rely on any single neuron or fragile combination of neurons.

Worked example: four activations (1, 1, 1, 1), drop rate p = 0.5. Suppose the random mask keeps the last two: (0, 0, 1, 1). The survivors are scaled by 1 / (1 − 0.5) = 2, giving (0, 0, 2, 2). The average is still 1, so the next layer sees the same size signal on average. At evaluation time, nothing is dropped and nothing is scaled.

flowchart LR
  subgraph Train["Training step"]
    a1((1)) --> k1((0))
    a2((1)) --> k2((0))
    a3((1)) --> k3((2))
    a4((1)) --> k4((2))
  end
  subgraph Eval["Evaluation"]
    b1((1)) --> e1((1))
    b2((1)) --> e2((1))
    b3((1)) --> e3((1))
    b4((1)) --> e4((1))
  end

import random
random.seed(0)
h = [1, 1, 1, 1]
p = 0.5
# m_i ~ Bernoulli(1 - p): a coin flip each
m = [1 if random.random() < 1 - p else 0 for _ in h]
m  # → [0, 0, 1, 1]
# (m ⊙ h) / (1 - p)
[m_i * h_i / (1 - p) for m_i, h_i in zip(m, h)]  # → [0.0, 0.0, 2.0, 2.0]

In code: dropout draws the mask and scales the survivors during training, and passes activations through unchanged at evaluation time.

Why it matters dropout was a key ingredient of the deep-learning revival and is still used in many models (the original transformer used p = 0.1). Forgetting to switch it off at evaluation (model.eval() in PyTorch) is a classic bug that makes predictions noisy.

Add a tax on the size of the weights to the loss, and the model must decide whether each weight earns its keep. L2 (ridge, "weight decay") taxes the square of each weight: big weights pay a lot, small ones almost nothing, so everything shrinks a little but nothing is eliminated. L1 (lasso) charges a flat rate per unit of size: small weights can't justify the fee, so they're wiped out entirely, to exactly zero.

Worked example: with independent (orthonormal) features, the penalized weights have a closed form. Start from the unpenalized weights (3, 0.5, −2) and use penalty strength 1:

import math
w = [3, 0.5, -2]
# λ ("lambda" is taken in Python)
lam = 1
# ‖w‖₂², ‖w‖₁
sum(w_i ** 2 for w_i in w), sum(abs(w_i) for w_i in w)  # → (13.25, 5.5)
# L2: shrink every weight
[w_i / (1 + lam) for w_i in w]  # → [1.5, 0.25, -1.0]
# L1: sign(w) max(|w| - λ, 0)
[math.copysign(max(abs(w_i) - lam, 0), w_i) for w_i in w]  # → [2.0, 0.0, -1.0]

In code: penalised_weights gives both closed forms from the table, and fit_sparse_problem fits the ten-feature problem in the figure (L1 by alternating gradient steps with soft_threshold).

Why it matters L2 (as weight decay; see AdamW in primer.ml.optimizers) is on by default when training transformers. L1 is the tool when you want a sparse, interpretable model that uses only a few features.

A student has practice questions (to learn from), a mock exam (to check progress and decide what to revise), and the real exam, sat once. Data is split the same way. Training data fits the weights. Validation data tunes choices like model size, learning rate and when to stop. Test data is touched once, at the end, to estimate real-world performance honestly. If you keep tuning against the test set, it quietly becomes a second validation set and stops being honest.

Worked example: 100 examples split 70 / 15 / 15, shuffled first, with no example in two sets. With only 10 examples, 5-fold cross-validation cuts them into 5 folds of 2; each fold takes a turn as the validation set while the model trains on the other 8, so every example is held out exactly once.

flowchart TB
  subgraph K["5-fold cross-validation"]
    f1["Run 1: [VAL] train train train train"]
    f2["Run 2: train [VAL] train train train"]
    f3["Run 3: train train [VAL] train train"]
    f4["Run 4: train train train [VAL] train"]
    f5["Run 5: train train train train [VAL]"]
  end
  K --> A[Average the 5 validation scores]

examples = list(range(10))
k = 5
# 5 folds of 2
folds = [examples[2 * i: 2 * i + 2] for i in range(k)]
# each model trains on the other 8
[len(examples) - len(fold) for fold in folds]  # → [8, 8, 8, 8, 8]
# score of the model trained without fold i, on fold i
scores = [1.0, 0.5, 1.0, 1.0, 0.5]
# (1/k) Σ_i score_i
sum(scores) / k  # → 0.8

In code: train_val_test_split shuffles and cuts the indices into three disjoint sets, and k_fold yields the training and validation indices for each fold in turn.

Why it matters with small datasets a single split is noisy, and cross-validation gives a more reliable estimate. With large datasets (or expensive models) one fixed validation set is enough.

A student who glimpsed the answer key scores perfectly on the practice exam and learns nothing. Data leakage is when information that won't be available at prediction time, or information from the test set, sneaks into training. Offline scores look superb; production scores collapse.

Worked examples:

• Duplicates across the split. Every row stored twice, labels 30% noisy (so no honest model can beat about 70%). Split before removing duplicates, and a model that just copies its nearest training row scores 87%: it's reading the answers off each test row's twin. Deduplicate first and the same model scores about 50%.
• A feature from the future. A fraud model is given chargeback_filed, a column filled in only after a customer disputes a charge, which is to say after the outcome. Offline accuracy: 98%. In production that column is always empty at decision time, and accuracy falls to 49%, worse than an honest model using only the legitimate features (61%).

flowchart LR
  subgraph T["Timeline of one transaction"]
    direction LR
    A[Transaction happens] --> P["Model must decide<br/>(prediction time)"]
    P --> O[Fraud confirmed?]
    O --> C[chargeback_filed recorded]
  end
  C -. "leaks backwards into<br/>the training table" .-> P

In code: duplicate_leakage_demo and future_feature_demo run both examples; the "model" in the first is a one-nearest-neighbour lookup (copy the label of the closest training row), the purest memoriser there is.

Why it matters leakage is the most common reason a model that looked great in evaluation fails in production. For language-model benchmarks the big version is contamination: test questions that appeared in the pretraining data. The defences are procedural: deduplicate before splitting, split by time or by user when the real task is predicting the future or new users, and ask of every feature "would I actually know this at prediction time?"

Answer each one out loud or on paper before you open it. If you can explain it, you know it.