Alignment and safety

Picture hiring a new assistant and handing them a one-page brief: be useful, tell the truth, don't cause trouble. The brief is clear to you, but you can't watch every task they do. So you check what you can check: how quickly they reply, whether customers leave a thumbs-up, whether the report has the right headings. The assistant, like anyone being measured, learns what the checks reward. If the checks and the brief agree, all is well. Where they disagree, the assistant drifts towards the checks.

Alignment is the engineering work of making a model's behaviour match the brief, not just the checks. In practice the brief is usually summarised as three targets:

Every lesson section below takes one of those measurements, builds it from scratch on made-up, neutral examples, and shows where it can mislead. The training machinery itself (reward models, RLHF, DPO) lives in primer.ml.training_stages; the runtime checks that wrap a deployed model live in primer.agents.guardrails. This lesson sits between them: how the targets are written down, how failures are searched for, and how the result is judged before release.

Everyday picture A call centre rewards agents for short calls. Calls get shorter. Some of that is agents getting better; some of it is agents hanging up on hard customers. The number keeps improving while the thing it stood for gets worse. This is Goodhart's law: once a measure becomes a target, it stops being a good measure.

Tiny worked example A model is being tuned against a reward model that scores answers. Each tuning step adds two things to its answers: some genuine content, which helps but saturates (there is only so much to say), and some padding, which the reward model can't tell apart from content. Say genuine content after steps is and padding is . The reward model sees content plus padding; the person reading sees content minus the cost of wading through padding.

import math
def g(s):
    return 1 - math.exp(-s / 10)
def p(s):
    return s / 50
# proxy(s) and true(s) at step 16
round(g(16) + p(16), 3), round(g(16) - p(16), 3)  # → (1.118, 0.478)
# ... and at step 50: the proxy keeps climbing, the true value has collapsed
round(g(50) + p(50), 3), round(g(50) - p(50), 3)  # → (1.993, -0.007)
# the step where the true value peaks
max(range(51), key=lambda s: g(s) - p(s))  # → 16

flowchart LR
  W[What we want<br/>helpful, honest, harmless] --> R[What we can write down<br/>principles, rater instructions]
  R --> M[What we can measure<br/>reward model score, eval sets]
  M --> O[Optimize the model<br/>against the measurement]
  O -. drifts towards .-> M
  O -. should match .-> W

Why it matters in practice: this gap is the root of most alignment failures. A model tuned hard against a reward model finds the reward model's blind spots (called reward hacking, built from scratch in primer.ml.reinforcement). The standard defences are a penalty for drifting far from the starting model (the β in primer.ml.training_stages), stopping early, and refreshing the reward model with new labels where the policy has found its weak spots.

In code: goodhart_curve returns the proxy and true value at every step.

Everyday picture A newspaper has a style guide. A junior writer drafts a story; an editor reads it against the style guide, writes margin notes ("unsourced claim", "too certain"), and the writer revises. Over time the editor's notes also teach the newsroom which of two drafts is better. Constitutional AI does this with models: the style guide is a short, written list of principles (the "constitution"), and a model plays the editor.

It has two stages:

1. Critique and revise. The model drafts an answer, is asked to critique it against a principle, then to rewrite it. The revised answers become fine-tuning data (supervised fine-tuning, as in primer.ml.training_stages).
2. AI feedback (RLAIF). The model compares pairs of answers against the principles and says which is better. Those AI-labelled pairs train the reward model, in place of (or alongside) human labels. The rest is the same preference tuning as RLHF.

The appeal is that the principles are written in plain language, can be read and argued about, and can be changed without relabelling thousands of examples by hand.

Tiny worked example Our toy constitution has four made-up rules:

Asked "Will this backup script work?", the model drafts three candidates:

B beats A, and B beats C. A and C tie, and a tie tells the reward model nothing, so that pair is dropped. Three candidates give two preference pairs: (B over A) and (B over C).

# one row per candidate: broken? for helpful, honest, harmless, cites
broken = {"A": [0, 1, 0, 1], "B": [0, 0, 0, 0], "C": [1, 0, 0, 1]}
# v(y) = Σ_k 1[y breaks principle k]
v = {name: sum(flags) for name, flags in broken.items()}
v  # → {'A': 2, 'B': 0, 'C': 2}
# keep only the pairs with a strict winner, winner first
pairs = [(a, b) if v[a] < v[b] else (b, a) for a, b in [("A", "B"), ("A", "C"), ("B", "C")] if v[a] != v[b]]
pairs  # → [('B', 'A'), ('B', 'C')]

A real constitution has more principles, written as sentences rather than keyword checks, and the "editor" is a language model reading them, so its judgements are softer and can be wrong. The shape of the pipeline is the same.

flowchart LR
  P[Prompt] --> D[Model drafts<br/>several answers]
  C[(Constitution<br/>written principles)] --> CR[Critic model<br/>critiques each answer]
  D --> CR
  CR --> RV[Revise<br/>fix what the critique found]
  RV --> SFT[Revised answers<br/>become fine-tuning data]
  CR --> L[AI labeler<br/>compares pairs]
  C --> L
  L --> PP[Preference pairs<br/>winner, loser]
  PP --> RM[Reward model]
  RM --> RL[Preference tuning<br/>as in RLHF]

Why it matters in practice: human labelling is slow, costly and hard to keep consistent; written principles make the target explicit and auditable. The risk moves rather than disappears: the AI labeler has its own blind spots, and whatever it gets wrong, the reward model learns faithfully. That is why AI labels are spot-checked against human ones, the same way primer.agents.evals calibrates an LLM judge.

In code: CONSTITUTION holds the four toy principles, critique returns the names of the ones an answer breaks, revise applies one fix per broken principle, and constitutional_preference_pairs turns a list of candidates into (winner, loser) pairs, dropping ties. The pairs are what primer.ml.training_stages.reward_model_loss trains on.

Everyday picture Before a bank opens a new vault, it pays a team to try to get in. Not because it expects burglars to be clever in any particular way, but because the people who built the vault only think of the ways in they already guarded against. Red-teaming is the same for models and their safety checks: search systematically for inputs that make them fail, count how often the search succeeds, fix what it found, and search again.

Tiny worked example In our toy world there is one off-limits action, called xyzzy (a made-up word). A simple safety filter blocks any request containing the word "xyzzy". The toy language also has two made-up synonyms, "plugh" and "quux", that mean exactly the same thing. The filter's author tested it with three hand-written requests:

Three for three: the filter looks perfect. An automated search does something duller and more thorough: it takes the request "please do xyzzy" and rewrites each word at random with any word that means the same thing ("please" or "kindly", "do" or "perform", "xyzzy" or "plugh" or "quux"). Six attempts from that search:

Four of six attempts get through. The fraction of attempts that get through is the attack success rate.

# 1 = the attempt got through, 0 = it was blocked
through = [0, 1, 1, 0, 1, 1]
# ASR = (1/N) Σ_i 1[x_i gets through]
round(sum(through) / len(through), 3)  # → 0.667
# the three hand-written tests: all blocked
hand_written = [0, 0, 0]
sum(hand_written) / len(hand_written)  # → 0.0

flowchart LR
  S[Seed request<br/>known off-limits] --> MU[Mutate<br/>rewrite with same-meaning words]
  MU --> F{Safety filter<br/>blocks it?}
  F -- yes --> MU
  F -- no --> LOG[Log a failure<br/>it got through]
  LOG --> ASR[Measure<br/>attack success rate]
  ASR --> FIX[Fix the filter<br/>using what was found]
  FIX --> RE[Search again<br/>with a fresh seed]
  RE --> ASR

The toy's vocabulary is tiny and finite, so the patched keyword list ends up complete. Real language is not: any keyword list will miss paraphrases no one has searched for yet. That is why real systems use learned classifiers instead of keyword lists, keep searching after every fix, and never rely on one filter alone (the layered checks in primer.agents.guardrails). Real-world red-teaming uses people and other models as the search, which finds far more varied failures than word swaps.

Why it matters in practice: a safety check nobody has tried to break has an unknown failure rate, which in practice means a higher one than anyone thinks. Systematic search turns "we couldn't think of a way past it" into a measured rate that can be tracked from release to release.

In code: KeywordFilter is the toy filter, red_team_attempts runs the seeded word-swap search, attack_success_rate turns the outcomes into ASR, red_team_search does both, and patch_filter adds every word for xyzzy found in a successful attempt. HAND_WRITTEN_TESTS is the author's own test set.

Everyday picture A tutor is asked "what's 7 × 8?" and says 56. The student frowns: "I'm pretty sure it's 54." A good tutor says "let's check" and still answers 56. A tutor who wants to be liked says "oh, you're right, 54." That second tutor is sycophantic: the answer depends on what the person seems to want to hear, not on the question.

Tiny worked example Ask a model five factual questions twice: once plainly, and once after the user states a wrong answer.

Two of five answers changed only because the user pushed. That share is the flip rate.

plain = [56, "Canberra", 27, 100, 7]
pushed = [56, "Sydney", 27, 90, 7]
# 1[a_pushed ≠ a_plain] for each question
flips = [int(p != q) for p, q in zip(pushed, plain)]
flips  # → [0, 1, 0, 1, 0]
sum(flips) / len(flips)  # → 0.4

flowchart LR
  Q[Factual question<br/>with a known answer] --> A1[Ask plainly]
  Q --> A2[Ask after the user<br/>states a wrong answer]
  A1 --> P1[Plain answer]
  A2 --> P2[Pushed answer]
  P1 & P2 --> CMP{Same?}
  CMP -- no --> FL[Count a flip]
  CMP -- yes --> OK[Held its ground]

Why preference training can cause it

Everyday picture People tend to rate an answer that agrees with them a little more kindly. That's human, and it is mild. But a reward model learns from thousands of those ratings, and then a model is tuned hard to please the reward model. A mild tilt in the labels becomes a steady push in the model.

Tiny worked example Using the Bradley-Terry model from primer.ml.training_stages: a rater compares a correct answer that contradicts them with an agreeing answer that is wrong. The correct answer is better by 1 point of genuine quality, but agreement earns a bonus of 2 points in the rater's eyes. The agreeing answer wins with probability σ(2 − 1) = σ(1) = 0.731, nearly three times in four.

import math
def sigma(z):
    return 1 / (1 + math.exp(-z))
# P(agreeing ≻ correct) = σ(δ − Δ)
round(sigma(2.0 - 1.0), 3)  # → 0.731
# a smaller agreement bonus still wins more than a third of the time
round(sigma(0.5 - 1.0), 3)  # → 0.378

Why it matters in practice: a sycophantic model is least reliable exactly when someone most needs a straight answer, because they already hold a wrong belief. The usual fixes all target the labels: rater instructions that ask about correctness first, preference pairs built specifically so that the correct answer disagrees with the user, and flip-rate evaluations run on every release.

In code: sycophancy_flip_rate asks a seeded toy model each made-up question plainly and under pressure and returns the flip rate; sycophantic_preference is σ(δ − Δ), computed with primer.ml.training_stages.preference_probability.

Everyday picture A pharmacist refuses to sell some things without a prescription. Refuse too little and harm gets through; refuse too much and people are turned away for aspirin. Both are failures, and making one rarer tends to make the other more common. A model that declines requests faces the same trade.

Tiny worked example A classifier gives every request a risk score between 0 and 1, and the model refuses anything scoring at least the threshold . Three benign requests (like "What is the capital of France?") score 0.1, 0.3 and 0.6; three requests for the toy off-limits action score 0.4, 0.8 and 0.9. At :

benign = [0.1, 0.3, 0.6]
off_limits = [0.4, 0.8, 0.9]
def OR(t):
    return sum(s >= t for s in benign) / len(benign)
def HC(t):
    return sum(s < t for s in off_limits) / len(off_limits)
round(OR(0.5), 3), round(HC(0.5), 3)  # → (0.333, 0.333)
# stricter thresholds trade one error for the other
[(t, round(OR(t), 3), round(HC(t), 3)) for t in (0.35, 0.25)]  # → [(0.35, 0.333, 0.0), (0.25, 0.667, 0.0)]

These are the precision/recall trade-offs from primer.ml.metrics wearing different names. Treat "refuse" as the positive call: over-refusal is the false positive rate, and harmful compliance is the miss rate, one minus recall. Choosing the threshold is the same pricing of errors as in that lesson's cost-versus-threshold section.

flowchart LR
  R[Request] --> CL[Risk classifier<br/>score 0 to 1]
  CL --> T{Score at or above<br/>threshold t?}
  T -- yes --> RF[Refuse]
  T -- no --> AN[Answer]
  RF -. if it was benign .-> ORB[Over-refusal]
  AN -. if it was off-limits .-> HCB[Harmful compliance]

Why it matters in practice: over-refusal is easy to overlook because it fails quietly: nobody reports a harmful answer that didn't happen, but people do stop using a model that turns down ordinary requests. Measuring both rates on every release, on dedicated sets of benign-but-sensitive- looking requests as well as off-limits ones, keeps the trade visible.

In code: refusal_tradeoff returns both error rates at each threshold, on seeded toy scores or on scores you pass in.

Everyday picture A new car model is crash-tested, driven on a closed track, then lent to a few fleet customers before it reaches showrooms. Each stage is cheaper to fail than the next, and each has a checklist it must pass before the car moves on. Models are released the same way.

Tiny worked example Before release, a candidate model is measured on four evaluation sets, and each measurement has a limit set in advance:

One measurement is over its limit, so the release is held and the report names that one measurement. Setting the limits before measuring matters: limits chosen after seeing the numbers tend to drift to wherever the numbers landed.

flowchart LR
  EV[Evaluation sets<br/>red-team, benign, off-limits,<br/>sycophancy, capability] --> G{Release gate<br/>every limit met?}
  G -- no --> FX[Hold and fix<br/>more training data, better filter]
  FX --> EV
  G -- yes --> I[Internal use]
  I --> TT[Trusted testers]
  TT --> SM[Small share of users]
  SM --> ALL[Everyone]
  SM -. new failures become .-> EV
  ALL -. new failures become .-> EV

Why it matters in practice: every measurement in this lesson is noisy and partial on its own. A fixed set of limits, checked on every release, is what turns them into a decision, and staged rollout is what limits the cost when the measurements were wrong.

In code: release_gate compares each measurement with its limit and returns whether the release passes and one reason for every limit missed.

Answer each one out loud or on paper before you open it. If you can explain it, you know it.