Reasoning models

Ask someone to multiply 37 by 48 in their head, instantly, and they will probably guess. Hand them a pencil and a minute and they will get it right. They didn't get smarter in that minute. They got room: somewhere to write 37 × 8 = 296 and 37 × 40 = 1,480, so each small step could lean on the last.

A language model is in the same position. It can do a fixed amount of work for each token it writes: one pass through all of its layers. A reasoning model is a model that has learned to use the pencil. Before it answers, it writes out intermediate steps, checks them, backs up when one is wrong, and only then commits to an answer. This lesson builds each piece from scratch:

1. why writing steps helps at all (chain of thought);
2. why more thinking can buy more accuracy (test-time compute);
3. sampling several attempts and voting on the answer (self-consistency);
4. checking attempts, either the final answer or every step (verifiers);
5. how reinforcement learning teaches a model to think this way;
6. what thinking costs, and where it still fails.

Everyday picture Picture a clerk who, each time they glance at a page, can do exactly one addition and write down one number. Hand them a column of four numbers and allow one glance, and they have to guess. Allow three glances and a margin to write in, and they get it right every time. The margin is the whole trick: what they write on one glance is there to read on the next.

Tiny example Our toy model is that clerk. Every token it writes costs one forward pass (one run of the whole model to produce one token), and each forward pass can do one addition. Ask it for 3 + 5 + 8 + 2, which needs three additions.

The weights are identical in both columns. The only difference is that the second column wrote its running totals down, where the next pass could read them. Writing intermediate steps before the answer is called chain of thought.

flowchart LR
  subgraph D["Answer at once: 1 token"]
    direction LR
    q1["3+5+8+2 ="] --> p1["pass 1<br/>3+5 = 8<br/>no pass left"] --> a1["17, a guess"]
  end
  subgraph C["Think first: 3 tokens"]
    direction LR
    q2["3+5+8+2 ="] --> s1["pass 1<br/>writes 3+5=8"] --> s2["pass 2<br/>reads 8<br/>writes 8+8=16"] --> s3["pass 3<br/>reads 16<br/>answers 18"]
  end

The rule behind the table: the number of steps a model can do one after another grows with the number of tokens it writes.

# the toy: 1 addition per pass; 3 + 5 + 8 + 2 needs 3 additions
L = 1
# answering at once: T = 1, so S = 1, not enough
L * 1  # → 1
# the one pass adds 3 + 5, then estimates the unreached 8 and 2 at 4.5 each
(3 + 5) + round(4.5 * 2)  # → 17
# thinking first: T = 3, so S = 3, exactly enough
L * 3  # → 3
t1 = 3 + 5  # → 8
t2 = t1 + 8  # → 16
t2 + 2  # → 18
# a 32-layer model: answering at once, then after a 1,000-token chain of thought
32 * 1  # → 32
32 * 1000  # → 32000

A layer is not literally one addition, so is a rough count, but the shape of the rule holds: a model with a fixed depth can only do a fixed amount of one-after-another work per token, and problems that need more must spread it over more tokens. Li et al. (2024) proved a version of this for transformers: with enough chain-of-thought tokens they can solve inherently serial problems that a fixed-depth transformer answering at once cannot.

That price is real: each token costs roughly floating-point operations for a model with parameters (see primer.ml.inference), so a 1,000-token chain costs a thousand answers' worth of compute. In practice this is why simply asking a model to "think step by step" (Kojima et al., 2022) or showing it worked examples with steps (Wei et al., 2022) improved accuracy on arithmetic and logic puzzles, and why every reasoning model writes a long chain before its answer.

In code: solve is the toy model: it spends one forward pass per written step, one on the answer, and estimates anything it never reached. It returns an Attempt holding the steps, the answer and the tokens used.

Everyday picture An exam has questions of very different difficulty. Give yourself one minute per question and you finish only the easiest. Two minutes, and the next tier falls. Each doubling of time unlocks one more tier, until you can finish everything and extra time buys nothing.

Tiny example Test-time compute is computation spent while answering, as opposed to while training. The simplest dial is a thinking budget: the most tokens the model may write before it must answer. Take six problems that need 1, 2, 4, 8, 16 and 32 additions, and a budget of 8 tokens. Four of them fit (1, 2, 4, 8). The other two must be guessed, and say a guess is right 10% of the time. Accuracy is 4/6 + 2/6 × 0.1 = 0.70.

There are two ways to spend test-time compute, and the rest of this lesson uses both.

flowchart LR
  P[Problem] --> LONG["Think longer<br/>one chain, bigger budget"]
  P --> WIDE["Think wider<br/>n chains side by side"]
  LONG --> A1[Answer]
  WIDE --> PICK["Pick one:<br/>vote or verifier"] --> A2[Answer]

needs = [1, 2, 4, 8, 16, 32]
g = 0.1
B = 8
# F(B): the share of problems that fit
F = sum(d <= B for d in needs) / len(needs)
round(F, 3)  # → 0.667
# acc(B) = F(B) + (1 − F(B)) · g
round(F + (1 - F) * g, 3)  # → 0.7
# each doubling of B lets one more of the six levels fit
round((1 / len(needs)) * (1 - g), 3)  # → 0.15

The straight line on a log axis is built into this toy, because its difficulties are spaced by doubling. Real problem sets are also spread over many scales of difficulty, and published reasoning models show the same shape over a useful range: accuracy rising roughly in proportion to the logarithm of thinking tokens, then leveling off. Snell et al. (2024) found that spending test-time compute adaptively (more on harder prompts) beats spending it evenly, and that on problems a small model can sometimes solve, extra test-time compute can stand in for a much larger model. In practice, model APIs expose this dial as a "reasoning effort" setting or a maximum number of thinking tokens.

In code: budget_accuracy is the formula, and budget_sweep runs solve on random sums at each budget and reports accuracy and tokens actually spent.

Everyday picture Unsure of an answer, you ask five friends separately. If four of them say the same thing, you trust it. Each friend can be wrong, but it is unlikely that most of them are wrong in the same way, unless they all read the same wrong article.

Tiny example A model that samples its tokens (see primer.ml.inference) writes a different chain each time. Ask the toy's noisy cousin for 3 + 5 + 8 + 2 three times and it answers 18, 17, 18. Keep only the final answers and take the most common one: 18. Sampling several chains of thought and returning the most common final answer is self-consistency (Wang et al., 2022), also called majority voting.

flowchart LR
  Q["3+5+8+2 = ?"] --> S1["chain 1 ends in 18"] & S2["chain 2 ends in 17"] & S3["chain 3 ends in 18"]
  S1 & S2 & S3 --> V["count final answers<br/>18: two votes, 17: one"] --> A["answer 18"]

How much does voting help? Take the simplest case, a yes/no question: every wrong vote lands on the same wrong answer, and each vote is right with the same chance , independently of the others.

import math
p, n = 0.6, 3
# one term per winning count k = 2, 3
terms = [math.comb(n, k) * p**k * (1 - p)**(n - k) for k in range(math.ceil(n / 2), n + 1)]
[round(t, 3) for t in terms]  # → [0.432, 0.216]
round(sum(terms), 3)  # → 0.648
# five votes
round(sum(math.comb(5, k) * 0.6**k * 0.4**(5 - k) for k in range(3, 6)), 3)  # → 0.683
# a 40% solver on a yes/no question: voting makes it worse
round(sum(math.comb(5, k) * 0.4**k * 0.6**(5 - k) for k in range(3, 6)), 3)  # → 0.317

When voting fails: shared mistakes. Everything above assumed the votes were independent. Samples from one model are not: they share its training, its blind spots and its reading of the question. Model that as a shared draw: each sample copies one common answer with probability (rho), and otherwise answers on its own. Each single sample is still right 60% of the time, so only the correlation changes.

In practice, self-consistency gives a solid gain for a few samples and then flattens, because a model's mistakes on a question are correlated. Diversity helps (different prompts, different models, a tool that computes instead of guessing), and voting only works where answers can be compared exactly.

In code: majority_vote picks the most common answer, majority_accuracy is the formula (with ties on even counted as a coin flip), and correlated_vote_accuracy simulates votes with scattered wrong answers and a shared-mistake rate.

Everyday picture One teacher looks only at the boxed answer at the bottom of the page. Another marks every line of working. The first can't tell a lucky guess from understanding, and when the answer is wrong, can't tell you where you went wrong. The second can do both.

Tiny example A verifier is anything that scores a candidate solution. Here are two chains the toy wrote for 3 + 5 + 8 + 2 (true answer 18), each with slips:

An outcome verifier looks only at the final answer: either a learned outcome reward model (ORM) that predicts whether it is right, or a check against a known answer. It gives the second chain full marks, although two slips just happened to cancel. A process verifier, or process reward model (PRM), scores every step. It catches the second chain's slip and points at exactly where the first one went wrong.

flowchart LR
  subgraph O["Outcome verifier"]
    direction LR
    o1["3+5=8"] --> o2["8+8=17"] --> o3["17+2=19"] --> oc{"final 19<br/>equals 18?"} --> orr["reward 0<br/>but where did it go wrong?"]
  end
  subgraph P["Process verifier"]
    direction LR
    p1["3+5=8<br/>ok"] --> p2["8+8=17<br/>wrong"] --> p3["17+2=19<br/>ok"] --> pr["first bad step: 2"]
  end

Verifiers power best-of-n: sample chains, keep the one the verifier scores highest. With a perfect verifier, best-of-n is right whenever any of the samples is right, a number called pass@n.

p, n = 0.3, 5
# (1 − p)^n: every one of the five is wrong
round((1 - p) ** n, 5)  # → 0.16807
round(1 - (1 - p) ** n, 5)  # → 0.83193
# the experiment's chains: 5 additions, each clean with chance 0.8
round(0.8 ** 5, 3)  # → 0.328

In practice, the best verifiers are programs: unit tests for code, an exact match for a math answer, a proof checker. Where no program can check, a learned verifier stands in. Lightman et al. (2023) trained a process reward model on 800,000 human labels of individual steps and found it picked correct solutions far more reliably than an outcome reward model. A learned verifier can be fooled, though, and more samples mean more chances to find a wrong answer it likes: Cobbe et al. (2021) saw best-of-n accuracy start to fall again after a few hundred samples.

In code: check_step is the toy's process verifier for one line, first_bad_step runs it over a chain, and outcome_reward compares only the final answer with a reference. noisy_chain writes chains whose slips carry forward, pass_at_n is the formula, and verifier_experiment compares one sample, voting, best-of-n with the step checker, and the pass@n ceiling.

Everyday picture A workbook with the answers printed in the back. Nobody shows you how to solve anything. You try each problem several ways, check the back, and do more of whatever worked. Over hundreds of problems you discover good habits on your own: write things down, double-check the tricky step, don't stop too early.

Tiny example Prompting a model to "think step by step" gets it to write steps; a reasoning model is trained to write good ones. The recipe behind models such as DeepSeek-R1 is reinforcement learning (see primer.ml.reinforcement) with two ingredients:

• Verifiable rewards. Train on problems whose answers a program can check: math with a known final answer, code with unit tests. The reward is 1 if the final answer is right and 0 if not. Nobody grades the chain itself.
• Group comparisons. For each problem, sample a group of attempts and judge each one against its own group. With and rewards 1, 0, 0, 1, the group averages 0.5, so the two right attempts are above average (+1) and the two wrong ones below (−1). The model is nudged towards whatever the right ones did, including how they reasoned. This is the heart of GRPO (group relative policy optimization).

flowchart LR
  P["problem with a checkable answer"] --> G["sample G attempts,<br/>each with its own chain of thought"]
  G --> R["check each final answer:<br/>reward 1 or 0"]
  R --> A["advantage: reward minus group mean,<br/>divided by group spread"]
  A --> U["make above-average attempts more likely,<br/>below-average ones less"]
  U -->|next batch| P

import statistics
r = [1, 0, 0, 1]
mu = statistics.mean(r)  # → 0.5
sigma = statistics.pstdev(r)  # → 0.5
[(r_i - mu) / sigma for r_i in r]  # → [1.0, -1.0, -1.0, 1.0]
# a group that all succeeded: spread 0, nothing to learn
statistics.pstdev([1, 1, 1, 1])  # → 0.0

The toy: learning how long to think. Our policy (the model's rule for choosing what to do) makes one choice per problem: how many tokens to think for, out of 1, 2, 4, 8, 16 or 32. It keeps a separate choice for each difficulty (problems needing 1, 2, 4 or 8 steps), and it starts out preferring short answers, like a model never rewarded for thinking: 63% of the time it answers in 1 token. A chain shorter than the steps needed must guess (right 10% of the time). Otherwise each step gets as many tries as the length allows, and a try slips 10% of the time: one pass through the steps, or two (the first attempt and a recheck that catches a slip), or more. The reward is 1 for a right answer and 0 for a wrong one. That is all.

In code: group_advantages is the formula, success_probability is the toy's chance of solving a problem at a given length, and train_reasoner runs the whole loop: sample a group of lengths per difficulty, reward, compute advantages, and nudge the policy.

The price of thinking

Rewarded for correctness alone, nothing ever tells the model to stop: any extra length that helps even slightly gets reinforced. Real reasoning models show this as overthinking: hundreds of tokens spent on "what is 2 + 3?". The usual remedy is to charge for length: subtract a small penalty per token from the reward. The expected reward for a chain of length on a problem needing steps (with ) becomes:

eps, lam = 0.1, 0.01
# an 8-step problem: chance of success at 8, 16 and 32 tokens
[round((1 - eps ** (L // 8)) ** 8, 3) for L in (8, 16, 32)]  # → [0.43, 0.923, 0.999]
# minus λL: 16 tokens wins
[round((1 - eps ** (L // 8)) ** 8 - lam * L, 3) for L in (8, 16, 32)]  # → [0.35, 0.763, 0.679]
# a 1-step problem at 1, 2 and 4 tokens: 2 tokens wins
[round((1 - eps ** (L // 1)) ** 1 - lam * L, 3) for L in (1, 2, 4)]  # → [0.89, 0.97, 0.96]

Now look back at the two figures. With the penalty and no division by the spread (green), training finds exactly those best lengths, 2 tokens for the easiest problems and 16 for the hardest. With GRPO's division by the spread (red), the easiest problems shrink to 1 token, and accuracy settles at 0.92, below the 0.96 of correctness alone. Why? In a group where every attempt succeeded, rewards differ only by the penalty: 0.99 for 1 token, 0.98 for 2. Their spread is tiny, so dividing by it inflates that 0.01 difference into advantages of +1 and −1, as loud as the difference between solved and failed. The penalty ends up far stronger than its size. Liu et al. (2025) analyse biases like this in GRPO; the lesson for anyone training or budgeting a reasoning model is that the exact shape of the reward decides how long the model thinks.

Everyday picture A long line of dominoes falls all the way only if every single one is placed right. Add more dominoes and the chance that one is misplaced grows, however careful you are with each.

Slips compound

Tiny example If each step of a chain slips 2% of the time, a 10-step chain is clean 82% of the time, and a 50-step chain only 36% of the time.

eps = 0.02
round((1 - eps) ** 10, 3)  # → 0.817
round((1 - eps) ** 50, 3)  # → 0.364
round((1 - eps) ** 200, 3)  # → 0.018

In code: steps_all_right is the formula.

What thinking costs

Tiny example Eight sampled chains of 2,000 tokens each, at an illustrative $10 per million output tokens, cost 16 cents per question. Run side by side at 50 tokens per second, the user waits 40 seconds whether you sample one chain or eight.

n, T, c, v = 8, 2000, 10.0, 50
round(n * T * c / 10**6, 2)  # → 0.16
T / v  # → 40.0
# one chain instead of eight: an eighth of the money, the same wait
round(1 * T * c / 10**6, 2)  # → 0.02

Thinking wider costs money; thinking longer costs money and time. See primer.agents.cost for pricing requests and measuring cost per successful task.

In code: reasoning_cost returns both numbers for one question.

Other ways reasoning fails

• The chain is not a transcript. The written reasoning need not be the real cause of the answer. Turpin et al. (2023) nudged models towards an answer with a hidden bias in the prompt; the answers followed the bias, and the written explanations never mentioned it. Treat a chain of thought as evidence about the model's reasoning, not a faithful record of it.
• Overthinking. Long chains on easy questions waste tokens and time, and a model can talk itself out of a right first answer.
• No checker, less progress. Reinforcement learning with verifiable rewards works where a program can check the answer. Open-ended writing, judgement calls and long projects have no cheap checker, and gains there are smaller.
• Checkers get gamed. A verifier with gaps is a target: code that special-cases the unit tests passes them. This is reward hacking; see primer.ml.reinforcement.

Budgeting it

Everyday picture You wouldn't convene a committee to decide what to have for lunch, and you wouldn't let one person sign off a bridge design alone. Match the effort to the stakes and to whether the result can be checked.

flowchart TD
  Q[Incoming task] --> E{"Easy, or<br/>latency-critical?"}
  E -->|yes| N["No thinking,<br/>or a small budget"]
  E -->|no| C{"Can a program<br/>check the answer?"}
  C -->|"yes: tests, math"| BV["Think, sample n,<br/>keep what passes the check"]
  C -->|no| B["Think with a capped budget;<br/>vote if answers compare exactly"]
  N & BV & B --> M["Measure accuracy and cost per task;<br/>raise budgets only where it pays"]

Answer each one out loud or on paper before you open it. If you can explain it, you know it.