Positional information

Level 2 starts with the problem itself, a bag of Scrabble tiles, and adds each family of positions in turn.

Everyday picture Pour the words of a sentence into a bag of Scrabble tiles. The bag for "dog bites man" and the bag for "man bites dog" hold exactly the same tiles. Anyone who only gets the bag cannot tell a news story from a routine dog attack. Attention, on its own, only ever gets the bag.

Tiny worked example Give each word a 2-number embedding: dog = (1, 0), bites = (0, 1), man = (1, 1). Average the words of each sentence, which is the simplest possible "sentence vector":

Addition doesn't care about order, and neither does attention: every token compares itself with every other by dot product, and nothing in that computation says who came first.

flowchart LR
  A["'dog bites man'"] --> E1[Embed each token]
  B["'man bites dog'"] --> E2[Embed each token]
  E1 --> S1["Set {dog, bites, man}"]
  E2 --> S2["Set {man, bites, dog}"]
  S1 --> ATT[Attention with no positions]
  S2 --> ATT
  ATT --> SAME[Same vectors, only reordered:<br/>the meaning difference is lost]

The math A permutation is a reshuffle of an ordered list. Written as a matrix (a grid of numbers), a permutation matrix is all zeros except one 1 in each row, and multiplying by it just reorders rows. Attention without positions is permutation-equivariant:

import math
# q = k = v = the word vector, no positions
def Attn(X):
    out = []
    for q in X:
        scores = [sum(a * b for a, b in zip(q, k)) / math.sqrt(2) for k in X]
        exps = [math.exp(s) for s in scores]
        weights = [e / sum(exps) for e in exps]
        out.append(tuple(round(sum(w * v[c] for w, v in zip(weights, X)), 3) for c in range(2)))
    return out
# dog, bites, man
X = [(1, 0), (0, 1), (1, 1)]
# P swaps rows 1 and 3: man, bites, dog
PX = [X[2], X[1], X[0]]
Attn(X)  # → [(0.802, 0.599), (0.599, 0.802), (0.752, 0.752)]
# the same rows, shuffled the same way
Attn(PX)  # → [(0.752, 0.752), (0.599, 0.802), (0.802, 0.599)]

Shuffle the input and you get the same outputs, shuffled the same way. encode_sentence(..., scheme="none") runs a real attention layer and shows the "dog" row is the same vector whether "dog" comes first or last.

In code: word_vector gives each word a fixed random embedding, and order_similarity pools each sentence's outputs and compares them, which gives the bars in the figure.

Why it matters Almost all meaning in language is carried by order: negation scope, who did what to whom, code. So position must be injected. There are three families, below.

Everyday picture A clock with many hands. The second hand moves fast and tells nearby moments apart; the hour hand moves slowly and tells morning from evening. Read all the hands together and every moment has a unique fingerprint. Sinusoidal codes give every position such a set of "hands" and add them to the token's embedding.

Tiny worked example With 4 dimensions there are two hands. The fast one turns 1 radian per position; the slow one turns 1/100 of a radian:

The fast pair already looks very different at positions 1 and 2; the slow pair barely moves, and it will only distinguish positions hundreds apart.

flowchart LR
  T["token id"] --> E["embedding lookup<br/>(what the word means)"]
  P["position 0,1,2..."] --> S["sin/cos at many frequencies<br/>(where the word is)"]
  E --> ADD(("+"))
  S --> ADD
  ADD --> B["transformer blocks"]

The math and the code Two functions do the work. Picture a point walking around a circle of radius 1. The angle it has turned is measured in radians (a full turn is 2π ≈ 6.28 radians). Cosine of the angle is how far right the point is, and sine is how far up; both always lie between −1 and +1, and both repeat every full turn.

import math
d, pos = 4, 1
PE = []
# one (sin, cos) pair per i
for i in range(d // 2):
    # ω_i: pair i's speed
    omega_i = 1 / 10000 ** (2 * i / d)
    # columns 2i and 2i+1
    PE += [math.sin(pos * omega_i), math.cos(pos * omega_i)]
[f"{v:.3f}" for v in PE]  # → ['0.841', '0.540', '0.010', '1.000']

sinusoidal_encoding(n, d) builds the whole (n × d) table in four lines. A shift of k positions rotates every (sin, cos) pair by the same angle k·ω_i wherever you start, so the dot product of two position codes depends only on their distance.

Why it matters Fixed codes need no training and exist for any position, and the distance property lets the model learn "look 3 tokens back" once and use it everywhere. But the code is mixed into the content vector itself, which later methods avoid.

Everyday picture Numbered theatre seats. Every seat has its own brass plate, and the theatre has a fixed number of seats.

Tiny worked example GPT-2 has a table of 1,024 position vectors, each 768 numbers long: 786,432 extra parameters, learned like any other weight. Position 5 looks up row 5. Position 1,024 has no row at all.

flowchart LR
  T["token id 318"] --> TE["token table<br/>(vocab × d)"]
  P["position 5"] --> PE["position table<br/>(max_len × d)"]
  TE --> ADD(("+"))
  PE --> ADD
  ADD --> B["transformer blocks"]
  P2["position ≥ max_len"] -.-> X["no row: cannot be encoded"]

The math and the code

E_dog = [0.5, -0.2]
# a position table with rows 0 to 3
P = [[0.0, 0.0], [0.0, 0.0], [0.0, 0.0], [0.1, 0.3]]
# x_3 = E_dog + P_3
[round(e + p, 2) for e, p in zip(E_dog, P[3])]  # → [0.6, 0.1]
# no row 1024: it cannot be encoded
len(P) > 1024  # → False

primer.ml.transformer.TinyGPT uses exactly this.

Why it matters It's simple and works well inside the trained length, but it cannot extrapolate. That limitation pushed the field to RoPE.

Everyday picture Two people point at a clock face, one at 1 o'clock and one at 4 o'clock. The angle between their arms is 90°. Move both of them on to 8 and 11 o'clock and the angle is still 90°. RoPE turns each token's query and key like clock hands, by an amount proportional to its position. When a query meets a key, only the angle between them (the distance between the tokens) affects the score, not where on the clock they started.

Tiny worked example Use 2 dimensions, so there is one pair turning 1 radian per position. Let q = k = (1, 0). Put the query at position 3 and the key at position 7:

• q turns by 3 rad: (cos 3, sin 3) = (−0.990, 0.141)
• k turns by 7 rad: (cos 7, sin 7) = (0.754, 0.657)
• score = −0.990·0.754 + 0.141·0.657 = −0.654 = cos(4)

Now move both 100 positions later, to 103 and 107. The score is still cos(107 − 103) = cos 4 = −0.654. Only the distance, 4, survived.

flowchart LR
  X[Token vector at position m] --> Q[q = x W_q]
  X --> K[k = x W_k]
  X --> V[v = x W_v]
  Q --> RQ["Rotate each pair of q<br/>by m·θ_i"]
  K --> RK["Rotate each pair of k<br/>by m·θ_i"]
  RQ --> S["Score = q'·k'<br/>depends only on distance"]
  RK --> S
  S --> SM[Softmax] --> W[Weighted sum of V<br/>V is NOT rotated]
  V --> W

The math and the code A rotation turns a point around the centre by some angle without changing its distance from the centre. Written as a 2 × 2 grid times a column of two numbers (a matrix-vector multiply: each output number is one row of the grid multiplied position-by-position with the input and added up):

import math
def rotate(x, m, theta_i=1.0):
    # the angle m θ_i
    a = m * theta_i
    # row 1 of the grid
    return (math.cos(a) * x[0] - math.sin(a) * x[1],
            # row 2 of the grid
            math.sin(a) * x[0] + math.cos(a) * x[1])
[f"{v:.3f}" for v in rotate((1, 0), m=3)]  # → ['-0.990', '0.141']
d, i = 2, 0
# θ_0 = 10000^(-2i/d)
10000 ** (-2 * i / d)  # → 1.0

Why only the distance survives: turning q by angle a and k by angle b and then taking their dot product (multiply matching numbers, add them up; large when the vectors point the same way) gives the same answer as turning k alone by b − a. So

import math
# turn a pair by an angle (θ = 1, so the angle is the position)
def R(angle, x):
    return (math.cos(angle) * x[0] - math.sin(angle) * x[1],
            math.sin(angle) * x[0] + math.cos(angle) * x[1])
def dot(a, b): return sum(a_i * b_i for a_i, b_i in zip(a, b))
q = k = (1, 0)
# ⟨R_m q, R_n k⟩ with m = 3, n = 7
round(dot(R(3, q), R(7, k)), 3)  # → -0.654
# ⟨q, R_(n-m) k⟩: only the distance
round(dot(q, R(7 - 3, k)), 3)  # → -0.654
# 100 positions later, the same score
round(dot(R(103, q), R(107, k)), 3)  # → -0.654

apply_rope does this for a vector or a whole sequence with three element-wise lines, and no matrix multiply.

In code: rope_frequencies returns each pair's turning speed θ_i, which apply_rope multiplies by the position to get every angle.

Why it matters Relative distance is what language actually uses ("the word two back"), rotation keeps vector lengths unchanged, and because position lives in angles, you can stretch it after training. That last point is the next section.

Everyday picture A ruler marked from 0 to 4,000. To measure something 32,000 long you can shrink the object by 8× so it fits on the ruler (interpolation), or re-draw only the long-distance marks while keeping the fine millimetre marks (NTK-aware scaling).

Tiny worked example A model trained on 4,096 tokens now reads 32,768. Interpolation multiplies every position by 4096/32768 = 1/8, so position 32,000 is treated as 4,000: an angle the model has seen. Neighbours are now 1/8 of a position apart, which a short fine-tune teaches it to resolve.

flowchart LR
  P["positions 0 ... 32k"] --> C{method}
  C -->|interpolation| PI["pos × 4k/32k<br/>all hands slowed 8×"]
  C -->|NTK-aware| NTK["raise the RoPE base<br/>slow hands slowed, fast hands kept"]
  PI --> R["RoPE with angles inside<br/>the trained range"]
  NTK --> R
  R --> F["short fine-tune on long text"]

The math and the code Position interpolation (interpolate_positions):

pos, L_train, L_new = 32_000, 4_096, 32_768
# pos' = pos · L_train / L_new
pos * L_train / L_new  # → 4000.0

NTK-aware scaling (ntk_scaled_base) changes the base instead:

base, s, d = 10_000, 8, 128
# s^(d/(d-2)): just above 8
round(s ** (d / (d - 2)), 2)  # → 8.27
# base' = base · s^(d/(d-2))
base_new = base * s ** (d / (d - 2))
round(base_new)  # → 82685
# θ_i for the last pair, 2i = 126
slowest = lambda b: b ** (-126 / 128)
f"{slowest(base):.2e}", f"{slowest(base_new):.2e}"  # → ('1.15e-04', '1.44e-05')
# exactly 8× slower
round(slowest(base) / slowest(base_new), 6)  # → 8.0

YaRN refines this per frequency band and is used by many long-context models. ALiBi skips position vectors entirely and subtracts a penalty proportional to distance from each attention score.

Why it matters It's how models trained on a few thousand tokens are cheaply stretched to 100k or more, instead of being pretrained again from scratch.

In a decoder, token i sees exactly i+1 tokens, so even with no position encoding ("NoPE") the model can partly infer where it is. Bidirectional attention (BERT-style) has no such crutch and is fully order-blind.

In code: encode_sentence with its causal flag set runs the same layer under primer.ml.attention.causal_mask; its pooled similarity is the "causal mask only" bar in the first figure.

Answer each one out loud or on paper before you open it. If you can explain it, you know it.