Clustering and everyday uses of embeddings

Level 2 builds each of those methods from nothing, starting with a sack of mail.

The everyday picture. Tip a sack of unlabeled mail onto a table and sort it into piles by what each letter is about. Nobody gave you the pile names; you notice that some letters are about passwords and others about holidays, and similar letters end up together. That's clustering: finding groups in data without being told what the groups are.

Embeddings make it possible for text. Every ticket, email or document gets coordinates on a map of meaning (primer.ml.embeddings.similarity), so "similar" becomes "close", and sorting mail becomes finding crowds of nearby points. The same closeness also powers four everyday jobs: spotting near-duplicates, routing requests to the right team, flagging anomalies, and caching answers to questions already asked.

This lesson uses 30 short IT, HR and finance support tickets (five kinds, six of each) and two off-topic ones, embedded with the repo's toy embedder (primer.common.embedder).

In code: ticket_embeddings returns the tickets' unit vectors and texts, and ticket_kinds their true kinds, which the clustering never sees.

Everyday picture a town wants k post boxes placed so that everyone's walk to their nearest box is as short as possible. Start with the boxes anywhere. Everyone walks to their nearest box; then each box moves to the middle of the people who chose it. Repeat until nobody switches box.

Tiny worked example four points, (0, 0), (0, 1), (10, 0), (10, 1), and k = 2. However the boxes start, the two left points pick one box and the two right points the other. Each box moves to the middle of its pair: (0, 0.5) and (10, 0.5). Every point is now 0.5 away from its box, so the total squared walk is 4 × 0.5² = 1.0, and nothing changes on the next round.

flowchart LR
  I["Place k centres<br/>(k-means++: spread out)"] --> A["Assign: each point<br/>joins its nearest centre"]
  A --> M["Move: each centre goes to<br/>the mean of its points"]
  M --> C{Did any centre move?}
  C -->|yes| A
  C -->|no| D[Done: labels + centres]

On the example: four points each 0.5 from their centre: J = 4 × 0.25 = 1.0.

In Python:

x = [(0, 0), (0, 1), (10, 0), (10, 1)]
# the two centroids
mu = [(0, 0.5), (10, 0.5)]
# c(i): the cluster each point joined
c = [0, 0, 1, 1]
# ‖x_i - μ_c(i)‖²
J = sum((x_i[0] - mu[c_i][0]) ** 2 + (x_i[1] - mu[c_i][1]) ** 2
        # Σ over every point
        for x_i, c_i in zip(x, c))
J  # → 1.0

In code: kmeans alternates assign and move from k-means++ starts and returns the labels, centres and inertia J of the best of several restarts; kmeans_history records one run round by round, which is what this figure draws.

Why it matters k-means is fast and simple, and it's inside things you use: IVF vector indexes cluster the corpus with it (primer.ml.embeddings.ann), and topic discovery over tickets or documents often starts with it. Its weaknesses are that you must choose k and that it assumes round, similar-sized clusters.

Everyday picture at a party, you're in the right group if the people in your group are much closer to you than the people in the next group over.

Tiny worked example the four points again, split into the two pairs. For (0, 0): its partner is a = 1 away; the other pair is 10 and √101 = 10.05 away, on average b = 10.025. Its score is (b − a) / b = 9.025 / 10.025 = 0.900, and by symmetry every point scores the same, so the silhouette is 0.900.

On the example: (10.025 − 1) / 10.025 = 0.900.

In Python:

import math
# a(i): distance to its partner
a = math.dist((0, 0), (0, 1))
# b(i): the other pair, averaged
b = (math.dist((0, 0), (10, 0)) + math.dist((0, 0), (10, 1))) / 2
a, round(b, 3)  # → (1.0, 10.025)
# s(i)
round((b - a) / max(a, b), 3)  # → 0.9

In code: silhouette averages s(i) over every point, and best_k_by_silhouette runs kmeans for each candidate k and keeps the one with the highest silhouette.

Everyday picture a festival seen from a drone. A crowd is wherever people stand shoulder to shoulder; a loner by the fence belongs to no crowd. You don't decide in advance how many crowds there are.

Tiny worked example points on a line at 0, 0.1, 0.2, 5.0, 5.1, 5.2 and 20, with reach ε = 0.15 and "a crowd needs at least 2". 0, 0.1 and 0.2 chain together (each within 0.15 of the next); so do 5.0, 5.1 and 5.2. 20 has nobody within 0.15. Result: two clusters and one noise point, labels 0 0 0 1 1 1 −1.

On the example: N₀.₁₅(0.1) = {0, 0.1, 0.2}, 3 ≥ 2, so 0.1 is core; N₀.₁₅(20) = {20}, 1 < 2, so 20 is noise.

In Python:

points = [0, 0.1, 0.2, 5.0, 5.1, 5.2, 20]
eps, minPts = 0.15, 2
# N_ε(x): every point within reach of x
def N(x):
    return [y for y in points if abs(x - y) <= eps]
# a core point
N(0.1), len(N(0.1)) >= minPts  # → ([0, 0.1, 0.2], True)
# nobody within reach: noise
N(20), len(N(20)) >= minPts  # → ([20], False)

flowchart TD
  P[Take an unvisited point] --> N{At least minPts<br/>within ε?}
  N -->|no| L[Leave it as noise for now]
  N -->|yes| S[Start a new cluster]
  S --> G[Add every neighbour;<br/>if a neighbour is also core,<br/>add its neighbours too]
  G --> G
  G --> P
  L --> P

In code: dbscan finds the core points, floods each cluster outward through them, and labels everything unreached −1 (noise).

That's the problem HDBSCAN solves. It effectively runs DBSCAN at every reach at once, builds a tree of how crowds merge as the reach grows, and keeps the crowds that persist over the widest range of reaches. It handles clusters of different densities, needs no ε, and is the usual choice for exploring messy real text such as support tickets.

Everyday picture a shadow on a wall. A 3-D object casts a 2-D shadow; turn the object and the shadow changes. PCA (principal component analysis) turns the object so its shadow is as spread out as possible, keeping as much of the original variation as a flat picture can.

Tiny worked example three points on a straight line, (1, 1), (2, 2), (3, 3). All their variation runs along the diagonal, so the first direction explains 100% of it and the second 0%. Along that direction the points sit √2 = 1.414 apart, exactly as in the original.

On the example: centred, the points are (−1, −1), (0, 0), (1, 1). Their singular values are 2 along the diagonal and 0 across it, so the shares are 2² / (2² + 0²) = 1 and 0 / 4 = 0.

In Python:

import math
centred = [(-1, -1), (0, 0), (1, 1)]
# along the diagonal
u = [(1 / math.sqrt(2), 1 / math.sqrt(2)),
     # across it (the SVD finds these; here we know them)
     (1 / math.sqrt(2), -1 / math.sqrt(2))]
# spread along u_j
sigma = [math.sqrt(sum((x * u_j[0] + y * u_j[1]) ** 2 for x, y in centred))
         for u_j in u]
[round(sigma_j, 3) for sigma_j in sigma]  # → [2.0, 0.0]
[round(sigma_j ** 2 / sum(sigma_k ** 2 for sigma_k in sigma), 3) for sigma_j in sigma]  # → [1.0, 0.0]

In code: pca centres the data, projects it onto the top n directions from the SVD, and returns each direction's explained share.

UMAP and t-SNE draw prettier maps by a different rule: keep each point's neighbours next to it, and let distances elsewhere stretch. Like a subway map, they're great for "what's near what" and misleading for "how far" or "how big": gaps between clusters and cluster sizes in those plots don't reflect the real space. Use them for intuition, never for measurement.

flowchart LR
  T[New text] --> E[Embed]
  E --> D{Nearly identical to<br/>something stored?}
  D -->|yes| DUP[Near-duplicate:<br/>merge or skip]
  E --> R{Closest route centroid<br/>above threshold?}
  R -->|yes| ROUTE[Send to that team/tool]
  R -->|no| HUMAN[Fallback: human or<br/>general assistant]
  E --> A{Far from every<br/>cluster centre?}
  A -->|yes| ANOM[Flag as anomaly]
  E --> C{Cached question, same context,<br/>not expired, above threshold?}
  C -->|yes| HIT[Return cached answer]

Near-duplicates. Photocopies with a sticky note on them: nearly identical vectors. "password reset link expired" and "The password reset link has expired!" score cosine 1.0 here; flag pairs above a threshold calibrated on labeled pairs (primer.ml.embeddings.similarity).

In code: near_duplicates embeds a list of texts and returns every pair whose cosine reaches the threshold.

Routing. A receptionist listening to a request and pointing to the right desk. Each route (a team, a tool, an agent) is represented by the centroid of a few example requests; a new request goes to the closest centroid, or to a fallback when nothing is close:

On the example: "my vpn tunnel drops when I work remote" scores highest against the IT helpdesk; "what is the capital of france" is near 0 against every route, below θ = 0.3, so it goes to the fallback.

With the numbers: a toy version with three routes in four dimensions: μ_it = (1, 0, 0, 0), μ_finance = (0, 1, 0, 0), μ_hr = (0, 0, 1, 0). The request q = (0.8, 0.6, 0, 0) has cosines 0.8, 0.6 and 0.0 with them; the best, 0.8, clears θ = 0.3, so it goes to it_helpdesk. The request q = (0.1, 0.2, 0, 1) points mostly where no route lies: its cosines are about 0.1, 0.2 and 0.0, all below θ, so it goes to the fallback.

In Python:

import math
def cos(a, b):
    dot = sum(a_k * b_k for a_k, b_k in zip(a, b))
    return dot / (math.sqrt(sum(a_k ** 2 for a_k in a)) * math.sqrt(sum(b_k ** 2 for b_k in b)))
mu = {"it_helpdesk": (1, 0, 0, 0), "finance": (0, 1, 0, 0), "hr": (0, 0, 1, 0)}
theta = 0.3
def route(q):
    scores = {r: cos(q, mu_r) for r, mu_r in mu.items()}
    # arg max_r cos(q, μ_r)
    best = max(scores, key=scores.get)
    return best if scores[best] >= theta else "fallback"
[round(cos((0.8, 0.6, 0, 0), mu_r), 2) for mu_r in mu.values()], route((0.8, 0.6, 0, 0))  # → ([0.8, 0.6, 0.0], 'it_helpdesk')
[round(cos((0.1, 0.2, 0, 1), mu_r), 2) for mu_r in mu.values()], route((0.1, 0.2, 0, 1))  # → ([0.1, 0.2, 0.0], 'fallback')

In code: Router holds one unit-length centroid per route; Router.scores gives a request's cosine with each, and Router.route applies θ and the fallback. support_router builds the lesson's three routes.

Anomaly detection. A stranger at a party is far from every group. Score each item by its distance to the nearest cluster centre; the largest scores are the unusual items. Here the coffee machine and the dog come out on top.

In code: anomaly_scores returns each point's distance to its nearest cluster centre.

Semantic caching. An FAQ desk that remembers answers. If a new question means the same as one already answered, return the stored answer and skip the model call, saving cost and latency. Three guards keep it safe:

• A strict, calibrated threshold. "How do I reset my VPN?" is only about 0.46 similar to "How do I reset my password?" here: related, but a different question.
• A context key. "What is my PTO balance?" means something different for Alice and Bob. Answers are only reused within the same user, tenant and permission scope.
• Expiry. A time-to-live, so answers age out when the facts change.

flowchart LR
  Q[Question + context] --> F[Keep entries with the<br/>same context, not expired]
  F --> S[Most similar entry]
  S --> T{cosine ≥ threshold?}
  T -->|yes| H[Hit: return stored answer]
  T -->|no| M[Miss: call the model,<br/>store the new answer]

In code: SemanticCache.put stores a question, its answer and its context; SemanticCache.lookup filters by context and expiry, then finds the most similar entry, and SemanticCache.get returns its answer only above the threshold.

Answer each one out loud or on paper before you open it. If you can explain it, you know it.