GANs

A forger wants to print banknotes that pass as real. A detective wants to catch every fake. At first the forger is hopeless: smudged ink, the wrong colour, and the detective spots every note. But every time the detective rejects a note, the forger learns what gave it away and fixes that. Every time the forger improves, the detective has to look more closely. Neither is ever told what a real banknote should look like. They only push against each other, and both keep getting better.

The contest ends when the forger's notes are so good that the detective can do no better than guess: "real" half the time, "fake" the other half. At that point the forger has learned to make banknotes, and nobody ever wrote down a rule for what a banknote is.

That is a generative adversarial network, a GAN. The forger is a neural network called the generator; the detective is a second network called the discriminator. Train them against each other on photos of faces and the generator learns to produce new faces that never existed.

This lesson builds both players from scratch in NumPy, trains them on a toy picture you can see at a glance, and then shows the three ways the contest goes wrong: the players chase each other in circles, the forger settles for copying a few examples (mode collapse), and the detective wins so completely that the forger stops learning. Each failure gets a fix you can run. The same goal, making new samples, is reached differently by primer.ml.generative.autoencoders (squeeze and rebuild) and primer.ml.generative.diffusion (remove noise a little at a time).

Everyday picture The forger is a recipe that turns a few dice rolls into a banknote: different rolls, a different note. The detective is a machine you feed a note into, and out comes a single number, how sure it is that the note is real.

Tiny worked example Our "banknotes" are points on a flat page. The real data is a ring of eight little clouds: eight centres evenly spaced on a circle of radius 2, each real point scattered a tiny amount (a spread of 0.05) around one centre, chosen at random. This toy picture comes from the research literature on GAN failures because you can see whether a forger has found all eight clouds.

• The generator G takes two random numbers z (the dice rolls, drawn from a bell curve) and returns one point on the page. It is a small neural network: 2 numbers in, two layers of 32 tanh units, 2 numbers out.
• The discriminator D takes a point and returns a probability that it is real. It is another small network: 2 numbers in, 64 tanh units, one score out, then squashed into the range 0 to 1.

The squashing is the sigmoid, the same one primer.ml.neural_net uses to turn a score into a probability:

import math
def sigma(a):
    return 1 / (1 + math.exp(-a))
# a one-neuron detective: score a(x) = x_1 - 1
def a(x):
    return x[0] - 1
round(sigma(a((2, 0))), 3)  # → 0.731
round(sigma(a((0, 0))), 3)  # → 0.269

flowchart LR
  Z["noise z<br/>2 random numbers"] --> G["Generator G<br/>the forger"]
  G --> F["fake point G(z)"]
  R["real point x<br/>from the ring"] --> D["Discriminator D<br/>the detective"]
  F --> D
  D --> P["D(point)<br/>probability it is real"]
  P -. "learns to call real real<br/>and fake fake" .-> D
  P -. "learns to make D say real<br/>about its fakes" .-> G

Why it matters that last point is the whole trick. Nobody writes down what a face, a voice or a banknote is. The detective discovers what separates real from fake, and its gradient (the direction that would make it less sure a fake is fake; see primer.ml.neural_net for gradients) tells the forger how to improve.

In code: ring_of_gaussians draws real points around ring_modes, MLP is both players (its MLP.backward also returns the gradient with respect to the input, the channel through which the forger learns), with shapes GENERATOR_SIZES and DISCRIMINATOR_SIZES, and sigmoid turns the detective's score into a probability.

Everyday picture Picture a scoreboard. The detective earns points for confident, correct calls and loses points, heavily, for confident mistakes. The detective wants the score as high as possible. The forger wants it as low as possible. One number, two players pulling it in opposite directions: that is a minimax game.

Tiny worked example The detective looks at two real notes and two fakes. The score uses the logarithm (log): log 1 = 0, and the log of a small number is a large negative number, so a confident mistake costs far more than a hesitant one (see primer.notation for logs from scratch).

The average over the real notes is −0.164, over the fakes −0.367, and the total is −0.53. A perfect detective (1 on every real note, 0 on every fake) would score log 1 + log 1 = 0, the ceiling. A detective reduced to a coin flip (0.5 on everything) scores log 0.5 + log 0.5 = −1.386.

import math
# D(x) on two real notes, D(G(z)) on two fakes
d_real, d_fake = [0.9, 0.8], [0.2, 0.4]
# E over real x of log D(x): an average
real_term = sum(math.log(d) for d in d_real) / len(d_real)
round(real_term, 3)  # → -0.164
# E over noise z of log(1 - D(G(z)))
fake_term = sum(math.log(1 - d) for d in d_fake) / len(d_fake)
round(fake_term, 3)  # → -0.367
round(real_term + fake_term, 2)  # → -0.53
# a coin-flip detective, D = 1/2 on everything
round(math.log(0.5) + math.log(0.5), 3)  # → -1.386

If this looks familiar, it should: −V is exactly the binary cross-entropy of a classifier that labels real points 1 and fakes 0 (see primer.ml.losses). The detective is an ordinary classifier. What is new is that its second class, the fakes, keeps changing underneath it.

Training alternates. Nobody can solve "min over G of max over D" directly, so each round takes one small step for each player:

flowchart TB
  S["sample a batch of real points<br/>and a batch of noise"] --> F["forger makes fakes G(z)"]
  F --> DS["detective step:<br/>climb V, so D(real) rises<br/>and D(fake) falls"]
  DS --> GS["forger step:<br/>change G so the new detective<br/>scores its fakes higher"]
  GS --> CHK{"trained enough?"}
  CHK -- no --> S
  CHK -- yes --> OUT["keep G;<br/>throw D away"]

Why it matters each player is trained with an ordinary optimizer on an ordinary loss, but the loss surface moves every time the other player steps. That is the root of everything that goes wrong later in this lesson.

In code: value_function computes V from a batch of verdicts, and train_gan runs the loop above on the ring, one detective step then one forger step, each with Adam (primer.ml.optimizers.Adam).

Everyday picture Suppose that, at one spot on the page, real points turn up three times as often as fakes. However clever the detective is, it cannot tell two identical-looking points apart, so the best it can do there is to say "75% real". It should match the local mix, no more and no less.

Tiny worked example At a point where the real data's density (how thickly its points cover that spot) is 0.3 and the forger's is 0.1, the best verdict is 0.3 / (0.3 + 0.1) = 0.75. Try its neighbours: the detective's expected score at that spot is 0.3 · log D + 0.1 · log(1 − D), which is −0.2274 at D = 0.7, −0.2249 at D = 0.75 and −0.2279 at D = 0.8. The middle one is the highest.

import math
p_data, p_g = 0.3, 0.1
round(p_data / (p_data + p_g), 2)  # → 0.75
# the detective's expected score at this spot, for a few verdicts D
def score(D):
    return p_data * math.log(D) + p_g * math.log(1 - D)
[round(score(D), 4) for D in (0.7, 0.75, 0.8)]  # → [-0.2274, -0.2249, -0.2279]
# a forger that matches the data exactly
round(0.3 / (0.3 + 0.3), 2)  # → 0.5

Where does the formula come from? At each point the detective is choosing one number D to maximise . The slope of that with respect to D is , and setting the slope to zero gives . This is the central result of the original GAN paper, and it tells you where the game ends. At the equilibrium the forger's distribution equals the data's, and the best detective says 1/2 everywhere, scoring V = −log 4 ≈ −1.386, the coin-flip score from the table above. Plug back into V and what remains is −log 4 plus twice the Jensen-Shannon divergence between the real and forged distributions, a measure of how different two piles of probability are that is zero only when they are identical. So a forger facing a perfect detective is really minimising that divergence. Keep that in mind: it returns as a problem in the fixes section.

Why it matters the detective never needs to model what real data looks like; it only estimates a ratio between two densities. That is why GANs could learn sharp images long before anyone could write down a probability for an image.

In code: optimal_discriminator is the formula, fit_discriminator_table trains the most flexible detective possible (one free score per point) against a fixed forger and arrives at the same verdicts without being told the formula, and detective_verdicts evaluates a trained detective anywhere on the page.

Everyday picture Early in training, the forger is terrible and the detective is sure every fake is fake. Now picture a game of "hotter, colder" where the helper stops talking once you are far enough away: the further off you are, the quieter the hints. That is the original forger loss. The fix is a helper who shouts louder the further off you are.

Tiny worked example The detective looks at a fake and is 99% sure it is fake: D(G(z)) = 0.01. How hard does each loss push the forger?

• The original loss, log(1 − D(G(z))), which the forger minimises: its slope with respect to the detective's score is −D = −0.01. Almost nothing.
• The non-saturating loss, −log D(G(z)), which the forger also minimises: its slope is −(1 − D) = −0.99. Ninety-nine times more.

When the detective is undecided, D = 0.5, both slopes are −0.5. The two losses agree when it doesn't matter and differ exactly when it does.

import math
def sigma(a):
    return 1 / (1 + math.exp(-a))
# a score that makes the detective 99% sure the fake is fake
a = math.log(0.01 / 0.99)
round(sigma(a), 2)  # → 0.01
# measure each slope by nudging the score a tiny bit either way
h = 1e-6
def slope(loss):
    return (loss(a + h) - loss(a - h)) / (2 * h)
def saturating(a):
    return math.log(1 - sigma(a))
def non_saturating(a):
    return -math.log(sigma(a))
round(slope(saturating), 3)  # → -0.01
round(slope(non_saturating), 3)  # → -0.99

Both losses want the same thing, a higher verdict on the fakes, and they share the same equilibrium. Only the strength of the push differs. The original GAN paper already recommended the switch, and essentially every GAN since uses the non-saturating loss or a Wasserstein-style loss (below).

Why it matters a forger that gets no gradient does not learn, and the detective's lead only widens. Early in training the detective always has the easy job, so this failure is the default, not an edge case.

In code: generator_logit_gradient returns both slopes, and head_start_experiment gives the detective a head start and measures the forger's gradient under each loss.

Everyday picture Two people share an office thermostat. One turns the heat up whenever it feels cold; the other opens a window whenever it feels hot. Each reacts to where the room was, so the temperature overshoots one way, then the other, and never settles. A GAN's players do the same: the forger moves to where the detective currently says "real", and by the time it arrives, the detective has moved.

Tiny worked example The smallest GAN there is, the Dirac GAN, has one number per player. The real data is a single point at x = 0. The forger has one number θ and always outputs x = θ. The detective has one number ψ and scores a point as ψ · x, so D(x) = σ(ψx). Start with the forger at θ = 1 and a detective with ψ = 0 (it cannot tell anything apart), and let both take steps of size 1 at the same time:

After step 1 the detective has learned "bigger x means fake" (ψ turned negative), and after step 2 the forger has moved towards the data at 0. Both moves were sensible, yet the pair got further from the equilibrium, the point (0, 0) where the forger sits on the data and the detective is indifferent.

import math
def sigma(a):
    return 1 / (1 + math.exp(-a))
theta, psi, lr = 1.0, 0.0, 1.0
s = sigma(psi * theta)
s  # → 0.5
# both move at once, each using the other's old position
theta, psi = theta + lr * s * psi, psi - lr * s * theta
(theta, psi)  # → (1.0, -0.5)
s = sigma(psi * theta)
round(s, 4)  # → 0.3775
theta, psi = theta + lr * s * psi, psi - lr * s * theta
(round(theta, 3), round(psi, 3))  # → (0.811, -0.878)
# distance from the equilibrium (0, 0): 1, then 1.118, then 1.195
round(math.hypot(theta, psi), 3)  # → 1.195

flowchart LR
  A["fake sits to the right<br/>of the data"] --> B["detective tilts:<br/>right means fake"]
  B --> C["forger slides left,<br/>overshoots past the data"]
  C --> D["detective tilts back:<br/>left means fake"]
  D --> E["forger slides right,<br/>overshoots again"]
  E --> A

Why it matters a real GAN has millions of numbers per player, and this rotation happens in many directions at once. It shows up as losses that swing up and down without trending, and samples that keep changing character instead of steadily improving.

In code: dirac_gan runs this two-number game with simultaneous or alternating steps.

Everyday picture The forger discovers that one particular note, say the twenty, always fools the detective. So it prints nothing but twenties. The detective eventually learns that twenties are suspicious, so the forger switches to printing only fifties, and so on. Each forgery is good, but the forger never produces the full variety of real money. Each distinct kind of real example is a mode, and settling on a few of them is mode collapse.

Tiny worked example To measure variety on the ring, generate 1,000 points and count, for each of the eight clouds, how many land within 0.15 of its centre (3 spreads). A cloud is covered if it gets at least 20 of the 1,000 (2%; a perfect forger gives each about 125). A point that lands near any centre counts as high quality. After 2,500 rounds with equal learning rates for both players, the counts for the eight clouds are (0, 68, 0, 20, 0, 51, 0, 17). Clouds 2, 4 and 6 are covered, cloud 8's 17 falls short, so 3 of 8 modes are covered, and only 156 of the 1,000 points are high quality: the rest are strung between clouds.

stateDiagram-v2
  direction LR
  EvenClouds: forger piles onto clouds 1, 3, 5, 7
  CatchEven: detective learns those clouds are suspicious
  OddClouds: forger jumps to clouds 2, 4, 6, 8
  CatchOdd: detective learns these are suspicious
  EvenClouds --> CatchEven
  CatchEven --> OddClouds
  OddClouds --> CatchOdd
  CatchOdd --> EvenClouds