12.1.1 GAN Architecture

Overview

Two networks, the same training data, opposite goals. One tries to forge convincing samples; the other tries to catch the forgeries. Train them together for long enough and a strange thing happens — both get good, and at the equilibrium point the forger is producing samples the detective can no longer reliably tell apart from real data.

That is the entire idea behind a Generative Adversarial Network. Ian Goodfellow and his co-authors sketched it out in 2014 [1], and a decade later — through DCGAN, StyleGAN [6], BigGAN [7], and the diffusion-era hybrids — it remains one of the most-studied frameworks in generative modelling. Before tackling those, the bare-minimum version is the cleanest place to start: a Generator that learns to produce numbers drawn from N(4.0, 1.25), and a Discriminator that scores them. No convolutions, no images, just the adversarial loop.

Learning objectives

1. Describe the Generator/Discriminator split and the role of each network.
2. Read the minimax loss and recognise it as a two-player zero-sum game.
3. Run a minimal PyTorch GAN and watch it learn a Gaussian distribution.
4. Diagnose the common failure modes — mode collapse, vanishing gradients, runaway oscillation — from training curves.

Quick start — meet an untrained Generator

Before training anything, look at what a fresh Generator produces. The point is to see how far the starting state is from the target.

import torch
import torch.nn as nn

# Target distribution the Generator will eventually learn
DATA_MEAN = 4.0
DATA_STDDEV = 1.25

class Generator(nn.Module):
    def __init__(self):
        super().__init__()
        self.layer1 = nn.Linear(1, 5)
        self.layer2 = nn.Linear(5, 5)
        self.layer3 = nn.Linear(5, 1)

    def forward(self, x):
        x = torch.tanh(self.layer1(x))
        x = torch.tanh(self.layer2(x))
        return self.layer3(x)

generator = Generator()
noise = torch.rand(100, 1)
output = generator(noise)
print(f"Generated {len(output)} numbers")
print(f"Mean: {output.mean():.2f}, Std: {output.std():.2f}")

A typical run prints something like Mean: 0.01, Std: 0.02 — random weights, random output, nowhere near N(4.0, 1.25). The full training script in Exercise 1 closes this gap by pairing the Generator with a Discriminator that supplies the feedback signal.

Core concepts

Concept 1 — Two networks, opposite jobs

A GAN is two neural networks trained in lock-step. The Generator G is a function from random noise z to a synthetic sample G(z). The Discriminator D takes a sample (real or generated) and outputs a probability between 0 and 1 that the sample came from the real dataset.

Notice that the two networks never share parameters, never share gradients, and never see each other’s internals. Their only contact is the scalar score D(G(z)) that flows back to G as a training signal. That tight, narrow channel is what makes GANs both surprisingly effective and notoriously fiddly to train.

Concept 2 — The adversarial game

Goodfellow’s original paper frames training as a minimax objective on a single value function V(D, G):

min_G max_D  V(D, G) = E_{x~p_data}[ log D(x) ] + E_{z~p_z}[ log(1 - D(G(z))) ]

Read it left-to-right: the Discriminator wants to maximise this expression (push D(x) toward 1 on real data, push D(G(z)) toward 0 on fakes); the Generator wants to minimise it (push D(G(z)) toward 1 so the second term shrinks). It is a zero-sum game in the game-theoretic sense — any gain for one side is a loss for the other [2].

In practice we don’t solve the minimax in closed form. We alternate gradient steps:

for epoch in range(num_epochs):
    real_data = sample_real_batch()
    fake_data = generator(sample_noise())

    # Step 1 — Discriminator: classify real as 1, fake as 0
    d_loss = bce(discriminator(real_data), ones) \
           + bce(discriminator(fake_data.detach()), zeros)
    d_loss.backward(); d_optimizer.step()

    # Step 2 — Generator: get the Discriminator to call its fakes real
    fake_data = generator(sample_noise())
    g_loss = bce(discriminator(fake_data), ones)
    g_loss.backward(); g_optimizer.step()

The .detach() on fake_data in the Discriminator step is not optional — without it, gradients leak back into the Generator during a phase that is supposed to update only D. Many first GAN implementations break here and produce nonsense.

If everything goes right, training converges to a Nash equilibrium where G reproduces the true data distribution and D outputs 0.5 regardless of input — it has stopped being able to tell real from fake [1].

Concept 3 — Why GAN training is hard

The adversarial setup is elegant on paper and unstable in practice. Three failure modes show up often enough that they have names.

Mode collapse. The Generator discovers one output that consistently fools the current Discriminator and starts producing only that, ignoring the rest of the data distribution. You’ll see the generated samples all look identical even as the loss stays low [3].

Vanishing gradients. If the Discriminator gets too good too fast, D(G(z)) saturates near 0, the log(1 - D(G(z))) term flattens, and the gradient the Generator receives goes to zero. The Generator stops improving even though there’s plenty of room to grow [3].

Oscillation without convergence. Both losses bounce around indefinitely. The networks are chasing each other in a cycle rather than settling into equilibrium. Some oscillation is normal, but persistent large swings usually mean the learning rates need rebalancing.

The DCGAN paper [4] codified a set of practical rules — Adam with β₁ = 0.5 [8], BatchNorm in most layers, LeakyReLU in the Discriminator, ReLU + Tanh in the Generator — that make training much more reliable. A parallel research line tackled the same instability from the loss-function side: Wasserstein GAN [5] swapped the original BCE objective for an earth-mover distance, removing the vanishing-gradient pathology almost entirely. You’ll see the DCGAN rules applied in Lesson 12.1.2.

Exercises

EXECUTE I.

Train a GAN on a Gaussian

Run the full training script. It pairs the 3-layer Generator from the Quick Start with a small Discriminator that compares the statistical moments (mean, standard deviation, skew, kurtosis) of a batch of generated numbers against the moments of a real batch from N(4.0, 1.25).

gan_architecture.py

bash Copy

python gan_architecture.py

What to watch for as it runs

• The Generator’s reported mean climbs from ~0 toward 4.0 over the first ~1500 epochs.
• The standard deviation lags behind the mean — getting std ≈ 1.25 typically takes another 1000–2000 epochs.
• Both losses oscillate. They don’t smoothly decrease the way a supervised classifier’s loss would.

Reflection

1. Roughly how many epochs in does the Generator first produce numbers with the right mean?
2. What are the final mean and standard deviation, and how close are they to the target (4.0, 1.25)?
3. Do the loss curves “converge” in the usual sense? Why or why not?

Discussion

1. The mean usually crosses 3.0 around epoch 1000–1500 and lands within a few tenths of 4.0 by epoch 3000. Yours will vary — the random seed determines a lot about the early trajectory.
2. After 5000 epochs the script typically reports something like mean=3.95, std=1.22. Not exactly the target, but close enough that the histograms visibly overlap.
3. Not in the usual sense. Both losses oscillate around values bounded away from zero. This is healthy: at equilibrium the Discriminator should be guessing at chance (~50% accuracy → log-loss around ln(2) ≈ 0.69), and if G’s loss kept dropping it would mean G is winning the game rather than reaching equilibrium with D.

CREATE III.

Bridge from numbers to pixels

The same adversarial loop you just trained on scalars works on images — the only thing that changes is the shape of the Generator’s output and the Discriminator’s input. To make that concrete, the bridge script trains a slightly larger fully-connected GAN to produce simple 8 × 8 line patterns.

gan_patterns.py

bash Copy

python gan_patterns.py

What changes compared to Exercise 1

• The Generator outputs 64 values (an 8 × 8 flattened image) instead of a single number.
• The Discriminator sees the full flattened image instead of four statistical moments.
• The Generator uses BatchNorm and LeakyReLU — borrowed early from the DCGAN recipe — because fully-connected 64 → 64 GANs without those tricks are particularly hard to train.

Why the generated patterns are imperfect

Fully-connected layers treat each output pixel as independent. The network can’t easily express “a horizontal line” as a unit; it has to coincidentally produce 8 bright pixels in a row out of 64 unrelated outputs. Convolutional networks solve this by encoding spatial locality directly into the architecture — which is exactly the leap Lesson 12.1.2 makes with DCGAN.

Implementation note

Summary

Common pitfalls

• Forgetting .detach() when training the Discriminator on fake data — gradients leak into G and training silently misbehaves.
• Letting one network run away. If the Discriminator hits ~100% accuracy in the first few epochs, you’ve already lost; rebalance learning rates or step counts.
• Reading the loss curves like a classifier’s. Smoothly-decreasing losses in a GAN usually mean one player is dominating, not that the system is converging.
• Trying to debug instability by adding capacity to the Generator. Most early-training failures come from input normalisation, learning-rate balance, or the missing .detach() — not from the network being too small.

References