12.3.1 DDPM Basics

Overview

By 2020, GANs had dominated generative image modelling for six years. Then Ho, Jain and Abbeel published “Denoising Diffusion Probabilistic Models” [1], achieved competitive image quality without adversarial training, and within two years diffusion-based architectures had displaced GANs as the dominant approach. Stable Diffusion [2], DALL-E 2 [3], Imagen, and Midjourney all use variants of the same core mechanism.

The trick is structurally elegant. Start with a clean image, add Gaussian noise in 1,000 small steps until the image is pure noise, and you have a forward process whose math is closed-form. Now train a neural network to reverse one step of that process: given a noisy image at timestep t, predict the noise component. Once trained, sample new images by running the reverse process from t = 999 down to t = 0, starting from pure noise and denoising step by step. No adversarial training, no mode-collapse risk, no carefully-balanced discriminator — just MSE on noise prediction.

This lesson trains a DDPM on the same African fabric dataset used in Lesson 12.1.2 (DCGAN) and Lesson 12.1.3 (StyleGAN), enabling a direct three-way comparison of the three dominant generative paradigms on identical data. Training takes 15–20 hours on a modern GPU; the pre-trained checkpoint is on a GitHub Release for everyone who’d rather skip to generation.

Learning objectives

1. Explain the forward diffusion process: what β_t controls, why ᾱ_t matters, and why the closed-form jump from x_0 to x_t is what makes training tractable.
2. Describe the reverse-process training objective in one sentence (predict the noise) and the architecture (U-Net with timestep conditioning, self-attention, residuals) that makes it work.
3. Sample new images by running the trained reverse process; tune sampling-step count via DDIM and predict the quality trade-off.
4. Compare DDPM’s training stability and mode coverage against DCGAN and StyleGAN on the same dataset.

Quick start — load checkpoint, sample 4 fabrics

Download the pre-trained checkpoint from the GitHub Release. The denoising-diffusion-pytorch library handles the U-Net plumbing; we just point it at the weights.

from denoising_diffusion_pytorch import Unet, GaussianDiffusion
from torchvision.utils import save_image
import torch

model = Unet(dim=64, dim_mults=(1, 2, 4, 8), channels=3)
diffusion = GaussianDiffusion(
    model, image_size=64, timesteps=1000, sampling_timesteps=250,
)

ckpt = torch.load('models/ddpm_african_fabrics.pt', map_location='cpu')
ema = {k.replace('ema_model.', ''): v
       for k, v in ckpt['ema'].items() if k.startswith('ema_model.')}
diffusion.load_state_dict(ema)
diffusion.eval()

with torch.no_grad():
    samples = diffusion.sample(batch_size=4)        # [4, 3, 64, 64]
samples_up = torch.nn.functional.interpolate(samples, scale_factor=4, mode='nearest')
save_image(samples_up, 'quick_start.png', nrow=2, padding=4, pad_value=1)

That’s four fabric patterns generated from pure noise via 250 DDIM steps. Each step is one U-Net forward pass; the whole sample takes ~10 seconds on CPU, sub-second on GPU.

Core concepts

Concept 1 — Why a reversed corruption process beats adversarial training

GANs train a generator against a discriminator and hope the equilibrium is favourable. In practice, that equilibrium is fragile: mode collapse, vanishing gradients, oscillation between the two networks (see Lesson 12.1.1). The trade-off you accept is fundamental — adversarial training is what gives GANs their sharpness, and it’s also what makes them temperamental.

Diffusion takes the opposite design stance. There’s no adversary. The training objective is mean squared error on a noise-prediction task, the same flavour of loss used in any regression problem. The training dynamics are correspondingly boring: loss decreases monotonically, no instabilities, no balance to maintain. The trade-off you accept is slow sampling — 250–1,000 forward passes through the network per generated image, versus a GAN’s one.

Sohl-Dickstein et al.’s 2015 paper [4] introduced the theoretical framework borrowing from non-equilibrium thermodynamics; Ho, Jain and Abbeel’s 2020 paper [1] made it work on natural images. The five-year gap is a useful reminder that “elegant theory” and “competitive system” are not the same thing.

Concept 2 — Forward diffusion: noise schedule and the closed-form shortcut

The forward process gradually adds Gaussian noise to a clean image x_0 over T = 1,000 timesteps. At each step:

q(x_t | x_{t-1}) = N(x_t; √(1 - β_t) · x_{t-1}, β_t · I)

β_t is the noise schedule — small per-step values (typically 0.0001 to 0.02) that control how much fresh noise gets mixed in. The √(1 - β_t) factor lightly shrinks the previous image to keep variance bounded.

The closed-form shortcut is what makes training efficient. Because the noise additions are independent Gaussians, you can sample x_t directly from x_0 without iterating through t - 1 intermediate steps:

q(x_t | x_0) = N(x_t; √(ᾱ_t) · x_0, (1 - ᾱ_t) · I)

where ᾱ_t = ∏ᵗ_{s=1}(1 - β_s) is the cumulative product. This is the reason training is fast: each training step samples a random t, jumps directly to x_t in one closed-form operation, asks the network to predict the noise, and computes one gradient. No sequential rollout.

The choice of schedule matters. The linear schedule in the original DDPM paper aggressively destroys signal by t ≈ 250; the cosine schedule introduced by Nichol and Dhariwal [5] preserves signal longer at middle timesteps, which means the model gets more informative gradients during training.

Concept 3 — Reverse diffusion: U-Net predicts the noise

Generation reverses the process. Start at x_T ~ N(0, I) (pure noise); at each timestep, ask the network to predict the noise component, subtract a scaled version, and arrive at x_{t-1}. Repeat 1,000 times — or, with DDIM acceleration, 250 times — and land at a clean image x_0.

The training objective is one of the cleanest in modern deep learning:

L = E_{t, x_0, ε} [ ‖ε - ε_θ(√(ᾱ_t)·x_0 + √(1-ᾱ_t)·ε, t)‖² ]

Read inside-out: take a clean x_0, corrupt it to timestep t using the closed-form shortcut, ask the network ε_θ to predict the noise ε that was added, and minimise the squared error. The trained network learns a noise-prediction function conditioned on the timestep.

The architecture is a U-Net [6] — encoder-decoder with skip connections at every resolution level. The skips matter because noise prediction requires pixel-level output detail; without them, the bottleneck would discard the high-frequency information the decoder needs to reconstruct.

Four architectural details matter for diffusion specifically:

1. Timestep conditioning — sinusoidal embeddings encode the integer timestep as a vector; every block receives this signal so the network can specialise behaviour by noise level.
2. Self-attention [7] — at middle and bottom resolutions, attention layers let each spatial position see every other; essential for long-range coherence (a fabric pattern’s colour palette must agree across the whole image).
3. Residual connections [8] — every conv block computes output = block(x) + x; gradients flow through the shortcut, training a deep network is tractable.
4. Group normalisation [9] — diffusion uses small batches due to memory; BatchNorm statistics become noisy below ~16 samples per batch, so GroupNorm (which normalises across channel groups within a single image) is the safer default.

Concept 4 — DDIM: turning 1000 steps into 250 (or 50)

Sampling 1,000 sequential U-Net forward passes per image is slow. Song, Meng and Ermon’s DDIM [10] (Denoising Diffusion Implicit Models) showed that a deterministic version of the reverse process can skip many of those steps. The same trained DDPM model works — only the sampling procedure changes.

The intuition: in the original DDPM sampler, each step has stochasticity that compounds over the trajectory. DDIM removes that stochasticity and chooses a sparser timestep schedule (e.g., every 4th step) to integrate the deterministic reverse process. With 250 DDIM steps you get quality comparable to 1,000 DDPM steps; with 50 steps you get usable but visibly softer output.

diffusion = GaussianDiffusion(
    model, image_size=64,
    timesteps=1000,                # training schedule (unchanged)
    sampling_timesteps=250,        # DDIM accelerated sampling
)

DDIM is what makes diffusion practical for interactive use. The newer Flow Matching family (Lesson 12.7.1) takes the trajectory-straightening idea further — straight paths from noise to data, integrable in 20–50 Euler steps without retraining the model differently.

Exercises

pip install denoising-diffusion-pytorch torch torchvision
python exercise1_generate.py

MODIFY II.

Sampling steps and denoising visualisation

exercise2_explore.py — step-count comparison + denoising animation

bash Copy

python exercise2_explore.py

Produces exercise2_sampling_comparison.png (quality at different step counts) and exercise2_denoising.gif (a single sample’s reverse process visualised step by step).

Goal 1 — Step count sweep. Try sampling_timesteps ∈ {50, 100, 250, 1000}. The relationship is sub-linear: doubling steps from 250 → 500 produces almost no visible improvement.

Goal 2 — Different seeds. Change RANDOM_SEED to see how the starting noise affects output. Two close seeds give similar outputs; very different seeds give unrelated ones.

Goal 3 — Custom DDIM schedules. Read the source of denoising_diffusion_pytorch.GaussianDiffusion.ddim_sample to see how the timestep selection works. Try a logarithmic instead of uniform spacing to see if it changes anything.

What to expect — step count

50 steps: usable, slightly soft. 100 steps: comfortable default for fast iteration. 250 steps: the production default. 1,000 steps: marginally sharper, 4× the compute. The diminishing returns curve flattens hard around 250.

What to expect — different seeds

Two seeds that differ by 1 produce visually similar but distinguishable patterns. Two distant seeds produce unrelated patterns. The noise space is continuous in the sense that nearby points decode to nearby images, which is what makes the morphing animation in Exercise 3 possible.

python exercise3_morph.py

python exercise3_train.py

Summary

Common pitfalls

• Confusing training timesteps with sampling timesteps. timesteps=1000 is fixed by training; sampling_timesteps=250 is your inference-time choice.
• Running with the wrong checkpoint key. The downloadable .pt contains both the trained weights and the EMA-tracked copy; you usually want the EMA version (ema_model.*), which is what the quick-start code extracts.
• Skipping the eval() mode call before sampling — BatchNorm/Dropout in training mode produces noisier outputs.
• Trying to interpret the trained model’s outputs as “average of training images.” It’s not — it’s samples from the learned distribution, which is structurally different from the empirical training distribution.
• Comparing FID scores between DDPM, DCGAN, and StyleGAN runs at different image resolutions or with different image-preprocessing pipelines. The numbers are only comparable under matched conditions.

References