12.1.4 Pix2Pix Applications

Overview

DCGAN Lesson 12.1.2 generates fabric patterns from random noise. StyleGAN Lesson 12.1.3 generates them with controllable disentangled latents. Both are unconditional — the input is noise, and the only thing controlling the output is which random seed you picked.

Pix2Pix, introduced by Isola, Zhu, Zhou and Efros at CVPR 2017 [1], does something structurally different. The input is no longer noise — it’s another image. The model learns to translate input images into output images while preserving spatial structure. Feed it a black-and-white line drawing, get a colourised version. Feed it a building’s segmentation label, get a photorealistic facade. Feed it a daytime satellite image, get the night-time version. The same architecture handles all of these tasks; what changes is the paired training data.

This lesson works through two demonstrations on the same architecture: a pre-trained facades model (label-to-photo, ready to use) for Exercises 1 and 2, and an optional from-scratch retrain on the Anime Sketch Colorization Pair dataset from Kaggle [2] for Exercise 3. The underlying idea — conditional GAN, U-Net generator, PatchGAN discriminator, L1+adversarial loss — generalises to almost any image-to-image translation task you can find paired data for.

Learning objectives

1. Distinguish conditional from unconditional GANs and explain what the conditioning input adds to the training signal.
2. Build a U-Net generator and explain why the skip connections are non-negotiable for pixel-aligned output.
3. Understand the PatchGAN discriminator’s patch-level real/fake scoring and predict when it outperforms a full-image discriminator.
4. Combine adversarial loss with L1 reconstruction loss and tune λ for sharpness vs accuracy trade-off.

Quick start — translate a facade label into a building photo

The pre-trained facades_label2photo weights are on the GitHub Release. Place under checkpoints/, load with the helper script, run on one of the sample facade labels.

import torch
from facades_generator import create_facades_generator
from PIL import Image
import numpy as np
import matplotlib.pyplot as plt

# Load pre-trained facades U-Net generator
generator = create_facades_generator('checkpoints/generator_weights.pth')
generator.eval()

# Load a CMP Facades segmentation label
label = Image.open('sample_facades/base/cmp_b0001.png').convert('RGB').resize((256, 256))
label_arr = np.array(label).astype(np.float32) / 255.0
x = torch.from_numpy((label_arr - 0.5) / 0.5).permute(2, 0, 1).unsqueeze(0)

with torch.no_grad():
    facade = generator(x)

facade = ((facade.squeeze().permute(1, 2, 0).numpy() + 1) / 2).clip(0, 1)
plt.imsave('quick_start_output.png', facade)

The input is a coloured segmentation label (different colours = walls, windows, doors, sky). The output is a photorealistic building photo that follows the label’s structure but fills it with realistic textures.

Core concepts

Concept 1 — Conditional vs unconditional generation

The structural shift from DCGAN to Pix2Pix is that the input vector is no longer arbitrary noise but a meaningful image. Mirza and Osindero’s 2014 paper [3] introduced the conditional GAN framework in the abstract — feed any conditioning information c alongside the noise — but Pix2Pix specialised it to a powerful case: when c is itself an image, you get image-to-image translation.

# Unconditional GAN (DCGAN)
G(z) → image                    # z = noise, output is random

# Conditional GAN with image conditioning (Pix2Pix)
G(x) → image                    # x = input image, output is structured translation

The conditioning shifts what the generator has to learn. Instead of “produce something that looks like a real image,” the task becomes “produce the output image that corresponds to this input image.” Training data becomes paired — every input must have a known correct output — but the supervision is much stronger as a result. The model can converge faster and produce more controllable results, in exchange for needing paired data.

This same idea later evolved into the conditioning patterns used by Stable Diffusion (text-to-image, where the condition is a text embedding rather than an image) and ControlNet (Lesson 12.3.2, where structural conditioning is layered onto an existing text-to-image model).

Concept 2 — U-Net generator and the necessity of skip connections

Pix2Pix’s generator is a U-Net [5] — the same encoder-decoder-with-skip-connections architecture used in Lesson 12.3.1’s DDPM for noise prediction. For image-to-image translation, the skip connections are not a tuning choice; they’re a structural requirement.

Consider what happens without skip connections. The encoder downsamples 256×256 → 128 → 64 → 32 → 16 → 8 → 4 → 2 → 1 (a 1×1 bottleneck with all the input information compressed into one vector). The decoder then has to reconstruct a 256×256 output that aligns pixel-for-pixel with the input. Without skip connections, every spatial detail has to survive the bottleneck round-trip — and most of it doesn’t. The output ends up blurry and misaligned with the input lines.

Skip connections fix this by piping each encoder layer’s output directly to the corresponding decoder layer at the same resolution. Fine detail bypasses the bottleneck entirely; the bottleneck only has to encode the high-level transformation (this label means “wall”, this means “window”), while the skip connections carry the spatial layout that the output must respect.

def forward(self, x):
    # Encoder — store every output for skip connections
    d1 = self.down1(x)    # 256 → 128
    d2 = self.down2(d1)   # 128 → 64
    d3 = self.down3(d2)   # 64 → 32
    d4 = self.down4(d3)   # 32 → 16
    d5 = self.down5(d4)   # 16 → 8
    d6 = self.down6(d5)   # 8 → 4
    d7 = self.down7(d6)   # 4 → 2
    d8 = self.down8(d7)   # 2 → 1 (bottleneck)

    # Decoder — concatenate corresponding encoder feature at each step
    u1 = self.up1(d8, d7)
    u2 = self.up2(u1, d6)
    u3 = self.up3(u2, d5)
    u4 = self.up4(u3, d4)
    u5 = self.up5(u4, d3)
    u6 = self.up6(u5, d2)
    u7 = self.up7(u6, d1)

    return self.final(u7)  # → 256x256

Concept 3 — PatchGAN: discriminator that scores patches, not images

A standard GAN discriminator outputs a single scalar per image — real or fake. For high-resolution image-to-image translation, this single-scalar signal is too coarse: the discriminator can be fooled by a globally plausible image that has local artefacts (smeared windows, misaligned tiles, fuzzy edges).

Pix2Pix’s PatchGAN discriminator outputs a grid of real/fake predictions — typically 30×30 — where each prediction corresponds to a 70×70 pixel patch in the input. The generator is now penalised on every patch, not just the overall image. This concentrates training pressure on local texture quality, which is exactly where the L1 loss (Concept 4) is weakest.

The discriminator receives both the input condition and the output image, concatenated along the channel dimension. This lets it check not just “is this output realistic?” but “is this output a valid translation of the given input?”

def forward(self, sketch, target):
    x = torch.cat([sketch, target], dim=1)   # 3 + 3 = 6 channels
    return self.model(x)                     # → (batch, 1, 30, 30)

Three practical advantages stack up: detail-awareness (more sensitive to local artefacts), denser training signal (one loss value per patch, not per image), and parameter efficiency (the discriminator can be smaller because it only ever processes a single patch’s receptive field at a time).

Concept 4 — L1 + adversarial loss: accuracy from L1, sharpness from the GAN

Pix2Pix’s loss combines two complementary terms:

L_total = L_GAN(G, D) + λ · L_L1(G)

Adversarial loss is the standard GAN objective: generator tries to fool the discriminator, discriminator tries to catch fakes. This term rewards perceptual realism — sharpness, plausible textures, no obvious artefacts.

L1 reconstruction loss is the pixel-wise absolute difference between the generator’s output and the ground-truth output: L_L1 = E[|y - G(x)|]. This term rewards literal pixel accuracy. The L1 norm (rather than L2) is chosen because it’s less sensitive to outliers and produces less blurry results than L2 would.

Why both? Each fixes the other’s weakness. Pure adversarial loss produces sharp but spatially-imprecise outputs — the generator can satisfy the discriminator with a realistic image that doesn’t match the input. Pure L1 loss produces blurry but spatially-precise outputs — the L1 minimum at any pixel is the average of all plausible colours, which is grey-ish. Combining the two with λ = 100 (the conventional default in the original paper) lets the L1 pin the output to the right location and the GAN sharpen the textures within that location.

Exercises

EXECUTE I.

Run the pre-trained facades model on sample labels

generator_weights.pth — 208 MB U-Net generator exercise1_observe.py — multi-facade comparison facades_generator.py — generator loader helper

Place the generator checkpoint at checkpoints/facades_generator.pth, then:

bash Copy

python exercise1_observe.py

Output: exercise1_output.png — a grid showing several facade segmentation labels alongside their generated photo equivalents. The CMP Facades dataset [6] is also on Kaggle if you want more inputs to try.

Reflection

1. The generated photos all share a particular visual style — warm beige walls, dark wooden doors, regular window grids. Where does that style come from?
2. The same facade label decoded twice with the same model produces the same output. How does this differ from DCGAN, where the same noise vector also produces the same output?
3. What would happen if you fed a label that wasn’t a facade (e.g. a random colour blob) into this generator?
4. The PatchGAN discriminator (Concept 3) outputs a 30×30 grid of patch scores. How does this granularity affect what the generator learns to do?

Discussion

1. The CMP Facades training dataset is dominated by buildings from a specific architectural style (mostly Central European 19th/20th-century apartment buildings). The model has learned the visual statistics of that dataset, not facades-in-general; train it on Japanese houses and you’d get a different style.

2. Mechanically the same — both are deterministic forward passes through a frozen network. But conceptually different: DCGAN’s input is meaningless noise that maps onto a random point in the learned distribution; Pix2Pix’s input is a meaningful label that maps onto a specific translation of that label.

3. The model would produce something — but it would be garbage. The model has been trained only on inputs from the segmentation-label distribution; outside that distribution its behaviour is undefined. You’d get textures vaguely reminiscent of facades but composed nonsensically.

4. The generator must produce output that looks realistic at every 70×70-patch scale, not just globally. This is why Pix2Pix outputs have crisp local textures (the patch discriminator catches local artefacts) but can have slight global inconsistencies (the patch discriminator can’t see across patches).

MODIFY II.

Modify labels and see how output changes

exercise2_explore.py — label-modification explorations

bash Copy

python exercise2_explore.py

Three explorations:

Goal 1 — Window count. Modify a facade label to add or remove windows; observe how the generator responds. The output should track the input’s structural changes.

Goal 2 — Colour palette injection. Try modifying the label colour-coding to see how strict the model is about mapping label colours to specific materials.

Goal 3 — Partial labels. Provide a label with large unlabelled regions; observe how the model fills them in.

What to expect — window modifications

The generator follows the input strictly — adding a window in the label adds a window in the output, in the right location. Spatial alignment is one of Pix2Pix’s strongest properties; the skip connections + L1 loss combination is what enforces it.

What to expect — colour palette

The colour-to-material mapping is learned from the training data, which used a specific colour code. If you stay within that code’s vocabulary, the model produces clean textures. Inject colours from outside the training distribution and you get ambiguous outputs — the model has no learned mapping for that input colour, so it produces something in-between materials.

What to expect — partial labels

Where labels are absent, the model produces output that approximates the average of plausible completions given the surrounding context. This is the same blurriness mechanism as VAE outputs (Lesson 12.2.1) — when the model is uncertain, MSE-like losses converge to the mean of valid options.

TRAIN III.

Train Pix2Pix from scratch on anime sketch colourisation

This exercise trains a full Pix2Pix from scratch on the Anime Sketch Colorization Pair dataset from Kaggle [2] — ~17,000 paired sketch/colour images.

exercise3_train.py — full Pix2Pix training (100 epochs, GPU) pix2pix_model.py — U-Net generator + PatchGAN discriminator preprocess_anime_sketches.py — Kaggle dataset preprocessor

Step 1 — Get the dataset. Visit kaggle.com/datasets/ktaebum/anime-sketch-colorization-pair, download (~14 GB compressed; free Kaggle account required), extract to anime_sketch_dataset/, then run:

bash Copy

python preprocess_anime_sketches.py

This splits paired images into anime_processed/sketches/ and anime_processed/colors/ so the training script can find them.

Step 2 — Train.

bash Copy

python exercise3_train.py

Configuration:

Hyperparameter	Value
Image size	256×256
Batch size	1 (the original paper’s choice)
Generator	U-Net with 8 down/up blocks
Discriminator	PatchGAN, 70×70 receptive field
Optimiser	Adam, lr=2e-4, β₁=0.5
Loss weight	λ_L1 = 100
Epochs	100
Estimated time	~3-5 hours on RTX 5070Ti

Sample images saved every epoch in training_results/. Loss curves and visual progression help diagnose training health.

What to watch for during training

• L1 loss should decrease monotonically; if it spikes, learning rate is too high.
• Adversarial losses should oscillate around a non-zero plateau — both reaching zero is a bad sign (discriminator dominated, generator stopped improving).
• Sample images at epoch 25 should already show recognisable colour assignment; if they don’t, dataset preprocessing may be off.
• Memory: ~6 GB GPU at batch size 1. Lower if needed; quality holds.

Issue triage

Dataset extraction creates wrong folder structure — the Kaggle archive sometimes nests deeper than expected; the preprocessor expects anime_sketch_dataset/data/train/ with paired sketch+colour images side-by-side.

Loss oscillates wildly — discriminator is too strong. Try training the generator twice per discriminator step.

Outputs are correctly coloured but blurry — λ_L1 is too high; try 50 instead of 100.

Outputs are sharp but spatially misaligned — λ_L1 is too low; raise it.

Make it your own

After training, run the model on sketches you draw yourself. The model has seen ~17,000 anime sketches and generalises reasonably well to similar styles. For very different sketch styles (e.g., realistic line drawings), expect degraded results — that’s the limit of supervised translation: it knows the distribution it was trained on, not the universe of possible sketches.

Summary

Common pitfalls

• Forgetting that paired data is required. Pix2Pix supervises pixel-by-pixel — without aligned ground truth, the L1 loss is undefined and the model can’t train.
• Resizing input and output to different dimensions. The U-Net expects matching input/output resolutions; mismatch breaks the skip-connection alignment.
• Setting λ_L1 = 0 to try “pure adversarial” — the result is sharp but spatially garbage. The L1 term is what enforces the correspondence.
• Training with a batch size that’s too large. Pix2Pix’s original paper used batch size 1; PatchGAN works best when each forward pass sees one example fully rather than averaging across a batch.
• Using a single-scalar discriminator. Without PatchGAN’s patch-level scoring, local artefacts go unpenalised and outputs look smeared up close.

References