12.4.1 Neural Style Transfer

Overview

In 2015, Gatys, Ecker and Bethge [1] published a result that helped popularise generative AI long before GANs, VAEs or diffusion models reached the broader public: feed a photograph and a painting through a frozen pre-trained classifier, optimise the photograph’s pixels to match the painting’s style while preserving its own content, and out comes the photograph reimagined as a painting. The technique requires no special training, no dataset, no generative model — just an off-the-shelf VGG-19 [2] and gradient descent on pixel values.

The trick rests on a structural observation about convolutional networks. As an image passes through a deep CNN, shallow layers capture local textures and edge statistics, while deep layers capture semantic content (what objects are present and where). The Gatys algorithm uses one deep layer’s feature map as the content representation and the Gram matrices of several shallow layers as the style representation. Compose a loss from both, optimise the input pixels, and the model finds an image that simultaneously matches one image’s content and another’s style.

This is the foundation of nearly every “art filter” app you’ve used since 2016 (Prisma, DeepArt, Lensa). The optimisation-based version this lesson walks through is slow (~30 seconds per image on GPU); the production apps use the feed-forward variant Johnson et al. introduced in 2016 [3], trained once per style for real-time inference.

Learning objectives

1. Explain how a CNN’s depth separates content from style — what shallow layers encode vs deep layers.
2. Compute a Gram matrix and explain why it captures style independently of spatial layout.
3. Build the optimisation-based NST loss (content + style + optional total-variation) and run it on a photograph + painting pair.
4. Predict the trade-off between optimisation-based NST (per-image, slow, flexible) and fast feed-forward NST (per-style, fast, fixed).

Quick start — run NST on any photo and painting

The full implementation is in neural_style_transfer.py. It uses VGG-19 (auto-downloaded by torchvision the first time), L-BFGS as the optimiser, and a content image as the starting point.

pip install torch torchvision pillow
python neural_style_transfer.py \
    --content my_photo.jpg \
    --style starry_night.jpg \
    --output styled.jpg \
    --steps 300

300 L-BFGS steps takes ~30 seconds on a GPU, ~3–5 minutes on CPU. The output is the content image re-rendered in the style of the painting.

Core concepts

Concept 1 — Content and style live at different depths

A trained image classifier has to recognise what is in an image regardless of how it’s painted. To do this, its early layers learn local features (edges, colour blobs, simple textures) that don’t depend on the object’s identity, while its deep layers learn high-level abstractions (face vs car vs cat) that don’t depend on local pixel statistics. Gatys’ insight was that this depth-separation between “how it looks” and “what it is” is exactly the dichotomy artistic style transfer needs.

The content representation: the activation map at one deep convolutional layer (the original paper uses conv4_2 of VGG-19). Two images that depict the same scene have similar feature maps at this depth, even if their pixel-level details differ. So content loss is just MSE between the target image’s deep features and the content image’s deep features.

The style representation: the Gram matrices (Concept 2) of feature maps at several shallow and middle layers (conv1_1 through conv5_1). These capture local texture statistics across spatial scales. Style loss is MSE between the target’s Gram matrices and the style image’s Gram matrices, summed across layers.

The total loss to minimise:

L = α · L_content + β · L_style + γ · L_tv
                       (optional regulariser)

α and β are mixing weights (typically α = 1, β = 10⁶ because Gram-matrix values are much smaller than feature-map values). γ (typically 1e-4) controls the optional total-variation regulariser that smooths high-frequency noise.

Concept 2 — Gram matrices: style as feature co-activation

The Gram matrix is what makes the style representation work. Given a feature map F of shape (C, H × W) — C channels flattened over spatial positions — the Gram matrix is G = F · Fᵀ of shape (C, C). Each entry G[i, j] is the inner product of channel i’s activations with channel j’s activations across all spatial positions.

What this captures: which channels tend to fire together. In a CNN trained on natural images, individual channels respond to specific visual features (a particular edge orientation, a particular colour, a particular texture). The Gram matrix captures the correlations between these features — which orientations co-occur with which colours, which textures appear alongside which others.

Crucially, the Gram matrix is spatially invariant. By summing over all H × W positions, it discards information about where a feature appears and keeps only how much of each pairwise co-activation the image contains. This is why two images with the same style but different content have similar Gram matrices: their local texture statistics are similar even though their objects are different.

def gram_matrix(features: torch.Tensor) -> torch.Tensor:
    """Channel-wise Gram matrix; captures feature co-activation statistics."""
    batch, channels, h, w = features.size()
    flat = features.view(batch * channels, h * w)
    return (flat @ flat.t()) / (batch * channels * h * w)

The normalisation by (batch · channels · h · w) makes the Gram matrix scale-invariant across different image sizes — important because the style image and the target image may not have the same spatial dimensions.

Concept 3 — Optimisation in pixel space (not weight space)

Here’s the part that distinguishes NST from every other “deep learning generates images” technique you’ve seen. There is no trained generator network. The VGG-19 is frozen — its weights never change. What gets optimised is the image pixels themselves.

target = content_image.clone().requires_grad_(True)
optimizer = torch.optim.LBFGS([target], max_iter=1)

for step in range(num_steps):
    optimizer.step(lambda: compute_loss(target, ...))

Every gradient step updates the pixel values of target directly to reduce the combined content+style loss. L-BFGS is the optimiser of choice because the problem is small (one image worth of pixels, not a network’s worth of weights), low-dimensional enough that second-order methods are tractable, and converges much faster than first-order methods like Adam for this specific formulation.

Initialisation matters. Starting from the content image (the choice in the reference implementation above) converges fastest and produces outputs that strongly preserve content. Starting from random noise produces more abstract, “painterly” outputs that emphasise style over recognisability. Starting from the style image itself rarely produces useful results — the optimiser gets stuck in a basin near the style image.

Exercises

python neural_style_transfer.py \
    --content path/to/landscape.jpg \
    --style   path/to/starry_night.jpg \
    --output  landscape_starry.jpg \
    --steps 300

import torch
import torch.nn.functional as F

def gram_matrix(features: torch.Tensor) -> torch.Tensor:
    """
    Compute the Gram matrix of a feature map.

    TODO:
    1. Get batch, channels, h, w from features.shape
    2. Reshape features to (batch*channels, h*w)
    3. Compute Gram: F @ F.T
    4. Normalise by (batch*channels*h*w)
    """
    pass

def style_loss(target_features: dict, style_grams: dict, layers: set) -> torch.Tensor:
    """
    Sum MSE between target's Gram matrices and reference style Gram matrices
    across all style layers.

    TODO:
    1. For each layer in `layers`:
       a. Compute target's Gram matrix from target_features[layer]
       b. Compute MSE between that and style_grams[layer]
    2. Return the sum
    """
    pass

# Quick test
if __name__ == "__main__":
    features = torch.randn(1, 64, 32, 32)
    g = gram_matrix(features)
    assert g.shape == (64, 64), f"Expected (64, 64), got {g.shape}"
    print("gram_matrix test passed")

    fake_target = {"conv_1": torch.randn(1, 64, 16, 16),
                   "conv_2": torch.randn(1, 128, 8, 8)}
    fake_style  = {"conv_1": torch.randn(64, 64),
                   "conv_2": torch.randn(128, 128)}
    loss = style_loss(fake_target, fake_style, {"conv_1", "conv_2"})
    print(f"style_loss test value: {loss.item():.4f}")

import torch
import torch.nn.functional as F

def gram_matrix(features: torch.Tensor) -> torch.Tensor:
    batch, channels, h, w = features.size()
    flat = features.view(batch * channels, h * w)
    return (flat @ flat.t()) / (batch * channels * h * w)

def style_loss(target_features: dict, style_grams: dict, layers: set) -> torch.Tensor:
    return sum(
        F.mse_loss(gram_matrix(target_features[k]), style_grams[k])
        for k in layers
    )

def total_variation_loss(img: torch.Tensor) -> torch.Tensor:
    dh = (img[:, :, 1:, :] - img[:, :, :-1, :]).pow(2).mean()
    dw = (img[:, :, :, 1:] - img[:, :, :, :-1]).pow(2).mean()
    return dh + dw

# In the main loop:
total = content_w * c + style_w * s + 1e-4 * total_variation_loss(target)

Summary

Common pitfalls

• Forgetting requires_grad_(True) on the target image — gradient descent has nothing to optimise.
• Forgetting requires_grad_(False) on the VGG-19 weights — you’ll waste memory tracking gradients you’ll never use.
• Style weight too small (e.g. 1.0 instead of 1e6) — output looks identical to the content image.
• Using a different CNN than VGG-19 without recalibrating the layers — ResNet or DenseNet have different layer numbering and feature statistics, and the standard conv_4/conv_1..5_1 choices won’t transfer.
• Starting optimisation from random noise expecting fast convergence — works eventually but takes ~3× longer than starting from the content image.

References