9.1.3 Activation Functions as Art

Big question

What does the activation function actually do? Without it, a neural network is just a stack of linear transforms — and any stack of linear transforms collapses into a single linear transform [1]. The activation introduces non-linearity, the one ingredient that lets the network learn curved decision boundaries, image features, and language structure. Choosing the activation function is one of the few neural-network decisions where the field has reached near-consensus: ReLU (or its variants) dominate, with sigmoid and tanh largely retired to recurrent networks and output layers [2].

Learning objectives

1. Plot the curve and derivative of each canonical activation function.
2. Explain why ReLU (max(0, x)) became standard around 2012 — sparsity, no saturation, fast gradient.
3. Apply each activation pixel-wise to an image and observe how it transforms the value distribution.
4. Connect the activation choice to the vanishing gradient problem in deep networks.

Part 1 — Why non-linearity matters

Without an activation, a multi-layer network reduces to a single matrix multiplication:

y = W₃ (W₂ (W₁ x)) = (W₃ W₂ W₁) x = W' x

The composition of linear functions is linear. To learn anything more interesting than a hyperplane separator, you need a non-linear function between layers. Any non-linear function works in principle; in practice, the seven canonical activations balance three concerns:

• Differentiability — for backpropagation.
• Saturation — sigmoid and tanh squash inputs into bounded ranges, which causes gradients to vanish for large |x|.
• Computational cost — max(0, x) is one comparison; tanh(x) is several transcendental operations.

Part 2 — The canonical functions

import numpy as np

def sigmoid(x):     return 1 / (1 + np.exp(-x))
def tanh(x):        return np.tanh(x)
def relu(x):        return np.maximum(0, x)
def leaky_relu(x, alpha=0.01): return np.where(x > 0, x, alpha * x)
def elu(x, alpha=1.0): return np.where(x > 0, x, alpha * (np.exp(x) - 1))
def gelu(x):        return 0.5 * x * (1 + np.tanh(np.sqrt(2/np.pi) * (x + 0.044715 * x**3)))
def swish(x):       return x * sigmoid(x)

The historical sequence is instructive:

The ReLU paper of Glorot, Bordes & Bengio (2011) was the inflection point — ReLU-based networks trained faster and reached lower loss than sigmoid networks, on the same data and architecture. By 2012’s AlexNet, ReLU was standard [3, 5].

Part 3 — Apply each to an image

A useful way to see what each activation does is to apply it pixel-wise to a normalised image. The transformations are striking:

The effect on real images:

• Sigmoid — compresses everything into the middle of the brightness range. The image looks low-contrast.
• Tanh — same shape, signed. Slightly more contrast than sigmoid.
• ReLU — anything below zero (after centring) → black. Sharp half-clip effect.
• Leaky ReLU — same as ReLU but with very dim negatives instead of pure black.
• ELU — like leaky ReLU but with smoother negative tail.
• GELU/Swish — a tiny negative slope that grows smoothly into positive identity. Hard to distinguish visually from ReLU at first glance, but training dynamics differ.

Synthesis project

CREATE III.

Aurora — the activation as art

Build an artistic composition using activation functions: generate a smooth field (e.g. fbm noise from Module 06), apply different activations to different regions, and combine into one image.

Approach

import numpy as np
# (assume fbm() helper available from Module 6)

H, W = 400, 800
noise = fbm((H, W), scale=120, octaves=6)
noise = (noise - 0.5) * 4   # centre around 0 with width

# Apply tanh to make smooth banded structure
shaped = np.tanh(noise * 2)

# Map to colour
rgb = np.zeros((H, W, 3))
rgb[..., 0] = np.clip(shaped * 0.3 + 0.2, 0, 1)
rgb[..., 1] = np.clip(shaped * 0.8 + 0.4, 0, 1)
rgb[..., 2] = np.clip(0.6 + shaped * 0.3, 0, 1)

The activation is the artistic knob: tanh gives smooth bands; ReLU gives sharp half-bands; sigmoid gives soft horizons. Each gives a different aurora-like aesthetic.

Downloads

References