9.2.1 Feedforward Networks

Big question

If a single hidden layer can approximate any continuous function (Universal Approximation Theorem, 1989), why does anybody build deeper networks? The answer is that width and depth are different resources. A shallow but wide network needs exponentially many neurons to express the same function a deep narrow network captures in polynomially many [1]. Depth gives you compositional structure: each layer learns features built from features below. This is the central insight of modern deep learning — the network “discovers” useful intermediate representations on its own.

Learning objectives

1. Generalise the 2-2-1 backprop code from 9.1.2 to a configurable layer list [2, 8, 8, 1].
2. Train a 2-8-8-1 network on non-XOR datasets (spirals, moons, concentric circles).
3. Visualise hidden-layer activations as a learned representation of the input.
4. Compare depth (more layers) vs width (more neurons per layer) on the same problem.

Part 1 — A configurable forward pass

The 2-2-1 network of 9.1.2 hard-coded two weight matrices. Generalise to any architecture:

class MLP:
    def __init__(self, layers, seed=0):
        rng = np.random.default_rng(seed)
        self.Ws, self.bs = [], []
        for i in range(len(layers) - 1):
            # He init for ReLU
            self.Ws.append(rng.normal(0, np.sqrt(2/layers[i]),
                                       size=(layers[i+1], layers[i])))
            self.bs.append(np.zeros(layers[i+1]))

    def forward(self, x):
        a = x
        cache = [x]
        for W, b in zip(self.Ws[:-1], self.bs[:-1]):
            z = W @ a + b
            a = np.maximum(0, z)        # ReLU hidden
            cache.append(a)
        # final layer with sigmoid for binary classification
        z = self.Ws[-1] @ a + self.bs[-1]
        y = 1 / (1 + np.exp(-z))
        cache.append(y)
        return y, cache

layers=[2, 8, 8, 1] gives a 2-input, two-hidden-layer, 1-output network with 8 neurons each in the hidden layers.

Part 2 — Depth as composition

Each hidden layer applies its own non-linear transformation. The first hidden layer carves the input space into half-spaces (just like a perceptron); the second hidden layer combines those half-spaces into more complex regions; and so on. With ReLU activations, each additional layer can roughly double the number of linear regions the network expresses [2].

Part 3 — Learned representations

The hidden activations are the learned representation. For a 2-8-8-1 network trained on a spiral dataset, plotting the 8-dimensional hidden activations (projected to 2D via PCA) reveals that the network has un-twisted the spiral — in hidden space the two classes are linearly separable, exactly the property the output layer needs.

Synthesis project

CREATE III.

Decision-boundary evolution GIF

Generate a sequence of decision-boundary images at epochs 0, 100, 300, 1000, 3000, and assemble into a GIF showing the network’s learning trajectory.

Implementation hint

import imageio

frames = []
for epoch in [0, 100, 300, 1000, 3000]:
    train_until(net, X, y, epoch)
    img = render_boundary(net)
    frames.append(img)
imageio.mimsave('evolution.gif', frames, fps=2)

The fun of this exercise is watching the boundary morph. Networks don’t go directly from “flat” to “perfect” — they hit recognisable intermediate stages: a single dividing line, a more complex curve, the rough spiral shape, then refinement.

Downloads

References