9.2.3 Recurrent Networks

Big question

How does a neural network handle data that arrives one item at a time, where the meaning of each item depends on what came before? Text is the canonical example: the meaning of “bank” depends on whether we previously said “river” or “money.” A feedforward network has no way to remember previous inputs. Recurrent Neural Networks (RNNs) maintain a hidden state vector that is updated at every time step, carrying information forward [1]. The vanilla RNN of 1986 became the LSTM of 1997 [2], the GRU of 2014 [3], and eventually the Transformer of 2017 [4]. The recurrent idea has evolved, but it began here.

Learning objectives

1. Express an RNN as a function h_t = tanh(W_x x_t + W_h h_{t-1} + b) that maps an input + previous state to a new state.
2. Unroll the recurrence in time and apply backpropagation through time (BPTT).
3. Train a character-level RNN on a small text corpus and generate plausible-sounding output.
4. Identify the vanishing-gradient problem that motivated LSTMs and GRUs.

Part 1 — The recurrent unit

A vanilla RNN cell maps an input vector x_t and a previous hidden state h_{t-1} to a new hidden state:

def rnn_step(x_t, h_prev, W_x, W_h, b):
    return np.tanh(W_x @ x_t + W_h @ h_prev + b)

The same weights W_x, W_h, b are reused at every time step. Computing the output sequence is a simple loop:

def forward(self, x_seq):
    h = np.zeros(self.hidden_size)
    states = [h]
    for x_t in x_seq:
        h = rnn_step(x_t, h, self.W_x, self.W_h, self.b)
        states.append(h)
    return states

The hidden state is the network’s memory — it carries information about everything seen so far. The capacity of that memory is limited by the dimensionality of h; bigger hidden vectors hold more information [5].

Part 2 — Unrolling and backpropagation through time

To train, we treat the RNN as a feedforward network with T layers (one per time step), all sharing the same weights. Backpropagation propagates gradients backward through every step:

∂L/∂W_h = Σ_t  ∂L_t/∂h_t · ∂h_t/∂W_h

This is Backpropagation Through Time (BPTT) [1]. The catch: gradients can vanish or explode exponentially as they propagate backward through many time steps, because the same recurrence matrix W_h is multiplied repeatedly. For T = 100 steps, even a small spectral radius error compounds.

Part 3 — Character-level language model

The classic RNN demo (Karpathy 2015) is a character-level language model: input one character, predict the next. Train on Shakespeare or Linux kernel source, and the network generates eerily plausible text [6].

# Forward pass for one input character
h = rnn_step(one_hot(char), h, W_x, W_h, b)
logits = W_y @ h + b_y
probs = softmax(logits)
next_char = np.random.choice(len(vocab), p=probs)

After enough training, the network learns spelling, basic grammar, and even some style. It does not learn semantics: the output is fluent gibberish, not coherent thought.

Synthesis project

# Task: input sequence of length T; first element is class 0 or 1;
# remaining T-1 elements are noise; predict the first element at step T.
def make_batch(T, batch_size=64):
    cls = rng.integers(0, 2, batch_size)
    xs = rng.normal(0, 1, (batch_size, T, 1))
    xs[:, 0, 0] = cls
    return xs, cls

Downloads

References