Recurrent Neural Networks (RNNs) for Sequence Modeling

Understand Recurrent Neural Networks (RNNs). Learn how they process sequential data, weight sharing, hidden states, and vanishing gradient limitations.

Introduction

If you watch a movie, you don't process each frame as an isolated image. Your brain remembers what happened in the previous frames, allowing you to understand the plot.

Standard feedforward neural networks (like MLPs and CNNs) cannot do this. They treat every input (and output) as completely independent of the others. To process sequential data—where the order of inputs matters, such as text sentences, stock prices, or audio signals—we need Recurrent Neural Networks (RNNs). RNNs have a "memory" (hidden state) that updates as they read inputs step-by-step.

What You Will Learn

  • Why standard neural networks fail on sequential data.
  • The architecture of an RNN and the concept of Hidden States.
  • The math of RNN computations.
  • The Vanishing and Exploding Gradient Problem in sequence learning.
  • How to implement a basic RNN cell in PyTorch.

Why This Topic Matters

Almost all real-world data has a temporal or sequential structure. Whether you are building an auto-complete text system, translating speech, forecasting weather, or predicting stock movements, RNNs are the foundational architecture that introduced temporal state tracking to deep learning.

Prerequisites

Detailed Explanation

The key innovation of an RNN is the feedback loop. At each time step $t$, the network processes the current input $x_t$ and the memory from the previous step $h_{t-1}$.

RNN Unrolling/Unfolding

An RNN can be visualized in two ways: as a single cell with a loop, or "unrolled" across time:

Folded:
   [x_t] ---> ( RNN Cell ) ---> [y_t]
                 ^   |
                 +---+ (loop)

Unrolled:
   [x_1] ---> ( RNN_1 ) ---> [y_1]
                 | (h_1)
                 v
   [x_2] ---> ( RNN_2 ) ---> [y_2]
                 | (h_2)
                 v
   [x_3] ---> ( RNN_3 ) ---> [y_3]

At each time step $t$:

  1. The network takes current input $x_t$ and previous hidden state $h_{t-1}$.
  2. It calculates the new hidden state $h_t$: $$h_t = \tanh(W_{hh} h_{t-1} + W_{xh} x_t + b_h)$$
  3. It computes the output $y_t$ (if needed): $$y_t = W_{hy} h_t + b_y$$

Where:

  • $W_{xh}$ is the weight matrix for input-to-hidden connections.
  • $W_{hh}$ is the weight matrix for hidden-to-hidden connections.
  • $W_{hy}$ is the weight matrix for hidden-to-output connections.
  • Weight Sharing: Crucially, the weights ($W_{xh}, W_{hh}, W_{hy}$) are shared (identical) across all time steps. This allows the network to process sequences of any length.

Types of RNN Architectures

Depending on the task, RNNs can be configured in different ways:

| Architecture | Input Shape | Output Shape | Use Case | | :--- | :--- | :--- | :--- | | One-to-Many | 1 vector | Sequence | Image Captioning (Image $\rightarrow$ Sentence) | | Many-to-One | Sequence | 1 vector | Sentiment Analysis (Sentence $\rightarrow$ Positive/Negative) | | Many-to-Many| Sequence | Sequence | Language Translation (English $\rightarrow$ Spanish) |

The Fatal Flaw: Vanishing Gradients

During backpropagation, we calculate gradients by moving backward through time (Backpropagation Through Time, or BPTT).

Since weights are shared, calculating the gradient at time step $t=1$ based on errors at $t=100$ requires multiplying the weight matrix $W_{hh}$ by itself 100 times.

  • If weights are $< 1$: The gradients shrink exponentially (Vanishing Gradient), causing the network to forget the beginning of the sentence.
  • If weights are $> 1$: The gradients grow exponentially (Exploding Gradient), causing the weights to overflow and return NaN.

This means standard RNNs have short-term memory and struggle to capture dependencies longer than 10-15 steps.

Visual Diagram (Mermaid)

graph LR
    subgraph RNN Step t
    X[Input x_t] -->|W_xh| H[Hidden State h_t]
    Hprev[Prev Hidden h_t-1] -->|W_hh| H
    H -->|W_hy| Y[Output y_t]
    end
    style H fill:#3B82F6,stroke:#fff,color:#fff

Python Code Examples

We will build a simple RNN sequence classifier using PyTorch.

import torch
import torch.nn as nn

class RNNClassifier(nn.Module):
    def __init__(self, vocab_size, embed_dim, hidden_dim, output_dim):
        super(RNNClassifier, self).__init__()
        # 1. Embedding Layer (maps word indices to dense vectors)
        self.embedding = nn.Embedding(vocab_size, embed_dim)
        
        # 2. Basic RNN Layer
        # batch_first=True means input shape is (batch, sequence_length, features)
        self.rnn = nn.RNN(embed_dim, hidden_dim, batch_first=True)
        
        # 3. Output Layer
        self.fc = nn.Linear(hidden_dim, output_dim)

    def forward(self, text):
        # text shape: [batch_size, seq_length]
        embedded = self.embedding(text) # shape: [batch, seq_len, embed_dim]
        
        # output: contains hidden states for all time steps
        # hidden: contains final hidden state (h_T)
        output, hidden = self.rnn(embedded)
        
        # We pass the final hidden state (hidden[-1]) through the linear layer
        return self.fc(hidden.squeeze(0))

# Instantiate model
# Vocab size=1000, Embedding=100, Hidden=64, Output=2 (Binary classification)
model = RNNClassifier(vocab_size=1000, embed_dim=100, hidden_dim=64, output_dim=2)
print(model)

# Simulate batch of 4 sentences, each containing 5 words
dummy_input = torch.randint(0, 1000, (4, 5))
output = model(dummy_input)
print("\nOutput Logits Shape:", output.shape) # Expected: [4, 2]

Industry Use Cases

  • Stock Price Forecasting: Modeling historical stock values sequentially to predict next-day values.
  • Speech Recognition: Converting continuous audio wave frames into text sequences.
  • Sentiment Analysis: Reading customer reviews word-by-word to predict if they are happy or frustrated.

Advantages & Limitations

Advantages

  • Handles Variable Sequence Lengths: Can process a sentence of 5 words or 50 words without modifying network parameters.
  • Maintains Temporal Order: Processes inputs step-by-step, preserving historical context.

Limitations

  • Short-Term Memory: Fails on long documents due to vanishing gradients.
  • No Parallel Processing: Because step $t$ depends on step $t-1$, calculations must run sequentially, making training on GPUs slower than CNNs or Transformers.

FAQs

Q: What is Backpropagation Through Time (BPTT)? A: It is the backpropagation algorithm applied to recurrent networks. The network is unrolled across all time steps, and gradients are calculated from the final loss backward through each temporal step.

Q: How do we fix exploding gradients in RNNs? A: A common technique is Gradient Clipping, which clips gradients to a maximum value if their magnitude exceeds a set threshold.

Summary

RNNs process sequence datasets by passing historic hidden states across consecutive computational steps. However, standard RNN architectures suffer from vanishing gradients, limiting their effective memory to short sequences.

Next Topic

How do we design networks that can store long-term memories and decide what to keep and what to forget? Let's check: LSTMs and GRUs: Gated Neural Networks for Long-Term Memory.

← Back to Syllabus