LSTMs and GRUs: Gated Neural Networks for Long-Term Memory

Learn LSTMs and GRUs. Understand forget gates, input gates, output gates, cell states, and how gated architectures prevent vanishing gradients.

Introduction

If you are reading a long novel, you don't need to remember every single word to understand the story. You read a paragraph, keep the important plot points, forget the descriptions of the background characters, and update your understanding of the story.

Standard RNNs cannot do this; they try to remember everything, which leads to vanishing gradients and short-term memory loss.

To solve this, researchers Sepp Hochreiter and Jürgen Schmidhuber introduced the Long Short-Term Memory (LSTM) network in 1997. LSTMs and their modern counterpart, the Gated Recurrent Unit (GRU), use mathematical "gates" to control exactly what information to add, keep, or discard from memory over time.

What You Will Learn

How LSTMs maintain long-term memory via the Cell State.
The roles of the three LSTM gates: Forget, Input, and Output.
The simplified structure of a Gated Recurrent Unit (GRU).
How LSTMs prevent the vanishing gradient problem.
How to implement LSTMs and GRUs in PyTorch.

Why This Topic Matters

Until the rise of Transformers, LSTMs were the state of the art in NLP, speech synthesis, and time-series forecasting. They remain highly popular for applications with limited training data or resource constraints because they are lightweight, robust, and capable of maintaining memories across hundreds of steps.

Prerequisites

Detailed Explanation

LSTMs solve the short-term memory problem by introducing two parallel data streams: the Hidden State ($h_t$) (short-term memory) and the Cell State ($C_t$) (long-term memory).

The LSTM Gates Architecture

The Cell State acts like a highway, passing information straight down the sequence with only minor linear modifications. Access to the Cell State is protected by three gates, which use Sigmoid functions (outputting $0$ to $1$) to scale information flow.

graph TD
    A[Input x_t + Prev Hidden h_t-1] --> B{Forget Gate}
    A --> C{Input Gate}
    A --> D{Output Gate}
    
    B -->|Decides what to forget| CellState[Cell State C_t]
    C -->|Decides what to write| CellState
    D -->|Decides what to show| HiddenState[Hidden State h_t]

1. The Forget Gate ($f_t$)

Decides how much of the old long-term memory ($C_{t-1}$) to throw away.

Formula: $$f_t = \sigma(W_f \cdot [h_{t-1}, x_t] + b_f)$$
Behavior: If $f_t = 0$, the old memory is completely erased. If $f_t = 1$, it is fully kept.

2. The Input Gate ($i_t$)

Decides what new information to add to the Cell State.

Formula: $$i_t = \sigma(W_i \cdot [h_{t-1}, x_t] + b_i)$$ $$\tilde{C}t = \tanh(W_c \cdot [h{t-1}, x_t] + b_c)$$
Updating the Cell State: The new Cell State ($C_t$) is computed by forgetting old memory and adding the new candidate values: $$C_t = f_t \odot C_{t-1} + i_t \odot \tilde{C}_t$$ (where $\odot$ denotes element-wise multiplication).

3. The Output Gate ($o_t$)

Decides what the new hidden state ($h_t$) should be.

Formula: $$o_t = \sigma(W_o \cdot [h_{t-1}, x_t] + b_o)$$ $$h_t = o_t \odot \tanh(C_t)$$

Gated Recurrent Unit (GRU)

Introduced by Kyunghyun Cho et al. in 2014, the GRU is a popular, simplified variant of the LSTM.

Differences:
- Merges the Cell State and Hidden State into a single state $h_t$.
- Uses only two gates: the Reset Gate ($r_t$) (decides how to combine new input with past memory) and the Update Gate ($z_t$) (acts as both forget and input gate).
Advantage: Fewer parameters make GRUs faster to train and less prone to overfitting on small datasets.

LSTM vs. GRU Comparison

Visual Diagram (Mermaid)

graph LR
    subgraph LSTM Cell
    Cprev[Cell State C_t-1] -->|Forget Gate ft| Ccurr[Cell State C_t]
    Cand[Candidate C_t] -->|Input Gate it| Ccurr
    Ccurr -->|\tanh| Hout
    Hout -->|Output Gate ot| Hcurr[Hidden State h_t]
    end
    style Ccurr fill:#10B981,stroke:#fff,color:#fff
    style Hcurr fill:#3B82F6,stroke:#fff,color:#fff

Python Code Examples

We will build an LSTM and GRU sequence model in PyTorch.

import torch
import torch.nn as nn

# Parameters
batch_size = 8
seq_len = 10
input_dim = 50  # Word embedding size
hidden_dim = 128

# Simulate input: 8 sentences, 10 words each, 50 features per word
dummy_input = torch.randn(batch_size, seq_len, input_dim)

# 1. Instantiate LSTM
lstm = nn.LSTM(input_size=input_dim, hidden_size=hidden_dim, batch_first=True)
lstm_out, (hn, cn) = lstm(dummy_input)

print("LSTM Output Shape:", lstm_out.shape) # [batch, seq_len, hidden_dim]
print("LSTM Hidden State hn Shape:", hn.shape) # [1, batch, hidden_dim]
print("LSTM Cell State cn Shape:", cn.shape) # [1, batch, hidden_dim]

# 2. Instantiate GRU
gru = nn.GRU(input_size=input_dim, hidden_size=hidden_dim, batch_first=True)
gru_out, hn_gru = gru(dummy_input)

print("\nGRU Output Shape:", gru_out.shape) # [batch, seq_len, hidden_dim]
print("GRU Hidden State hn Shape:", hn_gru.shape) # [1, batch, hidden_dim]
# Note: GRU has no Cell State (cn)!

Industry Use Cases

Machine Translation: Classic Google Translate was powered by sequence-to-sequence encoder-decoder LSTMs.
Time-Series Energy Forecasting: Predicting electric grid load for the next 24 hours based on temperature, history, and weekday.
Siri / Alexa Wake Word Detection: Tiny GRU models run on low-power device chips to continuously listen for "Hey Siri" or "Alexa".

Summary

LSTMs and GRUs resolve the short-term memory constraints of standard RNNs. By utilizing sigmoidal gates, they selectively filter, preserve, and transmit details across long sequence sequences. While LSTMs are highly accurate, GRUs present a faster, parameter-light alternative for sequence processing tasks.

Next Topic

While LSTMs process sequences one-by-step, how can we process entire images to locate edges, shapes, and complex structures? Let's check: Introduction to Computer Vision (CV).

← Back to Syllabus