Optimization Algorithms: SGD, Momentum, RMSprop, and Adam

Compare Deep Learning optimizers. Learn about Stochastic Gradient Descent (SGD), Momentum, RMSprop, and Adam with equations and convergence behaviors.

Introduction

Imagine you are a blindfolded hiker trapped at the top of a foggy mountain, trying to find your way down to the valley (the minimum loss).

If you take a step, check the slope, and immediately walk in that direction, you will move slowly and might get stuck in a small pothole (local minimum). If you start running, using your momentum, you can roll right over the potholes, but you might overshoot the valley. If you adjust your stride based on how steep the terrain has been, you can navigate rocks and flat spots effectively.

In deep learning, the strategies we use to walk down this mountain are called Optimization Algorithms (Optimizers). They govern how weights are updated based on calculated gradients.

What You Will Learn

  • The limitations of standard Gradient Descent.
  • Stochastic Gradient Descent (SGD) and Mini-batch SGD.
  • How Momentum accelerates convergence.
  • The mechanics of adaptive learning rate optimizers: RMSprop and Adam.
  • Which optimizer to choose for your project.

Why This Topic Matters

Training a deep network can take days or weeks. Choosing the right optimizer can reduce training time from days to hours. Optimizers like Adam have become the industry standard because they handle noisy gradients, sparse features, and complex loss landscapes with minimal manual hyperparameter tuning.

Prerequisites

Detailed Explanation

Optimizers update weights using gradients ($g_t = \nabla L(w_t)$). Let's trace their evolutionary history.

1. Stochastic Gradient Descent (SGD)

Instead of calculating gradients using the entire dataset (which is slow and memory-intensive), SGD computes gradients using a single random data point or a small batch (Mini-batch SGD).

  • Update Rule: $$w_{t+1} = w_t - \eta \cdot g_t$$
  • Limitation: Updates are highly noisy and oscillate wildly in steep valleys, slowing down convergence.

2. SGD with Momentum

To smooth out oscillations, Momentum mimics physical momentum. It keeps track of past gradients and adds a fraction ($\beta$) of the previous update vector ($v_{t-1}$) to the current update:

  • Velocity Update: $$v_t = \beta v_{t-1} + \eta g_t$$
  • Weight Update: $$w_{t+1} = w_t - v_t$$
  • Advantage: Accelerates descent in the correct direction and dampens oscillations.

3. RMSprop (Root Mean Squared Propagation)

RMSprop is an adaptive learning rate optimizer. Instead of using a fixed learning rate for all weights, it scales the learning rate for each weight based on the running average of its historical gradients ($s_t$):

  • Squared Gradient Average: $$s_t = \beta s_{t-1} + (1 - \beta) g_t^2$$
  • Weight Update: $$w_{t+1} = w_t - \frac{\eta}{\sqrt{s_t + \epsilon}} \cdot g_t$$
  • Advantage: If a feature has large, oscillating gradients, it gets scaled down (divided by a large $\sqrt{s_t}$). If it has tiny gradients, it gets scaled up. This prevents gradient explosion. ($\epsilon$ is a tiny number to avoid division by zero).

4. Adam (Adaptive Moment Estimation)

Adam is the combination of Momentum and RMSprop. It tracks both the first moment (mean of gradients, $m_t$) and the second moment (uncentered variance of gradients, $v_t$):

  • Momentum-like Term (First Moment): $$m_t = \beta_1 m_{t-1} + (1 - \beta_1) g_t$$
  • RMSprop-like Term (Second Moment): $$v_t = \beta_2 v_{t-1} + (1 - \beta_2) g_t^2$$
  • Bias Correction: Since $m_t$ and $v_t$ are initialized to zero, they are biased toward zero. We correct this bias: $$\hat{m}_t = \frac{m_t}{1 - \beta_1^t}, \quad \hat{v}_t = \frac{v_t}{1 - \beta_2^t}$$
  • Weight Update: $$w_{t+1} = w_t - \frac{\eta}{\sqrt{\hat{v}_t} + \epsilon} \cdot \hat{m}_t$$

Adam is the default optimizer for 95% of deep learning projects because it combines the benefits of both adaptive rates and momentum.

Visual Diagram (Mermaid)

graph TD
    A[Optimizer Selection] --> B[SGD: Simple updates, high noise]
    A --> C[Momentum: Tracks velocity to damp oscillations]
    A --> D[RMSprop: Scales step size by gradient magnitude]
    A --> E[Adam: Momentum + RMSprop adaptive tracking]
    
    style E fill:#10B981,stroke:#fff,color:#fff

Python Code Examples

We will configure SGD and Adam optimizers in a model training setup using standard syntax structures.

# Conceptualizing Optimizer Logic in Python
class SGD:
    def __init__(self, lr=0.01):
        self.lr = lr

    def update(self, w, grad):
        return w - self.lr * grad

class Adam:
    def __init__(self, lr=0.001, beta1=0.9, beta2=0.999, epsilon=1e-8):
        self.lr = lr
        self.beta1 = beta1
        self.beta2 = beta2
        self.epsilon = epsilon
        self.m = 0
        self.v = 0
        self.t = 0

    def update(self, w, grad):
        self.t += 1
        # Update first and second moments
        self.m = self.beta1 * self.m + (1 - self.beta1) * grad
        self.v = self.beta2 * self.v + (1 - self.beta2) * (grad ** 2)
        
        # Bias correction
        m_hat = self.m / (1 - self.beta1 ** self.t)
        v_hat = self.v / (1 - self.beta2 ** self.t)
        
        # Update weight
        w = w - (self.lr / (np.sqrt(v_hat) + self.epsilon)) * m_hat
        return w

# Test updates
w = 1.0
grad = 0.5
adam_opt = Adam()
print("Adam updated weight step 1:", adam_opt.update(w, grad))

Industry Use Cases

  • Natural Language Processing (Transformers): Models like BERT, GPT, and Claude are trained almost exclusively using AdamW (a variant of Adam with improved weight decay regularization).
  • Computer Vision (CNNs): While Adam is common, some state-of-the-art vision models use SGD with Momentum because it can sometimes achieve better final generalization accuracy when combined with careful learning rate scheduling.

Summary

Optimizers are mathematical engines that adjust neural network weights to minimize loss. While SGD is simple but noisy, adding Momentum dampens oscillations. Adaptive algorithms like RMSprop adjust step sizes feature-by-feature, and Adam combines both velocity and adaptive scaling to provide the most reliable default optimization strategy in modern AI.

Next Topic

How do we actually build, train, and manage these tensors, layers, and optimizers in production? Let's write our first tensor code in: Introduction to PyTorch.

← Back to Syllabus