Feedforward and Backpropagation: How Neural Networks Learn

Understand how Neural Networks learn. Explore the Feedforward pass, Loss Functions, Backpropagation, and the Calculus Chain Rule.

Introduction

If you have ever tried to learn a new language or play an instrument, your brain goes through a cycle: you try to speak or play (Feedforward), you hear the mistake you made (Loss calculation), and you adjust your muscle memory or thought process to fix it (Backpropagation).

In deep learning, neural networks learn in the exact same way. The entire training process is a continuous loop of making predictions, measuring errors, and updating weights using calculus.

What You Will Learn

  • How data moves forward through a network (Feedforward).
  • The role of Loss Functions in measuring errors.
  • The mathematics of Backpropagation using the Chain Rule.
  • How weights and biases are updated to reduce errors.

Why This Topic Matters

Backpropagation is the engine of Deep Learning. Without it, neural networks would be unable to learn. Understanding the calculus behind backpropagation allows you to diagnose why a model's loss is not decreasing, prevent gradients from exploding, and write customized training loops in frameworks like PyTorch.

Prerequisites

Detailed Explanation

Training a neural network consists of three fundamental stages:

graph TD
    A[Feedforward: Calculate Prediction] --> B[Loss Function: Calculate Error]
    B --> C[Backpropagation: Compute Gradients]
    C --> D[Weight Update: Gradient Descent]
    D --> A

1. The Feedforward Pass

In this step, input features pass forward through the layers to generate a prediction. For a single neuron:

  1. Compute the linear combination: $z = W^T X + b$
  2. Apply the activation function: $a = f(z)$ This output $a$ becomes the input $X$ for the next layer.

2. The Loss Function (Calculating Error)

Once the output layer produces a prediction ($\hat{y}$), we compare it to the ground-truth label ($y$) using a Loss Function ($L$).

  • Mean Squared Error (MSE) (for Regression): $$L = \frac{1}{2}(\hat{y} - y)^2$$
  • Binary Cross-Entropy (for Binary Classification): $$L = -[y \log(\hat{y}) + (1 - y) \log(1 - \hat{y})]$$

3. Backpropagation (The Chain Rule)

To minimize the Loss, we must find how changing our weights affects the Loss. Mathematically, we want to compute the partial derivative: $\frac{\partial L}{\partial w}$.

Because weights are nested deep within the network layers ($L \rightarrow \hat{y} \rightarrow z \rightarrow w$), we use the Chain Rule of calculus to unpack this derivative:

$$\frac{\partial L}{\partial w} = \frac{\partial L}{\partial \hat{y}} \cdot \frac{\partial \hat{y}}{\partial z} \cdot \frac{\partial z}{\partial w}$$

Let's break down each term for a single neuron using MSE and linear activation:

  1. How does Loss change with output? $$\frac{\partial L}{\partial \hat{y}} = (\hat{y} - y)$$
  2. How does output change with linear sum? (Assuming activation $f(z)$) $$\frac{\partial \hat{y}}{\partial z} = f'(z)$$
  3. How does linear sum change with weights? ($z = w \cdot x + b$) $$\frac{\partial z}{\partial w} = x$$

Combining them gives the gradient for the weight: $$\frac{\partial L}{\partial w} = (\hat{y} - y) \cdot f'(z) \cdot x$$

4. Updating the Weights (Gradient Descent)

Once the gradient is calculated, we update the weights in the opposite direction of the gradient to reduce the loss:

$$w_{new} = w_{old} - \eta \cdot \frac{\partial L}{\partial w}$$

Where $\eta$ (eta) is the Learning Rate—a hyperparameter controlling how big of a step we take down the error slope.

Visual Diagram (Mermaid)

graph LR
    subgraph Forward Pass
    X((Input X)) -->|W| H((Hidden H))
    H -->|V| Y((Prediction Y-hat))
    end
    
    subgraph Loss computation
    Y --> L[Loss Function]
    end
    
    subgraph Backward Pass
    L -->|\u2202L / \u2202Y-hat| Y
    Y -->|\u2202L / \u2202V| H
    H -->|\u2202L / \u2202W| X
    end

Python Code Examples

We will implement a simple feedforward and backpropagation pass for a single neuron in Python.

import numpy as np

# 1. Inputs, true label, weights, bias
X = np.array([2.0, 3.0])
y_true = 1.0

weights = np.array([0.1, -0.2])
bias = 0.5
learning_rate = 0.1

print("Initial Weights:", weights)

# 2. Forward pass (Linear activation for simplicity)
z = np.dot(X, weights) + bias
y_pred = z  # Linear output
loss = 0.5 * (y_pred - y_true) ** 2
print(f"Prediction: {y_pred}, Loss: {loss}")

# 3. Backward pass (Chain Rule calculation)
# dL/dy_pred = (y_pred - y_true)
# dy_pred/dz = 1
# dz/dw = X
dL_dypred = y_pred - y_true
dypred_dz = 1.0
dz_dw = X

# Gradient calculation
gradient_w = dL_dypred * dypred_dz * dz_dw
gradient_b = dL_dypred * dypred_dz * 1.0 # dz/db = 1

# 4. Weight update
weights = weights - learning_rate * gradient_w
bias = bias - learning_rate * gradient_b

print("Updated Weights after 1 step:", weights)
print("Updated Bias after 1 step:", bias)

Industry Use Cases

  • Autonomous Speech Recognition: Every speech model uses backpropagation over thousands of iterations to align the audio signals with text transcriptions.
  • Large Language Models (LLMs): Backpropagation runs across billions of parameters on clusters of H100 GPUs to train models like Llama or GPT.

Advantages & Limitations

Advantages

  • Highly Scalable: Calculates gradients for millions of parameters efficiently.
  • Calculus Precision: Provides exact direction of steepest descent.

Limitations

  • Local Minima: Gradient descent can get stuck in local minima or saddle points instead of the absolute global minimum.
  • Vanishing/Exploding Gradients: In very deep networks, multiplying small derivatives repeatedly causes gradients to shrink to zero, stopping learning. Stacking large values causes them to explode.

FAQs

Q: What is a Saddle Point? A: A point where the slope is zero in one direction, but not in others (like a horse saddle). High-dimensional spaces are full of saddle points, which can slow down gradient descent.

Q: What happens if the Learning Rate is too high? A: If the learning rate is too high, the updates will overshoot the minimum, causing the loss to fluctuate and diverge rather than converge.

Summary

Neural networks learn via a feedback loop. The Feedforward pass processes inputs to make predictions, which are evaluated by a Loss Function. Backpropagation calculates the influence of each weight on the error using the calculus Chain Rule, and Gradient Descent modifies weights to iteratively minimize the loss.

Next Topic

Standard gradient updates can be slow, noisy, and get stuck. How do modern networks optimize this process? Let's compare: Optimization Algorithms: SGD, Momentum, RMSprop, and Adam.

← Back to Syllabus