Activation Functions: Sigmoid, Tanh, ReLU, and Softmax

Master Activation Functions in Deep Learning. Learn about non-linearity, Sigmoid, Tanh, ReLU, Leaky ReLU, and Softmax with equations and Python code.

Introduction

If you connect multiple linear equations together, the result is always just another linear equation. If neural networks only performed linear additions ($W^T X + b$), they would be no more powerful than simple Linear Regression, no matter how many layers you stacked.

To learn complex, winding patterns like shapes, text context, or audio frequencies, neural networks need Activation Functions. These are mathematical formulas applied to a neuron's output that introduce non-linearity, allowing the network to approximate virtually any continuous function (a concept known as the Universal Approximation Theorem).

What You Will Learn

  • Why activation functions are mandatory in deep learning.
  • The mathematics and curves of Sigmoid, Tanh, ReLU, and Leaky ReLU.
  • The role of Softmax in multi-class classification.
  • How to implement and plot activation functions in Python.

Why This Topic Matters

Choosing the wrong activation function can lead to models that do not learn at all, a phenomenon known as the Vanishing Gradient Problem or the Dying ReLU Problem. As an AI engineer, understanding the trade-offs of each function helps you design architectures that converge quickly and accurately.

Prerequisites

Detailed Explanation

An activation function $f(z)$ takes the net input $z = W^T X + b$ and maps it to a new range. Here are the most important activation functions in deep learning:

1. Sigmoid Function

Maps any real-valued number to a probability value between $0$ and $1$.

  • Formula: $$\sigma(z) = \frac{1}{1 + e^{-z}}$$
  • Usage: Output layer for binary classification tasks.
  • Drawback: Vanishing Gradient: For very large or small inputs, the curve becomes extremely flat, meaning its derivative (gradient) is close to zero. This stops the weights from updating during backpropagation.

2. Tanh (Hyperbolic Tangent)

Similar to Sigmoid but maps values to a range between $-1$ and $+1$.

  • Formula: $$\tanh(z) = \frac{e^z - e^{-z}}{e^z + e^{-z}}$$
  • Usage: Often used in hidden layers.
  • Advantage: Zero-centered, meaning negative inputs are mapped to negative outputs, making optimization easier than Sigmoid. Still suffers from vanishing gradients.

3. ReLU (Rectified Linear Unit)

The absolute king of activation functions for hidden layers. If the input is positive, it returns the input; otherwise, it returns zero.

  • Formula: $$f(z) = \max(0, z)$$
  • Usage: Default choice for hidden layers.
  • Advantage: Computationally extremely fast and does not saturate for positive values (solves vanishing gradient).
  • Drawback: Dying ReLU: If neurons receive negative values, they output zero and their gradient becomes zero. These neurons "die" and never update.

4. Leaky ReLU

Fixes the dying ReLU problem by adding a small positive slope ($\alpha$) for negative inputs.

  • Formula: $$f(z) = \max(\alpha z, z) \quad (\text{usually } \alpha = 0.01)$$
  • Usage: GANs and deep networks where dying ReLUs are a bottleneck.

5. Softmax Function

Used in the output layer for multi-class classification. It takes a vector of raw scores (logits) and converts them into probabilities that sum up to $1$.

  • Formula: $$\text{Softmax}(z_i) = \frac{e^{z_i}}{\sum_{j} e^{z_j}}$$
  • Usage: Output layer of multi-class classification models.

Visual Diagram (Mermaid)

graph TD
    A[Neuron Sum: z] --> B{Choose Activation}
    
    B -->|Hidden Layer default| C[ReLU: f z = max 0, z]
    B -->|Hidden Layer zero-centered| D[Tanh: f z = -1 to 1]
    B -->|Binary Class Output| E[Sigmoid: f z = 0 to 1]
    B -->|Multi-Class Output| F[Softmax: Sum of outputs = 1.0]
    
    style C fill:#10B981,stroke:#fff,color:#fff
    style E fill:#3B82F6,stroke:#fff,color:#fff
    style F fill:#F59E0B,stroke:#fff,color:#fff

Python Code Examples

Let's implement these activation functions from scratch in Python using numpy.

import numpy as np

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

def tanh(z):
    return np.tanh(z)

def relu(z):
    return np.maximum(0, z)

def leaky_relu(z, alpha=0.01):
    return np.where(z > 0, z, z * alpha)

def softmax(z):
    exp_z = np.exp(z - np.max(z)) # Max subtraction prevents overflow
    return exp_z / np.sum(exp_z, axis=0)

# Test inputs
logits = np.array([2.0, 1.0, 0.1])
print("Raw Logits:", logits)
print("Softmax Probabilities:", softmax(logits))
print("Sum of Probabilities:", np.sum(softmax(logits)))

test_val = -5.0
print(f"\nInputs: {test_val}")
print(f"Sigmoid: {sigmoid(test_val)}")
print(f"ReLU: {relu(test_val)}")
print(f"Leaky ReLU: {leaky_relu(test_val)}")

Industry Use Cases

  • Image Classifiers: CNNs (like ResNet) use ReLU in hidden layers for speed and Softmax at the end to predict categories (e.g., Cat: 92%, Dog: 6%, Bird: 2%).
  • Sentiment Analysis: Output layer uses Sigmoid to output scores near $1.0$ (Positive) or $0.0$ (Negative).
  • Generative Adversarial Networks (GANs): Leaky ReLU is widely used in Discriminator networks to prevent dead neurons.

Summary

Activation functions introduce non-linearity, enabling deep neural networks to learn complex decision boundaries. While Sigmoid and Tanh are useful for bounded ranges, they suffer from vanishing gradients. ReLU is the standard for hidden layers due to its speed, with Leaky ReLU resolving its "dying" states, and Softmax rounding out output layers for multi-class classification.

Next Topic

Now that we know how inputs are processed through activation functions, how does a network calculate its errors and update its weights? Let's explore: Feedforward and Backpropagation.

← Back to Syllabus