Principal Component Analysis (PCA): Dimensionality Reduction

Understand Principal Component Analysis (PCA). Learn how to reduce dimensionality, find eigenvectors and eigenvalues, and implement PCA in Python.

Introduction

Imagine you are looking at a 3D sculpture of a horse, and you want to photograph it. A photo is a 2D projection of a 3D object. To take the best photo, you will walk around the sculpture to find the angle that captures the maximum detail (or variance) of the horse. If you shoot it straight from the front, you might just get a flat oval; if you shoot it from the side, you get the legs, head, and tail clearly.

This is the goal of Principal Component Analysis (PCA). In machine learning, datasets often contain dozens or hundreds of features (columns). PCA is a dimensionality reduction technique that finds the directions of maximum variance in high-dimensional data and projects it onto a lower-dimensional space (e.g., converting 100 features into 2 or 3 principal components) with minimal loss of information.

What You Will Learn

  • The curse of dimensionality and the need for reduction.
  • How PCA works mathematically (covariance, eigenvectors, eigenvalues).
  • What Principal Components are and how to interpret them.
  • How to implement PCA in Python using scikit-learn.

Why This Topic Matters

As data grows, models face the Curse of Dimensionality—training becomes slow, models overfit, and visualization becomes impossible. PCA is the most popular technique to combat this. It compresses feature spaces, accelerates model training, and enables you to plot complex multi-dimensional datasets onto simple 2D or 3D scatter plots.

Prerequisites

Detailed Explanation

The main goal of PCA is to identify patterns in data and detect correlation between variables. If two variables are highly correlated (e.g., height in inches vs. height in centimeters), one of them is redundant. PCA collapses these redundant dimensions.

Step-by-Step Mathematical Algorithm

  1. Standardize the Data: PCA is sensitive to variance scales. We must scale features to have a mean of $0$ and a standard deviation of $1$: $$z = \frac{x - \mu}{\sigma}$$
  2. Compute Covariance Matrix: Calculate a matrix representing how all variables vary together: $$\Sigma = \frac{1}{N} X^T X$$
  3. Compute Eigenvectors and Eigenvalues: Solve the equation: $$\Sigma v = \lambda v$$ Where:
    • $v$ is the Eigenvector (direction of the new axes / Principal Components).
    • $\lambda$ is the Eigenvalue (magnitude / amount of variance explained by that Component).
  4. Sort and Select: Sort the eigenvectors in descending order of their eigenvalues. The top $k$ eigenvectors represent the $k$ Principal Components that capture the most variance.
  5. Project Data: Multiply the original standardized data matrix by the top $k$ eigenvectors to get the new projected dataset.

Explained Variance Ratio

The Explained Variance Ratio tells us how much information (variance) is captured by each principal component.

  • If PC1 explains 70% of the variance, and PC2 explains 20%, keeping just these two components retains 90% of the original information, allowing us to safely drop all other features.

Visual Diagram (Mermaid)

graph TD
    A[Original High-Dimensional Data] --> B[Standardize Features]
    B --> C[Compute Covariance Matrix]
    C --> D[Extract Eigenvalues & Eigenvectors]
    D --> E[Sort Eigenvectors by Eigenvalue Size]
    E --> F[Select Top K Components]
    F --> G[Project Data onto New K-Dimensions]
    
    style A fill:#3B82F6,stroke:#fff,color:#fff
    style G fill:#10B981,stroke:#fff,color:#fff

Python Code Examples

Let's use PCA to compress the famous Iris dataset (4 features) down to 2 principal components for easy 2D plotting.

import numpy as np
from sklearn.datasets import load_iris
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA

# 1. Load the Iris data (features: sepal length/width, petal length/width)
iris = load_iris()
X = iris.data

# 2. Standardize features (mean=0, variance=1)
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

# 3. Apply PCA (reduce from 4 dimensions to 2)
pca = PCA(n_components=2)
X_pca = pca.fit_transform(X_scaled)

# 4. View results and variance explained
print("Original dimensions shape:", X.shape)
print("Compressed PCA dimensions shape:", X_pca.shape)
print("\nExplained Variance Ratio for [PC1, PC2]:")
print(pca.explained_variance_ratio_)
print(f"Total variance retained: {round(sum(pca.explained_variance_ratio_)*100, 2)}%")

Industry Use Cases

  • Facial Recognition (Eigenfaces): Representing pixel arrays of faces using a few principal components (eigenfaces) instead of thousands of individual pixels.
  • Gene Expression Analysis: Reducing thousands of genetic markers down to a handful of indicators to identify patient subcategories.
  • Anomaly Detection: Projecting network traffic features to PCA space. Deviations along the lower-variance components indicate anomaly behaviors.

Advantages & Limitations

Advantages

  • Reduces Overfitting: Fewer features reduce model complexity.
  • Improves Speed: Compressed datasets speed up training of downstream models (like SVM or Random Forests).
  • Enables Visualization: Reduces dimensions to 2D or 3D for plotting.
  • Eliminates Collinearity: Principal components are mathematically perpendicular (orthogonal), meaning they are completely uncorrelated.

Limitations

  • Loss of Interpretability: The new principal components are linear combinations of original features. You can no longer easily tell which specific real-world feature is driving a prediction.
  • Sensitive to Scaling: If variables are not standardized first, features with large scales will dominate.
  • Assumes Linearity: PCA only finds linear combinations. For complex non-linear relations, you need methods like Kernel PCA or t-SNE.

FAQs

Q: Should I run PCA before or after splitting into Train/Test sets? A: You should fit the StandardScaler and PCA on the training set only, and then transform both train and test sets. Fitting them on the full dataset causes data leakage because information from the test set would influence the mean and covariance matrix calculation.

Q: How many principal components should I keep? A: Use a Scree Plot (plotting cumulative explained variance vs. number of components) and select the number of components where the cumulative variance reaches a target threshold (typically 90% or 95%).

Summary

PCA is an unsupervised linear dimensionality reduction technique that standardizes data, builds a covariance matrix, and uses eigenvalues/eigenvectors to identify axes of maximum variance. By projecting the dataset onto these orthogonal axes, it discards redundant correlated noise, paving the way for faster model training and intuitive data visualizations.

Next Topic

Unsupervised learning is also the perfect tool for identifying rare events or fraudulent actions. Let's study how we can detect outliers using: Anomaly Detection.

← Back to Syllabus