Matrices and Vectors: Core Operations for AI

Introduction

In the previous tutorial, we established that AI data is stored in Matrices and Vectors. However, an AI doesn't just hold data; it transforms it. A Neural Network is simply a long sequence of matrix multiplications. Understanding how to perform basic operations on these structures—specifically addition, multiplication, and the dot product—is essential for debugging AI code.

What You Will Learn

• How to add and subtract Vectors and Matrices.
• Scalar multiplication vs. Matrix multiplication.
• The Dot Product (the most important calculation in Deep Learning).
• Matrix Transposition.

Why This Topic Matters

When you build a Deep Learning model in PyTorch, you define layers (e.g., a layer with 128 neurons connecting to a layer with 64 neurons). If you try to pass data through this network but your input Matrix has the wrong dimensions, PyTorch will instantly throw a RuntimeError: mat1 and mat2 shapes cannot be multiplied. Knowing matrix math rules prevents this.

Prerequisites

• Linear Algebra Basics
• NumPy

Detailed Explanation & Examples

1. Vector and Matrix Addition

To add two vectors or matrices, they must have the exact same dimensions. You simply add the numbers in matching positions.

[1, 2] + [3, 4] = [(1+3), (2+4)] = [4, 6]

2. Scalar Multiplication

If you multiply a Matrix by a single number (a Scalar), every individual element inside the Matrix is multiplied by that number.

2 * [1, 3] = [(2*1), (2*3)] = [2, 6]

3. The Dot Product (Matrix Multiplication)

This is the core engine of Neural Networks. Unlike addition, you do not just multiply matching positions.

The Golden Rule of Matrix Multiplication: To multiply Matrix A by Matrix B, the number of columns in Matrix A MUST equal the number of rows in Matrix B. If A is (3 x 2) and B is (2 x 4), you can multiply them. The inner numbers (2) cancel out, and the resulting Matrix will have the shape of the outer numbers (3 x 4).

If A is (3 x 5) and B is (2 x 4), the operation is mathematically impossible. This is the cause of 90% of beginner errors in PyTorch.

4. Transposition

Transposing a matrix simply means swapping its rows with its columns. If a matrix is shape (3 x 2), its transpose is shape (2 x 3). This is often done to force two matrices to align with the Golden Rule of Matrix Multiplication.

Visual Diagram (Mermaid)

graph TD
    A[Matrix Operations] --> B(Addition/Subtraction)
    A --> C(Scalar Multiplication)
    A --> D(Dot Product)
    A --> E(Transposition)
    
    B -.-> B1[Shapes MUST match exactly]
    D -.-> D1[Columns of A MUST match Rows of B]
    E -.-> E1[Rows become Columns]
    
    style D fill:#EF4444,stroke:#fff,color:#fff

Python Code Examples

We rely heavily on NumPy's np.dot() function to perform Matrix Multiplication.

import numpy as np

# 1. Addition (Element-wise)
matrix_A = np.array([[1, 2], [3, 4]])
matrix_B = np.array([[5, 6], [7, 8]])

print("Addition:\n", matrix_A + matrix_B)

# 2. Scalar Multiplication
print("Scalar Multiplication:\n", matrix_A * 10)

# 3. The Dot Product (Matrix Multiplication)
# A is (2x3)
A = np.array([
    [1, 2, 3],
    [4, 5, 6]
])

# B is (3x2)
B = np.array([
    [7, 8],
    [9, 1],
    [2, 3]
])

# Because columns of A (3) == rows of B (3), we can multiply!
# The result will be a (2x2) matrix.
result = np.dot(A, B)
print("Dot Product Result:\n", result)

# 4. Transposition (using .T)
print("Original A shape:", A.shape)
print("Transposed A:\n", A.T)
print("Transposed A shape:", A.T.shape) # Output: (3, 2)

Industry Use Cases

• Forward Propagation in Neural Networks: In a neural network, the "Input Data" (a vector) is multiplied by the "Weights" (a matrix) using the Dot Product to calculate the prediction of the next layer. This happens millions of times a second during training.

Advantages

• The Dot Product allows us to calculate the weighted sum of thousands of inputs simultaneously in a single, highly optimized hardware operation on a GPU.

Common Mistakes

• Element-wise vs. Dot Product: Using the * operator in NumPy performs Element-wise multiplication (multiplying matching positions). This is NOT standard Matrix Multiplication. To do mathematical matrix multiplication, you must use np.dot(A, B) or the @ operator A @ B.

FAQs

Q: Do I need to memorize the formula for the Dot Product? A: No, NumPy handles the arithmetic. However, you absolutely must memorize the "Golden Rule" regarding matrix shapes, or you will not be able to debug your code.

Summary

Matrices and Vectors are manipulated using standard arithmetic and the powerful Dot Product. By ensuring the shapes of your matrices align properly, you can leverage these operations to pass massive datasets through complex Artificial Intelligence models in milliseconds.

Next Topic

Linear Algebra allows us to structure and multiply our data. But how does an AI know if its multiplication was "correct", and how does it learn to improve? For that, we need the math of change: Calculus Basics.