Generative AI and the Transformer Architecture

Learn the foundations of Generative AI. Explore the Transformer architecture, the Self-Attention mechanism (Queries, Keys, Values), and encoder-decoder models.

Introduction

Before 2017, the gold standard for language translation was the LSTM. While LSTMs were a massive step up from basic RNNs, they had a critical bottleneck: they had to process text word-by-word sequentially. This made training slow and prohibited models from scaling to billions of parameters.

In June 2017, Google researchers published a landmark paper titled "Attention Is All You Need". They introduced the Transformer—a deep learning architecture that abandoned recurrence entirely in favor of Self-Attention. This breakthrough enabled parallel processing, laid the foundation for modern Generative AI, and birthed Large Language Models (LLMs) like GPT and Claude.

What You Will Learn

  • The core components of the Transformer.
  • How the Self-Attention Mechanism works (Queries, Keys, Values).
  • The difference between Encoder and Decoder modules.
  • Why Transformers can process text in parallel.

Why This Topic Matters

The Transformer is the foundational engine of modern artificial intelligence. It is not just used for text; it has been adapted for images (Vision Transformers), audio (Whisper), and biology (AlphaFold). To work with modern foundation models, you must understand the mathematical mechanics that drive the self-attention calculations.

Prerequisites

Detailed Explanation

The Transformer's key innovation is the ability to look at all words in a sentence simultaneously and determine which words are most relevant to one another, regardless of their distance apart.

The Self-Attention Mechanism

Self-Attention allows a model to associate each word in the input with every other word. For example, in the sentence: > "The animal didn't cross the street because it was too tired."

A human knows that "it" refers to the "animal", not the "street". Self-attention calculates this association mathematically.

The Query, Key, and Value Analogy

Imagine searching for a video on YouTube:

  • Query ($Q$): The search term you type (e.g., "AI tutorial").
  • Key ($K$): The titles/tags of all videos in the database.
  • Value ($V$): The actual video content you want to watch.

The search engine compares your Query against all Keys in the database to calculate similarity scores (attention weights). It then returns a weighted sum of the Values matching the highest similarity scores.

The Mathematical Formula of Scaled Dot-Product Attention:

$$\text{Attention}(Q, K, V) = \text{Softmax}\left(\frac{Q K^T}{\sqrt{d_k}}\right) V$$

Where:

  • $Q, K, V$ are matrices representing Queries, Keys, and Values.
  • $d_k$ is the dimension of the keys (used as a scaling factor $\sqrt{d_k}$ to prevent gradients from vanishing in the Softmax).
  • $QK^T$ measures the raw similarity (dot product) between every pair of words.

Encoder vs. Decoder

The original Transformer features two main blocks:

graph LR
    A[Input Text] --> B[Encoder]
    B -->|Context Vectors| C[Decoder]
    C --> D[Output Text]
  1. The Encoder: Processes the input sequence and constructs a dense representation (context vector) containing the meaning of the entire text.
    • Architecture: Bidirectional self-attention (looks at both left and right context).
    • Example models: BERT (good for analysis, classification, search).
  2. The Decoder: Takes the encoder's context representation and generates outputs token-by-token.
    • Architecture: Masked self-attention (can only look at past generated tokens, preventing it from "peeking" at future words).
    • Example models: GPT series, Llama (good for text generation, conversation).

Positional Encoding

Since Transformers process all tokens in parallel rather than step-by-step, they have no inherent concept of word order. To fix this, we add a mathematical wave function (sine and cosine waves) to the input embeddings. This is called Positional Encoding. It injects coordinate tags so the model knows the exact position of each word in the sentence.

Visual Diagram (Mermaid)

graph TD
    subgraph Transformer Architecture
    A[Input Embeddings] --> B[Positional Encoding]
    B --> C[Multi-Head Attention]
    C --> D[Add & Normalize]
    D --> E[Feed Forward Network]
    E --> F[Add & Normalize]
    end
    style C fill:#3B82F6,stroke:#fff,color:#fff
    style E fill:#10B981,stroke:#fff,color:#fff

Python Code Examples

We will write a simplified PyTorch script demonstrating the math of Scaled Dot-Product Attention.

import torch
import torch.nn.functional as F

# 1. Simulate Query, Key, and Value matrices for a sentence of 3 words
# Sequence length = 3, Hidden dimension = 4
Q = torch.randn(3, 4)
K = torch.randn(3, 4)
V = torch.randn(3, 4)

print("Queries Matrix (Q):\n", Q)

# 2. Step 1: Calculate Dot-Product similarity (Q * K_transpose)
scores = torch.matmul(Q, K.transpose(0, 1))
print("\nRaw Similarity Scores (Q K^T):\n", scores)

# 3. Step 2: Scale the scores by sqrt(d_k) where d_k = 4
d_k = K.shape[-1]
scaled_scores = scores / (d_k ** 0.5)

# 4. Step 3: Apply Softmax to get Attention Weights (rows sum to 1)
attention_weights = F.softmax(scaled_scores, dim=-1)
print("\nAttention Weights (Probability distribution):\n", attention_weights)

# 5. Step 4: Multiply by Values (V) to get the final representation
output = torch.matmul(attention_weights, V)
print("\nFinal Attention Output:\n", output)
print("Output Shape:", output.shape) # Expected: [3, 4]

Industry Use Cases

  • Machine Translation: Translating documents across languages in parallel, making systems like DeepL or Google Translate fast and contextual.
  • DNA Sequence Analysis: Re-purposing self-attention to process protein structures (AlphaFold) to predict molecular folds.
  • Code Autocompletion: Powering code completion tools like GitHub Copilot by analyzing surrounding code context.

Summary

The Transformer architecture replaced recurrent designs with self-attention, permitting models to calculate token associations in parallel. By representing inputs as Queries, Keys, and Values, and applying Positional Encodings to preserve sequence locations, Transformers enabled the training of the massive foundation networks that drive modern Generative AI.

Next Topic

How do we scale these Transformer networks to billions of parameters, pre-train them on the internet, and align them to follow human instructions? Let's check: Large Language Models (LLMs): Pre-Training, Fine-Tuning, and RLHF.

← Back to Syllabus