Large Language Models (LLMs): Pre-Training, Fine-Tuning, and RLHF

Understand Large Language Models (LLMs). Explore pre-training objectives, Supervised Fine-Tuning (SFT), and Reinforcement Learning from Human Feedback (RLHF).

Introduction

If you take a raw Transformer Decoder and train it on 10 terabytes of internet text, what do you get? You get a highly sophisticated auto-complete engine. If you prompt it with: "Write a cover letter," it might not give you a cover letter—instead, it might write: "Write a resume, Write an application form" because it is simply trying to predict what text comes next on a web page.

To convert a raw text predictor into a helpful, harmless, and honest assistant like ChatGPT or Claude, we must pass it through the LLM Alignment Pipeline. This training pipeline transforms raw probability models into conversational agents.

What You Will Learn

  • The three core phases of the LLM lifecycle.
  • How Pre-Training builds base language capabilities.
  • How Supervised Fine-Tuning (SFT) teaches instructions.
  • The mechanics of RLHF (Reinforcement Learning from Human Feedback).
  • Parameter-efficient training using LoRA (Low-Rank Adaptation).

Why This Topic Matters

As an AI practitioner, you will rarely train a base model from scratch because it costs millions of dollars in GPU compute. However, you will constantly need to fine-tune open-source models (like Llama 3 or Mistral) on company data. Understanding the alignment pipeline is crucial for customizing models to match corporate tones, code formats, or domain-specific terminology.

Prerequisites

Detailed Explanation

The lifecycle of a modern Large Language Model consists of three consecutive training phases:

graph TD
    A[Phase 1: Pre-training <br> Raw text -> Base Model] --> B[Phase 2: Supervised Fine-Tuning SFT <br> Instruction pairs -> Instruction Model]
    B --> C[Phase 3: RLHF Alignment <br> Human preferences -> Aligned Assistant Model]

Phase 1: Pre-Training (The Foundation)

  • Data: Terabytes of unstructured raw text (Wikipedia, books, GitHub code, web scrapings).
  • Objective: Self-supervised learning. The model reads text, masks the next word, and predicts it: $$P(w_t | w_1, w_2, \dots, w_{t-1})$$
  • Result: A Base Model (e.g., Llama-3-Base). It understands grammar, syntax, world facts, and reasoning, but does not know how to follow directions or hold a conversation.

Phase 2: Supervised Fine-Tuning (SFT)

To teach the base model to behave like an assistant, we train it on a high-quality, curated dataset of instruction-response pairs:

  • Instruction: "Explain photosynthesis."
  • Target: "Photosynthesis is the process used by plants..."
  • Result: An Instruction-tuned Model. It knows how to answer questions, write code, and summarize documents.

Phase 3: RLHF (Reinforcement Learning from Human Feedback)

Instruction models can still output toxic text, reveal private details, or hallucinate. To align the model to human values:

  1. Generate Responses: Ask the model a query and generate multiple candidate answers.
  2. Human Labeling: Ask human evaluators to rank the answers from best to worst.
  3. Train a Reward Model: Train a secondary neural network to predict the human preference score for any response.
  4. PPO Optimization: Use Reinforcement Learning (Proximal Policy Optimization) to adjust the LLM's weights so it maximizes the score predicted by the Reward Model.

Parameter-Efficient Fine-Tuning (PEFT) & LoRA

Fine-tuning all 70 billion parameters of an LLM requires massive GPU clusters. LoRA (Low-Rank Adaptation) solves this:

  • Instead of updating the original weight matrix $W$ (size $d \times k$), we freeze $W$ and train two smaller rank matrices, $A$ and $B$: $$\Delta W = B \times A$$
  • If the rank $r = 8$, we reduce the parameters we need to train by over 99%, allowing engineers to fine-tune LLMs on a single consumer GPU.

Visual Diagram (Mermaid)

graph LR
    subgraph LoRA Optimization
    X[Input Vector x] -->|Path 1: Frozen| W[Original Weights W_0]
    X -->|Path 2: Trainable| A[LoRA Down-Projection matrix A]
    A --> B[LoRA Up-Projection matrix B]
    W & B --> Add((+))
    Add --> Out[Output Vector y]
    end
    style W fill:#EF4444,stroke:#fff,color:#fff
    style A fill:#10B981,stroke:#fff,color:#fff
    style B fill:#10B981,stroke:#fff,color:#fff

Python Code Examples

We will write a conceptual script demonstrating how LoRA parameter reduction works mathematically.

import torch
import torch.nn as nn

# 1. Simulate a standard Linear Layer in an LLM (e.g., projection layer)
# Input features = 4096, Output features = 4096
in_dim = 4096
out_dim = 4096
original_linear = nn.Linear(in_dim, out_dim, bias=False)

# Calculate parameters to update (traditional fine-tuning)
original_params = sum(p.numel() for p in original_linear.parameters())
print(f"Original Parameters to update: {original_params:,}")

# 2. Configure LoRA parameters
# Rank r = 8 (highly compressed low-rank space)
r = 8

# Freeze original weights
for param in original_linear.parameters():
    param.requires_grad = False

# Define LoRA matrices
lora_A = nn.Parameter(torch.randn(in_dim, r))
lora_B = nn.Parameter(torch.zeros(r, out_dim)) # Initialized to zero so Delta W is 0 at start

# Calculate parameters to update with LoRA
lora_params = lora_A.numel() + lora_B.numel()
print(f"LoRA Parameters to update: {lora_params:,}")
print(f"Parameter reduction: {100 - (lora_params / original_params * 100):.4f}%")

Industry Use Cases

  • Legal Document Summarizers: Fine-tuning open-source LLMs on proprietary contracts and court transcripts to extract clauses.
  • Enterprise Customer Support Chatbots: Training models using company databases to answer support queries while suppressing hallucinations.
  • Toxicity Filtering: Deploying aligned LLMs to moderate forum posts and flag hate speech or private information.

Summary

The development of Large Language Models transitions from self-supervised pre-training (next-token prediction on internet text) to Supervised Fine-Tuning (answering instruction sets), and final RLHF alignment (optimizing human preferences). Using techniques like LoRA, developers can compress training requirements and customize open-source LLMs on budget hardware.

Next Topic

Once an LLM is aligned, how do we write inputs to guide its reasoning, prevent errors, and get structured outputs? Let's check: Prompt Engineering: Techniques, Formats, and Best Practices.

← Back to Syllabus