Retrieval-Augmented Generation (RAG): Architecture and Vector DBs

Learn Retrieval-Augmented Generation (RAG). Understand vector databases, document embeddings, retrieval mechanisms, and context injection.

Introduction

If you ask a pre-trained LLM: "What did our company decide in yesterday's board meeting?", it cannot answer. It doesn't have access to your private company data, and its knowledge is frozen at the time of its training.

We could fine-tune the model on yesterday's documents, but that is slow, expensive, and risks leaking private information. Instead, we use Retrieval-Augmented Generation (RAG). RAG is an architectural pattern where we search a database for documents relevant to the user's query, inject those documents into the prompt as context, and ask the LLM to generate an answer based only on the retrieved text.

What You Will Learn

The core pipeline of a RAG system.
How Document Chunking and Embeddings prepare data.
The role of Vector Databases in semantic search.
The step-by-step retrieval and generation flow.
How RAG prevents LLM hallucinations.

Why This Topic Matters

RAG is the most deployed architecture in enterprise AI today. It allows companies to connect LLMs to their internal wikis, PDFs, databases, and customer records. By referencing source files directly, RAG systems provide highly accurate, auditable answers with source citations, making them safe for commercial use.

Prerequisites

Detailed Explanation

A RAG pipeline consists of two distinct stages: the Ingestion Pipeline (offline) and the Retrieval/Generation Flow (online).

1. The Ingestion Pipeline (Offline)

Before we can search documents, we must convert them into a format searchable by computers:

graph TD
    A[Company PDFs & Wikis] --> B[Document Chunking: Split into paragraphs]
    B --> C[Embedding Model: Convert chunks to 1536-dim vectors]
    C --> D[Vector Database: Index and store vectors]

Document Chunking: LLMs have context window limits. We split large PDFs or files into smaller, overlapping sections called Chunks (e.g., 500 characters with 50 characters overlap).
Generating Embeddings: We pass each chunk through an embedding model (like OpenAI's text-embedding-3-small) to generate a dense vector representing its semantic meaning.
Vector Database: We store these vectors in specialized indexes (like Pinecone, Milvus, Chroma, or pgvector) for fast mathematical lookup.

2. The Retrieval & Generation Flow (Online)

When a user asks a query:

[User Query]
     |
     v
(Generate Query Embedding)
     |
     v
(Search Vector DB: Cosine Similarity) ----> [Top 3 Relevant Chunks]
                                                    |
                                                    v
[Injected Prompt: Context + User Query] ------------+
     |
     v
  [LLM] ---> [Response based purely on Context]

Query Embedding: Convert the user's query into a vector using the same embedding model.
Semantic Search: Query the Vector Database using Cosine Similarity to find the $K$ closest chunks to the query vector.

Context Injection: Insert these retrieved text chunks into the prompt:

Answer the question using ONLY the context provided below.
Context:
---
[Retrieved Chunk 1]
[Retrieved Chunk 2]
---
Question: [User Query]

Answer Generation: The LLM reads the injected prompt and generates a factual response anchored in the context.

Vector Databases vs. Relational Databases

Python Code Examples

We will write a python script simulating a complete vector similarity search block using NumPy.

import numpy as np

# 1. Simulate a Vector Database containing 3 document chunks
# Represented as simple 3-dimensional vectors for visualization
document_db = {
    "Doc 1 (Benefits Policy)": np.array([0.9, 0.1, 0.0]),
    "Doc 2 (Office Address)": np.array([0.1, 0.9, 0.0]),
    "Doc 3 (SQL database guide)": np.array([0.0, 0.1, 0.9])
}

# 2. User Query: "Where is the office located?"
# The embedding model maps this query close to Doc 2
query_vector = np.array([0.15, 0.85, 0.05])

def cosine_similarity(v1, v2):
    dot_product = np.dot(v1, v2)
    norm_v1 = np.linalg.norm(v1)
    norm_v2 = np.linalg.norm(v2)
    return dot_product / (norm_v1 * norm_v2)

# 3. Perform semantic search (calculate cosine similarity)
search_results = {}
for doc_name, doc_vector in document_db.items():
    sim = cosine_similarity(query_vector, doc_vector)
    search_results[doc_name] = sim

# Sort results by similarity
sorted_results = sorted(search_results.items(), key=lambda x: x[1], reverse=True)

print("Query Vector:", query_vector)
print("\nSearch Results (ranked by relevance):")
for doc, score in sorted_results:
    print(f" - {doc}: Similarity Score = {round(score, 4)}")
# Expected: Doc 2 has the highest score, representing the most relevant context.

Industry Use Cases

Internal Enterprise Search Bots: Letting employees query HR handbooks, sales figures, and technical Wikis.
Customer Support Automation: Answering user queries about product features by referencing user manuals.
Financial Research Agents: Scanning thousands of PDF earnings reports to extract fiscal data for portfolio managers.

Summary

RAG combines the database search capabilities of Vector Databases with the text synthesis power of LLMs. By chunking files, converting them into semantic embeddings, and injecting matches directly into LLM prompts as grounded context, RAG solves the constraints of frozen knowledge and hallucinations in corporate deployments.

Next Topic

How do we build complex multi-step RAG pipelines, manage conversations, and orchestrate agent actions in Python? Let's check: Building LLM Applications with LangChain and LlamaIndex.

← Back to Syllabus