Retrieval-Augmented Generation (RAG) is an architecture pattern where you enhance LLM responses by retrieving relevant information from external data sources and injecting it into the prompt at inference time. The model generates its response grounded in this retrieved context — not just its pre-trained knowledge.

This separation is critical to understand:

LLMs have powerful language understanding and generation capabilities, but they ship with three fundamental limitations that RAG directly addresses:

The most common question: "Should I fine-tune or use RAG?" This is not an either/or — they solve different problems and can be combined. Here's how to decide:

A production RAG system has 6 core stages. Each stage has multiple design choices — and getting all of them right is what separates "RAG that works in demos" from "RAG that works in production."

Retrieval finds relevant chunks. But the LLM doesn't understand "relevance" — it only processes tokens. This means how you assemble those chunks into the prompt matters enormously.

User queries are often vague, incomplete, or poorly structured. A single word like "refund" is not a good search query. Production systems improve retrieval by transforming queries before search.

Techniques include: rewriting for clarity, expanding keywords, generating multiple search queries from one question, and using the LLM itself to reformulate. Chapters 5 and 7 cover these strategies in detail.

You can build a working RAG demo in 20 lines of code. You can also watch it fail spectacularly in production. The gap between "it works on my laptop" and "it works for 1000 users on real data" is filled with engineering challenges that naive implementations ignore.

RAG is not free. Each query involves multiple steps — embedding lookup, vector search, optional re-ranking, larger prompt construction — each adding latency and cost.

You can't improve what you can't measure. RAG quality is measured in two independent dimensions: retrieval quality (did we find the right documents?) and generation quality (did we produce a correct, grounded answer?).

RAG is powerful, but it's not a universal solution. Some problems are better solved with other approaches — and forcing RAG where it doesn't fit leads to complex systems that underperform simpler alternatives.

Before you can retrieve, you need to ingest. The ingestion pipeline transforms raw documents — PDFs, web pages, Notion exports, database dumps — into indexed, searchable chunks. Each step has failure modes that propagate downstream.

Your knowledge lives in dozens of formats. Each format has quirks that affect text extraction quality. Use the right loader for each source — and always validate output before proceeding.

Chunking determines what your retriever can find. Too large, and irrelevant content dilutes the signal. Too small, and context is lost. The right strategy depends on your document type and query patterns.

Chunk size is a tradeoff between precision and context. There's no universal answer — it depends on query type, document structure, and embedding model capabilities.

Production RAG systems often use more sophisticated chunking patterns that decouple what you search from what you retrieve. These add complexity but can significantly improve quality.

Metadata enables filtering before semantic search — dramatically improving precision. Every chunk should carry metadata that answers: where did this come from, when, and who can see it?

An embedding is a fixed-length vector (array of numbers) that represents the semantic meaning of text. Embedding models are trained so that semantically similar text produces similar vectors — enabling "meaning-based" search rather than keyword matching.

There are dozens of embedding models available. They differ in quality (MTEB benchmarks), dimensionality (affects storage/speed), cost, and whether they can run locally. Here are the ones that matter in 2026:

Higher dimensions capture more nuance but cost more to store and search. OpenAI's embedding-3 models support dimension reduction via Matryoshka Representation Learning — you can truncate vectors without retraining.

Queries and documents are fundamentally different: queries are short questions, documents are long answers. Some embedding models handle this asymmetry explicitly — and they perform significantly better for retrieval tasks.

General-purpose embeddings work well for most text. But for specialized domains (legal, medical, code), fine-tuning on your data can improve retrieval quality significantly — 10–30% gains are common.

MTEB (Massive Text Embedding Benchmark) is the industry standard for comparing embedding models. It evaluates models across 56+ datasets in retrieval, classification, clustering, and more. But MTEB alone isn't enough — you must test on your own data.

You could store vectors in PostgreSQL as arrays. But when you have 10 million vectors, computing cosine similarity against all of them takes minutes. Vector databases use specialized indices that trade perfect accuracy for 100–1000× faster search.

The vector database market has exploded. Here's an honest comparison of the major options — there's no single "best" choice, only tradeoffs for your use case.

ANN search works by building an index — a data structure that allows skipping most vectors during search. The two dominant index types are HNSW and IVF. Most modern vector databases default to HNSW.

Here's how to set up the most common vector databases. All examples use Python and store 1536-dimensional embeddings with metadata.

Vector databases scale differently than traditional databases. Memory is usually the bottleneck, not disk. Here's what to expect:

There are fundamentally two ways to find relevant documents: dense retrieval (semantic similarity via embeddings) and sparse retrieval (keyword matching via inverted indices). Both have strengths; the best systems use both.

Hybrid search runs both dense and sparse retrieval, then combines the results. The key question: how do you merge two ranked lists? The most common approach is Reciprocal Rank Fusion (RRF).

User queries are often poor search queries: too short, ambiguous, or phrased as questions when documents are statements. Query enhancement transforms the user's question into something more likely to retrieve relevant documents.

Sometimes you know constraints before search: only documents from this year, only from the engineering team, only product docs. Metadata filtering narrows the search space, improving both precision and speed.

Real applications have multiple document types: FAQs, documentation, support tickets, code. Each may benefit from different chunking, embeddings, or ranking. Multi-index retrieval queries multiple specialized indices and merges results.

Embedding-based retrieval is fast but shallow — it computes similarity independently for each document without comparing query and document together. Re-ranking takes the top-k candidates and applies a more powerful (but slower) model that reads query + document jointly.

Research shows LLMs pay more attention to information at the beginning and end of the context window, but tend to miss information in the middle. This "lost in the middle" effect means document ordering matters.

If your top 5 results are 5 chunks from the same document saying the same thing, you waste context and miss other relevant information. Diversity ranking ensures retrieved results cover different aspects of the query.

Every RAG prompt has four parts. Getting each part right — and getting the ordering right — is what separates good RAG from great RAG.

Citations serve two purposes: they let users verify answers and they reduce hallucination by forcing the model to ground statements in specific sources. Without citations, you can't tell if the model invented something.

Even with 128K-token context windows, more context is not always better. Cost increases linearly, latency increases, and the "lost in the middle" effect gets worse. Smart context management is about using the window efficiently.

The hardest part of RAG isn't answering questions — it's knowing when not to answer. When retrieved context doesn't contain the answer, the LLM should say "I don't know" rather than hallucinate.

In conversation, follow-up questions reference earlier context: "What about for enterprise plans?" requires knowing the previous question was about pricing. Query rewriting transforms follow-ups into standalone queries for retrieval.

RAG failures fall into two categories: retrieval failures (wrong documents found) and generation failures (wrong answer produced from correct documents). Each requires different fixes.

RAGAS (Retrieval Augmented Generation Assessment) is an open-source framework that automates RAG evaluation. It computes faithfulness, answer relevance, and context metrics using LLM-as-judge.

Standard RAG uses whatever the retriever returns, even if the results are irrelevant. CRAG adds a self-correction step: evaluate retrieval quality, and if it's poor, try alternative strategies before generating.

Self-RAG teaches the LLM to decide: (1) whether retrieval is needed, (2) whether retrieved docs are relevant, and (3) whether the generated answer is grounded. The model outputs special reflection tokens during generation.

Standard RAG retrieves individual chunks. GraphRAG first builds a knowledge graph from documents (entities and relationships), then traverses the graph during retrieval. This enables multi-hop reasoning across documents.

In standard RAG, the pipeline is fixed: retrieve → generate. In Agentic RAG, the LLM acts as an agent that decides what to retrieve, when, and how many times. It can reformulate queries, request more context, or search different sources.

Many RAG queries are repetitive or semantically similar. Caching avoids re-running expensive retrieval and LLM calls for questions you've already answered.