What is RAG? The complete guide to Retrieval-Augmented Generation

What is RAG?

RAG was introduced by Meta AI in 2020 as a way to combine parametric knowledge (what the LLM learned during training) with non-parametric knowledge (live documents in an external store). The key insight: instead of baking all knowledge into model weights, you retrieve it on demand.

In a naive setup, an LLM answers from training data alone — stale, unverifiable, and prone to hallucination. With RAG, every prompt is augmented with the top-K most relevant document chunks pulled from a vector store. The model now has actual evidence to ground its answer.

Two patterns dominate in practice. Naive RAG chunks documents at a fixed size, runs a single vector search, and hands results to the LLM — quick to build, fine for prototypes. Advanced RAG layers hybrid search (BM25 + vector), cross-encoder reranking, query rewriting, and retrieval monitoring — required for enterprise-grade accuracy and trust.

RAG is a data pipeline, not a chat feature. Chunking, embedding, indexing, and reranking all need the same care as any production pipeline: versioning, monitoring, and regression tests on retrieval quality.

Why does RAG matter?

• Answers are grounded in your actual documents — not the model's training cutoff
• Citations include page numbers and similarity scores so users can verify
• Knowledge updates require re-indexing, not retraining — minutes vs days
• Context window is used efficiently: only the top relevant chunks, not the whole corpus
• Retrieval explainability — you can show exactly why each chunk was chosen
• Compliance is tractable — sensitive documents stay in your control plane

How does RAG work?

RAG is a four-stage pipeline. Documents flow in at ingest time; queries flow through at inference time.

1. Ingest — parse documents, split into chunks, embed each chunk with an embedding model (OpenAI text-embedding-3-small, Cohere embed-v3, or open-source bge-large), store vectors + metadata in a vector DB (Pinecone, Weaviate, Chroma, pgvector)
2. Retrieve — embed the user query with the same model, run vector similarity search for the top-K chunks (typically K=20), optionally add BM25 keyword search and merge the two ranked lists
3. Rerank — pass the top-20 retrieved chunks through a cross-encoder reranker (Cohere Rerank, BGE reranker) to re-score for actual relevance and select the top 3–5
4. Generate — build a prompt containing the user query plus the top reranked chunks, send to the LLM, stream the response, return citations linking each claim back to its source chunk

A minimal RAG pipeline in LangChain looks like this:

from langchain.vectorstores import Chroma
from langchain.embeddings import OpenAIEmbeddings
from langchain.chains import RetrievalQA
from langchain.chat_models import ChatOpenAI

# 1. Ingest: chunk + embed + store
vectordb = Chroma.from_documents(
    documents=chunks,
    embedding=OpenAIEmbeddings(model="text-embedding-3-small"),
)

# 2. Retrieve: top-4 relevant chunks
retriever = vectordb.as_retriever(search_kwargs={"k": 4})

# 3. Generate: LLM answers with context + citations
qa = RetrievalQA.from_chain_type(
    llm=ChatOpenAI(model="gpt-4o"),
    retriever=retriever,
    return_source_documents=True,
)
result = qa.invoke({"query": "What is our refund policy?"})

RAG vs fine-tuning

The single most common RAG question is: when do I retrieve vs when do I fine-tune?

Use RAG when knowledge changes. Use fine-tuning to change how the model responds, not what it knows. Many production systems combine both — fine-tune the model on the company's tone and structured output format, then use RAG to inject up-to-date facts.

RAG vs prompt engineering vs agents

RAG is one of four mainstream approaches to inject knowledge into an LLM. Each has a clear best-fit:

• Prompt engineering — paste static context into every prompt. Works for small, stable knowledge sets that fit in the context window. Breaks down past a few dozen pages.
• RAG — dynamically fetch the relevant context per query. Scales to millions of documents. The default for enterprise knowledge.
• Fine-tuning — train the model on your domain. Better for changing model behavior than model knowledge.
• Agents with tools — the LLM calls APIs (search, SQL, calculators) and reasons over the results. Best for multi-step reasoning over live data.

Most production AI features combine two or three. A customer-support copilot might use RAG for product docs, a fine-tuned model for response style, and a tool-using agent for live order lookups.

Common mistakes (and what to do instead)

• Chunks too large or too small — 4,000-token chunks dilute relevance scores; 50-token chunks lose sentence context. Sweet spot is 256–512 tokens with 10–20% overlap. Tune per document type.
• Vector search alone, no BM25 — vector search misses exact keyword matches like product names, error codes, and version numbers. Hybrid search (BM25 + vector) improves precision by 30–40% on enterprise corpora.
• Skipping the reranker — vector similarity scores are noisy at retrieval time. A cross-encoder reranker re-scores the top-20 against the query and selects the actual top-4. Without reranking, quality drops noticeably.
• No retrieval evaluation — most teams ship RAG without measuring retrieval quality. Track hit rate (was the answer chunk in the top-K?), MRR, and NDCG. Without metrics, you are flying blind.
• Treating RAG as a chat feature, not a pipeline — chunking, embedding, indexing, and reranking all need versioning, monitoring, and regression testing — exactly like any production data pipeline.
• Mixing embedding models across versions — re-embedding documents with a different model invalidates the entire index. Pin the embedding model + version in metadata.

Who is RAG for?

RAG is for AI engineers, ML engineers, and data engineers building grounded, citation-backed LLM features for production. If your product needs to answer questions from a knowledge base that changes more than once per quarter, RAG is almost certainly the right starting point.

Teams that benefit most:

• Enterprises shipping internal Q&A copilots over wikis, runbooks, and policy docs
• Support teams giving agents suggested answers with verifiable source links
• Legal and compliance teams making contracts, regulations, and case law searchable in natural language
• Data platform teams indexing logs, metrics, and runbooks for on-call engineers to query during incidents