RAG (Retrieval-Augmented Generation): How LLMs Use External Knowledge
RAG (Retrieval-Augmented Generation) combines a search system with a language model. The model retrieves relevant passages from your knowledge base, then generates an answer grounded in those passages — instead of relying only on what it memorized during training.
Why RAG exists
A standalone LLM has two problems for production:
- Knowledge cutoff. Models are trained on data up to a fixed date. They cannot answer questions about events, products, or documents that appeared after that date.
- Hallucination. Without grounding, models confidently invent plausible-sounding but false details.
RAG addresses both: the LLM only sees text you supplied, and you can update that text without retraining the model.
The architecture in 5 steps
- Index your corpus. Split documents into chunks (300-1000 tokens typical). Encode each chunk as a vector embedding using a model like
text-embedding-3-largeorbge-large. - Store vectors. Save embeddings in a vector database (pgvector, Pinecone, Qdrant, Weaviate, Chroma).
- Query time: embed the question. Use the same embedding model.
- Retrieve top-K. Approximate nearest neighbor search returns the K most relevant chunks (typically K = 4-20).
- Generate with context. Pass the retrieved chunks + the user question to the LLM in a single prompt: “Given these documents: [chunks], answer: [question].”
When RAG beats fine-tuning
| Scenario | RAG | Fine-tuning |
|---|---|---|
| Frequently changing knowledge (docs, prices, news) | ✓ | ✗ |
| Need to cite sources for compliance/trust | ✓ | ✗ |
| Inject company-specific terminology | partial | ✓ |
| Change model behavior/style | ✗ | ✓ |
| Cost per query | higher (retrieval + larger context) | lower |
| Setup cost | low | high (training run + dataset prep) |
Rule of thumb: RAG for facts that change, fine-tuning for skills that persist. Often you want both.
Where RAG fails in practice
- Bad retrieval = bad answer. If the top-K chunks don’t contain the needed information, the LLM will either hallucinate or refuse. Spend time on chunking strategy, embedding quality, and reranking.
- Long-context dilution. Stuffing 50 chunks into a prompt degrades reasoning quality. Use reranking (e.g., Cohere Rerank, BGE Reranker) to filter top-K down to top-3.
- Cross-document reasoning. RAG is good at “find the answer in one of these docs.” It’s bad at “synthesize a conclusion from contradictions across 20 docs.” Use multi-hop retrieval or graph-RAG for that.
Production checklist
- Evaluation set with question → expected source mapping
- Hybrid search (vector + BM25 keyword) — pure vector misses exact terms
- Reranker between retrieval and generation
- Source citations in the response (every claim → which chunk)
- Negative test: question outside corpus → model says “I don’t know” instead of hallucinating
RAG is not a model technique. It is an information architecture decision. The LLM is interchangeable; the corpus, the chunking, and the retrieval pipeline are where the work lives.