RAG in production: what actually breaks

You can build a RAGdefinitionRetrieval-augmented generation, or RAG. A search first pulls relevant passages from your documents, and a language model writes the answer from those passages. demo in an afternoon. A RAG system takes an engineering program. The demo never has to answer one question. When the retriever hands back the wrong context, does anyone notice before the user does? Everything that separates a dependable assistant from a liability lives in that gap.

We've watched the same three failure modes take down systems that were otherwise well built. In none of them was the model short on cleverness. Each time, somebody was making a claim that nobody measured.

Hallucination gets the attention. In production we more often see a confident answer built on the wrong passages. Bad chunkingdefinitionCutting documents into pieces before indexing. Cut by a fixed token count and a policy clause can end up in one chunk while its exceptions sit in the next one. splits a policy clause from its exceptions. Pure dense retrieval misses the exact identifiers, part numbers and acronyms keyword search would have caught. A stale index serves last quarter's pricing with this quarter's confidence. The model then writes fluently over the wrong evidence, and the answer looks perfectly plausible.

The fixes are unglamorous and well understood. Use hybrid retrieval that combines BM25definitionA classic keyword-ranking method. It reliably finds exact part numbers and acronyms that embedding search can miss. with dense embeddings, and put a rerankerdefinitionA second model that re-scores whatever the search returned, so the passages that matter most reach the language model first. over the candidate set. Chunk along the document's structure instead of a fixed token count. Treat index freshness as an SLO with an owner. Watch for embedding driftdefinitionSwitch embedding models without re-indexing and this is what you get. The stored vectors and the new queries no longer share a coordinate system. as well. Swap the embedding model without re-indexing the corpus and every stored vector now sits in a foreign coordinate system. None of this is research. It's what turns recall you assume into recall you can state.

Without an eval harness, every change to a RAG system is a guess ratified by a demo. The harness is a golden setdefinitionA fixed collection of real questions from users, with known source passages and reference answers. It's what every system change is scored on.: a few hundred real questions with known source passages and reference answers, drawn from actual user queries rather than what the team imagines users ask. Each one is scored on two independent axes: retrieval quality (recall@k, precision of the retrieved context) and generation quality (faithfulness to the retrieved passages, answer relevance). Score them separately, or you'll spend a week tuning prompts to fix what is actually a chunking bug.

Then wire the harness into CI. A new embedding model runs the golden set, and so does a chunking change, a prompt edit or a reranker swap, and each one blocks on regression the way a failing test would. LLM-as-judge scoring keeps this affordable, with the usual caveats. Calibrate the judge against a human-labeled sample, pin its version and audit where it disagrees. The harness won't tell you the system is good. It will tell you it just got worse, and it'll tell you before your users do.

# golden set runs on every change, like tests, because it is one
$ rag-eval run --golden-set v9 --diff main
retrieval  recall@5: 0.87  context precision: 0.74
generation faithfulness: 0.91  vs main: −0.04
regression: 12 questions worse · chunking suspected
✗ merge blocked · evidence attached to PR

What makes a RAG assistant dependable is everything around the model. Start with permission-aware retrieval. The index has to respect document ACLs at query time, or the assistant becomes the most efficient data-leak mechanism in the company, and syncing those ACLs from the source systems is its own pipeline with its own freshness SLO. Then ingestion. Documents change, so the pipeline re-embeds on update and writes tombstones for deletions, which means a retracted policy stops being retrievable the day it's retracted. Without that, what you have is an archive.

Third, the system should be easy to check. Each answer cites the chunks it used, so when an answer is wrong someone can see why within minutes. If retrieval confidence is low, the system replies "I don't have a good source for that" and leaves it there. In every deployment we've reviewed, that scripted refusal has beaten a fluent guess. We also log each query, the chunks retrieved and the citation clicks, and the failures go back into the golden set. That feedback is what improves the system month after month, more than the choice of model.