RAG that actually works in production

A demo works because the person asking the question already knows the answer is in the document they just uploaded. The query and the answer live in the same paragraph, retrieval is trivial, and the model looks brilliant.

Production traffic is nothing like that. Documents are stale or contradictory, queries are ambiguous or multi-part, and users phrase things in ways nobody anticipated. The same system that aced the demo now confidently makes things up, and you only find out when a customer does.

The first move is to stop testing on the happy path. Pull 100 to 200 real questions from support tickets, search logs, or a pilot group, write down the correct answer and its source for each, and treat that set as the thing you are actually building against.

If the right chunk never makes it into the context window, no model, however capable, can answer correctly. That makes retrieval quality the single biggest lever, and the one teams most often neglect because it is not the exciting part.

We treat retrieval as its own system with its own metrics. The key one is recall@k: for each question in the evaluation set, did at least one passage containing the answer appear in the top k results? Aim for recall@5 above 90 percent before you spend any time on prompt wording. If it is at 60 percent, the model is not your problem.

Evaluating retrieval separately from generation means that when answer quality drops, you know which half to fix instead of guessing.

Most retrieval gains come from a handful of unglamorous decisions. In rough order of impact:

• Chunk size: 300 to 800 tokens, split on structure (headings, paragraphs) not a fixed character count, with 10 to 20 percent overlap so answers that straddle a boundary are not lost.
• Hybrid search: combine BM25 keyword scoring with vector similarity. Pure vector search misses exact terms like error codes, product names, and acronyms; pure keyword misses paraphrase.
• Re-ranking: run the top 20 to 50 candidates through a cross-encoder re-ranker and keep the top 5. This is usually the single biggest quality jump for the least effort.
• Metadata filtering: attach source, date, product, and access level to every chunk, and filter before ranking so a user never retrieves a document they should not see or a version that is out of date.
• Embedding model: pick one sized to your domain and budget, and re-embed everything when you change it. Do not mix embeddings from two models in the same index.
• Deduplication: near-identical chunks from boilerplate (headers, legal footers) crowd out real content. Strip or collapse them at ingest.

Every answer should cite its sources. Citations let users verify a response, and they let you debug one, because you can see exactly which passages the model was reasoning over. Render them as links back to the source document and section, not just a footnote number.

Guardrails handle the cases retrieval cannot. When nothing useful comes back, the system should say 'I do not have that information' rather than invent a plausible answer. Set a minimum retrieval score threshold below which the system refuses to answer at all. A confident wrong answer is far more damaging than an honest 'I do not know', especially in regulated domains.

For high-stakes answers, add a second check: a lightweight model or rule pass that verifies the response only makes claims supported by the retrieved text, and flags or blocks anything that drifts.

A RAG call is several steps: embed the query, search, re-rank, assemble the prompt, generate. Each adds latency and, for the model calls, cost. Decide the budget up front. For an interactive assistant, aim for a p95 end-to-end latency under 3 seconds and stream the answer so the user sees progress sooner.

The re-ranker and the generation step dominate. You can cut cost meaningfully by caching embeddings, caching answers to common questions, using a smaller model for straightforward lookups and reserving the frontier model for genuinely hard queries, and trimming the context to the passages that actually matter rather than stuffing the window.

Track cost per query and p95 latency on a dashboard from day one. They creep up quietly as the corpus and traffic grow.

An evaluation set, the collection of real questions with known-good answers, turns 'it feels better' into a number you can track. Without it, every change is a gamble.

Run the eval on every meaningful change: a prompt tweak, a new embedding model, a chunking change, a corpus update. Score two things separately: retrieval (recall@k) and answer quality (faithfulness to the retrieved text, and correctness against the known answer). A change that helps one question but quietly breaks five others is caught before it ships, not after.

Add lightweight production monitoring too: log queries that returned nothing above threshold, answers users marked unhelpful, and cases where the model refused. Those logs are your next batch of evaluation questions.

RAG is for questions answered by retrieving and summarizing existing text. It is a poor fit for tasks that need computation, multi-step reasoning over structured data, or an action in another system. If the real question is 'how many open invoices does this customer have', that is a database query with a natural-language wrapper, not RAG.

It is also the wrong tool when the knowledge changes faster than you can re-index, or when correctness requires reconciling many sources rather than quoting one. In those cases a smaller, more deliberate pipeline, or a different architecture entirely, will serve better.

Knowing where the technique stops working is part of using it well.

A RAG system is only as current as its index. Decide up front how documents get in and how updates propagate: a nightly full rebuild is simple and fine for a slow-moving corpus, but a knowledge base that changes hourly needs incremental ingestion triggered by the source system.

Handle deletions explicitly. A document removed from the source but left in the index is a landmine, the model will happily cite policy that no longer applies. Track a stable document id, and on each sync reconcile: add new, update changed, remove deleted.

Version the chunks with an ingested-at timestamp so you can answer 'why did the assistant say that last Tuesday' and so retrieval can prefer the newest version when duplicates exist.

If different users are allowed to see different documents, access control cannot be an afterthought bolted on after generation. It has to happen at retrieval time: filter the candidate set to what this user is permitted to see before ranking, using the metadata attached at ingest.

Doing it after generation is both a leak risk and a quality problem, the model may have already reasoned over a passage it then has to redact, producing a mangled answer.

Test this deliberately with an evaluation set per role, and treat a cross-tenant retrieval as a security incident, not a bug.