Production RAG: Beyond the Demo

Retrieval-Augmented Generation is the most over-demoed and under-engineered pattern in AI. The demo is trivial: embed some documents, stuff the top matches into a prompt, generate an answer. The production system is where the real work lives.

The demo that lies to you

Here is the canonical demo, and it works beautifully — on ten clean documents:

from openai import OpenAI
 
client = OpenAI()
 
def answer(question: str, chunks: list[str]) -> str:
    context = "\n\n".join(chunks)
    resp = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "Answer using only the context."},
            {"role": "user", "content": f"Context:\n{context}\n\nQ: {question}"},
        ],
    )
    return resp.choices[0].message.content

Scale that to ten thousand messy documents and the cracks appear immediately: retrieval returns near-duplicates, the model answers confidently from the wrong chunk, and nobody notices until a user does.

What actually moves the needle

In our experience, three things separate a demo from a system.

1. Chunking is a modeling decision, not a config value

Splitting on a fixed token count destroys the exact structure that makes a document retrievable. Chunk on semantic boundaries — headings, sections, logical units — and carry metadata (source, section, timestamp) through the pipeline so you can filter and cite.

2. Evaluation before optimization

You cannot improve what you cannot measure. Build a labelled question set and score retrieval separately from generation:

def recall_at_k(retrieved: list[str], relevant: set[str], k: int) -> float:
    hits = sum(1 for doc in retrieved[:k] if doc in relevant)
    return hits / max(len(relevant), 1)

If recall@k is low, no amount of prompt engineering will save the answer. Fix retrieval first.

3. An honest failure mode

The worst RAG failure is a confident answer with no supporting context. Give the model an escape hatch — "say you don't know if the context doesn't contain the answer" — and enforce it with a grounding check before you show the response to a user.

The takeaway

RAG is not a library you install; it is a system you evaluate. The teams who win treat retrieval quality as a metric they own, not a side effect of a vector database. That is the entire game.