The gap between a RAG demo and a RAG product is wide and unforgiving. A weekend prototype that answers questions over a handful of documents looks like the hard part is done; then you point it at millions of documents and thousands of users, and every shortcut you took surfaces at once. The lessons that separate a production retrieval-augmented-generation system from a demo are consistent, and most of them are about retrieval quality, not the language model. Here's where the real work is.

The architecture, briefly

A production RAG pipeline has four stages that each deserve real engineering: chunk the documents, embed the chunks, retrieve the relevant ones for a query, and construct a prompt that lets the model answer from them. Most teams over-invest in the model and under-invest in the first three - which is exactly backwards, because the model can only answer as well as retrieval lets it.

Chunking: where RAG silently fails

Chunking gets treated as a trivial preprocessing step and quietly determines whether the whole system works. Chunk too large and retrieval returns a wall of text where the relevant sentence is buried and diluted; chunk too small and you lose the context that makes a passage meaningful. Naive fixed-size splitting cuts through the middle of ideas. The fixes that matter: respect document structure (split on sections and paragraphs, not character counts), use overlap so a fact spanning a boundary isn't lost, and keep metadata (source, section) on each chunk for filtering and citation. When answers are wrong, bad chunking is the most common silent cause.

Embeddings: pick one and standardize

The embedding model decides what "relevant" means, and switching it means re-embedding everything - so choose deliberately and standardize. The practical lesson is less "which model is best" and more "use the same model for documents and queries, evaluate it on your data, and don't change it casually." A model that tops a public benchmark may underperform a humbler one on your domain; measure retrieval quality on real questions before committing.

Vector store: start simple, scale when forced

A common trajectory is starting on pgvector (Postgres) because the data and the team are already there, then moving to a dedicated vector database as scale demands. The honest framing: pgvector is the right default for modest corpora - one fewer system, transactional, familiar - and you move to Pinecone, Weaviate, or Qdrant when you hit tens of millions of vectors, need lower latency, or want advanced filtering and indexing pgvector strains under. Don't reach for a specialized store before the scale justifies it; don't cling to pgvector once it's clearly the bottleneck.

Reranking: the upgrade that punches above its weight

The single change that most often transforms retrieval quality is adding a reranking step. Vector search is fast but approximate - it returns plausibly-relevant chunks, not the best-ordered ones. A reranker (a cross-encoder that scores each retrieved chunk against the query more precisely) reorders the top candidates so the genuinely best context lands first. Retrieve a broader set with cheap vector search, then rerank to a tight, high-quality few. For many teams this is the difference between "usually right" and "reliably right."

Prompt construction: less is more

The instinct is to stuff every retrieved chunk into the prompt. Resist it. Too much context dilutes the signal, raises cost and latency, and invites the model to wander. Pass the few highest-quality chunks (post-reranking), instruct the model to answer only from them, and tell it to say so when the context doesn't contain the answer - which prevents the confident hallucinations that destroy user trust. Tighter context, clearer instruction, better answers.

The takeaway

Shipping RAG to real users teaches one lesson repeatedly: the quality of the system is set by retrieval, and retrieval is set by the unglamorous stages everyone rushes. Chunk with care, choose and freeze an embedding model evaluated on your data, use the simplest vector store that fits your scale, add reranking, and keep the prompt tight and grounded. The language model is the part you can mostly take off the shelf; the pipeline around it is the part you have to engineer - and it's where every production RAG system is won or lost.