Retrieval-augmented generation usually arrives with a shopping list: a vector database, an embedding service, an orchestration framework, an LLM API. For a lot of real use cases - answering questions over a few thousand internal documents - that stack is more than the problem needs. If your documents already live in Snowflake, you can build a working RAG pipeline entirely inside the warehouse using Snowflake Cortex, with no external services and no data leaving your governance boundary. Here's the shape of it.

What RAG is, briefly

RAG answers a question by first retrieving the most relevant chunks of your own documents, then handing those chunks to a language model as context so it answers from your data rather than its training. The two halves are retrieval (find the right text) and generation (write the answer grounded in it). Vector embeddings power the retrieval: text is turned into numbers such that similar meanings sit close together, so "find relevant chunks" becomes "find nearby vectors."

Why build it in the warehouse

Keeping RAG in Snowflake removes the parts that make RAG projects heavy. The documents are already there and governed, so there's no export step and no second copy to secure. Cortex provides embedding and LLM functions you call like SQL, so there's no separate embedding service or model API to integrate. And similarity search runs against a table you already know how to manage. For document sets in the thousands-to-low-millions of chunks, this is enough - you trade some specialized vector-database performance for radically less moving parts.

The pipeline in four steps

1. Chunk the documents. Split each document into passages of a few hundred tokens with some overlap, stored as rows with their source metadata. Chunking is what lets retrieval return a relevant paragraph instead of a whole document.

2. Embed the chunks. Generate an embedding for each chunk with a Cortex embedding function and store the vector alongside the text:

UPDATE doc_chunks
SET embedding = SNOWFLAKE.CORTEX.EMBED_TEXT_768('e5-base-v2', chunk_text);

3. Retrieve by similarity. At query time, embed the question and find the nearest chunks with vector cosine similarity:

SELECT chunk_text
FROM doc_chunks
ORDER BY VECTOR_COSINE_SIMILARITY(
  embedding,
  SNOWFLAKE.CORTEX.EMBED_TEXT_768('e5-base-v2', :question)
) DESC
LIMIT 5;

4. Generate the answer. Pass the retrieved chunks and the question to a Cortex LLM function with a prompt that says "answer using only this context," and return the grounded answer with its sources.

A concrete use case

Compliance Q&A is a clean fit: a few thousand policy and regulatory documents, questions like "what's our retention requirement for transaction records," and a hard requirement that the data never leave the governed environment. The in-warehouse pipeline answers from the actual documents, cites them, and keeps everything inside Snowflake's access controls - exactly what a compliance team needs and exactly where external-API RAG gets blocked.

When to stay in Snowflake vs go external

Stay in the warehouse when your corpus is modest (thousands to low millions of chunks), latency in the hundreds of milliseconds is fine, and governance or simplicity outweigh peak performance. Move to a dedicated vector database - Pinecone, Weaviate, Qdrant, or pgvector at the lighter end - when you have tens of millions of vectors, need single-digit-millisecond retrieval, or want advanced indexing and filtering the warehouse doesn't offer. The honest dividing line is scale and latency, not capability: in-warehouse RAG is correct and simple until volume forces specialization.

Operating it

Re-embed chunks when documents change, watch retrieval quality (are the right chunks coming back?) as much as generation quality, and keep an eye on the compute the embedding and LLM functions consume. The failure mode is usually retrieval, not the model: if answers are wrong, check whether the relevant chunks were retrieved before blaming the LLM.

The takeaway

For document-grounded question answering over data that already lives in Snowflake, a Cortex-based RAG pipeline gives you the whole loop - chunk, embed, retrieve, generate - without a single external service. It won't outperform a purpose-built vector stack at massive scale, but for the common case of a few thousand internal documents and a governance requirement, the right answer is often the simplest one: keep the RAG where the data is.