"Should we fine-tune or just use RAG for this?" comes up on most LLM projects, and the honest answer is that it depends. But after shipping both approaches across several production systems, a clear mental model emerges for when each makes sense, where they overlap, and why the cost conversation is more nuanced than most write-ups suggest. This is a decision framework backed by production numbers rather than theory.

Note: The cost figures and latency measurements below reflect production systems as of late 2025. Pricing changes frequently, especially for GPU compute and API calls. Treat these as directional guidance, not gospel.

The core question most people get wrong

The biggest mistake is framing this as fine-tuning versus RAG, as if they solve the same problem. They don't. Fine-tuning changes how a model behaves. RAG changes what a model knows. These are fundamentally different interventions, and confusing them leads to expensive dead ends.

Fine-tuning teaches a model a new skill, style, or reasoning pattern - "write SQL in our company's specific dialect" or "produce responses in our brand voice." You modify the model's weights to internalize a behavior. RAG gives a model access to information it does not have - "answer questions about our internal docs" or "ground responses in today's data, not the training cutoff." You augment the model's context at inference time.

When fine-tuning wins

Fine-tuning is the clear winner in a surprisingly narrow set of cases, but when it wins, it wins big.

1. Consistent style and tone

Consider an LLM generating medical report summaries in a specific clinical format: exact section ordering, particular phrasing conventions, abbreviation rules. Few-shot prompting with RAG, feeding examples into the context window, got the format right roughly 70% of the time. Fine-tuning on 2,000 annotated examples raised accuracy to 96%. Prompting can nudge style; fine-tuning can nail it.

2. Specialized reasoning patterns

When a task requires a domain-specific reasoning chain that general models struggle with, fine-tuning helps. For financial covenant analysis - extracting specific clauses, applying conditional logic, outputting structured JSON - a base model could approximate the task with a large prompt but was brittle and slow. A fine-tuned model was faster (shorter prompts), cheaper (fewer tokens), and more reliable.

3. Latency-sensitive applications

Fine-tuned models need shorter prompts because the behavior is baked into the weights. In the covenant analysis system, moving from a 2,800-token prompt (examples plus instructions) to a fine-tuned model with a 400-token prompt cut p50 latency from 1.9 seconds to 0.6 seconds. At thousands of requests per second, that difference is significant.

4. Cost at scale (counterintuitive)

Fine-tuning has a high upfront cost, but at millions of API calls with long few-shot prompts, the per-request savings add up. Shorter prompts mean fewer input tokens, which means lower cost per call. The math is below.

When RAG wins

RAG is the right default for most enterprise use cases.

1. Rapidly changing information

If the knowledge base updates daily, weekly, or even monthly, RAG is the obvious choice. Fine-tuning on today's data means the model is stale by next week. A customer support system that ingests new product docs and release notes every sprint would be impractical to retrain on each update; with RAG, you re-index and the model immediately has the latest information.

2. Source attribution and trust

When users need to verify answers, RAG provides citations for free - the response points back to the specific document, page, or paragraph. A fine-tuned model cannot do this; the knowledge is dissolved into the weights with no traceability. For legal, compliance, medical, and financial applications, attribution is often a hard requirement.

3. Large or growing knowledge bases

A fine-tuned model has a fixed amount of knowledge baked in at training time. A corpus of millions of documents that keeps growing cannot be fine-tuned into coverage. For one client with 4.2 million internal documents, RAG with pgvector handles this cleanly; fine-tuning on that corpus would be impractical and would not guarantee recall of specific facts.

4. Fast prototyping

Setting up a basic RAG pipeline takes an afternoon. A fine-tuning pipeline with proper data preparation, training, evaluation, and deployment takes weeks. To explore whether an LLM can solve a problem at all, start with RAG; you can add fine-tuning later.

The cost breakdown: real numbers

Here is an actual cost comparison for a document Q&A system handling 50,000 queries per day.

Cost Component	RAG Approach	Fine-Tuned Approach
Upfront training / indexing	~$120 (embedding 500K docs)	~$800 - $2,400 (LoRA fine-tune)
Embedding model (monthly)	~$340 (50K queries/day × $0.00002/query)	$0
Vector DB hosting (monthly)	~$200 (pgvector on RDS)	$0
LLM inference per query	~2,100 tokens avg (context-heavy)	~600 tokens avg (short prompt)
LLM cost (monthly, GPT-4o-mini)	~$940	~$430
Reranker cost (monthly)	~$150 (self-hosted cross-encoder)	$0
Total monthly (steady state)	~$1,630	~$430
Data update frequency	Re-index: minutes	Re-train: hours + $$$

At first glance fine-tuning looks like the clear cost winner. But the table omits ongoing retraining: if data changes monthly and requires retraining, add another $800 - $2,400 per month. If it changes weekly, the math flips entirely. Also omitted is the engineering time to maintain a fine-tuning pipeline with data prep, evaluation suites, model versioning, and A/B testing infrastructure. For most teams, RAG's predictable, low-maintenance cost profile wins unless they operate at very high query volumes with stable data.

Latency comparison

Measured across production systems:

Pipeline Stage	RAG (p50 / p95)	Fine-Tuned (p50 / p95)
Query embedding	12ms / 25ms	-
Vector search	8ms / 18ms	-
Reranking	35ms / 65ms	-
LLM generation	1,400ms / 2,800ms	520ms / 980ms
Total end-to-end	1,455ms / 2,908ms	520ms / 980ms

LLM generation dominates in both cases, but RAG adds retrieval overhead and the longer context means more tokens to process. For real-time, user-facing applications where sub-second latency matters, fine-tuning has a significant edge. Aggressive caching can close the gap: semantic caching (embed the query, check whether a similar query was recently answered) cuts p50 from 1,455ms to about 180ms for cache hits, and roughly 55% of queries are near-duplicates in one production system.

LoRA fine-tuning: a working example

A practical fine-tuning pipeline. LoRA (Low-Rank Adaptation) is dramatically cheaper than full fine-tuning and works well for most use cases. This example fine-tunes a Mistral 7B model for structured output generation.

import torch
from datasets import load_dataset, Dataset
from transformers import (
    AutoModelForCausalLM,
    AutoTokenizer,
    TrainingArguments,
    BitsAndBytesConfig,
)
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
from trl import SFTTrainer


def prepare_training_data(examples_path: str) -> Dataset:
    """Load and format training examples as instruction-response pairs."""
    raw = load_dataset("json", data_files=examples_path, split="train")

    def format_example(example):
        return {
            "text": (
                f"### Instruction:\n{example['instruction']}\n\n"
                f"### Input:\n{example['input']}\n\n"
                f"### Response:\n{example['output']}"
            )
        }

    return raw.map(format_example, remove_columns=raw.column_names)


def fine_tune_with_lora(
    base_model: str = "mistralai/Mistral-7B-v0.3",
    dataset_path: str = "training_data.jsonl",
    output_dir: str = "./fine_tuned_model",
    num_epochs: int = 3,
    learning_rate: float = 2e-4,
    lora_rank: int = 16,
    lora_alpha: int = 32,
):
    """Fine-tune a model using QLoRA (4-bit quantized LoRA)."""

    # 4-bit quantization config — cuts VRAM from ~28GB to ~6GB
    bnb_config = BitsAndBytesConfig(
        load_in_4bit=True,
        bnb_4bit_quant_type="nf4",
        bnb_4bit_compute_dtype=torch.bfloat16,
        bnb_4bit_use_double_quant=True,
    )

    # Load base model with quantization
    model = AutoModelForCausalLM.from_pretrained(
        base_model,
        quantization_config=bnb_config,
        device_map="auto",
        trust_remote_code=True,
    )
    model = prepare_model_for_kbit_training(model)

    tokenizer = AutoTokenizer.from_pretrained(base_model)
    tokenizer.pad_token = tokenizer.eos_token
    tokenizer.padding_side = "right"

    # LoRA configuration — these are the adapter weights we actually train
    # rank=16 is a good default. Higher = more capacity but slower training
    lora_config = LoraConfig(
        r=lora_rank,
        lora_alpha=lora_alpha,
        target_modules=[
            "q_proj", "k_proj", "v_proj", "o_proj",
            "gate_proj", "up_proj", "down_proj",
        ],
        lora_dropout=0.05,
        bias="none",
        task_type="CAUSAL_LM",
    )

    model = get_peft_model(model, lora_config)

    # Print trainable parameters — should be ~0.5-1% of total
    trainable, total = model.get_nb_trainable_parameters()
    print(f"Trainable: {trainable:,} / {total:,} ({100 * trainable / total:.2f}%)")

    dataset = prepare_training_data(dataset_path)

    training_args = TrainingArguments(
        output_dir=output_dir,
        num_train_epochs=num_epochs,
        per_device_train_batch_size=4,
        gradient_accumulation_steps=4,
        learning_rate=learning_rate,
        weight_decay=0.01,
        warmup_ratio=0.03,
        lr_scheduler_type="cosine",
        logging_steps=25,
        save_strategy="epoch",
        bf16=True,
        optim="paged_adamw_8bit",
        report_to="wandb",
    )

    trainer = SFTTrainer(
        model=model,
        tokenizer=tokenizer,
        train_dataset=dataset,
        dataset_text_field="text",
        max_seq_length=2048,
        args=training_args,
    )

    trainer.train()
    trainer.save_model(output_dir)
    print(f"LoRA adapter saved to {output_dir}")


if __name__ == "__main__":
    fine_tune_with_lora(
        dataset_path="covenant_analysis_examples.jsonl",
        output_dir="./covenant_lora_adapter",
        num_epochs=3,
        lora_rank=16,
    )

A few notes. QLoRA (quantized LoRA) runs on a single A100 or even an A10G; full Mistral 7B fine-tuning requires 4x A100s. The adapter weights are tiny - usually 50-100MB versus 14GB for the full model - so you can store dozens of adapters and swap them at inference time. Training on 2,000 examples takes about 45 minutes on an A100, roughly $3 - $5 per run on a cloud GPU provider. The real expense is preparing those 2,000 examples, which often requires weeks of expert annotation.

RAG retrieval chain: a working example

A production RAG pipeline - no LangChain, just straightforward Python with the libraries that matter.

import asyncio
import hashlib
import json
from dataclasses import dataclass

import asyncpg
import numpy as np
from openai import AsyncOpenAI

client = AsyncOpenAI()

EMBEDDING_MODEL = "text-embedding-3-small"
GENERATION_MODEL = "gpt-4o-mini"
SIMILARITY_THRESHOLD = 0.72
TOP_K = 10
RERANK_TOP_N = 4


@dataclass
class RetrievedChunk:
    content: str
    source: str
    similarity: float
    rerank_score: float = 0.0


class RAGPipeline:
    """Production RAG pipeline with pgvector, reranking, and semantic caching."""

    def __init__(self, db_pool: asyncpg.Pool):
        self.db_pool = db_pool
        self._cache: dict[str, str] = {}

    async def embed_query(self, text: str) -> list[float]:
        """Generate embedding for a query string."""
        response = await client.embeddings.create(
            model=EMBEDDING_MODEL,
            input=text,
        )
        return response.data[0].embedding

    async def retrieve(self, query_embedding: list[float]) -> list[RetrievedChunk]:
        """Hybrid search: vector similarity + BM25 keyword matching."""
        embedding_str = json.dumps(query_embedding)

        async with self.db_pool.acquire() as conn:
            rows = await conn.fetch(
                """
                WITH vector_results AS (
                    SELECT id, content, source,
                           1 - (embedding <=> $1::vector) AS similarity
                    FROM document_chunks
                    WHERE 1 - (embedding <=> $1::vector) > $2
                    ORDER BY embedding <=> $1::vector
                    LIMIT $3
                ),
                keyword_results AS (
                    SELECT id, content, source,
                           ts_rank(search_vector, plainto_tsquery('english', $4)) AS kw_rank
                    FROM document_chunks
                    WHERE search_vector @@ plainto_tsquery('english', $4)
                    LIMIT $3
                )
                SELECT COALESCE(v.id, k.id) AS id,
                       COALESCE(v.content, k.content) AS content,
                       COALESCE(v.source, k.source) AS source,
                       COALESCE(v.similarity, 0) AS similarity,
                       COALESCE(k.kw_rank, 0) AS kw_rank,
                       (COALESCE(v.similarity, 0) * 0.7 + COALESCE(k.kw_rank, 0) * 0.3)
                           AS combined_score
                FROM vector_results v
                FULL OUTER JOIN keyword_results k ON v.id = k.id
                ORDER BY combined_score DESC
                LIMIT $3
                """,
                embedding_str,
                SIMILARITY_THRESHOLD,
                TOP_K,
                "",  # query text for BM25 — passed separately
            )

        return [
            RetrievedChunk(
                content=row["content"],
                source=row["source"],
                similarity=row["similarity"],
            )
            for row in rows
        ]

    def rerank(self, query: str, chunks: list[RetrievedChunk]) -> list[RetrievedChunk]:
        """Cross-encoder reranking using a local model.

        In production, we run ms-marco-MiniLM-L-12-v2 on a GPU sidecar.
        This simplified version shows the interface.
        """
        from sentence_transformers import CrossEncoder

        reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-12-v2")

        pairs = [(query, chunk.content) for chunk in chunks]
        scores = reranker.predict(pairs)

        for chunk, score in zip(chunks, scores):
            chunk.rerank_score = float(score)

        ranked = sorted(chunks, key=lambda c: c.rerank_score, reverse=True)
        return ranked[:RERANK_TOP_N]

    def _cache_key(self, query: str) -> str:
        normalized = query.strip().lower()
        return hashlib.sha256(normalized.encode()).hexdigest()

    async def generate(self, query: str, chunks: list[RetrievedChunk]) -> str:
        """Build prompt with retrieved context and generate a response."""
        context_parts = []
        for i, chunk in enumerate(chunks, 1):
            context_parts.append(
                f"[Source {i}: {chunk.source}]\n{chunk.content}"
            )

        context_block = "\n\n---\n\n".join(context_parts)

        response = await client.chat.completions.create(
            model=GENERATION_MODEL,
            messages=[
                {
                    "role": "system",
                    "content": (
                        "You are a helpful assistant that answers questions based on "
                        "the provided context. Always cite your sources using [Source N] "
                        "notation. If the context does not contain enough information to "
                        "answer, say so explicitly. Do not make up information."
                    ),
                },
                {
                    "role": "user",
                    "content": (
                        f"Context:\n{context_block}\n\n"
                        f"Question: {query}\n\n"
                        "Answer the question based on the context above."
                    ),
                },
            ],
            temperature=0.1,
            max_tokens=1024,
        )

        return response.choices[0].message.content

    async def query(self, user_query: str) -> str:
        """Full RAG pipeline: embed → retrieve → rerank → generate."""
        # Check semantic cache
        cache_key = self._cache_key(user_query)
        if cache_key in self._cache:
            return self._cache[cache_key]

        # Embed the query
        query_embedding = await self.embed_query(user_query)

        # Retrieve relevant chunks
        chunks = await self.retrieve(query_embedding)
        if not chunks:
            return "I could not find relevant information to answer your question."

        # Rerank for precision
        top_chunks = self.rerank(user_query, chunks)

        # Generate answer
        answer = await self.generate(user_query, top_chunks)

        # Cache the result
        self._cache[cache_key] = answer

        return answer


# Usage
async def main():
    pool = await asyncpg.create_pool("postgresql://user:pass@localhost/mydb")
    rag = RAGPipeline(pool)
    answer = await rag.query("What is our refund policy for enterprise contracts?")
    print(answer)


if __name__ == "__main__":
    asyncio.run(main())

This is a simplified version of a full pipeline, but it covers the important parts: hybrid search (vector + keyword), cross-encoder reranking, and basic semantic caching. A production system would also add query rewriting (expanding acronyms, decomposing multi-part questions), guardrails for prompt injection, and a feedback loop that logs user thumbs-up/down for evaluation.

The decision matrix

Factor	Favors Fine-Tuning	Favors RAG
Knowledge freshness	Static, rarely changes	Dynamic, updates frequently
Task type	Style, format, reasoning patterns	Knowledge retrieval, Q&A
Source attribution needed	No	Yes (citations required)
Knowledge base size	Small, focused domain	Large or growing corpus
Latency requirements	Sub-second critical	1-3 seconds acceptable
Query volume	High (>100K/day), amortizes cost	Low-medium, predictable cost
Team ML expertise	Strong (can manage training pipeline)	Moderate (can manage retrieval infra)
Data preparation effort	High (curated instruction pairs)	Low-medium (chunk and embed)
Time to production	Weeks to months	Days to weeks
Hallucination tolerance	Moderate (no grounding)	Low (grounded in sources)

The hybrid approach: why not both?

In practice, the most effective systems use both techniques together. This is not hedging - it is genuinely the best architecture for many real-world applications. The pattern:

1. Fine-tune for behavior. Train the model to follow your output format, tone, and reasoning style. This replaces the massive system prompt and few-shot examples.
2. RAG for knowledge. Retrieve relevant context at query time so the model has access to current, specific information.
3. Shorter prompts, better results. Because the fine-tuned model already knows how to behave, the RAG prompt only needs the retrieved context and the user query - no five-paragraph system prompt, no eight few-shot examples.

One demanding application - automated compliance checks for a fintech company - uses this hybrid pattern. The model is fine-tuned to understand financial regulation structure and output structured findings in a specific JSON schema; RAG supplies the actual regulation text and the client's internal policy documents. The fine-tuned model with RAG context outperforms both standalone approaches by a significant margin.

"""
Hybrid approach: fine-tuned model + RAG retrieval.

The fine-tuned model already 'knows' our output format and reasoning style,
so we only need to inject the retrieved context — no few-shot examples needed.
"""

HYBRID_SYSTEM_PROMPT = (
    "You are a compliance analyst. Analyze the provided regulation excerpts "
    "against the company policy and output findings in the required format."
)
# ↑ That is the ENTIRE system prompt. The fine-tuned model knows the format.
# Compare to 2,800 tokens of instructions needed for the base model.


async def hybrid_query(
    rag_pipeline: RAGPipeline,
    fine_tuned_model: str,
    regulation_query: str,
    company_policy: str,
) -> str:
    """Combine fine-tuned model with RAG-retrieved regulation context."""

    # Step 1: Retrieve relevant regulation chunks via RAG
    query_embedding = await rag_pipeline.embed_query(regulation_query)
    chunks = await rag_pipeline.retrieve(query_embedding)
    top_chunks = rag_pipeline.rerank(regulation_query, chunks)

    regulation_context = "\n\n".join(
        f"[Reg {i+1}: {c.source}]\n{c.content}"
        for i, c in enumerate(top_chunks)
    )

    # Step 2: Generate using the fine-tuned model (short prompt!)
    response = await client.chat.completions.create(
        model=fine_tuned_model,  # ft:gpt-4o-mini-2024-07-18:org:compliance:abc123
        messages=[
            {"role": "system", "content": HYBRID_SYSTEM_PROMPT},
            {
                "role": "user",
                "content": (
                    f"Regulations:\n{regulation_context}\n\n"
                    f"Company Policy:\n{company_policy}\n\n"
                    f"Query: {regulation_query}"
                ),
            },
        ],
        temperature=0.0,
        max_tokens=2048,
    )

    return response.choices[0].message.content

The token savings are real. The compliance system went from ~3,200 input tokens per query (base model with massive prompt) to ~1,100 tokens (fine-tuned model with just the context). At 80,000 queries per day, that is a meaningful cost reduction.

Real-world use cases

Customer support chatbot (RAG)

Knowledge base of 15,000 support articles that changes weekly; users need links to source docs. RAG was the obvious choice: chunk articles into ~300-token segments, embed with text-embedding-3-small, store in pgvector, rerank with a cross-encoder. Response quality improved 40% over a fine-tuned prototype because the model always had access to the latest articles.

Code review assistant (fine-tuning)

The model needed to follow specific review conventions: flag security issues in a particular format, suggest fixes using internal library conventions, output structured annotations. The knowledge (the style guide) is small and stable. Fine-tuning GPT-4o-mini on 1,500 annotated code-review examples dropped latency from 2.1s to 0.7s by eliminating the few-shot examples from the prompt.

Legal document analysis (hybrid)

The model needed to extract and compare clauses across contracts in a specific output format (fine-tuning) while accessing a constantly growing library of precedent documents and regulatory updates (RAG). Pure fine-tuning could not handle the dynamic knowledge; pure RAG could not nail the output format consistently. The hybrid approach solved both.

Internal knowledge base Q&A (RAG)

A question-answering system over 50,000 internal wiki pages, Confluence docs, and Slack threads. Data changes daily; users need to see the source document. A classic RAG use case - fine-tuning was not considered.

Common pitfalls to avoid

• Fine-tuning to inject knowledge. This almost never works well. Models are bad at memorizing specific facts through fine-tuning; they are good at learning patterns and styles. To make a model know that "Product X was released on March 15," use RAG.
• RAG with 20+ retrieved chunks. More context is not always better. Past 4-6 chunks, the model gets confused and latency balloons. Use a reranker to pick the best 3-5 chunks instead of dumping everything into the context window.
• Skipping evaluation before choosing. Build a test set of 100-200 representative queries with expected answers. Run both approaches (even quick prototypes) and measure. The answer is often different from intuition.
• Ignoring maintenance cost. Fine-tuning pipelines need data-drift monitoring, periodic retraining, model versioning, and rollback. RAG pipelines need index maintenance, embedding-model updates, and retrieval-quality monitoring. Factor this into the decision.
• Over-engineering early. Start with RAG and a good prompt. If you hit a wall on style, format, or latency, then add fine-tuning. Don't start with fine-tuning because it feels more sophisticated.

A practical decision flowchart

Walk through these questions in order:

1. Does the data change more than monthly? If yes, start with RAG.
2. Do users need source citations? If yes, you need RAG (at least for retrieval).
3. Is the main problem style/format/reasoning, not knowledge? If yes, fine-tuning is likely the path.
4. Is latency under 1 second a hard requirement? If yes, fine-tuning helps (shorter prompts).
5. Are there fewer than 500 labeled examples? If yes, start with RAG + good prompting - there's probably not enough data to fine-tune yet.
6. Can the team maintain a training pipeline? If not, stick with RAG. A poorly maintained fine-tuned model is worse than a well-maintained RAG system.

Answering "yes" across both categories points to a hybrid approach.

Wrapping up

The fine-tuning vs RAG decision is not about which technology is better - it's about understanding the problem being solved. To change how the model behaves, fine-tune. To change what the model knows, use RAG. If you need both, combine them. Start simple, measure everything, and add complexity only when the numbers justify it. And if it's still unclear, start with RAG: you can always add fine-tuning later, but you cannot un-train a model that learned the wrong things from bad training data.