A typical naive LLM serving stack performs poorly: a fine-tuned Llama 2 70B behind a plain FastAPI wrapper can show median time-to-first-token of 214ms and p99 tail latency of 680ms. For a code completion product where developers expect sub-100ms responses, that is a dealbreaker. With the right optimizations, the same model runs at 47ms median TTFT with 3.2x higher throughput and a 40% lower GPU bill. This walks through the exact sequence of changes, in order, with the benchmark captured at each step.

Environment: All benchmarks were run on NVIDIA A100 80GB and H100 80GB instances, serving Llama 2 70B and Mixtral 8x7B. Input sequences averaged 512 tokens, output 256 tokens. Numbers are from a production cluster as of January 2026.

Key Takeaways (TL;DR)

• Quantization alone cut memory 55% and latency 35%. AWQ 4-bit on Llama 70B gave us nearly identical quality with massive throughput gains.
• KV cache is the real bottleneck at scale. PagedAttention in vLLM eliminated OOM crashes and improved batch utilization from 40% to 92%.
• Continuous batching is non-negotiable. Static batching wastes 60-70% of GPU cycles waiting for the longest sequence to finish.
• Speculative decoding is underrated. A 7B draft model boosted our 70B throughput by 2.1x with no quality loss.
• Self-hosted beats API pricing above ~50M tokens/month. We break even at 47M tokens/month on 2x H100s versus GPT-4 API.

The Starting Point: Why Naive Serving Is So Slow

It helps to understand why default LLM inference is slow. A standard HuggingFace pipeline behind Uvicorn looks something like this:

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-70b-chat-hf",
    torch_dtype=torch.float16,
    device_map="auto"  # naive sharding across GPUs
)
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-70b-chat-hf")

def generate(prompt: str) -> str:
    inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
    outputs = model.generate(**inputs, max_new_tokens=256)
    return tokenizer.decode(outputs[0], skip_special_tokens=True)

This approach has three fatal problems in production. First, the model sits in FP16 consuming 140GB of VRAM for a 70B model - at least two A100 80GB GPUs just to load it. Second, every request is processed one at a time because there is no batching. Third, the KV cache grows without bounds until an OOM crashes the entire process.

Step 1: Quantization - the biggest single win

Quantization gives the single largest improvement. It reduces model weights from 16-bit floating point to 4-bit integers, cutting memory usage roughly in half while preserving most of the model quality.

There are three quantization formats worth knowing in 2026:

GPTQ (GPU-Optimized Post-Training Quantization)

GPTQ was the first mainstream approach. It uses calibration data to find optimal rounding for weight quantization. The main advantage is wide ecosystem support - nearly every serving framework can load GPTQ models. The downside is that quantization itself is slow (several hours for a 70B model) and quality can degrade on certain tasks.

AWQ (Activation-Aware Weight Quantization)

AWQ is a strong default. Instead of treating all weights equally, it identifies the 1% of "salient" weights that matter most for activation magnitudes and preserves them at higher precision. In these benchmarks, AWQ-quantized Llama 70B scored within 0.3% of the FP16 baseline on an internal eval suite while using 55% less memory, and quantization runs in under 30 minutes on a single GPU.

GGUF (llama.cpp Format)

GGUF is the format used by llama.cpp and its ecosystem. It supports mixed quantization levels (Q4_K_M, Q5_K_S, etc.) and can offload layers between CPU and GPU. GGUF is excellent for edge deployment and local inference, but for multi-GPU datacenter serving, AWQ or GPTQ with a proper serving engine wins on throughput - a reasonable split is GGUF on dev laptops, AWQ in production.

Here is the impact quantization had on our setup:

Configuration	VRAM Usage	TTFT (p50)	Tokens/sec	Quality (eval)
FP16 (baseline)	140 GB	214 ms	32 t/s	100%
GPTQ 4-bit	38 GB	148 ms	51 t/s	98.4%
AWQ 4-bit	36 GB	139 ms	55 t/s	99.7%
GGUF Q4_K_M	37 GB	155 ms	46 t/s	99.1%

Quantization alone dropped TTFT from 214ms to 139ms - a 35% reduction just by swapping the model checkpoint. The quality hit was negligible; users could not tell the difference in blind A/B tests across 10,000 completions.

Step 2: KV cache optimization and PagedAttention

After quantization, the KV cache is the next target. In autoregressive generation, every token needs to attend to all previous tokens. The key-value pairs from each layer's attention computation are cached so they do not need to be recomputed. For a 70B model with 80 layers and 64 attention heads, the KV cache for a single 2048-token sequence consumes about 5GB of VRAM.

The naive approach pre-allocates a contiguous block of memory for the maximum sequence length. With max_seq_len set to 4096 but an average request of 800 tokens, 80% of that allocation is wasted. Multiply by a batch of 32 concurrent requests and that is over 100GB of wasted VRAM - the reason the naive setup keeps running out of memory.

PagedAttention (vLLM)

PagedAttention, introduced by the vLLM project, solved this problem elegantly. Instead of contiguous pre-allocation, it manages KV cache in fixed-size "pages" (like virtual memory in operating systems). Pages are allocated on demand and can be non-contiguous. When a request finishes, its pages go back to a free pool.

The impact is large. Effective batch size goes from 8 (before OOM) to 64+ on the same hardware, memory utilization from ~40% effective to over 92%, and because more requests can be batched, throughput nearly triples.

from vllm import LLM, SamplingParams

# vLLM handles PagedAttention automatically
llm = LLM(
    model="TheBloke/Llama-2-70B-Chat-AWQ",
    quantization="awq",
    tensor_parallel_size=2,       # 2x A100 80GB
    max_model_len=4096,
    gpu_memory_utilization=0.92,  # leave headroom for KV cache pages
    enable_prefix_caching=True,   # reuse KV cache for common prefixes
)

# Prefix caching is critical for our use case
# System prompts share the same prefix across all requests
# vLLM detects this and reuses the computed KV cache pages
sampling_params = SamplingParams(
    temperature=0.1,
    top_p=0.95,
    max_tokens=256,
)

outputs = llm.generate(["Explain quicksort in Python:"], sampling_params)
print(outputs[0].outputs[0].text)

One often-overlooked feature is prefix caching. With a shared system prompt (for example a 400-token system prompt for code completions), vLLM caches those KV entries and reuses them across requests. That saves ~80ms on TTFT for every request because the model never recomputes the system-prompt attention.

Step 3: Continuous Batching

Traditional static batching waits until N requests arrive, pads them all to the same length, processes them as a batch, and returns results when the longest sequence finishes. If you batch 16 requests and 15 of them generate 50 tokens but one generates 500 tokens, those 15 finished requests sit waiting while the straggler completes. Utilization is terrible.

Continuous batching (also called "iteration-level batching") processes each iteration of the generation loop independently. When a request finishes its output, its slot is immediately freed and a waiting request takes its place. No padding, no waiting.

vLLM implements this by default. So does TGI (Text Generation Inference) from Hugging Face. The effect on tail latency is striking - p99 drops from 680ms to 190ms because short requests are no longer hostage to long ones.

Step 4: Flash Attention

Flash Attention (and its successor Flash Attention 2) restructures the attention computation to minimize reads and writes to GPU high-bandwidth memory (HBM). Standard attention materializes the full N x N attention matrix in HBM, which is slow and memory-hungry. Flash Attention tiles the computation so it stays in SRAM (the fast on-chip memory), never materializing the full matrix.

In practice, Flash Attention 2 gives a 1.5-2x speedup on the attention computation specifically, which translates to about a 20-25% improvement in end-to-end generation speed for long sequences. For short sequences (under 512 tokens), the gains are smaller because attention is a smaller fraction of total compute.

vLLM, TGI, and TensorRT-LLM all use Flash Attention by default on supported hardware (Ampere and Hopper GPUs). No configuration is needed beyond a recent version and a compatible GPU.

Step 5: Speculative decoding

Speculative decoding is an underrated optimization. The idea: use a small, fast "draft" model to generate K candidate tokens, then verify all K in a single forward pass through the large target model. Since LLM inference is memory-bandwidth-bound (not compute-bound), verifying K tokens costs nearly the same as generating one.

Pairing Llama 70B (the target) with Llama 7B (the draft), the draft model generates 5 candidate tokens at a time. If 4 out of 5 are accepted (about 75-85% of the time for code completion), you effectively generate 4 tokens for the cost of one target-model forward pass plus the cheap draft passes.

from vllm import LLM, SamplingParams

# vLLM supports speculative decoding natively since v0.4
llm = LLM(
    model="TheBloke/Llama-2-70B-Chat-AWQ",
    quantization="awq",
    tensor_parallel_size=2,
    speculative_model="TheBloke/Llama-2-7B-Chat-AWQ",
    num_speculative_tokens=5,
    speculative_max_model_len=2048,
    gpu_memory_utilization=0.90,  # need extra VRAM for draft model
)

sampling_params = SamplingParams(
    temperature=0.0,  # speculative decoding works best with greedy
    max_tokens=256,
)

# The API is identical — speculative decoding is transparent
outputs = llm.generate(
    ["Write a Python function to merge two sorted lists:"],
    sampling_params
)
print(outputs[0].outputs[0].text)

With speculative decoding enabled, throughput jumps from 55 tokens/sec to 118 tokens/sec. TTFT is unaffected (still ~139ms), but token generation speed more than doubles. The caveats: it adds about 3-4GB VRAM for the draft model and works best at temperature=0 or very low temperatures. At high temperatures, the draft model's token acceptance rate drops and the benefit shrinks.

Step 6: Tensor parallelism and GPU selection

For models above 30B parameters, you need multiple GPUs. Tensor parallelism shards the model's weight matrices across GPUs so each GPU computes a slice of every layer. This differs from pipeline parallelism (which shards by layer) - tensor parallelism keeps latency low because all GPUs work in parallel on every token.

The key constraint is interconnect bandwidth. Tensor parallelism requires an all-reduce operation after every transformer layer, so the GPUs need fast links. NVLink (600 GB/s on H100) makes this practical. PCIe (64 GB/s) introduces a bottleneck that can negate the parallelism gains.

GPU comparison: A100 vs H100 vs L40S

Tested across the same workload:

GPU	VRAM	FP16 TFLOPS	Memory BW	Llama 70B AWQ (tokens/sec)	Hourly Cost (cloud)	Cost per 1M tokens
A100 80GB (2x)	160 GB	624	4.0 TB/s	118 t/s	$6.40	$15.05
H100 80GB (2x)	160 GB	1,979	6.6 TB/s	215 t/s	$9.80	$12.66
L40S 48GB (4x)	192 GB	1,452	3.6 TB/s	98 t/s	$5.20	$14.74

The H100 wins on cost per token despite the higher hourly rate because its memory bandwidth is 65% higher than the A100. LLM inference is memory-bandwidth-bound during token generation - FP16 compute is almost irrelevant. The L40S looks attractive on price but its PCIe interconnect (no NVLink) limits tensor parallelism scaling, so throughput suffers when sharding across 4 cards.

Recommendation: if H100s are available at reasonable rates, use them. Otherwise two A100 80GB with NVLink is the sweet spot for 70B models. Avoid mixing GPU types or using PCIe-only interconnects for tensor parallelism - the performance cliff is real.

Serving framework comparison

Four serving frameworks benchmarked head-to-head on identical hardware (2x A100 80GB, NVLink) with the same AWQ-quantized Llama 70B model:

Feature	vLLM	TGI	TensorRT-LLM	Triton + vLLM
TTFT (p50)	47 ms	62 ms	39 ms	51 ms
TTFT (p99)	89 ms	124 ms	71 ms	95 ms
Throughput	118 t/s	95 t/s	142 t/s	115 t/s
Continuous Batching	Yes	Yes	Yes	Yes
PagedAttention	Yes	Yes (v2+)	No (custom)	Via vLLM
Speculative Decoding	Yes	Limited	Yes	Via vLLM
Quantization	AWQ, GPTQ, FP8	AWQ, GPTQ, BnB	FP8, INT8, INT4	Via backend
Setup Complexity	Low	Low	High	High
Model Support	Broad	Broad	NVIDIA models	Broad
OpenAI API Compat	Yes	Yes	Via Triton	Yes

vLLM

vLLM is the best default for most teams: the best balance of performance, ease of use, and model support. Setup is a single pip install, the OpenAI-compatible API server works out of the box, and the community moves fast. The one downside is that it is not quite as fast as TensorRT-LLM on raw throughput, but the gap narrows with each release.

TGI (Text Generation Inference)

Hugging Face's TGI is solid and easy to deploy via Docker, and integrates well with the HF Hub ecosystem. Performance is good but consistently 15-20% behind vLLM in these benchmarks, primarily due to less aggressive memory management. TGI is a good fit for teams already deep in the Hugging Face ecosystem that want minimal ops overhead.

TensorRT-LLM

NVIDIA's TensorRT-LLM delivers the best raw numbers - 20% higher throughput than vLLM in these tests. The catch is that setup requires building model-specific TensorRT engines, which takes hours and breaks when the model is updated. The workflow is "compile model into engine, deploy engine." For a stable model that will not change for months and a team comfortable with NVIDIA tooling, TensorRT-LLM is the performance leader. For everyone else, the engineering overhead is not worth the 20%.

Triton Inference Server

NVIDIA Triton is not a serving engine itself - it is an orchestrator that can host vLLM, TensorRT-LLM, or other backends. It adds model versioning, A/B testing, and ensemble pipelines (e.g., tokenizer → model → post-processor). It suits a staging environment for model-comparison experiments, but running bare vLLM in production avoids the ~4ms of latency and operational complexity the Triton abstraction layer adds.

Production vLLM setup: complete code

A production vLLM configuration with all the optimizations above:

"""
Production vLLM serving configuration.
Deploy with: python -m vllm.entrypoints.openai.api_server \
    --config config.yaml
Or use this script for programmatic control.
"""
from vllm import LLM, SamplingParams
from vllm.engine.arg_utils import AsyncEngineArgs
from vllm.engine.async_llm_engine import AsyncLLMEngine

# Engine configuration — every flag here was tuned from benchmarks
engine_args = AsyncEngineArgs(
    model="TheBloke/Llama-2-70B-Chat-AWQ",
    quantization="awq",
    dtype="auto",
    tensor_parallel_size=2,
    max_model_len=4096,
    gpu_memory_utilization=0.92,

    # KV cache optimization
    enable_prefix_caching=True,       # reuse KV cache for shared prefixes
    block_size=16,                     # PagedAttention block size

    # Speculative decoding
    speculative_model="TheBloke/Llama-2-7B-Chat-AWQ",
    num_speculative_tokens=5,

    # Batching
    max_num_batched_tokens=32768,      # max tokens per batch iteration
    max_num_seqs=256,                  # max concurrent sequences

    # Performance
    enforce_eager=False,               # allow CUDA graph capture
    enable_chunked_prefill=True,       # overlap prefill and decode
    disable_log_stats=False,           # keep stats for monitoring
)

engine = AsyncLLMEngine.from_engine_args(engine_args)

A benchmarking script to validate changes before deploying:

"""
LLM inference benchmarking script.
Measures TTFT, token generation speed, and throughput under load.
"""
import asyncio
import time
import statistics
from openai import AsyncOpenAI

client = AsyncOpenAI(
    base_url="http://localhost:8000/v1",
    api_key="not-needed",  # vLLM does not require auth by default
)

PROMPTS = [
    "Write a Python function that implements binary search.",
    "Explain the difference between TCP and UDP.",
    "Create a SQL query to find duplicate rows in a table.",
    "Write a Kubernetes deployment YAML for a FastAPI app.",
    # ... 100 more prompts from our production distribution
]

async def benchmark_single(prompt: str) -> dict:
    """Measure latency for a single request."""
    start = time.perf_counter()
    first_token_time = None
    token_count = 0

    stream = await client.chat.completions.create(
        model="TheBloke/Llama-2-70B-Chat-AWQ",
        messages=[{"role": "user", "content": prompt}],
        max_tokens=256,
        temperature=0.0,
        stream=True,
    )

    async for chunk in stream:
        if chunk.choices[0].delta.content:
            if first_token_time is None:
                first_token_time = time.perf_counter()
            token_count += 1

    end = time.perf_counter()
    ttft = (first_token_time - start) * 1000  # ms
    total_time = end - start
    tps = token_count / (end - first_token_time) if first_token_time else 0

    return {"ttft_ms": ttft, "tokens_per_sec": tps, "total_sec": total_time}

async def run_benchmark(concurrency: int = 32, num_requests: int = 200):
    """Run benchmark with specified concurrency level."""
    semaphore = asyncio.Semaphore(concurrency)
    results = []

    async def limited_benchmark(prompt):
        async with semaphore:
            return await benchmark_single(prompt)

    tasks = [limited_benchmark(PROMPTS[i % len(PROMPTS)])
             for i in range(num_requests)]
    results = await asyncio.gather(*tasks)

    ttfts = [r["ttft_ms"] for r in results]
    tps_values = [r["tokens_per_sec"] for r in results]

    print(f"=== Benchmark Results (concurrency={concurrency}) ===")
    print(f"TTFT  p50: {statistics.median(ttfts):.1f} ms")
    print(f"TTFT  p99: {sorted(ttfts)[int(0.99*len(ttfts))]:.1f} ms")
    print(f"TPS   p50: {statistics.median(tps_values):.1f} tokens/sec")
    print(f"Total throughput: {sum(tps_values):.0f} tokens/sec aggregate")

if __name__ == "__main__":
    # Run at multiple concurrency levels to find saturation point
    for c in [1, 4, 16, 32, 64]:
        asyncio.run(run_benchmark(concurrency=c, num_requests=200))
        print()

Cumulative impact

The combined effect of every change, in the order applied:

Optimization	TTFT (p50)	Throughput (t/s)	VRAM Used	Cumulative Improvement
Baseline (FP16, naive)	214 ms	32 t/s	140 GB	-
+ AWQ 4-bit quantization	139 ms	55 t/s	36 GB	1.7x faster
+ vLLM (PagedAttention)	82 ms	89 t/s	68 GB*	2.8x faster
+ Continuous batching	71 ms	105 t/s	68 GB	3.3x faster
+ Speculative decoding	68 ms	118 t/s	72 GB	3.7x faster
+ Prefix caching	51 ms	118 t/s	72 GB	4.2x faster
+ Chunked prefill	47 ms	122 t/s	72 GB	4.6x faster

* VRAM increased here because vLLM dynamically allocates KV cache pages - the model itself is 36 GB but the rest is active cache serving 64 concurrent sequences.

Cost analysis: self-hosted vs API

The reason to do this optimization work instead of just calling the OpenAI API is cost. At a volume of roughly 80M tokens/month, self-hosting saves over $8,000 per month. The math:

Option	Monthly Cost	Cost per 1M tokens	Latency Control	Privacy
GPT-4 API	$16,000	$20.00 (blended in/out)	None	Data leaves network
GPT-4o API	$6,000	$7.50 (blended)	None	Data leaves network
Claude 3.5 Sonnet API	$7,200	$9.00 (blended)	None	Data leaves network
2x A100 (self-hosted)	$4,608	$1.60	Full	On-prem
2x H100 (self-hosted)	$7,056	$0.91	Full	On-prem

The breakeven point depends on your volume. At under 20M tokens/month, APIs are cheaper when you factor in engineering time. Between 20M and 50M tokens/month, it depends on your team's ops capability. Above 50M tokens/month, self-hosting almost always wins - and the gap widens as volume grows because GPU costs are fixed while API costs scale linearly.

The cost table above does not include engineering salaries, on-call burden, or the risk of GPU hardware failures. Maintaining serving infrastructure costs roughly 5 hours per week of engineering time. Without someone comfortable with CUDA drivers, GPU monitoring, and model deployment pipelines, the simplicity of an API has real value.

Lessons learned and common pitfalls

Non-obvious lessons from optimizing LLM inference:

1. Profile before optimizing. Two weeks spent speeding up a tokenization pipeline can turn out to address only 0.3% of total latency. Use torch.profiler or NVIDIA Nsight to find the actual bottleneck before writing any code.
2. CUDA graph capture is fragile. vLLM captures CUDA graphs for the decode phase, eliminating kernel-launch overhead. But if input shapes vary wildly, graph capture creates hundreds of variants and eats VRAM. Set max_num_seqs to a power of 2 and keep max_model_len reasonable.
3. Monitor GPU memory fragmentation. After 48 hours of continuous serving, throughput can drop 15% from KV-cache memory fragmentation - a daily zero-downtime rolling restart fixes it. vLLM defragmentation is in progress but not yet production-ready.
4. Test quantization quality on your data. Generic benchmarks (MMLU, HellaSwag) do not predict how quantization affects a specific task. AWQ can degrade code-completion accuracy by 2% on a private eval set despite showing no degradation on public benchmarks - measure it before accepting the tradeoff.
5. Network overhead is real. A vLLM server and application backend in different availability zones add a 2ms round-trip that compounds to ~8ms per streaming response (4 round trips for setup). Co-locating them in the same zone reclaims those milliseconds for free.

Where this goes next

A 47ms TTFT is good enough for code completion, but three improvements are worth tracking. First, FP8 quantization on H100s promises another 15-20% throughput boost over INT4 AWQ with potentially better quality, since 8-bit preserves more precision. Second, disaggregated prefill-decode architectures (Splitwise, DistServe) separate the compute-heavy prefill phase from the memory-heavy decode phase onto different GPU pools for better hardware utilization. Third, smaller fine-tuned models that match 70B quality on specific tasks - Llama 8B fine-tunes already beat Llama 70B on some code-completion benchmarks.

LLM inference optimization is not a one-time project. Every few months a new technique or framework shifts the performance frontier. The fundamentals - quantization, efficient memory management, smart batching, and hardware-aware deployment - remain relevant even as the specific tools change. Start with vLLM and AWQ quantization, measure everything, and optimize from there. The benchmarking script above works with any OpenAI-compatible endpoint: point it at a vLLM server, run it at multiple concurrency levels, and you have a clear picture of serving performance in under 10 minutes.