PagedAttention & Continuous Batching: How vLLM Achieves 24x Serving Throughput
Eliminating KV cache memory fragmentation using OS virtual memory paging principles.
Table of Contents (30 Concepts)
1. The KV Cache Memory Bottleneck in Transformer Inference
In transformer autoregressive generation, keys and values of previous tokens are cached to avoid redundant computation. Traditional engines allocate contiguous memory blocks for the maximum context window, wasting 60-80% of GPU VRAM.
"Reserving a 50-passenger tour bus for every single person who might book a ticket, even if only 3 passengers actually show up."
[ Traditional KV Cache: 80% Internal Fragmentation ] ┌───────────────────────────────────────────────────────────┐ │ [Tokens 1-20] │ UNUSED VRAM (Wasted) │ └───────────────────────────────────────────────────────────┘
2. Static vs Dynamic Context Length Memory Allocation
Static allocation requires pre-allocating the maximum sequence length (e.g. 8k tokens) per request, severely capping concurrency.
3. Virtual Page Tables for Non-Contiguous KV Allocation
PagedAttention divides the KV cache into fixed-size physical blocks (e.g. 16 tokens each) and tracks them through logical page tables, completely eliminating internal fragmentation.
4. Continuous Iteration-Level Batching vs Static Request Batching
Instead of waiting for an entire batch of requests to finish generating before starting new ones, iteration-level batching inserts incoming requests into the GPU pipeline after every individual token step.
5. Prefix Caching: Reusing KV Tensors Across Shared Prompts
Caching the common system prompt KV states across hundreds of user queries avoids redundant transformer evaluations.
6. Tensor Parallelism (TP) vs Pipeline Parallelism (PP)
Splitting weight matrices across multiple GPU VRAMs using Megatron-LM styles vs splitting layers sequentially.
7. Speculative Decoding with Small Draft Models
Using a lightweight 1B draft model to generate 5 candidate tokens in parallel, which the 70B target model verifies in a single forward pass.
Subscribe to continue reading the full masterclass
You’ve finished the first 7 concepts. Join 120,000+ senior engineers to unlock the remaining 23 concepts, deep-dive trade-off diagrams, and our 120+ edition archive.