The inference slowdown you don’t see at first
You run a quick demo: the model starts responding fast, then the pace quietly drifts. Nothing “breaks,” but the time between tokens grows as the answer gets longer. It can feel like network jitter or a busy GPU, yet it’s often a built-in property of autoregressive decoding.
Each new token is generated only after the model reprocesses the entire prompt plus everything it has already written. As the sequence length grows, attention has more past tokens to look at, so the work per step increases. Unless you do something to avoid recomputing what hasn’t changed, token 200 is simply more expensive than token 20, and that cost shows up as latency and lower throughput under load.
KV cache, explained in plain model terms
Think of attention as the model repeatedly asking: “Given what I’ve seen so far, which past tokens matter right now?” To answer that, every token in the sequence gets turned into three things: a query (what I’m looking for), a key (what I offer), and a value (the information to pass along if I’m selected). During generation, the past tokens’ keys and values don’t change, but without caching you’d keep rebuilding them at every step.
The KV cache is simply a stored copy of those past keys and values for each attention layer. When you generate the next token, the model only computes the new token’s query (and its new key/value) and attends against the saved history, rather than recomputing the whole history again. The catch is memory: the cache grows with sequence length, batch size, number of layers, and hidden size, so longer contexts or bigger batches can quickly consume GPU RAM.
What changes between the first token and next
The first generated token is the expensive one because there’s no history packaged up yet. The model has to run the full prompt through every layer, and in each attention layer it must produce keys and values for every prompt position so later steps can reference them. That “prompt processing” (often called prefill) is mostly parallel across tokens, but it’s still a lot of work, and it also allocates the initial KV cache.
After that, each step is incremental. You still run all layers again, but only for the single new token: compute its query, key, and value, then compare that query against the stored keys for the entire past and mix the stored values into an updated hidden state. The compute shifts from “build everything again” to “one token plus a growing lookup,” which is why decode can feel steady at first and then gradually slow as the cache (and attention span) grows.
Why KV cache matters for latency and throughput
In practice, KV cache is the difference between “recompute the whole prompt every token” and “reuse what’s already known.” Without it, per-token latency balloons quickly because every step rebuilds keys and values for all prior positions across all layers. With it, the model pays a one-time prefill cost to populate the cache, then decode steps are mostly “one new token” work plus attention over an ever-growing history. That usually makes the time per generated token far more predictable, especially for longer answers.
Throughput is where the serving math gets real. If you can keep KV cache in memory, you can generate many tokens per second because the GPU spends less time doing redundant compute and more time on incremental decoding. But the same cache also limits how many sequences you can serve at once: longer contexts and larger batches eat VRAM, forcing smaller batches or shorter max context. Under load, that can reduce overall tokens/sec even if each individual token step is efficient.
The tradeoff: speed bought with memory
You see the tradeoff when you try to scale beyond a single chat. Caching keys and values makes decoding cheaper, but it turns “serving a request” into “reserving memory for the lifetime of that request.” Every new token extends the cache, and that cache is duplicated per layer and per sequence. A handful of long conversations can consume enough VRAM that the GPU can’t fit an efficient batch, even if compute is available.
That pressure shows up as practical constraints: you cap max context, limit max new tokens, or lower concurrency to avoid out-of-memory errors. You might also shrink the cache footprint with lower-precision storage or more compact attention implementations, but those choices can add engineering complexity or affect quality. Offloading KV cache to CPU memory can recover VRAM, yet it often costs latency because each decode step now depends on moving cache data across a slower link. The “fast path” is simple: keep KV on GPU. The “affordable path” is usually a set of limits and trade-offs you feel in product behavior.
How KV cache shapes serving real user traffic
You notice KV cache effects most when traffic shifts from “one long answer” to “many overlapping sessions.” Each active request keeps its own cache alive across layers, so concurrency becomes a memory scheduling problem: ten short chats can be cheaper than two long ones that keep generating. That’s why systems often prioritize admitting more new requests (prefill-heavy) versus continuing existing ones (decode-heavy), and why “time to first token” and “tokens per second” can trade places depending on queueing.
Batching also changes meaning. In prefill, you can batch lots of prompts efficiently. In decode, each sequence advances one token at a time, often at different lengths, so the server either waits to align steps (higher throughput, higher tail latency) or runs smaller, more frequent batches (lower throughput, better responsiveness). Practical constraints show up fast: long-context users can reduce global concurrency, and stopping rules, max new tokens, and context trimming become capacity controls, not just UX choices.
Practical takeaways for choosing an inference setup
If you’re picking an inference setup, start by deciding what you’re optimizing: time to first token is mostly prefill and benefits from prompt batching, while steady tokens/sec is decode and is dominated by how many KV caches you can keep resident. Set max context and max new tokens as capacity levers, not just product defaults, because they bound cache growth and protect concurrency. When you need more parallel sessions, reduce KV footprint (lower precision cache, smaller model, shorter contexts) before chasing more batching. If you must offload KV to CPU, plan for higher latency and more variance under load.
