Systems · 2026-09-06 · 986 words · 4 min read

KV cache arithmetic: memory per token, batch size, and context cost

Bytes of KV cache per token from a config.json, worked for Qwen3-8B and Qwen3-32B at four context lengths on an 80 GB device, with GQA and int8 KV as the two levers.

KV cache inference GQA memory

Every token a transformer has seen so far costs memory for as long as the request is alive, and that memory is the KV cache. It is why a serving box that holds a model comfortably at 4k context falls over at 128k, why a 32B model on one 80 GB GPU can serve one long request or nine short ones, and why grouped-query attention was worth retraining models for. The arithmetic is one line, and this article works it for two published models with nothing but their config.json files. The listing is code/kv-cache-arithmetic.py; its output table is datasets/kv-cache-arithmetic.csv. No GPU is touched; every number is a calculation on published shapes.

The one line

For each layer, attention stores a key and a value vector per token per KV head. So:

KV bytes per token = 2 (K and V) x layers x kv_heads x head_dim x bytes per element

The four shape numbers come straight from config.json: num_hidden_layers, num_key_value_heads, head_dim, and the dtype (bf16 is 2 bytes). Note that num_attention_heads, the query head count, does not appear. That is the point of grouped-query attention (Ainslie et al., arXiv:2305.13245): several query heads share one KV head, so the cache scales with the smaller number.

def kv_bytes_per_token(m: dict, bytes_per_elem: int = 2, kv_heads: int | None = None) -> int:
    kvh = m["kv_heads"] if kv_heads is None else kv_heads
    return 2 * m["layers"] * kvh * m["head_dim"] * bytes_per_elem

Two models

From the Qwen3-8B and Qwen3-32B model cards and config files, read on 2026-09-06:

Both cards state 32,768 tokens of native context and 131,072 with YaRN. Both use 8 KV heads, which means the 32B model, with twice the query heads, has the same KV heads per layer as the 8B and pays for its extra layers only.

Per token, at bf16: Qwen3-8B stores 2 x 36 x 8 x 128 x 2 = 147,456 bytes, 144 KiB. Qwen3-32B stores 2 x 64 x 8 x 128 x 2 = 262,144 bytes, 256 KiB. A 32k-token request is therefore 4.83 GB of cache on the 8B and 8.59 GB on the 32B. A 128k request is 19.3 GB and 34.4 GB.

Batch capacity on one 80 GB device

Take an 80 GB device, subtract the bf16 weights, subtract a 4 GB reserve for activations, CUDA context and fragmentation (a placeholder; measure yours), and divide what is left by the per-request cache. The script prints:

=== Qwen3-8B: 36 layers, 32 heads, 8 KV heads, head_dim 128 ===
weights bf16 ~16.4 GB; free for KV on 80 GB after 4 GB reserve: 59.6 GB
  bf16 KV, GQA (as shipped)               144.0 KiB/token
      ctx   4096:   0.60 GB per request ->   98 concurrent
      ctx  32768:   4.83 GB per request ->   12 concurrent
      ctx 131072:  19.33 GB per request ->    3 concurrent
  int8 KV, GQA                             72.0 KiB/token
      ctx  32768:   2.42 GB per request ->   24 concurrent

=== Qwen3-32B: 64 layers, 64 heads, 8 KV heads, head_dim 128 ===
weights bf16 ~65.6 GB; free for KV on 80 GB after 4 GB reserve: 10.4 GB
  bf16 KV, GQA (as shipped)               256.0 KiB/token
      ctx   4096:   1.07 GB per request ->    9 concurrent
      ctx  32768:   8.59 GB per request ->    1 concurrent
      ctx 131072:  34.36 GB per request ->    0 concurrent

Concurrent requests at full context length, upper bounds. Three readings.

The 32B model on one 80 GB device is a one-request server at 32k context, and it cannot hold a single 128k request at all: 34.4 GB of cache does not fit in the 10.4 GB left after weights. That is not a software problem. Serving that model at long context means tensor parallelism across devices or a quantised checkpoint, and the arithmetic says so before you install anything.

The "no GQA" row is what the 8B model would cost with a KV head per query head, the pre-GQA design: 576 KiB per token, four times the shipped figure, and 3 concurrent 32k requests instead of 12. The GQA paper's claim that an intermediate number of KV heads gives most of the memory saving of multi-query attention is, in this configuration, a 4x saving in cache per token.

Int8 KV halves everything, which is the second lever and the one available at deployment time rather than training time. Whether the quality cost is acceptable is a per-model measurement, not something the arithmetic can tell you.

Why context length costs what it does

Two costs scale with context. The cache above scales linearly with tokens in the request, and it is per-request, so it also scales with concurrency; that is the memory bill. Attention compute at each decode step reads the whole cache, so per-token decode time also grows linearly with context, and prefill of a long prompt grows quadratically in the attention term; that is the latency bill. Providers price long-context tiers higher because both bills are real, and a request at 128k context occupies 32 times the cache of a 4k request while it is alive, which is 32 short requests' worth of capacity it is displacing.

The consequence for anyone building on top of a served model: the cheapest way to cut inference cost is often to send less context, because a request's cost is proportional to what it holds resident, not just to what it reads once.

The dataset

datasets/kv-cache-arithmetic.csv is the full grid, 24 rows.

_README: - model: Qwen3-8B or Qwen3-32B - variant: bf16 KV with GQA as shipped; bf16 KV with kv_heads set to the query head count; int8 KV with GQA - kv_bytes_per_token: from the formula above - context: 4096, 8192, 32768, or 131072 tokens - kv_gb_per_request: kv_bytes_per_token x context / 1e9 - concurrent_requests_80gb: floor of free memory over per-request cache, with free memory = 80 GB minus bf16 weights minus a 4 GB reserve

What the arithmetic does not include

Real servers do not hold every request at full context; requests arrive at different lengths and grow as they decode, and paged allocation lets a server pack far more requests than the full-length count suggests. The reserve is a guess; activations for a large prefill batch can be several GB. Weights at bf16 are the assumption; a 4-bit checkpoint of the 32B model frees roughly 50 GB for cache and changes the table entirely. And the formula is for standard multi-head or grouped-query attention; architectures with latent-compressed KV or sliding-window layers store less per token and need their own line. The formula is still the place to start, because it is the one number you can compute from a config file before spending anything.