How much GPU memory a self-hosted deployment needs. Model weights are the part everyone estimates. The KV cache is the part that decides whether it survives concurrent users, because it grows with context length and with every request in flight.
From the model's config.json:
num_hidden_layers, num_key_value_heads, and
hidden_size / num_attention_heads.
Requests in flight at once, not daily users. A support tool with 200 daily users may only see three at a time.
Weights KV cache Overhead
| Component | GB |
|---|---|
| Model weights | -- |
| KV cache (all requests) | -- |
| Runtime overhead | -- |
| Total | -- |
| GPU | Nameplate | Usable | Max concurrency | Verdict |
|---|---|---|---|---|
| RTX 3060 | 12 GB | 10.8 GB | -- | -- |
| RTX 4060 Ti | 16 GB | 14.4 GB | -- | -- |
| RTX 3090 | 24 GB | 21.6 GB | -- | -- |
| RTX 4090 | 24 GB | 21.6 GB | -- | -- |
| NVIDIA L4 | 24 GB | 21.6 GB | -- | -- |
| NVIDIA A10G | 24 GB | 21.6 GB | -- | -- |
| RTX 5090 | 32 GB | 28.8 GB | -- | -- |
| 2 x RTX 3090 | 48 GB | 43.2 GB | -- | -- |
| NVIDIA L40S | 48 GB | 43.2 GB | -- | -- |
| NVIDIA A100 | 80 GB | 72.0 GB | -- | -- |
| NVIDIA H100 | 80 GB | 72.0 GB | -- | -- |
| 4 x RTX 4090 | 96 GB | 86.4 GB | -- | -- |
Usable memory is 90% of nameplate VRAM, which is vLLM's
default gpu_memory_utilization. Multi-GPU rows assume tensor parallelism and
do not model its communication overhead.
Weights are parameters multiplied by bytes per parameter. The KV cache holds two tensors, keys and values, for every layer, for every token in context, for every request in flight:
kv_bytes = 2 × layers × kv_heads × head_dim × precision × context × concurrency
Overhead covers the CUDA context, activations and allocator fragmentation, estimated at 1 GB plus 6% of weights.
Two results tend to surprise people. Models using grouped-query attention have far fewer KV heads than attention heads, so Llama 3.1 8B (8 KV heads) holds a longer context than Phi-3 mini 3.8B (32 KV heads) despite being twice the size. And quantising weights from FP16 to INT4 shrinks the weights by four but leaves the cache untouched, so on a high-concurrency deployment the saving is smaller than the headline suggests.
Background on the decisions this feeds into: choosing inference infrastructure, GPU options for local inference, and what self-hosted inference costs. (If you want this sized against a real workload, get in touch.)