Why is training VRAM so much higher than inference?

During full fine-tuning, the GPU must store: model weights, gradient tensors of the same size, and Adam optimizer state — first moment (m1), second moment (m2), and FP32 master weights — adding roughly 12 extra bytes per parameter. For a 7B model in BF16, inference needs ~14 GB but full training needs ~60–80 GB.

What is the KV cache and why does it scale with sequence length?

The key-value cache stores attention keys and values for all previous tokens. Its size = 2 × layers × KV-heads × head-dim × sequence-length × batch-size × bytes. KV cache grows linearly with context length — extending from 4K to 128K tokens increases it by 32×. GQA reduces KV-heads, substantially cutting cache size.

What does GGUF quantisation mode calculate?

GGUF mode calculates weight memory as parameters × (quantBits / 8) bytes. The KV cache is always stored in FP16 regardless of weight quantisation. Q4_K_M (≈4 bits) is the most popular balance of quality and memory efficiency for llama.cpp and Ollama users.

What is Grouped Query Attention (GQA) and why does it matter for VRAM?

GQA reduces the number of key and value heads to a fraction of query heads. Since KV cache scales with KV heads, GQA can reduce KV cache by 4× or more. Llama 3.1 uses 8 KV heads vs 32 query heads. Enter num_key_value_heads from config.json for accurate KV cache estimates.

How many GPUs do I need for a 70B model?

At FP16, a 70B model requires approximately 140 GB for weights alone, fitting on two H100 80GB cards. At INT4, weights drop to ~35 GB, fitting on a single H100. The GPU compatibility table in the calculator shows minimum GPU count for every card in your configuration.

Is the VRAM estimate accurate for vLLM, Transformers, and llama.cpp?

The calculator uses the standard theoretical formula all runtimes are based on. vLLM paged attention may add 5–15% overhead. llama.cpp memory-maps weights and can run larger models. The 7% framework overhead buffer accounts for most runtime variance. Always validate with a small test run before production deployment.

VRAM Calculator

Estimate GPU VRAM requirements for LLM inference, full fine-tuning, LoRA training, and GGUF quantisation. Calculate model weights, KV cache, optimizer states, and framework overhead from first principles — then check which GPUs fit your workload. Free, instant, and fully private.

VRAM Calculator

Estimate GPU VRAM requirements for LLM inference, full fine-tuning, LoRA training, and GGUF quantisation. Model weights, KV cache, activations, optimizer states, and framework overhead — all calculated from first principles in your browser.

Model Preset

32 layers · 8 KV heads · head dim 128

Model Architecture

Parameters (Billions)

Transformer Layers

KV Heads (GQA)

Head Dimension

Workload Mode

Weights + KV cache + minimal activations. Smallest footprint.

Weight Precision

Context & Batch

Sequence Length

tokens

51264K128K

Batch Size

reqs

13264

Compare Against GPU

Total VRAM Required

16.57 GB

for inference

8.03B params · FP16 · seq 4,096 · batch 1

Model Weights

14.96 GB

KV Cache

512 MB

Activations

31 MB

Overhead

1.08 GB

NVIDIA RTX 4090 (24 GB) (24 GB)

✅ Fits on a single GPU (69.0% VRAM used)

Single GPU ✓

VRAM Usage16.57 GB / 24 GB (69.0%)

On-prem dev/inference, small models

VRAM Breakdown

Model Weights14.96 GB (90.3%)

KV Cache512 MB (3.0%)

Activations31 MB (0.2%)

Framework Overhead1.08 GB (6.5%)

Precision Comparison (Same Config)

Precision	Weights	Total VRAM	Fits 24 GB?
FP32	29.91 GB	33.14 GB	✗ No
FP16 Selected	14.96 GB	16.57 GB	✓ Yes
BF16	14.96 GB	16.57 GB	✓ Yes
FP8	7.48 GB	8.55 GB	✓ Yes
INT8	7.48 GB	8.55 GB	✓ Yes
INT4	3.74 GB	4.54 GB	✓ Yes

GPU Compatibility

GPU	VRAM	GPUs Needed	Status
NVIDIA RTX 4090 (24 GB)Selected On-prem dev/inference, small models	24 GB	1×	Single GPU
NVIDIA A10G (24 GB) Inference & ML serving	24 GB	1×	Single GPU
NVIDIA L4 (24 GB) Cost-efficient inference	24 GB	1×	Single GPU
NVIDIA RTX 3090 (24 GB) Budget on-prem inference	24 GB	1×	Single GPU
Apple M4 Pro (24 GB unified) Mac inference, small models	24 GB	1×	Single GPU
NVIDIA A100 40 GB Training medium models, inference	40 GB	1×	Single GPU
NVIDIA L40S (48 GB) Inference, multi-modal workloads	48 GB	1×	Single GPU
NVIDIA A40 (48 GB) Inference & fine-tuning	48 GB	1×	Single GPU
Apple M4 Max (48 GB unified) Mac inference, light fine-tuning	48 GB	1×	Single GPU
NVIDIA H100 / H100 PCIe (80 GB) LLM training & production inference	80 GB	1×	Single GPU
NVIDIA A100 80 GB LLM fine-tuning & inference	80 GB	1×	Single GPU
AMD MI250X (128 GB) Large-scale HPC & LLM training	128 GB	1×	Single GPU
Apple M4 Ultra (128 GB unified) On-device Mac inference	128 GB	1×	Single GPU
NVIDIA H200 SXM (141 GB) Largest LLM inference & training	141 GB	1×	Single GPU
AMD MI300X (192 GB) Very large model inference & training	192 GB	1×	Single GPU
NVIDIA RTX 3080 (10 GB) Small model inference only	10 GB	2×	2× Multi-GPU
NVIDIA RTX 4070 (12 GB) Light inference, LoRA tuning	12 GB	2×	2× Multi-GPU
NVIDIA V100 (16 GB) Legacy workloads	16 GB	2×	2× Multi-GPU
NVIDIA RTX 4080 (16 GB) Consumer dev, small inference	16 GB	2×	2× Multi-GPU

Estimation Notes: VRAM estimates use the standard formula: weights × bytes/param + KV cache (2 × layers × KV-heads × head-dim × seq-len × batch × FP16 bytes) + framework overhead (7%). Training estimates include Adam optimizer states and gradients. Actual usage varies by runtime (vLLM, llama.cpp, Transformers), quantisation method, and model architecture specifics. Always leave 10–15% VRAM headroom. All calculations run locally in your browser.

Why Use Our VRAM Calculator?

First-Principles VRAM Estimates

Every calculation uses the actual VRAM formula: model weights in your chosen precision, KV cache from architecture parameters, activations, optimizer states, gradients, and framework overhead — not guesswork.

Fully Private — Runs in Browser

Your model configurations, parameter counts, and context settings are computed locally on your device. Nothing is sent to any server — your infrastructure planning data stays 100% confidential.

Inference, Training & GGUF Modes

Switch between inference, full fine-tuning (with Adam optimizer states and gradients), LoRA adapter training, and GGUF quantised inference — each mode applies the correct VRAM formula for that workload.

GPU Compatibility Table

See at a glance which GPUs can run your model on a single card and which require multi-GPU tensor parallelism. Compare 19 GPUs including H100, A100, RTX 4090, AMD MI300X, and Apple Silicon.

Common Use Cases for VRAM Calculator

Choosing a GPU for LLM Inference

Before renting or buying hardware, use the VRAM calculator to determine whether a 70B model fits on a single A100 80GB or requires two GPUs. Avoid over-provisioning and unnecessary cost.

Planning Fine-Tuning Runs

Full fine-tuning with Adam stores model weights, gradients, and optimizer states simultaneously — often 3–6× the inference footprint. Estimate total VRAM before starting a training job to prevent OOM crashes.

Optimising KV Cache for Long Contexts

KV cache grows linearly with sequence length and batch size. Use the vram calculator to see exactly how much additional VRAM is consumed when extending context from 4K to 32K or 128K tokens.

Selecting the Right GGUF Quantisation

Compare Q4, Q5, and Q8 quantisation levels side-by-side. See the exact VRAM trade-off between model quality and memory footprint to pick the right llama.cpp or Ollama quantisation for your hardware.

Running Models on Consumer Hardware

Determine which quantised models fit in 12 GB, 16 GB, or 24 GB VRAM on consumer GPUs like RTX 4090, RTX 4080, or RTX 4070 Ti. Plan your on-prem inference setup without expensive trial and error.

LoRA Training Budget Planning

LoRA freezes the base model and only trains adapter matrices, dramatically reducing training VRAM. Use the VRAM calculator with LoRA mode to see how rank selection affects your GPU memory requirements.

Understanding LLM VRAM Requirements

What is a VRAM Calculator?

A VRAM calculator estimates the GPU memory required to load and run large language models. Unlike system RAM, VRAM (Video RAM) is on-chip GPU memory — all model weights, the KV cache, activations, and framework buffers must fit within this fixed budget simultaneously. The amount of VRAM required depends on: the number of parameters (directly scales weight storage), the numerical precision (FP32 = 4 bytes/param, FP16/BF16 = 2 bytes/param, INT4 = 0.5 bytes/param), the sequence length and batch size (scales the KV cache), and the workload type (inference, training, or LoRA). Our VRAM calculator models all four components using the same formulas that vLLM, Transformers, and llama.cpp use internally.

How VRAM Is Calculated

Model Weights: The dominant component for inference. Calculated as parameters × bytes-per-parameter. A 7B FP16 model = 7 × 10⁹ × 2 bytes = 14 GB. INT4 quantisation halves this to ~3.5 GB. The vram calculator applies the exact bytes-per-parameter for each selected precision.
KV Cache: The key-value attention cache grows with context length and batch size. Formula: 2 × layers × KV-heads × head-dim × seq-len × batch × bytes-per-activation. Models using Grouped Query Attention (GQA) — like Llama 3.1 — have far fewer KV heads than attention heads, dramatically reducing KV cache size.
Training Overhead: Full fine-tuning requires storing gradients (same size as weights) and Adam optimizer states — first-moment (m1), second-moment (m2), and FP32 master weights — totalling roughly 12 extra bytes per parameter on top of the working model copy. LoRA confines optimizer states to the small adapter matrices only.
Framework Overhead: PyTorch, CUDA runtime, cuDNN, and memory fragmentation consume a further 5–10% of the working set. The calculator adds a 7% overhead buffer to every estimate. Always leave at least 10% free headroom in practice.

Precision and Quantisation Reference

FP32 (4 bytes/param): Full precision. Rarely used for inference; sometimes required for the master weight copy in mixed-precision training. A 7B model = 28 GB.
FP16 / BF16 (2 bytes/param): Standard precision for modern LLM inference and mixed-precision training. BF16 has a wider dynamic range and is preferred on Ampere and Hopper GPUs. A 7B model = 14 GB.
INT8 / FP8 (1 byte/param): 8-bit quantisation (bitsandbytes, FP8 on H100). Acceptable quality degradation. A 7B model ≈ 7 GB.
INT4 / GGUF Q4 (0.5 bytes/param): 4-bit quantisation for consumer hardware. Q4_K_M and Q4_K_S are the most popular GGUF formats. A 7B model ≈ 3.5–4 GB, fitting comfortably on a 6 GB consumer GPU.

Accuracy Notes and Limitations

This VRAM calculator produces engineering-grade estimates, not exact measurements. Actual VRAM usage can vary by ±10–20% depending on: the serving runtime (vLLM uses paged attention with different KV cache management; llama.cpp uses memory-mapped weights; Transformers pre-allocates static caches), the specific model implementation (some models use sliding window attention or sparse MLP reducing activation memory), CUDA graph capture overhead, and memory fragmentation across allocation lifetimes. Use this calculator for hardware selection and capacity planning — always validate with a small test run before committing to a production deployment. All calculations run locally in your browser; your model configurations are never sent to any server.

Related Tools

OpenAI Cost Calculator

Estimate API costs across OpenAI models using input/output tokens. Include model selector, token estimator, monthly usage projections, and pricing breakdown.

Claude Cost Calculator

Calculate Anthropic Claude API usage costs online. Estimate prompt and completion expenses with support for Claude 3.5, Opus, Sonnet, Haiku, prompt caching, Batch API, and model cost comparisons.

Gemini Cost Calculator

Estimate Google Gemini API costs based on input and output tokens. Support model comparisons (Gemini 1.5, 2.5, 3.x) and calculate monthly, daily, and annual API pricing forecasts.

DeepSeek Cost Calculator

Estimate DeepSeek API costs based on input, output, and cached prompt tokens. Compare DeepSeek-V4-Flash and DeepSeek-V4-Pro pricing structures online.

Frequently Asked Questions About VRAM Calculator

A VRAM calculator estimates the GPU memory required to load and run a large language model. It computes model weight storage (parameters × bytes-per-precision), KV cache (from sequence length, batch size, and attention architecture), activations, and — for training — optimizer states and gradients. It runs entirely in your browser with no data uploads required.

The parameter count is usually in the model name (7B, 70B, 405B). For exact values, check the model's config.json on Hugging Face — look for the "num_parameters" field, or multiply hidden_size × num_layers × ~12 as a rough estimate. Our presets include accurate parameter counts for popular models like Llama 3.1, Qwen2.5, Mistral, and Gemma 2.

During full fine-tuning, the GPU must store: the model weights (same as inference), gradient tensors of the same size, and Adam optimizer state — which includes FP32 master weights, first moment (m1), and second moment (m2) tensors, adding roughly 12 extra bytes per parameter. For a 7B model in BF16, inference needs ~14 GB but full training needs ~60–80 GB. LoRA training dramatically reduces this by limiting optimizer states to the small adapter matrices only.

The key-value cache stores the attention keys and values computed for all previous tokens in a sequence. This avoids recomputing them at each generation step. The cache size = 2 × layers × KV-heads × head-dim × sequence-length × batch-size × bytes. Since sequence length is in the formula linearly, extending context from 4K to 128K tokens increases KV cache by 32×. Models with Grouped Query Attention (GQA) reduce KV-heads to 8 or fewer, substantially cutting KV cache size.

GGUF (used by llama.cpp and Ollama) quantises model weights to 2–8 bits per parameter, dramatically reducing weight storage. GGUF mode calculates weight memory as parameters × (quantBits / 8) bytes. The KV cache is always stored in FP16 regardless of weight quantisation, as it changes dynamically with each generation step. Q4_K_M (≈4 bits) is the most popular balance of quality and memory efficiency.

GQA reduces the number of key and value attention heads to a fraction of query heads (e.g., 8 KV heads vs 32 query heads in Llama 3.1). Since KV cache memory scales with KV heads, GQA reduces KV cache by 4× in that example. This is why modern models like Llama 3.1 can serve longer contexts without proportionally exploding VRAM. In the vram calculator, enter the actual num_key_value_heads from config.json for accurate KV cache estimates.

At FP16, a 70B model requires approximately 140 GB for weights alone, plus KV cache. This fits on two NVIDIA H100 80GB or A100 80GB GPUs (160 GB combined) using tensor parallelism. At INT4, weights drop to ~35 GB, fitting on a single H100 or two A100 40GB cards. Use the GPU compatibility table in the calculator to see the minimum GPU count for each card in your configuration.

The calculator uses the standard theoretical formula that all runtimes are based on. vLLM uses paged attention which manages KV cache in pages — actual peak VRAM may be 5–15% higher due to page overhead. llama.cpp memory-maps weights on CPU and only loads active layers to GPU, so it can run larger models with less VRAM. Transformers pre-allocates the full KV cache up front. The 7% framework overhead buffer in the calculator accounts for most of this variance.

Yes. The VRAM calculator runs entirely client-side in your web browser. Your model architecture parameters, precision choices, and context settings are processed locally on your device and never sent to any server. No account or signup is required.