How to Choose an LLM for Your Hardware
Stop relying on parameter count guesses. Learn how to calculate the four real memory consumers: weights, KV cache, activation, and runtime overhead, so you can avoid out-of-memory crashes.
If you've ever downloaded an open-weight model, watched it crash with an out-of-memory error, downloaded a smaller one, and watched that crash too, you already know the problem.
Model cards tell you parameter counts: 7B, 14B, 32B, 70B. They almost never tell you whether that model will actually run on the GPU sitting on your desk.
Most people solve this by trial and error: download 20 GB of weights, run it, watch it fail, and repeat with another quant. It works eventually, but it burns hours of time and bandwidth, and it teaches you nothing you can reuse next time.
There is a better way. In this guide, we'll break down the exact math of LLM inference: what actually happens inside your GPU and how to calculate whether a model will fit before downloading anything.
It's not just "the model size"
The headline number on a model card only describes the parameter count. That is just one part of what has to fit in memory.
The real total your GPU needs looks like this:
Skip any one of these terms and your estimate will fail, usually in the direction of: "it fit in my calculation, but crashed my GPU."
1. Weights: the majority VRAM consumer
When you run models locally, the weights take up most of your VRAM, usually 65% to 85% of your available memory.
This size comes down to two things: parameter count and precision. And this is where quick mental math usually falls apart.
Why the simple parameters × precision formula fails
Most guides suggest a quick shortcut:
For unquantized models (like a 7B model at FP16: 7B parameters × 2 bytes ≈ 14 GB), that estimate works reasonably well. But this formula fails completely on modern mixed-precision models and k-quants.
Here is why:
- •Mixed-precision k-quants: Popular formats like
Q4_K_Mare not uniformly 4 bits. They keep critical attention tensors (likev_projoro_proj) at 5-bit or 6-bit precision, while compressing feed-forward layers at 4-bit. The true average lands around ~4.8 bits per parameter. - •High-precision embeddings and output heads: Layers like token embedding tables (
token_embd) and normalization weights are often kept at FP16, Q8_0, or Q6_K. On modern models with 150k-token vocabularies (like Qwen or DeepSeek), these layers alone add gigabytes that a naive 4-bit calculation ignores. - •Mixture of experts (MoE): Router gates and shared layers often stay at higher precision than the routed expert blocks.
This is why asking "Can my GPU run a 7B model?" is the wrong question. A 7B model at FP16 needs ~14.5 GB of VRAM just for weights. That same 7B model in Q4_K_M takes only ~4.3 GB. The quantization format and its mixed-precision distribution matter just as much as the parameter count.
2. KV cache: the part that grows quietly
Here is where most back-of-the-envelope estimates fall apart. As a model generates text, it caches the Key and Value tensors for every token in the conversation so it doesn't have to recompute self-attention from scratch each time.
That cache grows linearly with your context length. Depending on the model's attention architecture, it grows at very different rates:
- •Multi-head attention (MHA): Older architectures (like original LLaMA-1) allocate separate KV heads for every query head. At 16k or 32k context, the KV cache alone can take 10 GB to 15 GB of VRAM.
- •Grouped-query attention (GQA): Modern models (LLaMA 3, Qwen 2.5, Mistral) share key-value heads across query groups (often 8:1), cutting KV cache size by 4x to 8x.
- •Multi-head latent attention (MLA) and sliding window (SWA): DeepSeek compresses key-values into low-rank latent vectors (~9x reduction), while Gemma and Mistral use sliding windows to cap local attention memory.
Two models with the exact same parameter count can have wildly different KV cache costs. A model might run smoothly for short prompts, then crash halfway through an extended coding session or document analysis.
3. Activation memory: the temporary spike
While the model generates text, it needs scratch space for intermediate calculations (layer norms, projections, attention logits) separate from the weights and KV cache.
Activation memory scales with your prompt chunk size (batch or micro-batch ubatch) and the model's hidden dimension size. While temporary, it is a real allocation spike:
The long prompt trap: Activation memory is often the difference between a model that idles comfortably, and one that crashes the moment you paste an 8,000-token document into your prompt.
4. Runtime overhead
Before you load a single weight, your GPU driver and OS claim memory just to exist:
- •CUDA context: On an NVIDIA card or Google Colab instance, the CUDA runtime reserves around 500 MB of VRAM before running any operations.
- •Desktop display server: Windows DWM or macOS WindowServer takes 0.8 GB to 1.5 GB of dedicated VRAM just to render your monitors and open windows.
If you have an 8 GB or 16 GB GPU and don't subtract this overhead upfront, your math will be off by a predictable 1.0 to 1.5 GB, turning an expected fit into an out-of-memory crash.
The practical checklist
Instead of guessing and hoping for the best, run this sequence before you pull any model:
- 1.Check what memory is actually free right now. Don't look at the number printed on your GPU box. Open Task Manager, run
nvidia-smi, or check your OS activity monitor to see what VRAM or unified memory you actually have free after your desktop and browser take their cut. - 2.Get the true file size from metadata. Don't estimate weights by multiplying parameter count by 4. Look at the exact GGUF or safetensors file size. Quantized files (especially k-quants) package tensors at different bit depths and keep embedding tables in higher precision, so always check the real byte count.
- 3.Subtract activation memory and runtime overhead. Take off about 500 MB for the CUDA context, another slice for your display server if you're on a single desktop GPU, and scratch space for the prompt chunk (activation buffer).
- 4.See what's left for your KV cache. Whatever memory remains after weights and overhead is your real budget for conversation history.
- 5.Set your context window to fit that remaining space. If you have 2 GB left, a model with Grouped-Query Attention (GQA) or sliding-window attention might easily give you 16k or 32k tokens. An older model with full multi-head attention might only give you 4k before crashing.
How whichllmmodel automates this math
None of this math is exotic, but the details are scattered everywhere. Quantization constants live in one repo, attention shapes are buried in 40-page papers, and your actual free VRAM changes depending on what apps you have open.
Doing this by hand on scrap paper for every new model gets old fast.
That is why I built whichllmmodel.com. You pick your hardware (or let the tool detect it), and it calculates the exact weights + KV cache + activation + overhead breakdown for you. You can slide your context length around and see exactly where the memory ceiling hits before downloading a single gigabyte.
If you want to check your current machine right from your terminal:
npx whichllmmodel -o
This scans your active hardware and opens the calculator tuned directly to your machine.
Written by Zubair Tahir
Founder & Lead Developer
Building independent, empirical evaluation tools to eliminate model decision fatigue for AI engineers and developers.