Methodology: How We Calculate Hardware Fit

1. The formula

Text models are autoregressive: they generate one token at a time, and each new token attends back over every previous token in the sequence. That decode loop is why text models need a KV cache term that image/diffusion models don't — the cache grows as generation proceeds, and at long context lengths it can dwarf the weights themselves. Getting hardware-fit right for text models means getting all three terms right, not just the weights.

2. How we calculate model weights

3. How we calculate KV cache

This is where a single generic formula breaks down, because different attention architectures allocate KV cache completely differently. We maintain separate, independently validated formulas for each family:

• •GQA (grouped-query attention) — multiple query heads share a smaller set of key/value heads, cutting cache size versus attention where every head keeps its own K/V.
• •MLA (multi-head latent attention) — keys and values are compressed into a shared low-rank latent representation before being cached, shrinking cache size well below standard multi-head attention. Not every inference engine implements this optimization — some fall back to the standard (uncompressed) cache formula for MLA models instead. You can select whether to assume MLA optimization is active in the filters.
• •MLA KDA — a hybrid that interleaves MLA layers with Kimi Delta Attention, a gated linear-attention layer with a fixed-size recurrent state, so most layers don't accumulate a growing cache at all.
• •GQA Delta Net — a hybrid that interleaves GQA layers with Gated DeltaNet linear-attention layers, similarly replacing most of the network's cache growth with a fixed-size recurrent state.
• •Sliding window attention — each token attends only to a fixed-size local window of recent tokens rather than the full sequence, capping cache size at that window regardless of context length.
• •Sliding window with KV sharing — sliding-window attention where cached key/value state is shared across layers, reducing memory further on top of the windowing.
• •Sliding window with global attention — mixes local sliding-window layers with occasional full-attention layers, so most of the network gets the cache savings while a few layers retain long-range context.
• •Mamba (state-space, no traditional KV cache) — replaces attention with a fixed-size recurrent state that doesn't grow with sequence length, so there's no KV cache term to calculate at all.

The architecture family determines which formula applies — treating them all as "standard attention" is the single biggest source of error we see in generic VRAM calculators.

4. How we calculate overhead memory

Overhead covers everything beyond weights and KV cache that inference actually consumes: CUDA workspace, intermediate activation buffers, and inference-engine scratchpads. Rather than treating these as fixed constants, we model overhead as dynamic — scaling with the rest of the model's footprint:

The 8% multiplier applies to the combined weights + KV cache size rather than being a flat number, because as either grows — a bigger model or a longer context — the intermediate activations and CUDA buffers required grow roughly proportionally too. A fixed constant would undersell the real cost at scale; scaling it with weights + KV cache keeps the estimate accurate as models and context windows get larger.

5. How we test on real GPU runs

6. How we sort and filter models by hardware fit

Once total VRAM is calculated for a given model, quantization, and context length, we compare it against your hardware and assign one of three categories:

• •Fits in VRAM — the full model and context fit within your GPU's dedicated memory; nothing spills over.
• •CPU Offloaded — VRAM cannot hold the whole model, so the runtime allocates the KV Cache and runtime overhead into GPU VRAM first, fills the remaining VRAM with model weights, and offloads only the remaining weights into System RAM.
• •Too Large — either the KV Cache and overhead exceed your VRAM entirely, or the offloaded weights exceed your usable System RAM.