whichLlmmodel

Methodology

whichllmmodel does not guess whether a model runs on your machine. We calculate it layer by layer. This page explains how each feature on our platform works under the hood.

01.

The Master Equation of Memory

A model does not run on weight files alone. When local inference crashes with an Out-Of-Memory (OOM) error, it is almost always because the KV cache and runtime scratchpads pushed usage over your physical hardware limit.

Total Memory Required = Model Weights + KV Cache + Runtime Overheads

Every calculation on this site breaks down into these three distinct components:

  • Model Weights:The static bytes required to hold the model's tensor layers in memory.
  • KV Cache: The persistent attention memory that stores past token history across your conversation. It scales directly with context length.
  • Runtime Overheads: The CUDA driver context and prompt micro-batch scratchpad memory allocated while evaluating tokens.

KV Cache vs. Activation Memory: Why They Scale Differently

A common misconception is that running a long context window (like 64k or 128k) exponentially blows up your activation scratchpads. In reality, they are governed by two entirely separate parameters:

  • KV Cache scales with Total Context Length (n_ctx): It stores the persistent attention history (keys and values) for all previously generated tokens. This memory stays in VRAM for the entire duration of your session and grows linearly with context.
  • Activation Memory scales with Batch Size (ubatch): It is a transient scratchpad used during matrix multiplication. When processing a 100,000-token prompt, the engine evaluates it in chunks (e.g. --ubatch 512 or 256). These activations exist only for milliseconds while that chunk calculates, and are discarded immediately after.

Why this matters: If you are on an 8GB or 12GB GPU and want to ingest massive prompts, your KV cache will expand, but you can prevent sudden activation OOM spikes simply by lowering --ubatch to 256 or 128.

02.

Model Weight Sizing: Why Naive Math Fails

The most common reason local AI developers experience unexpected Out-Of-Memory (OOM) crashes is relying on naive parameter arithmetic. On paper, multiplying parameter count by bit depth seems straightforward—in practice, it miscalculates memory by gigabytes.

The Naive 4-Bit Myth

A common rule of thumb seen across forums calculates 4-bit model size as:

Naive Size = Parameters (B) × 4 bits ÷ 8 = 0.5 × Parameters (GB)

For a 70-billion parameter model, this formula predicts exactly 35.0 GB. Yet when you download the standard Llama-3.1-70B-Instruct-Q4_K_M.gguf, the physical file on disk is ~42.5 GB—over 7 gigabytes larger. If your hardware budget relied on 35 GB, your GPU will instantly crash.

Why Real Quantized Files Are Significantly Larger

  • Mixed-Precision K-Quants: High-quality quantizations like Q4_K_M are not 4.0 bits across the board. Sensitive layers—such as token embeddings (token_embd) and the output projection head (output.weight)—are kept at 6-bit, 8-bit, or FP16 precision to protect model intelligence from collapsing.
  • Block Scales & Offsets: Modern quant formats divide weights into small blocks (e.g. 32 or 256 weights). Each block requires its own floating-point scale and minimum offset so the engine can dequantize tensors on the fly.
  • Tensor Alignment & GGUF Overhead: GGUF enforces strict 32-byte memory alignment for every tensor to enable direct zero-copy memory mapping (mmap), adding container padding.

When accounting for these factors, the effective bits-per-weight (BPW) of Q4_K_M is actually ~4.80 to 4.92 BPW (not 4.0), and Q8_0 is ~8.50 BPW (not 8.0).

Our Ground-Truth Approach

whichllmmodel never guesses model weights. We do not use approximation formulas or theoretical multipliers in our calculations. Every weight size across all models and quantization formats is pulled directly from verified upstream binary artifacts:

  • GGUF Quantizations: We index the exact, byte-for-byte physical file sizes of verified GGUF releases from trusted community builders (such as Bartowski, Unsloth, and official model maintainers). The memory figure shown is the exact payload that will occupy your disk and VRAM.
  • FP16 / BF16 Base Weights: For unquantized checkpoints, we extract the exact byte footprint directly from the official Hugging Face model.safetensors.index.json metadata.
  • Auxiliary Sub-Module Isolation: For modern hybrid architectures, we explicitly parse auxiliary components—such as 10+ GB speculative N-Gram lookup tables or Multi-Token Prediction (MTP) heads—so users can see exactly what can be offloaded to NVMe SSD or system RAM.
03.

KV Cache Scaling Across Architectures

Not all KV caches scale identically. In early models, running a 64k or 128k context window could easily require more VRAM for the token history than for the entire model itself. Today, architectural innovations have drastically reshaped how context memory behaves in hardware.

1. Multi-Head (MHA) vs. Grouped-Query Attention (GQA)

In original Transformer architectures using Multi-Head Attention (such as LLaMA 1), every query head maintained an independent key and value head. At long context lengths, this caused memory consumption to balloon steeply.

Modern open-weights models (including LLaMA 3, Mistral, and Qwen) standardise on Grouped-Query Attention (GQA). By sharing a single key-value head across a group of query heads (typically an 8:1 ratio), GQA achieves roughly an 75% to 85% reduction in KV cache memory with practically zero loss in generation quality.

2. Multi-Head Latent Attention (MLA) — DeepSeek

DeepSeek-V2 and DeepSeek-V3 introduced Multi-Head Latent Attention. Instead of retaining full uncompressed key-value tensors for every token, MLA projects attention inputs into a compact, low-rank latent vector before storing them in the cache.

During inference, the latent vectors are projected back up on the fly. This architecture cuts total KV cache footprint by over 85% to 90% compared to conventional attention, making 128k context windows practical on modest hardware setups.

3. Sliding Window Attention (SWA) — Gemma & Mistral

Rather than having every layer attend to the entire document history, models with Sliding Window Attention restrict intermediate layers to look back across a fixed token horizon (for example, the last 4,096 tokens).

In engines like llama.cpp, these sliding layers are allocated as fixed-size circular ring buffers. As your context grows beyond the window, older tokens in sliding layers are overwritten rather than expanding VRAM indefinitely. Only global layers scale linearly, creating a distinct memory inflection point that flattens your overall context footprint.

4. Linear & Recurrent Attention (GDN, DeltaNet, Mamba)

Hybrid models and state-space architectures (such as Google's Gated DeltaNet in Gemma 2 or Mamba) do not store an expanding historical tape of individual tokens. Instead, they continuously compress history into a fixed-size internal recurrent state token by token.

Their memory footprint scales with O(1) constant complexity. Whether processing 1,000 tokens or 1,000,000 tokens, the recurrent memory requirement remains effectively flat.

KV Cache Quantization (FP16 vs. Q8 vs. Q4)

By default, runtime engines store key-value tokens in 16-bit floating point (FP16, 2 bytes per value). Modern local runtimes support quantizing the cache itself:

  • Q8_0 Cache: Reduces KV memory by ~50% with imperceptible perplexity change.
  • Q4_0 Cache: Reduces KV memory by ~70% to 75%, unlocking massive contexts on cards with 8 GB to 16 GB of VRAM.
04.

Discrete GPU vs. Apple Unified Memory

Selecting between a PC discrete GPU and an Apple Silicon Mac fundamentally changes the underlying memory physics and offload rules:

Discrete GPU (PC / Windows / Linux)

Discrete systems split execution across two completely separate physical memory pools connected by the PCIe bus:

  • Dedicated VRAM: Extremely fast (500 to 1,000 GB/s on GDDR6/X). This is where active model layers and the primary KV cache need to live for real-time speed.
  • System Host RAM: Much slower (40 to 100 GB/s on DDR4/DDR5). Used as overflow memory when model layers exceed VRAM capacity.

Fitting 100% of layers into VRAM yields 40–80+ tokens/sec. Spilling dense layers into system RAM forces continuous PCIe round-trips that drag generation speeds down.

Apple Silicon (Unified Memory): Capacity vs. Bandwidth

Apple Silicon shares a single memory pool across CPU and GPU cores, eliminating PCIe transfer bottlenecks and allowing a laptop to hold massive models that would otherwise require multi-GPU setups. However, two crucial realities govern Mac inference:

1. The 25% macOS Safety Reserve

Usable Unified RAM = Total Mac RAM × 0.75

macOS reserves roughly 25% of total memory for system graphics, display compositing, and OS tasks. Exceeding 75% triggers aggressive memory compression and SSD swapping. whichllmmodel automatically enforces this 75% usable limit in its calculations.

2. The Bandwidth Bottleneck: Why "Fitting" Isn't Always "Fast"

LLM text generation (the decoding phase) is strictly memory-bandwidth bound. To generate a single token, the chip must read every active parameter from memory into its compute cores. Generation speed scales directly with memory bandwidth:

  • Base & Pro Chips (100–200 GB/s): Base M-series and Pro chips have modest memory bus widths (100 to 150 GB/s on Base; ~150 to 200 GB/s on Pro). While an M3 Pro with 36 GB RAM can easily fit a 32B model, its ~150 GB/s bandwidth means generation crawls at ~6 to 10 tokens/sec.
  • Max & Ultra Chips (300–800 GB/s): Only Max (~300–400 GB/s) and Ultra (~800 GB/s) chips provide the memory throughput needed to run 32B+ models at fluid, conversational speeds.
  • Discrete GPUs by Comparison (750–1,000+ GB/s): A desktop RTX 4080 or 4090 offers 716 to 1,008 GB/s on GDDR6X—roughly 5x to 8x the bandwidth of a base Mac.

Unified memory solves the capacity barrier, but memory bandwidth governs your actual generation speed.

05.

llama.cpp vs. Transformers Engine

The engine you choose directly impacts how memory is structured for model weights, attention buffers, and context history:

llama.cpp Engine (GGUF)

The standard runtime for local consumer inference (powers Ollama, LM Studio, Jan, and llama-server):

  • Deterministic Pre-Execution: Before running a single token, llama.cpp constructs a static computation graph that plans exact byte requirements across all layers, preventing surprise runtime allocation spikes.
  • 256-Cell Ring Buffers: Aligns KV cache allocations to 256-token blocks. Models with sliding window attention (like Gemma) reuse ring cells so context memory stops growing past the window size.
  • Native KV Quantization: Natively compresses context memory down to Q8_0 (1.06 bytes/element) or Q4_0 (0.56 bytes/element).

Transformers & vLLM (Safetensors)

The standard runtime for Python server deployments and batched API serving:

  • Dynamic CUDA Heap: PyTorch dynamically allocates tensors during generation on the CUDA heap, which can cause transient memory spikes.
  • Full Precision Baseline: Default KV cache typically runs unquantized (FP16 or BF16 at 2.0 bytes/element).
  • PagedAttention (vLLM): Allocates 16-token virtual pages on demand, virtually eliminating fragmentation in multi-user serving.

Key Takeaway: An 8B parameter model running at 32k context can take under 9 GB of VRAM in llama.cpp with Q4_0 KV cache, but easily exceeds 16 GB under standard unquantized Transformers FP16.

06.

Total vs. Available Memory

Across whichllmmodel, you can toggle between Total and Available memory modes. Here is how they change our hardware fit calculations under the hood:

Total Memory Mode

Designed for normal everyday desktop workstations with screens attached. Operating systems (Windows DWM, macOS WindowServer, Linux desktop environments) consume memory for display compositing, open browser tabs, and background apps. In this mode:

  • In the Transformers Engine: We subtract a flat 1.5 GB safety buffer from your VRAM for display compositing and PyTorch CUDA context.
  • In the llama.cpp Engine: We do not deduct a flat 1.5 GB. Instead, its exact overhead formula explicitly budgets 500 MiB for the desktop display buffer, 488 MiB for CUDA context, and the micro-batch (ubatch) activation workspace.
  • Host System RAM (Both Engines): We reserve 3.0 GB of RAM for operating system background tasks, browsers, and services.

When to use:When you enter your card's advertised box spec (e.g. 16 GB RTX 4080 or 36 GB Mac) and want to be sure launching an app won't crash your desktop.

Available Memory Mode

Designed for users who know their exact live unallocated free memory. Evaluates the model strictly 1:1 against your hardware with zero safety deductions:

  • Zero VRAM Display Deduction: For llama.cpp, the 500 MiB display buffer is zeroed out. For Transformers, no 1.5 GB buffer is subtracted.
  • Zero Host RAM Deduction: Evaluates against 100% of the host RAM you input.

When to use: On dedicated headless cloud instances (RunPod, Lambda, Vast.ai) or when entering live free memory numbers directly from system resource monitor.

07.

CPU Layer Offloading (-ngl)

When a model is slightly too large for your GPU, you don't have to abandon it. The CPU Layer Offload Slider lets you split model layers across GPU VRAM and host system RAM using the -ngl flag.

How llama.cpp Places Layers: Top-Down Allocation

Most users assume llama.cpp offloads from Layer 0 upwards. In reality, llama.cpp allocates layers from the top down:

  1. The Output Head Gets First Priority: Total offloadable units equal total_layers + 1. The very first piece placed on the GPU is the Output Head (lm_head), ensuring vocabulary sampling runs at full GPU speed.
  2. Layers Fill Top-Down: Next, layers are assigned to the GPU backwards—starting from the last layer down toward the first, for as many layers as your VRAM can hold.
  3. Base Layers Remain in RAM: The remaining bottom layers (Layer 0, 1, 2...) stay in CPU host RAM if VRAM runs out.

KV Cache Follows Layer Placement

KV cache memory is allocated on the exact device where that specific layer computes. If Layer 24 is offloaded to GPU, its KV cache is stored in GPU VRAM. If Layer 4 remains on CPU, its KV cache is stored in Host RAM.

08.

Mixture of Experts (MoE) Offloading

Mixture of Experts (MoE) models (like Qwen 35B or Mixtral 8x7B) behave fundamentally differently from dense models during CPU offloading. How you offload them determines whether you get 4 tokens/sec or 25 tokens/sec.

Naive -ngl vs. Dedicated MoE Offloading (--cpu-moe)

When an MoE model exceeds your VRAM, you have two distinct approaches:

  • Approach 1: Naive Layer Splitting (-ngl <N>) — Slow (3–6 t/s): Treats the MoE model like a dense model by chopping entire layers. For every offloaded layer, its Attention heads, Layernorms, and KV Cache are booted into slow CPU host RAM. The CPU is forced to compute heavy multi-head attention sequentially, causing severe generation bottlenecks.
  • Approach 2: Dedicated MoE Expert Offload (--cpu-moe) — Fast (15–25+ t/s): Maxes out -ngl 99 so that 100% of Attention, Layernorms, and the KV Cache stay in GPU VRAM. Only the expert weights (~85% of total weight bytes) are placed in host RAM. Because only 2 to 8 experts activate per token, PCIe traffic remains minimal and token rates stay practical.

The Two llama.cpp Offloading Flags

  • Full Offload (--cpu-moe): Moves 100% of expert layers into system RAM while keeping all attention mechanisms and KV cache accelerated in GPU VRAM:
    llama-cli -m model.gguf -ngl 99 --cpu-moe
    Enables running massive 30B+ MoE models on 8GB to 12GB VRAM cards at practical reading speeds.
  • Partial Offload (--n-cpu-moe <N>): Selectively moves only N expert layers to system RAM, leaving all remaining expert layers accelerated directly in VRAM:
    llama-cli -m model.gguf -ngl 99 --n-cpu-moe 14
    Maximizes performance by utilizing every megabyte of available VRAM before spilling into RAM.

Our Analytical Minimum-Expert Solver

whichllmmodel implements an analytical solver that calculates the minimum number of expert layers (N_cpu) that must be offloaded to prevent an OOM crash:

N_cpu = ceil( (Full VRAM Required - Available VRAM) / Expert Size Per Layer )

If N_cpu is less than the total number of layers, we output the exact flag so you keep maximum experts accelerated on the GPU.

09.

N-Gram & Auxiliary SSD Offloading

Modern architectures increasingly incorporate auxiliary embedding matrices—such as speculative N-Gram lookup tables (8–12 GB) and Per-Layer Embeddings (PLE). Because these tables can easily double memory requirements, we model two distinct offloading tiers:

Types of Auxiliary Tables

  • N-Gram Speculative Tables: Used in speculative decoding models to predict candidate token sequences ahead of time. While these tables can reach 8–12 GB, each token generation step only queries a few sparse keys.
  • PLE (Per-Layer Embeddings): Certain specialized models assign distinct embedding matrices across different network depths. Rather than filling VRAM, these matrices can be dynamically accessed as each layer computes.

1. Host RAM Offload: Protecting GPU VRAM

Dense model weights require high-bandwidth memory (500–1,000+ GB/s) because every matrix parameter participates in token generation. In contrast, auxiliary tables are sparse lookups. By default, runtimes place these tables in Host System RAM, keeping 100% of your GPU VRAM dedicated to active transformer weights and KV cache.

2. NVMe SSD Streaming: Protecting Host RAM

On workstations with 16 GB or 32 GB of system RAM, allocating an extra 8–14 GB table in host memory can push the OS into heavy disk swapping or trigger an OOM crash.

Because each token generation step only queries tiny slices (~4 to 16 KB) from the multi-gigabyte table, modern runtimes can memory-map (mmap) the file directly from a fast NVMe SSD.

When you toggle Auxiliary SSD Offload on whichllmmodel, we safely drop the host RAM requirement for these auxiliary tables to 0 GB, letting you run these models without needing a 64 GB RAM workstation.