GPU Advisor — Local Inference Optimization

Detect your GPU and get actionable recommendations for running LLMs locally with llama.cpp.

When to Use

• User asks "what model can I run on my GPU?"
• User asks "will this model fit in my VRAM?"
• User is getting OOM errors with llama.cpp
• User wants to maximize context length on their hardware
• User asks about KV cache quantization or TurboQuant

Prerequisites

• AMD GPU: rocm-smi available (ROCm installed)
  • Verify: rocm-smi --showinfo
• NVIDIA GPU: nvidia-smi available (CUDA drivers installed)
  • Verify: nvidia-smi
• Python 3.8+ (stdlib only, no dependencies)
  • Verify: python3 --version

Commands

Detect GPU

python3 scripts/gpu_advisor.py detect

Returns: GPU name, arch, VRAM total/used/free, driver version, compute platform.

Recommend model configuration

python3 scripts/gpu_advisor.py recommend --model Qwen3.5-27B --quant Q5_K_M

Returns: whether it fits, recommended KV cache type (f16/q8_0/q4_0/turbo3/turbo4), max context length.

Check if a specific config fits

python3 scripts/gpu_advisor.py check --model-size 18.5 --ctx 80000 --kv-type turbo3

Returns: VRAM breakdown (model + KV + overhead), fit/no-fit verdict.

How It Works

The advisor uses a simple VRAM budget model:

Total VRAM = Model weights + KV cache + Compute overhead (~500MB)

KV cache size = 2 × n_layers × n_kv_heads × head_dim × ctx_length × bytes_per_element
  f16:    2 bytes/element (baseline)
  q8_0:   1 byte/element (2× compression)
  q4_0:   0.5 bytes/element (4× compression)
  turbo3: 0.375 bytes/element (5.3× compression)
  turbo4: 0.25 bytes/element (8× compression)

For SWA models (Gemma 4), the advisor recommends f16 on SWA layers + turbo on global layers, because turbo on SWA causes catastrophic quality loss.

Example Output

GPU: AMD Radeon RX 7900 XTX (gfx1100)
VRAM: 24576 MiB total, 22891 MiB free
Platform: ROCm 6.4

Model: Qwen3.5-27B Q5_K_M (18.5 GiB)
Context: 80,000 tokens

  KV type   | KV size  | Total VRAM | Fits? | Max context
  ----------|----------|------------|-------|------------
  f16       | 5,120 MB | 24,444 MB  |  NO   | ~62K
  q4_0      | 1,280 MB | 20,604 MB  |  YES  | ~250K
  turbo3    |   960 MB | 20,284 MB  |  YES  | ~330K
  turbo4    |   640 MB | 19,964 MB  |  YES  | ~500K

Recommendation: turbo3 — best quality/compression tradeoff.
  llama-server -m model.gguf -ngl 99 --ctx-size 80000 \
    --cache-type-k turbo3 --cache-type-v turbo3

Known Model Profiles

The advisor includes VRAM profiles for popular models:

Guardrails

• VRAM Safety: Do not exceed 90% of free VRAM; OOM errors may crash system processes.
• Thermal Limits: Monitor GPU temperatures during long context inference to prevent throttling.
• Driver Integrity: Ensure ROCm/CUDA drivers match GPU architecture to avoid kernel panics.
• Model Verification: Validate GGUF checksums to prevent loading corrupted weights.

Limitations

• SWA detection is based on known model names, not parsed from GGUF metadata
• Model size database covers popular models; unknown models need --model-size via check command
• TurboQuant requires a compatible llama.cpp build (upstream or domvox/llama.cpp-turboquant-hip)