size-local-model
Skill7 starsChanged 31 days agoMITView on GitHub
Estimate the GPU or unified memory needed to run an open-weight model locally, covering weights at a given precision or quantization plus KV cache for a chosen context length and concurrency, and compare it with available memory. Use whenever the user asks whether a model fits their GPU, Mac, or server, how much VRAM or RAM a model needs, what quantization to use, how context length or batch size affects memory, or how to size hardware for Ollama, llama.cpp, vLLM, or LM Studio, even if they don't say "memory". Gives estimates with stated assumptions, never a guarantee of fit.
In this repository4 other files it gives its agents.
- AGENTS.mdAGENTS.md
- eval-modelsSkill
- find-modelSkill
- route-modelsSkill
The file
We split this skill into the rules it gives an agent
3 rules. About 0 are common, 1 is distinctive, and 2 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈1
Distinctive · estimate
This author's own choices. Worth a look.
2
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 13
Read the model card or the model's config.json from its official repository.
- Very specific: line 34
Add --json for machine-readable output.
- Distinctive: line 52
If the total exceeds available memory, suggest the levers in order of typical impact: shorter context, lower concurrency, KV-cache …
≈ “Common” is an estimate: at least 1 in 5 of the 350,495 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Estimate local model memory
Gather inputs from official sources
Read the model card or the model's config.json from its official repository. Do not guess architecture values.
| Input | Where it comes from |
| Total stored parameters (billions) | Model card. For mixture-of-experts models use total parameters, not active ones: all experts are loaded. |
| Effective bits per weight | 16 for BF16/FP16, 8 for 8-bit, about 4–5 for common 4-bit formats. Most accurate: file size in bytes × 8 ÷ parameter count from the actual download. |
| Layers | num_hidden_layers |
| KV heads | num_key_value_heads; if absent, the model uses standard multi-head attention, so use num_attention_heads. |
| Head dimension | head_dim if present, otherwise hidden_size ÷ num_attention_heads. |
| Context | Tokens the user plans to use per sequence, not the advertised maximum. |
| Concurrency | Sequences served at the same time (1 for a single user). |
| KV precision | 16 by default; 8 if the runtime is set to quantize the KV cache. |
Run
python3 scripts/estimate_memory.py --params-b 8.03 --bits 16 \
--layers 32 --kv-heads 8 --head-dim 128 --context 8192 \
--concurrency 1 --overhead-pct 10 --available-gib 24Add --json for machine-readable output. Without the KV inputs, only weights are estimated and the output says so.
The modelfinder_estimate_memory MCP tool runs the same calculation with the same inputs.
Formulas (binary GiB):
- weights = parameters × bits ÷ 8
- KV cache = 2 × layers × KV heads × head dimension × context × concurrency × KV bits ÷ 8
Report
Give weights, KV cache, overhead, total, and headroom, then the assumptions. Always state:
- The KV formula assumes standard full attention (MHA or GQA). It does not apply to multi-head latent attention, sliding-window, hybrid, or linear attention; for those, use the runtime's documentation or measure.
- The overhead percentage is an allowance chosen by the user, not a measurement. Runtimes add buffers, activations, and fragmentation.
- Fitting in memory says nothing about speed or quality. Quantization can reduce quality; recommend testing the chosen quantization on the user's own examples with eval-models (local servers work through the
openai_compatibleprovider). - On shared or unified memory (for example Apple silicon), the operating system and other apps also need memory.
If the total exceeds available memory, suggest the levers in order of typical impact: shorter context, lower concurrency, KV-cache quantization, lower weight precision, a smaller model, or splitting across devices if the runtime supports it. Do not guarantee that any configuration will run.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.