agentleFS
Sign inStart free
Hugging FaceKnown publisherhuggingface/mlclaw/assets/hf-tooling/skills/huggingface-local-models/SKILL.md

huggingface-local-models

Skill10 starsChanged 3 months agoMITView on GitHub

Use to select models to run locally with llama.cpp and GGUF on CPU, Mac Metal, CUDA, or ROCm. Covers finding GGUFs, quant selection, running servers, exact GGUF file lookup, conversion, and OpenAI-compatible local serving.

  • Installs packages

We split this skill into the rules it gives an agent

12 rules. 3 are common, 0 are distinctive, and 9 are very specific to this project.

  • 3

    Common

    Most of the 3 edited versions we read keep these.

  • 0

    Distinctive

    This author's own choices. Worth a look.

  • 9

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    Search the Hugging Face Hub for llama.cpp-compatible GGUF repos, choose the right quant, and launch the model with llama-cli or …

    line 8
  2. Very specific:

    Search the Hub with apps=llama.cpp.

    line 12
  3. Common:

    Prefer the exact HF local-app snippet and quant recommendation when it is visible.

    line 14
  4. Very specific:

    Prefer the exact quant that HF marks as compatible on the ?local-app=llama.cpp page.

    line 94
  5. Very specific:

    Keep repo-native labels such as UD-Q4_K_M instead of normalizing them.

    line 95
  6. Very specific:

    Default to Q4_K_M unless the repo page or hardware profile suggests otherwise.

    line 96
  7. Very specific:

    Prefer Q5_K_M or Q6_K for code or technical workloads when memory allows.

    line 97
  8. Very specific:

    Consider Q3_K_M, Q4_K_S, or repo-specific IQ / UD-* variants for tighter RAM or VRAM budgets.

    line 98
  9. Very specific:

    Treat mmproj-*.gguf files as projector weights, not the main checkpoint.

    line 99
  10. Common:

    Read hub-discovery.md for URL-first workflows, model search, tree API extraction, and command reconstruction.

    line 103
  11. Very specific:

    Read quantization.md for format tables, model scaling, quality tradeoffs, and imatrix.

    line 104
  12. Common:

    Read hardware.md for Metal, CUDA, ROCm, or CPU build and acceleration details.

    line 105

“Common” is measured: kept by at least 6 in 10 of the 3 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Hugging Face Local Models

Search the Hugging Face Hub for llama.cpp-compatible GGUF repos, choose the right quant, and launch the model with llama-cli or llama-server.

Default Workflow

  1. Search the Hub with apps=llama.cpp.
  2. Open https://huggingface.co/<repo>?local-app=llama.cpp.
  3. Prefer the exact HF local-app snippet and quant recommendation when it is visible.
  4. Confirm exact .gguf filenames with https://huggingface.co/api/models/<repo>/tree/main?recursive=true.
  5. Launch with llama-cli -hf <repo>:<QUANT> or llama-server -hf <repo>:<QUANT>.
  6. Fall back to --hf-repo plus --hf-file when the repo uses custom file naming.
  7. Convert from Transformers weights only if the repo does not already expose GGUF files.

Quick Start

Install llama.cpp

brew install llama.cpp
winget install llama.cpp
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
make

Authenticate for gated repos

hf auth login

Search the Hub

https://huggingface.co/models?apps=llama.cpp&sort=trending
https://huggingface.co/models?search=Qwen3.6&apps=llama.cpp&sort=trending
https://huggingface.co/models?search=<term>&apps=llama.cpp&num_parameters=min:0,max:24B&sort=trending

Run directly from the Hub

llama-cli -hf unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q4_K_M
llama-server -hf unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q4_K_M

Run an exact GGUF file

llama-server \
    --hf-repo unsloth/Qwen3.6-35B-A3B-GGUF \
    --hf-file Qwen3.6-35B-A3B-UD-Q4_K_M.gguf \
    -c 4096

Convert only when no GGUF is available

hf download <repo-without-gguf> --local-dir ./model-src
python convert_hf_to_gguf.py ./model-src \
    --outfile model-f16.gguf \
    --outtype f16
llama-quantize model-f16.gguf model-q4_k_m.gguf Q4_K_M

Smoke test a local server

llama-server -hf unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q4_K_M
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer no-key" \
  -d '{
    "messages": [
      {"role": "user", "content": "Write a limerick about exception handling"}
    ]
  }'

Quant Choice

  • Prefer the exact quant that HF marks as compatible on the ?local-app=llama.cpp page.
  • Keep repo-native labels such as UD-Q4_K_M instead of normalizing them.
  • Default to Q4_K_M unless the repo page or hardware profile suggests otherwise.
  • Prefer Q5_K_M or Q6_K for code or technical workloads when memory allows.
  • Consider Q3_K_M, Q4_K_S, or repo-specific IQ / UD-* variants for tighter RAM or VRAM budgets.
  • Treat mmproj-*.gguf files as projector weights, not the main checkpoint.

Load References

  • Read hub-discovery.md for URL-first workflows, model search, tree API extraction, and command reconstruction.
  • Read quantization.md for format tables, model scaling, quality tradeoffs, and imatrix.
  • Read hardware.md for Metal, CUDA, ROCm, or CPU build and acceleration details.

Resources

  • llama.cpp: https://github.com/ggml-org/llama.cpp
  • Hugging Face GGUF + llama.cpp docs: https://huggingface.co/docs/hub/gguf-llamacpp
  • Hugging Face Local Apps docs: https://huggingface.co/docs/hub/main/local-apps
  • Hugging Face Local Agents docs: https://huggingface.co/docs/hub/agents-local
  • GGUF converter Space: https://huggingface.co/spaces/ggml-org/gguf-my-repo

Download now (ZIP)

Trending

Steady: 0 new projects took it in the last 4 weeks, none the 4 weeks before.thin data

Week of Aug 30: 0 new projectsWeek of Sep 6: 0 new projectsWeek of Sep 13: 0 new projectsWeek of Sep 20: 0 new projectsWeek of Sep 27: 0 new projectsWeek of Oct 4: 0 new projectsWeek of Oct 11: 0 new projectsWeek of Oct 18: 0 new projects
Aug 2026Sep 2026Oct 2026
New projects taking a version of this file each week, over the 8 weeks our latest run read: only 0 in all, so this is a hint, not a trend.
  • Other versions5 projects use a version of this.

    Word for word · 2

    1. huggingface/skills★ 11k100% same
    2. ailessonio/open-source-skills★ 0100% same

    Edited · 3

    1. waybarrios/opencode-power-pack★ 54796% same
    2. huggingface/context-course★ 112100% same
    3. PracticalSwan/agent-skills★ 2262% same

    Wrote their own version · 0

    None found.

    How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.

  • Used alongsideProjects with it also use huggingface-paper-publisher and transformers-js.

    Other files people use alongside this one.

    Files found in the same public projects as a version of this one, more often than they turn up anywhere else. Each project counts once, large collections count for less, and this file's own repository is left out.
    • Hugging FaceSkill

      huggingface-paper-publisher

      Publish and manage research papers on Hugging Face Hub. Supports creating paper pages, linking papers to models/datasets, claiming authorship, and generating professional markdown-based research articles.

      Also in 3 projects that use this

    • Hugging FaceSkill

      transformers-js

      Run state-of-the-art machine learning models directly in JavaScript. `@huggingface/transformers` supports text, vision, audio, and multimodal tasks in browsers and Node.js / Bun / Deno, with WebGPU or WASM execution.

      Also in 3 projects that use this

  • In this repository12 other files it gives its agents.

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

Check your setup, free