Populate a Loom HF org

The job: make an org such as huggingface.co/loom-fun host one correct, fully assessed Loom package for each of the top models that fit the device, every card generated by the current skill. This skill is the campaign around the per-model skills. It orchestrates; the per-model work is loom-port-llm, loom-optimize and loom-benchmark, and skill fixes follow loom-improve's rules. Read the loom umbrella's hard rules first.

Roles

• Orchestrator (the agent running this skill): plans, keeps the worklist, dispatches subagents, reviews their PRs and merges the good ones. It does no porting itself, so its context holds conclusions, not kernel logs.
• Workers (subagents, one model or one fix each): port, optimize, publish, fix. Give each a brief that names the model, its Q4_K_M source, its reference (step 1), the gate set, the GPU rules and the deliverable (a published repo, or a PR). A worker's report is not visible to the user; the orchestrator relays what matters.

Step 0: pick the models and see what the org has

K=${CLAUDE_SKILL_DIR}/scripts
python3 $K/rank_models.py --top 50 --max-gb 80 -o ranked.json      # draft target list
python3 $K/org_status.py --org loom-fun --targets ranked.json -o status.json

rank_models.py ranks the Hub's GGUF repos (by downloads, plus the trending list) per model, by downloads per day since the model's first GGUF (at most 30 days), so a model released last week isn't undercounted against one that has been out for months. It keeps chat LLMs, groups every quantizer's repos (unsloth, bartowski, ggml-org, ...) and variants (-GGUF, -BF16, -qat) of one model into one entry, drops user-account finetunes, and keeps models whose plain Q4_K_M is at most --max-gb (80 GB leaves about 45 GB of Strix Halo's 128 GB for KV cache and the OS). It sends the HF token from HF_TOKEN or ~/.cache/huggingface/token; the anonymous API is rate-limited.

Review the draft by hand before acting on it. Remove what isn't a chat model the org should host: video, music or encoder repos the Hub mislabels (e.g. DeepBeepMeep/MiniMax-H3), finetunes whose card hides their base (e.g. schneewolflabs/B1-9B), and derived_from entries that only re-tune a listed model (e.g. ThinkingCap on Qwen3.8-27B). Keep a lab's own product built on another lab's base when it adds something (NVIDIA's Bolt-E4B on Gemma 4: drafter, vision). One plain Q4_K_M per model; never Q8_0 or UD quants unless the user asks. Show the list and the status counts to the user before a campaign, as a page if they'll share it.

org_status.py reads each org repo's own files and classifies it:

A B card says nothing about the model's age: LFM3-Audio-2B was weeks old when its card was B. Only A repos are references. For each target it assigns one action, and lists repos that duplicate a better one or match no target:

"Applicable" is the card generator's own rule (card_lib.gate_applies / metric_applies): vision only with a projector, spec only with drafting, a long-document metric unless the Loom server refused that document and its prompt is longer than the model's native context (card_lib.over_context; a measured value always counts), and so on. Never delete or retire a repo yourself: list duplicates and non-targets for the user to decide. --targets takes rank_models.py output ({"models": [...]}) or a plain list (entries or model ids; rank by order when absent). Each repo's generator is the loom-skills commit that generated its programs: llm_config.json "generator" (build.py's stamp), else the card's recovered one (generator_source says which).

Order: references first

Group the targets by GGUF architecture and family. In each group, port the model that will serve as the reference first (usually the smallest, so its loop is fastest), publish it as category A, then derive the others from it: Qwen3.5-4B, then 9B, 2B and 0.8B; Qwen3.6-35B-A3B, then Ornith-1.5-35B-A3B and 122B (same qwen35moe shape). Adjacent families reuse building blocks (hybrid attention, MoE routing, vision tower, chat format, drafter), so after every publish, re-run the reference lookup (find_reference.py rank) for the queue. Within that order, go by action (fix, remeasure, investigate, then update and create by rank, then sweep). New architectures with no reference (org_status and the lookup both say so) are the most expensive; start them early so they run in parallel with derived ports.

The loop

Repeat until org_status.py says done for every target.

Every card records local, DevLab and roofline performance. DevLab Dispatch runs the gates (step 3), scripted ports (step 1) and a DevLab perf session; a local session (both engines, a quiet local GPU, the repo's published programs) gives the headline numbers. A local/DevLab gap is noted on the card and never debugged ("Local and DevLab performance", below). The two are decoupled on purpose: the local GPU never sits idle waiting for DevLab. When a model is handed to DevLab (step 3), the local GPU moves on to the next item in the queue (a port, a fix, a re-measure, an investigation). DevLab results come back as gate and DevLab perf evidence on the card and as an action in org_status.py; they are picked up on a later pass of the loop, not awaited.

1. Port and optimize (worker subagent, local GPU)

One worker per model runs loom-port-llm then loom-optimize, starting from the best category A reference (never a B repo). A reference saves time; it never lowers the bar.

• The reference lookup is loom-port-llm/find_reference.py, which port.sh runs as stage 1b by default (REFERENCE=off skips it, REFERENCE=loom-fun/REPO picks one). It queries the org live, so a package published a minute ago is a reference right away; it records llm_config.json "reference" (repo, commit, reasons, what was taken) and seeds draft/tune.py with the reference's measured drafting table. An A repo built from a quant other than plain Q4_K_M is still used as a reference, flagged in the ranking (the repo itself is update).
• Scripted DevLab ports (no agent, no local GPU): .github/workflows/devlab-port.yml runs on the devlab-port label, one matrix job per entry of .github/devlab/port.json, each .github/scripts/port_e2e.sh: port -> qualify -> finalize with NEEDLE_FULL=0 -> card evidence -> publish/ (the package without weights, in the artifact). qualify still runs LEVEL=quick, its half-context needle included. Its gates are evidence; its performance session is the card's DevLab session: see the next bullet.
• Campaign waves (practice): put each wave on its own branch from main with only .github/devlab/port.json changed and a draft PR based on main (GitHub creates no pull_request runs while a PR has merge conflicts). Start the next wave by removing and re-adding the label through the REST API; gh pr edit label edits fail silently in this repo: gh api -X DELETE repos/loom-fun/loom-skills/issues/N/labels/devlab-port, then gh api -X POST repos/loom-fun/loom-skills/issues/N/labels -f 'labels[]=devlab-port'.
• A DevLab-ported model: publish it from the job's evidence (make_info.py ... --source-kind devlab, as port_e2e.sh writes it), also when the job's result is failed gates (step 2). The gates count, and the DevLab performance session stays on the card (its DevLab columns); org_status marks the card remeasure (devlab-primary) until a short local session (Loom and llama.cpp, quiet GPU, the same programs) adds the headline numbers next to it.
• Local GPU: one job at a time. One model loaded, one gate run, benchmark or kernel-test batch at any moment. Several workers may run their CPU phases (inspection, program generation, card rendering) in parallel, but every GPU command takes the lock: flock -o /tmp/loom-gpu.lock CMD (-o: a server CMD leaves running, e.g. port.sh's, must not inherit the lock, or the next locked command waits on it forever). Check pgrep -a 'loom-serve|llama-server' first and stop only the servers you started. Follow the user's GPU instruction for the session (some sessions forbid the local GPU; then the worker runs .github/workflows/devlab-port.yml instead).
• Gates in the loop: the correctness set only. Kernel tests, golden compare vs llama.cpp (divergences are near-ties), consistency, spec (drafting packages), agent tool loop, sampling, vision (with a projector), and the over-capacity 400: LEVEL=dev where gates.sh has it, otherwise LEVEL=quick without the needle. Needle-in-a-haystack (half and full context) and other very expensive gates belong to step 3, not the critical path.
• Performance for the card's headline: one local quiet-GPU session, Loom and llama.cpp on the same GGUF and the programs being published, nothing else running. The DevLab session comes from step 3 and never replaces it.
• Speculative decoding stays automatic (opt-out): the optimizer picks the method and settings.

2. Publish the initial result

The worker publishes the package and its generated card (loom-benchmark "Hugging Face model card"; never hand-written): card/publish_package.py prepare PKG DIR --source REPO (the GGUF repo), then make_info.py build and render_card.py DIR, then publish_package.py upload DIR --repo loom-fun/NAME (--card-only for a card update: README.md, info.json and REPORT.md only; --parent-commit SHA with the repo commit whose info.json the card was built from, so a concurrent publish fails instead of being overwritten). Gates not yet run render as "no evidence"; that's expected and honest. The repo is now category A and org_status shows it as sweep until step 3 fills it.

Publish every port, done or not, as soon as it has a package and gate evidence: failing gates, a failing audit or perf flags don't hold it back. The card is the record of progress: a failing gate renders as fail, never dropped or loosened, and org_status marks the repo fix (step 4). Add the card to .github/nightly-models.json in the same PR, and republish as gates go green. Only done (below) needs every gate passing.

Pausing a model: list what would be done next, one step per line in a file, and republish the card only (DIR as in step 3.2): make_info.py build DIR --from-info DIR/info.json --status paused --next-steps FILE, render_card.py DIR, publish_package.py upload DIR --repo loom-fun/REPO --card-only --parent-commit $sha. The README then ends with a "Next steps" section. org_status keeps the repo's action and reports paused: true. To resume, rebuild with --status active --next-steps /dev/null.

3. Sweep on DevLab Dispatch: missing gates, the roofline and the DevLab perf session

DevLab is an elastic pool of Strix Halo runners; sweep many cards' gates at once there, never on the local GPU. Hand the card over and move the local GPU on to the next item.

1. The card is in .github/nightly-models.json (card, memory, kit, llama.cpp build; step 2). Run nightly-gates.yml (it runs on schedule, or gh workflow run nightly-gates.yml -f models=loom-fun/REPO,... -f level=full, or label a PR nightly-gates).
2. When it finishes, per card, in a directory holding the repo's current model.json, model.json.lock, llm_config.json and info.json (re-read info.json from the Hub right before, at a known commit: sha=$(curl -s https://huggingface.co/api/models/loom-fun/REPO/revision/main | python3 -c 'import json,sys; print(json.load(sys.stdin)["sha"])'), then curl -L -o DIR/info.json https://huggingface.co/loom-fun/REPO/resolve/$sha/info.json):
   • card/pull_devlab.py loom-fun/REPO --package DIR -o ev_devlab.json: the gates, with the run, job URL and runner (skips jobs that measured an older revision of the package), and the spec gate's acceptance stamped with the job's programs: acceptance depends on the model, drafter and text, not the machine, so a sweep on the published programs replaces a stale spec_acceptance (speed stays local-only);
   • card/make_info.py roofline --package DIR -o ev_roofline.json (tools venv, CPU only: reads the GGUF), once per card or after a session changed its acceptance (a first card, with no info.json yet: add --evidence ev_perf.json ev_gates.json). A drafted row's ceiling counts the acceptance measured in the row's own session (perf.py, c4_steady.py), else the spec gate's, and the card says which;
   • card/make_info.py build DIR --from-info DIR/info.json --evidence ev_devlab.json ev_roofline.json ... (the repo comes from info.json, the generator commit from the checkout), render_card.py DIR, and publish_package.py upload DIR --repo loom-fun/REPO --card-only --parent-commit $sha (fails if another worker published since: re-read and rebuild).
3. A card missing devlab-perf (local values with no DevLab value on the published programs): one devlab-perf job per card (.github/devlab/perf.json: card, memory_bytes, kit and llamacpp from nightly-models.json, llama_spec from the card's llama.cpp arguments, its drafter as {drafter} or, when llama.cpp used another file, {hf:OWNER/REPO/FILE}; variant {"name": "card"}, steps perf:card:2 perf:card:2:long llama:2 llama:2:long c4:card llamac4; several cards go in one experiment, one job each with its own card fields), then make_info.py perf --loom-jsonl perf_card.jsonl --llama-jsonl perf_llamacpp_nN.jsonl --c4-loom c4_card.txt --c4-llama c4_llamacpp_nN.txt --package DIR --source-kind devlab --run ID --url JOB_URL --device none --hrx-loom none on the job's artifact (no --baseline-version: the card keeps its local baseline), and rebuild the card as above. Low priority: it never blocks other work.
4. Missing local metrics are not swept: a local GPU worker measures them in one quiet local session with both engines (step 5).

4. Fix what the sweep found

A failing gate is a skill bug until proven otherwise (a model-specific workaround is the last resort). Dispatch a worker that reproduces the failure, finds the root cause, fixes it in loom-skills under loom-improve's rules (general fix, no KPI regression, tests), and opens a PR. The orchestrator reviews it against the failure. Then every published model the fix touches is regenerated with the skill by a worker (rebuild programs, re-run the affected gates, regenerate the card); hand-patching a published package is never allowed. Merge the PR once the regenerated models pass. Re-run org_status.py: anything still red goes round again.

5. Re-measure and investigate performance (local GPU worker)

First, re-measure (action remeasure; org_status lists the reasons): a card value whose session measured programs other than the repo's (stale: card_lib.stale_metrics, the session's programs_sha256 vs the published program/*.loom; the card marks them †) describes another build, and a metric only DevLab measured (devlab-primary: card_lib.devlab_metrics) has no headline value. A local GPU worker runs a fresh local session (Loom and llama.cpp, quiet GPU, the repo's programs: make_info.py perf, --source-kind local) and republishes the card, keeping its DevLab session; only then is the far flag below worth acting on.

org_status.py flags a category A card (action investigate, details under perf.far) when a metric is far from its roofline: its Loom value is below half the efficiency Loom ports have demonstrated for that kind of metric on this hardware (card_lib.FAR_FRACTION x the card's roofline.metrics.*.efficiency, which is roofline.py's EFF, or EFF_DRAFTED for drafted rows). Then:

1. Dispatch a local GPU worker with loom-optimize on the flagged metrics: a far-from-roofline metric is a performance problem to find and fix (ablate, compare with the roofline's gap budget, apply the lever). For a small drafted model far from its decode roofline, the lever is the verify pass and the drafter's cost (the ceiling does not count the drafter); the best depth depends on the workload's acceptance, so judge it on the session's own acceptance, not the spec gate's.
2. The worker republishes the card with a fresh local session (step 2). Move the local GPU to the next item meanwhile.
3. Deep debugging session when a metric is still far from its roofline after an optimization round (the card comes back investigate with the same metric after a loom-optimize worker's republish): the worker stops tuning and finds out why the gap stays, rather than accepting it:
   • a per-kernel profile of the flagged metric against the roofline's gap budget (which kernels, launches or host-side waits take the time the ceiling does not account for);
   • llama.cpp's efficiency on the same metric and machine (loom-roofline): a ceiling both engines are far from points at the model or the roofline, one only Loom is far from points at Loom;
   • the local machine's settings (power profile, CPU governor and EPP, clocks, kernel and firmware, GPU memory split, other load). The far flag is judged on the local value only; the DevLab value is not part of this investigation. Record the cause and the fix (or a skill fix under step 4) in the PR; a gap that turns out to be the machine, not Loom, goes into the card's evidence through a re-measured local session, never by hand.

6. Repeat

Re-rank occasionally (new releases arrive weekly). Stop when every target is done, then hand the user the duplicate and non-target lists to keep or retire.

Far from roofline: how the threshold was set

• FAR_FRACTION 0.5: the category A cards that are healthy (Qwen3.8-27B, Muse-Glimmer-30B, Bolt-E4B) sit at 0.51 or more of the demonstrated efficiency on every metric (the lowest is Bolt-E4B's 32k prefill, 0.17 of the roofline against 0.33 demonstrated); Flint's 1k-16k prefill sits at 0.24-0.46 of it and is flagged.
• Recalibrate it when a port moves EFF / EFF_DRAFTED (loom-roofline). EFF_DRAFTED was set against ceilings that counted the spec gate's acceptance; once the healthy cards' sessions record their own acceptance (perf.py), recompute it from those ceilings.

Local and DevLab performance (the user's rule)

Loom on DevLab often measures well below a local session on the same programs while llama.cpp does not (e.g. prefill -25 to -52 % on several cards, matched run 42300147814). The rule: if local and DevLab differ significantly, note it and move on. Every card records all three, each labeled by source: the local session (the headline Loom vs llama.cpp numbers, judged against the roofline), the DevLab session (its DevLab columns, with the local ÷ DevLab ratio of each engine) and the roofline. A DevLab session never replaces a local one, nor the other way round (card_lib.card_metrics). The gap is data: it is never a flag, never blocks done, and is never a reason to investigate. Diagnosing it is out of scope for port, optimize and campaign work; a separate effort will use the recorded data. The Loom-only DevLab sweeps older cards recorded stay in their info.json (valid loom-card/2), unrendered.

Done

A model is done when its repo is category A, built from the target's plain Q4_K_M, every applicable gate passes (including needle and full-context, from the sweep), every applicable metric is measured in one local session (Loom and llama.cpp, the published programs) and in a DevLab session on the same programs, the card carries its roofline, no metric's local value is far from it (whatever the local/DevLab gap), and the card was regenerated after the last skill fix that touched it. The org is done when every target is done and the user has decided on the rest.

Scripts