Improve the loom skills

The job: make loom (and its sub-skills) better at an end-to-end request, "get MODEL working on loom-serve, optimize it, and report vs llama.cpp", on models it has never seen, without making any KPI worse. This skill is evaluation-driven: measure a baseline first, change one thing, re-measure on the same models, and keep the change only if no KPI regressed (Anthropic's skill-authoring and agent eval guidance; sources at the end).

1. KPIs (all three, every time)

Secondary (diagnostic, not goals): manual minutes (hand-work phases), friction notes (each # line a trial agent logs is a skill gap), time per phase, tool calls.

A change ships only if, on every model its checks covered (section 3, step 5): it stays within its budget, wall clock doesn't rise > 10 %, no gate goes PASS -> FAIL, compare max gap doesn't rise > 0.3, and quiet-GPU perf margins don't drop beyond noise (kpi.py compare checks the first three; eval.sh the last). Never "fix" time by dropping a check that carries weight: a gate may only be removed if another gate provably covers the same failure.

2. Reference models (the eval suite)

No model runs by default and there is no full-suite run before a PR: each change gets the checks it needs (section 3, step 5). Each model stresses something different; don't tune to one.

Add a model when a user hits a failure none of these reproduce (a new architecture family, chat format, quant mix). Before trusting a new eval model, confirm a passing run is possible at all. Baselines to beat: reference/results.md (append every trial).

3. The loop

Progress:
- [ ] 0. State the goal (which KPI, which models) and freeze a baseline snapshot
- [ ] 1. Baseline trial(s) on the current skills (or reuse a row of reference/results.md for the same rev)
- [ ] 2. Profile: kpi.py summarize; read phases.log friction notes AND the agent's final report
- [ ] 3. Rank causes by minutes (or by gate failures); pick the biggest general one
- [ ] 4. Change the skills on a branch (one cause per change; prefer scripts over prose)
- [ ] 5. Plan the checks (what the change can affect, the cheapest evidence for each), write the plan down, run the deterministic ones
- [ ] 6. Run the planned agent trials, if any, on a NEW snapshot; kpi.py compare
- [ ] 7. Fold the trial's own discoveries back (bugs it fixed in its kit, tools it wrote)
- [ ] 8. Repeat 3-7 until the goal is met or the top remaining cause is not fixable in the skill
- [ ] 9. Quiet-GPU eval.sh on the packages the checks produced; results.md rows (with the plans); PR

0. Snapshots. Trials read a frozen copy, never the working tree: you will edit files while trials run, and bash reads scripts as it executes them (an edit can corrupt a running gates.sh).

git worktree add ~/lt/skills-vN <branch> --detach      # one per trial wave; never edit it

1-2. Run a trial (section 4). Profile with

scripts/kpi.py summarize TRIAL --label NAME --skill-rev vN --log kpis.jsonl
scripts/kpi.py compare --log kpis.jsonl BASE NEW        # exit 1 on a regression

Then read the trial's report and every # note. Time estimates in notes are unreliable (an agent wrote "45 min" for a 6-minute step): use the timestamps.

3. Rank by minutes, and classify each cause:

Prefer, in order: deleting a step, scripting it, caching it, documenting it. Put deterministic work in scripts (low freedom); leave judgment (kernel design, debugging) as guidance (high freedom).

4. Change. One cause per commit, message with the measured reason. Keep SKILL.md files concise: the agent already knows how to code; tell it what it can't know (this hardware, this SDK, the gates, past failures). A new reference example goes under loom-port-llm/reference/<family>-example/: its new files, a starter.patch for the starter files it changes (diff -u against the current starter), the reviewed cfg.json (or, once the port is published as a category A loom-fun package, an examples.json entry pinning that package instead: its llm_config.json is the reviewed config) and a README stating what it covers and what was left open. Don't ship what a category A package already carries. Move features that any model could use into the starter instead (default off, so the Qwen programs stay byte-identical). After any starter change, reference/rebase_examples.sh OLD_STARTER and check_examples.sh.

5. Plan the checks. An agent trial costs $2-6 in API tokens and 20-85 min; a deterministic check costs GPU time only. Choose by judgment, not a fixed suite. Before running anything, write down what the change can affect, which models exercise that path, the cheapest evidence for each, and what result would count as a regression; it goes in the results.md row. Reason from what each model's trial actually uses: an example's files only by its model; the starter path only by models without an example (Qwen3.5 today); a doc only by trials that read it (past trial transcripts show which: only flint read porting-notes.md and loom-ir-cookbook.md). The maker of a change wants it to pass: plan before seeing results, and don't shrink the plan afterwards. Cheapest evidence first:

• Syntax: bash -n, python3 -m py_compile.
• Benchmark, gate or perf scripts: replay the changed script (or LEVEL=dev gates.sh, or eval.sh) on finished trial packages of every model the change can reach; qualify.sh + finalize.sh only when the change is in those wrappers. No agent needed (the one-model-at-a-time change was validated this way on all five).
• Kernels, generators, the starter: kernel tests for every new or changed kernel, chat_probe --verify when the chat glue changes, then LEVEL=dev gates.sh on each affected model's rebuilt package after each kept change. Not quick/full: they run nightly on the published cards (nightly-gates.yml). In the issue #26–#29 sessions, 24+ quick runs (6.6 GPU-hours) caught 0 bugs; every real bug was caught by consistency (3), agent (3–4) or compare at first bring-up (2), which dev keeps (loom-benchmark, "Which level when").
• Starter or a reference example: rebuild every affected model's package (apply_example.sh + port.sh, or port.sh alone on the starter path), diff it against the last trial's package, and run that example's own checks (chat_probe --verify, kernel tests, gates). An identical package is not enough: glimmer's chat_probe --verify CLI broke while its package stayed byte-identical, and an example's stale build.py broke the chat placeholders at run time. Chat glue: both Qwen chat blocks still derive 12/12 and pass agent_golden.py. A fallback you add must be tested on a model that takes it (a split-K prefill fallback once produced garbage).
• Where checks run. The dev machine's GPU is scarce and sequential; AMD DevLab Dispatch is a large elastic pool. The dev machine runs only what the inner loop needs (kernel tests, chat_probe --verify, LEVEL=dev on the package you just changed). Long or deep checks, and anything that runs against the published cards (quick/full, gate-script or workflow changes, timing a gate level across models), run in DevLab CI: label the PR nightly-gates (runs dev and full on every card against the PR branch) or dispatch nightly-gates.yml.
• An agent trial only where the change can alter what the agent does: instructions, workflow, hints, the interface or output of a script the agent reads. Use the cheapest model that takes the changed path; add another only for a different path (starter vs example) or a risk specific to a model (memory and context changes: the largest-context model; the auto memory budget ran the machine out of RAM only on flint). A doc tweak usually needs no trial.

Misses surface on their own: every trial runs the whole current skill tree, so a regression on a model you didn't trial shows at its next trial against its results.md baseline. Trace it to the change and add what the plan missed to this step.

6. Run the agent trials. Same model first (did the change work?); a different family only if the plan says the change should generalize there. A re-test that benefits from a reference example of the same model measures the template, not generalization: say so.

7. Fold back. Each trial's kit diff is free engineering: generator fixes, missing helpers, a chat format. Diff TRIAL/work/kit (or skills-local) against the snapshot; merge what is general and verified (gates passed on it), with the trial's measurement in the commit message.

8. Stop when the goal's models finish within their roofline budgets with all gates passing, or the top remaining cause is outside the skill (hardware speed, model size, upstream SDK/llama.cpp bugs): report it with the minutes.

4. Running trials with subagents

Brief. Fill trial_prompt.md exactly; it states the objective, the output format (phases, numbers, gates, friction list, deliverables), the tools and sources allowed, and the boundaries. A vague brief produced duplicated work and contaminated results.

• Fresh general-purpose agent, skills from a frozen snapshot, its own install root, work dir, ports and state dirs. Isolated trials: no shared caches beyond the HF and pip caches. TRIAL dirs live on disk (e.g. ~/lt/trials/NAME), never under /tmp (tmpfs), and launch inside a memory-capped systemd-run --user --scope.
• Contamination: forbid reading the skills repo, other trials and old work dirs, and require the same of any subagent it spawns (a trial's chat subagent once read the working branch).
• Instrumentation: phases.log with epoch timestamps per phase and # friction notes.

Supervise, don't wait. Start a watchdog for every trial, in the background, right after launch:

scripts/watch_trial.sh TRIAL 25 300    # exits DONE, STALLED (nothing changed for 25 min) or TIMEOUT

A completion notification is not a liveness signal: an agent blocked on its own background loop never completes (one sat idle 16 h). On STALLED: look at the last phase and processes; message the agent or stop it, record the result so far, move on. Check every running trial at least every 30 min.

Concurrency. One GPU: at most two trials at once, and only if their models fit together (each gate run holds llama.cpp + Loom: 2x weights per trial; GPU queues run out at ~6 processes). Contention inflates every trial's wall clock and makes in-trial perf meaningless: compare wall clock only between runs with similar contention, and take perf from eval.sh. Stop finished trials' servers promptly (they hold GPU queues and memory).

Reading results. Grade outcomes, not paths: the gates and the wall clock, not whether the agent followed your expected steps. Read the transcript summary for failures; a failure must be clearly the skill's or the model port's fault, otherwise fix the eval (e.g. a sampling check that fails because the prompt is near-deterministic for that model, or llama.cpp disagreeing with itself).

5. Measuring correctly

• In-trial perf is informational. Comparable perf comes from scripts/eval.sh OUT PKG GGUF on a quiet GPU (no other loom-serve/llama-server), same llama.cpp build for every revision, one model at a time, medians. Re-serve a finished package from its trial dir to evaluate it later.
• Quality is re-checked by eval.sh with a frozen skill tree (EVAL_SKILLS=), so a skill change can't relax the gates it is graded by.
• The llama.cpp reference must be the fastest available build (latest lemonade release vs installed); a weak baseline inflates every margin.
• Two llama.cpp runs can disagree at near-ties; the gates handle it (on-demand re-record). Before blaming Loom for a divergence, check llama.cpp against itself.

6. Pitfalls that cost real time (don't repeat them)

• Big files in /tmp (tmpfs: RAM) and model jobs without a memory-capped scope (loom hard rules): leftover kernel-test dumps plus a 27B tune got systemd-oomd to kill the editor and every session in it. Trial work dirs go on disk; clean dumps and ablation packages as you go.
• Editing a script while a trial executes it (use snapshots). pkill -f/pgrep -f matching your own shell (kill by PID file). Waiting on notifications instead of watching liveness.
• Running perf or A/B while another measurement runs (qualify stages 4/6, finalize stage 1).
• Measuring a change on a contended GPU and keeping or reverting it on that number.
• Adding a fallback or generalization without a model that exercises it.
• Treating a template or reference win on the same model as generalization.
• Writing docs for imagined problems: every addition should trace to a trial note or measurement.
• Declaring a root cause before the gate confirms it (flint: "llama.cpp nondeterminism" was written up, then re-recording showed llama.cpp stable on those ids; the real cause was a sink mismatch between two attention kernels). Prove it with the failing gate, then write it down.
• Loosening a gate rule to make a model pass. Bisect first (config variants that switch one feature off, comparing Loom with Loom), then fix the kernel or document the evidence.

Files

• trial_prompt.md: the brief for a trial agent (fill the placeholders).
• scripts/kpi.py: trial KPIs (summarize) and regression check between two trials (compare).
• scripts/watch_trial.sh: trial watchdog (DONE / STALLED / TIMEOUT).
• scripts/eval.sh: fixed quiet-GPU evaluation of a package (frozen gates + perf, one model at a time); LEVEL=dev re-checks in minutes when the trial's own finalize already ran full.
• reference/results.md: every trial so far: the baselines to beat, and the lessons behind each rule.

Sources

Anthropic: Skill authoring best practices, Equipping agents with Agent Skills, Demystifying evals for AI agents, How we built our multi-agent research system.