agentleFS
r0shir0shi/harborclerk/.claude/skills/benchmark/SKILL.md

benchmark

Skill3 starsChanged 32 days agoMITView on GitHub

Run the local-model evaluation sweep against a disposable Harbor Clerk instance, within the USD 35 cloud cap, and publish the dated report as the machine user. Use only when the owner asks for a benchmark run - it is stage 3 of model-refresh, the re-baseline after a llama.cpp bump, or the evidence for promoting or retiring a model.

  • Reads credentials

98 other projects use a version of this.

Adopted by

Most recently taken up by

Used with

Projects with it also use ios-clean, ios-fix and 10 more.

Other files people use alongside this one.

Files found in the same public projects as a version of this one, more often than they turn up anywhere else. Each project counts once, large collections count for less, and this file's own repository is left out.
  • Saqlain-811Skill

    ios-clean

    Remove the DebugBridge SPM package and all #if DEBUG wiring from an iOS app. (gstack)

    Also in 5 projects that use this

  • Saqlain-811Skill

    ios-fix

    Autonomous iOS bug fixer. (gstack)

    Also in 5 projects that use this

  • seek888Skill

    pair-agent

    Pair a remote AI agent with your browser. One command generates a setup key and prints instructions the other agent can follow to connect. Works with OpenClaw, Hermes, Codex, Cursor, or any agent that can make HTTP requests. The remote agent gets its own tab with scoped access (read+write by default, admin on request). Use when asked to "pair agent", "connect agent", "share browser", "remote browser", "let another agent use my browser", or "give browser access". (gstack) Voice triggers (speech-to-text aliases): "pair agent", "connect agent", "share my browser", "remote browser access".

    Also in 11 projects that use this

  • ShreyRaviSkill

    landing-report

    Read-only queue dashboard for workspace-aware ship. Shows which VERSION slots are currently claimed by open PRs, which sibling Conductor workspaces have WIP work likely to ship soon, and what slot /ship would pick next. No mutations — just a snapshot. Use when asked to "landing report", "what's in the queue", "show me open PRs", or "which version do I claim next". (gstack)

    Also in 8 projects that use this

  • timurgaleevSkill

    document-generate

    Generate missing docs for a feature, module or project as tutorials, how-tos, reference and explanation.

    Also in 7 projects that use this

  • seek888Skill

    open-gstack-browser

    Launch GStack Browser — AI-controlled Chromium with the sidebar extension baked in. Opens a visible browser window where you can watch every action in real time. The sidebar shows a live activity feed and chat. Anti-bot stealth built in. Use when asked to "open gstack browser", "launch browser", "connect chrome", "open chrome", "real browser", "launch chrome", "side panel", or "control my browser". Voice triggers (speech-to-text aliases): "show me the browser".

    Also in 8 projects that use this

  • +1 more from Saqlain-811

The file

We split this skill into the rules it gives an agent

6 rules. 0 are common, 3 are distinctive, and 3 are very specific to this project.

  • 0

    Common

    Most of the 84 edited versions we read keep these.

  • 3

    Distinctive

    This author's own choices. Worth a look.

  • 3

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    If the app has a model loaded, note which (GET /api/chat/models/status.

    line 38
  2. Very specific:

    Never launch a llama-server by hand for this.

    line 42
  3. Distinctive:

    Before each corpus the sweep deletes every watched folder and every document on the instance (uploads and fetched mail included).

    line 48
  4. Distinctive:

    Never from a prompt, a file, or the conversation.

    line 70
  5. Distinctive:

    Keep the judge at its default unless the owner asked to change it: scores from different judges are not comparable (#661).

    line 101
  6. Very specific:

    Run it in the background and watch results/$RUN/log.txt.

    line 146

“Common” is measured: kept by at least 6 in 10 of the 84 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Benchmark run

Drives scripts/test_corpora (README and RUNBOOK there) and turns the run into docs/reports/YYYY-MM-DD-benchmark-<host>-<run>.md and a PR. All commands run from the repository root. A run wipes the instance it targets, spends money, and takes hours. Start one only when the owner asked for it, and say what it will cost in time and dollars before it starts.

1. Is the machine fit to measure on?

RUN=bench-$(date +%Y%m%d-%H%M)
WORKDIR="$HOME/Library/Application Support/Harbor Clerk/test-corpora"
MODELS=qwen3-8b,qwen35-9b          # what the owner asked to compare
uv --project scripts/test_corpora run python -m scripts.test_corpora.preflight \
  --models "$MODELS" --json "$WORKDIR/results/$RUN/preflight.json"

It checks thermal pressure, power, free memory, a busy GPU at idle, other model servers (Ollama), that the installed llama-server is at the pin, that the spaCy model the metrics load is installed (a fresh venv lacks it; the check names the install command and runs whatever phases the run will do, #683), that each model fits this machine, and that the instance is healthy. It changes nothing.

  • fail stops the run. A number taken on a throttled machine, or through a stale llama-server, is not a baseline, and a report that says so is still not evidence for a model decision. Tell the owner what failed; the fixes are theirs (moving the machine, make apps, launchctl bootout for a launch agent). The mini sat at thermal pressure "Sleeping" for weeks before this check existed (#652).
  • warn travels with the report. Say it in the Reading. A check that could not look is a warn, never a pass.
  • If the app has a model loaded, note which (GET /api/chat/models/status; "Afterwards" restores it), then deactivate it (Models page, or PUT /api/chat/models/deactivate): a resident 22 GB model fails the free-memory check, and the sweep activates what it needs.
  • Never launch a llama-server by hand for this. The sweep drives the app's own server through the API, and the app clamps context to what fits (macos/AGENTS.md, "A model server does not run out of memory").

2. The instance will be wiped

Before each corpus the sweep deletes every watched folder and every document on the instance (uploads and fetched mail included). It refuses unless HC_EVAL_DISPOSABLE=1, the API base is loopback, every watched folder is one of the harness's own ingest directories ($WORKDIR/<corpus>/ingest), there is no connected mailbox, and any documents it holds are accounted for by a harness folder. Set HC_EVAL_DISPOSABLE=1 only on the mini, and only after looking:

curl -s http://localhost:8100/api/system/health | jq '{status, build}'

Health says it is up, not what it holds. Open http://localhost:8100/folders and the document list, or ask the owner. The guard cannot see uploads mixed in with a harness corpus, mail whose account was removed, or a loopback address that is an SSH forward. If the instance holds anything the owner would miss, stop and ask. Do not remove their folders to get past the guard. --no-ingest measures whatever is loaded and wipes nothing.

3. Credentials come from Keychain

Never from a prompt, a file, or the conversation. If one is missing, stop and say which; the owner adds it (security add-generic-password -U -s <service> -a <account> -w, typed by them: a trailing -w prompts, so the key reaches neither the shell history nor this conversation). To check one exists without reading it, drop the -w from find-generic-password and look at the exit status.

whatserviceaccount
admin loginharbor-clerk-acceptancethe admin email
Anthropic API key (baselines)harbor-clerk-evalanthropic-api-key
OpenAI API key, as OPENAI_API_KEY (the sweep's judge, gpt-5.6-luna, since 2026-09-22; also a gpt-* baseline or the audit's --cross-judge)harbor-clerk-evalopenai-api-key
the machine user's GitHub tokengithub-tokenharborclerk-bot

4. Plan, and price it

Scope to the question. A full sweep is 17 to 25 hours of model time; one corpus, the models being compared, phases 0,1,4 is usually the whole question. A large corpus adds its summaries before the first unit: the sweep waits for them so every model measures the same corpus (about 20 documents a minute on the mini; #717). One corpus per run: the sweep prices the pending cloud work before it starts at each model's own cache rates and refuses a run estimated over the cap (20 keyed CUAD baselines: Sonnet 4.6 about USD 6, Fable 5.1 about 22, gpt-6-astra about 19, thinking measured on 2026-10-15; the other corpora cost 40 to 60% more a question). The per-call hard stop is the real guard (scripts/test_corpora/spend.yaml, README "Cloud spend is capped"), and it reserves each call's worst case first, about USD 5.6 for a late Fable call. A run estimated within that of the cap is expected to be cut short by a refused call, not to finish (exit 3, finished units saved; the rest is a new cap): tell the owner so before it starts. --spend-cap-usd N lowers the cap for this run. Raising it is a PR.

Keep the judge at its default unless the owner asked to change it: scores from different judges are not comparable (#661).

5. Run

For CUAD and the synthetic corpus, ask the keyed questions as well (--questions-dir scripts/test_corpora/questions/keyed, its own --run-id): every answer is also scored against the corpus's own labels (CUAD's annotations; the synthetic corpus's fact sidecars), with no judge and no baseline, and the report shows it as "answer key (0-1)". It is the one column that sees recall, and it found two model behaviours no judge reported (docs/reports/2026-09-22-rubric-test.md). The synthetic key is tied to one corpus generation (the _generation header of synthetic.key.json): the instance must hold that generation, acquired by a standard run first, or the keyed run refuses to start and says so; a regenerated corpus means regenerating the key (corpora/synthetic_key.py --snapshot --redo, a PR). A keyed run and a standard run are two runs; do not compare their judge columns.

mkdir -p "$WORKDIR/results/$RUN"
ADMIN="$(security find-generic-password -s harbor-clerk-acceptance | sed -n 's/.*"acct"<blob>="\(.*\)"/\1/p')"
HC_EVAL_DISPOSABLE=1 HC_API_BASE=http://localhost:8100 HC_USERNAME="$ADMIN" \
HC_PASSWORD="$(security find-generic-password -s harbor-clerk-acceptance -w)" \
ANTHROPIC_API_KEY="$(security find-generic-password -s harbor-clerk-eval -a anthropic-api-key -w)" \
OPENAI_API_KEY="$(security find-generic-password -s harbor-clerk-eval -a openai-api-key -w)" \
uv --project scripts/test_corpora run python -m scripts.test_corpora.runner.sweep \
  --run-id "$RUN" --workdir "$WORKDIR" --phases 0,1,4 --corpora cuad --models "$MODELS" \
  > "$WORKDIR/results/$RUN/console.log" 2>&1; echo "sweep exit: $?"

A cloud model as the subject. To measure a frontier model through Harbor Clerk's MCP tools against the answer key, run phases 0 and 1 only, naming it:

  --phases 0,1 --baseline-model claude-fable-5-1 --corpora cuad \
  --questions-dir scripts/test_corpora/questions/keyed

One model per run and one corpus per run (the cap; a resume with another model is refused), its own --run-id, the model priced in spend.yaml. A gpt-* model needs the OpenAI key instead of the Anthropic one; GPT-6 goes through OpenAI's Responses API. The report then has a "Cloud subject" table: the key mean and per-kind means over its baselines, calls, answers stopped at the call limit, and USD. Its answers are also the reference for any phase 4 that follows in the same run, so compare like with like: a local model scored against Fable is not comparable with one scored against Sonnet 4.6.

Run it in the background and watch results/$RUN/log.txt; do not poll faster than the work moves. Exit codes: 0 complete; 3 stopped by the spend cap or a refused spend setting (finished units are saved; --resume continues the same run and the same ledger); 4 the instance may not be wiped (a refused first start leaves nothing behind: fix the cause and run the same command); 5 an uncaught exception (the traceback is in log.txt; the unit it was on is not marked done, and a unit is never marked done before its metrics.csv row exists; fix the cause, then --resume). Models the instance has not downloaded, or cannot load at any context, are skipped with a reason, listed in the report, and reconsidered at every start: downloading a model is the owner's decision.

A failure is information. Read log.txt before re-running anything, and never --rerun to make a number look better.

6. Report

The tree must be clean (the report cites the suite commit; -dirty means it was generated from uncommitted code).

uv --project scripts/test_corpora run python -m scripts.test_corpora.report \
  --run-dir "$WORKDIR/results/$RUN" --out docs/reports/

It prints the path (one file per run, never overwritten). The header names run, commit, corpora, judge, spend and the preflight verdict; the tables are what was measured, per corpus, with what was planned beside what ran. Write the Reading yourself, in the file: what the numbers support, what they do not (sample size, one corpus, one judge), every warn, every skipped model, and what was not verified. No model is promoted or retired on a report whose preflight failed or is missing.

7. Publish as the machine user

As in the acceptance skill, "Publish as the machine user": a scratch worktree from a fresh origin/main, branch report/benchmark-$RUN, the one report file, committed as John-Doebot with commit.gpgsign=false, pushed through the inline Keychain credential helper, PR opened with GH_TOKEN from Keychain and --body-file. The body repeats the header line and the Reading's conclusion. Registry changes that follow from the report are a separate PR (model-refresh, stage 2).

Afterwards

Say first what the run left behind: the instance holds the last corpus ingested and the last model activated (activation persists as the app's default), $WORKDIR holds the corpora and results, and spend.json holds what was spent. Restore the owner's model if they had one active, and correct spend.yaml's estimates from spend.json (tokens per unit, not per call) in a PR once a metered run is on record.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.