benchmark
Skill3 starsChanged 32 days agoMITView on GitHub
Run the local-model evaluation sweep against a disposable Harbor Clerk instance, within the USD 35 cloud cap, and publish the dated report as the machine user. Use only when the owner asks for a benchmark run - it is stage 3 of model-refresh, the re-baseline after a llama.cpp bump, or the evidence for promoting or retiring a model.
- Reads credentials
98 other projects use a version of this.
Adopted by
Most recently taken up by
- TsukiNi22/skillstheir own version, first seen Nov 16, 2026
- TsukiNi22/ai-utilstheir own version, first seen Nov 16, 2026
- AndresArreguy/ecc-pi-porttheir own version, first seen Nov 16, 2026
- akshaypimprikar/pragmatheir own version, first seen Nov 11, 2026
- Rojios/everything-claude-codetheir own version, first seen Nov 8, 2026
- accso/ai-context-mcptheir own version, first seen Nov 8, 2026
Used with
Projects with it also use ios-clean, ios-fix and 10 more.
Other files people use alongside this one.
Files found in the same public projects as a version of this one, more often than they turn up anywhere else. Each project counts once, large collections count for less, and this file's own repository is left out.
Saqlain-811Skillios-clean
Remove the DebugBridge SPM package and all #if DEBUG wiring from an iOS app. (gstack)
Also in 5 projects that use this
Saqlain-811Skillios-fix
Autonomous iOS bug fixer. (gstack)
Also in 5 projects that use this
seek888Skillpair-agent
Pair a remote AI agent with your browser. One command generates a setup key and prints instructions the other agent can follow to connect. Works with OpenClaw, Hermes, Codex, Cursor, or any agent that can make HTTP requests. The remote agent gets its own tab with scoped access (read+write by default, admin on request). Use when asked to "pair agent", "connect agent", "share browser", "remote browser", "let another agent use my browser", or "give browser access". (gstack) Voice triggers (speech-to-text aliases): "pair agent", "connect agent", "share my browser", "remote browser access".
Also in 11 projects that use this
ShreyRaviSkilllanding-report
Read-only queue dashboard for workspace-aware ship. Shows which VERSION slots are currently claimed by open PRs, which sibling Conductor workspaces have WIP work likely to ship soon, and what slot /ship would pick next. No mutations — just a snapshot. Use when asked to "landing report", "what's in the queue", "show me open PRs", or "which version do I claim next". (gstack)
Also in 8 projects that use this
timurgaleevSkilldocument-generate
Generate missing docs for a feature, module or project as tutorials, how-tos, reference and explanation.
Also in 7 projects that use this
seek888Skillopen-gstack-browser
Launch GStack Browser — AI-controlled Chromium with the sidebar extension baked in. Opens a visible browser window where you can watch every action in real time. The sidebar shows a live activity feed and chat. Anti-bot stealth built in. Use when asked to "open gstack browser", "launch browser", "connect chrome", "open chrome", "real browser", "launch chrome", "side panel", or "control my browser". Voice triggers (speech-to-text aliases): "show me the browser".
Also in 8 projects that use this
- +1 more from Saqlain-811
Other versions98 projects use a version of this.
Word for word · 0
None found.
Edited · 0
None found.
Wrote their own version · 98
- garrytan/gstack★ 142ksame name
- dragonflydb/dragonfly★ 32ksame name
- xtermjs/xterm.js★ 21ksame name
- udecode/plate★ 17ksame name
- Z3Prover/z3★ 13ksame name
- androidx/androidx★ 6.1ksame name
- samchon/typia★ 6ksame name
- PaddlePaddle/FastDeploy★ 3.7ksame name
- zalando/skipper★ 3.3ksame name
- samchon/nestia★ 2.2ksame name
88 more not listed
How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.
In this repository9 other files it gives its agents.
- AGENTS.mdAGENTS.md
- alembic/AGENTS.mdAGENTS.md
- frontend/AGENTS.mdAGENTS.md
- macos/AGENTS.mdAGENTS.md
- acceptanceSkill
- build-macosSkill
- model-refreshSkill
- verifySkill
- harbor-clerkSkill
The file
We split this skill into the rules it gives an agent
6 rules. 0 are common, 3 are distinctive, and 3 are very specific to this project.
0
Common
Most of the 84 edited versions we read keep these.
3
Distinctive
This author's own choices. Worth a look.
3
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 38
If the app has a model loaded, note which (GET /api/chat/models/status.
- Very specific: line 42
Never launch a llama-server by hand for this.
- Distinctive: line 48
Before each corpus the sweep deletes every watched folder and every document on the instance (uploads and fetched mail included).
- Distinctive: line 70
Never from a prompt, a file, or the conversation.
- Distinctive: line 101
Keep the judge at its default unless the owner asked to change it: scores from different judges are not comparable (#661).
- Very specific: line 146
Run it in the background and watch results/$RUN/log.txt.
“Common” is measured: kept by at least 6 in 10 of the 84 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Benchmark run
Drives scripts/test_corpora (README and RUNBOOK there) and turns the run into
docs/reports/YYYY-MM-DD-benchmark-<host>-<run>.md and a PR. All commands run
from the repository root. A run wipes the instance it targets, spends money,
and takes hours. Start one only when the owner asked for it, and say what it
will cost in time and dollars before it starts.
1. Is the machine fit to measure on?
RUN=bench-$(date +%Y%m%d-%H%M)
WORKDIR="$HOME/Library/Application Support/Harbor Clerk/test-corpora"
MODELS=qwen3-8b,qwen35-9b # what the owner asked to compare
uv --project scripts/test_corpora run python -m scripts.test_corpora.preflight \
--models "$MODELS" --json "$WORKDIR/results/$RUN/preflight.json"It checks thermal pressure, power, free memory, a busy GPU at idle, other model
servers (Ollama), that the installed llama-server is at the pin, that the spaCy
model the metrics load is installed (a fresh venv lacks it; the check names the
install command and runs whatever phases the run will do, #683), that each model
fits this machine, and that the instance is healthy. It changes nothing.
failstops the run. A number taken on a throttled machine, or through a stalellama-server, is not a baseline, and a report that says so is still not evidence for a model decision. Tell the owner what failed; the fixes are theirs (moving the machine,make apps,launchctl bootoutfor a launch agent). The mini sat at thermal pressure "Sleeping" for weeks before this check existed (#652).warntravels with the report. Say it in the Reading. A check that could not look is awarn, never apass.- If the app has a model loaded, note which (
GET /api/chat/models/status; "Afterwards" restores it), then deactivate it (Models page, orPUT /api/chat/models/deactivate): a resident 22 GB model fails the free-memory check, and the sweep activates what it needs. - Never launch a
llama-serverby hand for this. The sweep drives the app's own server through the API, and the app clamps context to what fits (macos/AGENTS.md, "A model server does not run out of memory").
2. The instance will be wiped
Before each corpus the sweep deletes every watched folder and every
document on the instance (uploads and fetched mail included). It refuses
unless HC_EVAL_DISPOSABLE=1, the API base is loopback, every watched folder is
one of the harness's own ingest directories ($WORKDIR/<corpus>/ingest), there
is no connected mailbox, and any documents it holds are accounted for by a
harness folder. Set HC_EVAL_DISPOSABLE=1 only on the mini, and only
after looking:
curl -s http://localhost:8100/api/system/health | jq '{status, build}'Health says it is up, not what it holds. Open http://localhost:8100/folders
and the document list, or ask the owner. The guard cannot see uploads mixed in
with a harness corpus, mail whose account was removed, or a loopback address
that is an SSH forward. If the instance holds anything the owner would miss,
stop and ask. Do not
remove their folders to get past the guard. --no-ingest measures whatever is
loaded and wipes nothing.
3. Credentials come from Keychain
Never from a prompt, a file, or the conversation. If one is missing, stop and
say which; the owner adds it (security add-generic-password -U -s <service> -a
<account> -w, typed by them: a trailing -w prompts, so the key reaches neither
the shell history nor this conversation). To check one exists without reading
it, drop the -w from find-generic-password and look at the exit status.
| what | service | account |
| admin login | harbor-clerk-acceptance | the admin email |
| Anthropic API key (baselines) | harbor-clerk-eval | anthropic-api-key |
OpenAI API key, as OPENAI_API_KEY (the sweep's judge, gpt-5.6-luna, since 2026-09-22; also a gpt-* baseline or the audit's --cross-judge) | harbor-clerk-eval | openai-api-key |
| the machine user's GitHub token | github-token | harborclerk-bot |
4. Plan, and price it
Scope to the question. A full sweep is 17 to 25 hours of model time; one corpus,
the models being compared, phases 0,1,4 is usually the whole question. A large
corpus adds its summaries before the first unit: the sweep waits for them so every
model measures the same corpus (about 20 documents a minute on the mini; #717).
One corpus per run: the sweep prices the pending cloud work before it starts
at each model's own cache rates and refuses a run estimated over the cap (20
keyed CUAD baselines: Sonnet 4.6 about USD 6, Fable 5.1 about 22, gpt-6-astra
about 19, thinking measured on 2026-10-15; the other corpora
cost 40 to 60% more a question). The per-call hard stop is the real guard
(scripts/test_corpora/spend.yaml, README "Cloud spend is capped"), and it
reserves each call's worst case first, about USD 5.6 for a late Fable call. A
run estimated within that of the cap is expected to be cut short by a refused
call, not to finish (exit 3, finished units saved; the rest is a new cap): tell
the owner so before it starts. --spend-cap-usd N lowers the cap for this run.
Raising it is a PR.
Keep the judge at its default unless the owner asked to change it: scores from different judges are not comparable (#661).
5. Run
For CUAD and the synthetic corpus, ask the keyed questions as well (--questions-dir
scripts/test_corpora/questions/keyed, its own --run-id): every answer is also scored against the corpus's own
labels (CUAD's annotations; the synthetic corpus's fact sidecars), with no judge and no baseline, and the report
shows it as "answer key (0-1)". It is the one column that sees recall, and it found two model behaviours no judge
reported (docs/reports/2026-09-22-rubric-test.md). The synthetic key is tied to one corpus generation (the
_generation header of synthetic.key.json): the instance must hold that generation, acquired by a standard run
first, or the keyed run refuses to start and says so; a regenerated corpus means regenerating the key
(corpora/synthetic_key.py --snapshot --redo, a PR). A keyed run and a standard run are two runs; do not compare
their judge columns.
mkdir -p "$WORKDIR/results/$RUN"
ADMIN="$(security find-generic-password -s harbor-clerk-acceptance | sed -n 's/.*"acct"<blob>="\(.*\)"/\1/p')"
HC_EVAL_DISPOSABLE=1 HC_API_BASE=http://localhost:8100 HC_USERNAME="$ADMIN" \
HC_PASSWORD="$(security find-generic-password -s harbor-clerk-acceptance -w)" \
ANTHROPIC_API_KEY="$(security find-generic-password -s harbor-clerk-eval -a anthropic-api-key -w)" \
OPENAI_API_KEY="$(security find-generic-password -s harbor-clerk-eval -a openai-api-key -w)" \
uv --project scripts/test_corpora run python -m scripts.test_corpora.runner.sweep \
--run-id "$RUN" --workdir "$WORKDIR" --phases 0,1,4 --corpora cuad --models "$MODELS" \
> "$WORKDIR/results/$RUN/console.log" 2>&1; echo "sweep exit: $?"A cloud model as the subject. To measure a frontier model through Harbor Clerk's MCP tools against the answer key, run phases 0 and 1 only, naming it:
--phases 0,1 --baseline-model claude-fable-5-1 --corpora cuad \
--questions-dir scripts/test_corpora/questions/keyedOne model per run and one corpus per run (the cap; a resume with another model
is refused), its own --run-id, the model priced in spend.yaml. A gpt-*
model needs the OpenAI key instead of the Anthropic one; GPT-6 goes through
OpenAI's Responses API. The report then has a "Cloud subject" table: the key
mean and per-kind means over its baselines, calls, answers stopped at the call
limit, and USD. Its answers are also the reference for any phase 4 that follows
in the same run, so compare like with like: a local model scored against Fable
is not comparable with one scored against Sonnet 4.6.
Run it in the background and watch results/$RUN/log.txt; do not poll faster
than the work moves. Exit codes: 0 complete; 3 stopped by the spend cap
or a refused spend setting (finished units are saved; --resume continues the
same run and the same ledger); 4 the instance may not be wiped (a refused
first start leaves nothing behind: fix the cause and run the same command);
5 an uncaught exception (the traceback is in log.txt; the unit it was on
is not marked done, and a unit is never marked done before its metrics.csv
row exists; fix the cause, then --resume). Models the instance has not downloaded, or cannot load at any
context, are skipped with a reason, listed in the report, and reconsidered at every start:
downloading a model is the owner's decision.
A failure is information. Read log.txt before re-running anything, and never
--rerun to make a number look better.
6. Report
The tree must be clean (the report cites the suite commit; -dirty means it
was generated from uncommitted code).
uv --project scripts/test_corpora run python -m scripts.test_corpora.report \
--run-dir "$WORKDIR/results/$RUN" --out docs/reports/It prints the path (one file per run, never overwritten). The header names run,
commit, corpora, judge, spend and the preflight verdict; the tables are what was
measured, per corpus, with what was planned beside what ran. Write the
Reading yourself, in the file: what the numbers support,
what they do not (sample size, one corpus, one judge), every warn, every
skipped model, and what was not verified. No model is promoted or retired on a
report whose preflight failed or is missing.
7. Publish as the machine user
As in the acceptance skill, "Publish as the machine user": a scratch worktree
from a fresh origin/main, branch report/benchmark-$RUN, the one report file,
committed as John-Doebot with commit.gpgsign=false, pushed through the
inline Keychain credential helper, PR opened with GH_TOKEN from Keychain and
--body-file. The body repeats the header line and the Reading's conclusion.
Registry changes that follow from the report are a separate PR (model-refresh,
stage 2).
Afterwards
Say first what the run left behind: the instance holds the last corpus
ingested and the last model activated (activation persists as the app's
default), $WORKDIR holds the corpora and results, and spend.json holds what
was spent. Restore the owner's model if they had one active, and correct
spend.yaml's estimates from spend.json (tokens per unit, not per call)
in a PR once a metered run is on record.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

