update-optimal-blog
Skill212 starsChanged 7 months agoApache-2.0View on GitHub
Refresh the data tables in optimal-blog.md from master.db. Checks the data for integrity problems FIRST, then runs the generator that picks per-language winners and splices every GEN-marked table, then reconciles the surrounding prose. Use after new experiment results land, or when the optimal-blog numbers are stale.
In this repository6 other files it gives its agents.
- CLAUDE.mdCLAUDE.md
- compare-runsSkill
- diagnose-failed-runSkill
- evaluate-runSkill
- file-run-issuesSkill
- run-summarySkill
The file
Files
Update optimal-blog.md
Overview
optimal-blog.md records what to run today, per language and task size. Its data
tables are not hand-written — they are generated from master.db by
the retort report optimal subcommand (code in
src/retort/reporting/optimal.py) and live between
<!-- GEN:<key> START/END --> markers. This skill is the safe procedure for refreshing
them: check the data before you trust it, regenerate, verify the round-trip, then fix
any prose whose numbers moved.
The order matters. master.db does not record the full stack/config (see the health gaps below), so a blind regenerate can silently publish wrong numbers. Always run the health check first and stop if it reports anything new.
Why per-language, not aggregates
A single cross-language reliability number is misleading — it blends a stack's strong languages with its weak ones (local Qwen passes Python/Go but fails Rust; Opus 4.8 dips on Java). The generator's centrepiece is the per-language success-rate matrix; the leading-stacks routine aggregate is explicitly labelled the least-useful number. Keep it that way — do not "promote" an aggregate back into the recommendation.
Steps
1. Health-check master.db FIRST (gate)
retort report optimal --healthCompare against the known, accepted gaps (already documented in the blog's Keeping this current section):
- ⚠️ No sampling columns /
max_context_tokensunpopulated — the qualified config is curated inFEATURED_STACKS, not filtered from data. - ⚠️ ~250 rows have a blank
model(local provenance bug) — attributed by experiment slug. - ⚠️
experiment-11,experiment-29not ingested.
You MUST STOP and surface to the user if the report shows anything beyond those:
- "Unmapped model strings" — a new model appeared that no featured/legacy entry
covers. Decide whether it's a new featured stack (add to
FEATURED_STACKS) or legacy (add toKNOWN_NONFEATURED) before regenerating. Publishing without deciding would drop it silently. - New experiment dirs not in master.db — results exist on disk but aren't ingested; the refresh would omit them. Re-ingest first, or note the omission to the user.
- A jump in the blank-model count — the harness may have regressed; a run recording no
model is invisible to the tables. (The fix landed in
src/retort/playpen/runner.pystack_metadata(); if new blanks appear, that path is being bypassed.)
2. Regenerate the tables
retort report optimal --write optimal-blog.mdThis splices every GEN:* block (leading stacks, per-language matrix, per-language
winners, prompt method). It only touches text between markers.
3. Verify the round-trip
retort report optimal --write optimal-blog.md # run twice
git diff --stat optimal-blog.mdA second --write MUST produce no further change — the tables are idempotent. If
git diff shows table cells changed, that's the real new data; if it shows nothing, the
blog was already current.
4. Reconcile the prose
The generator owns the tables, not the sentences around them. After a refresh, read the diff and fix any prose that quotes a number that moved:
- The bullets under Leading stacks (e.g. "Opus 4.8 ~0.59 on hard").
- The per-language recommendation table (the curated one with the prompt/testing column)
and its
†/‡footnotes — reconcile its picks against the generated matrix and winner table. A cell that flipped qualified↔unqualified (e.g. a language gaining local support) changes the recommendation. - The decision procedure list.
- The language split sentence if a language moved between local and cloud.
You MUST NOT edit numbers inside the GEN markers by hand — re-run the generator instead.
5. Report
Tell the user: which table cells changed, any prose you reconciled, and — if step 1 found anything — what you stopped on. If nothing changed, say the blog was already current.
Constraints Summary
- You MUST run
--healthand clear it against the known gaps before--write. - You MUST NOT hand-edit between
GEN:*markers; the generator is the source of truth. - You MUST NOT introduce or re-elevate a cross-language aggregate as a recommendation — per-language success rates are the point.
- You MUST verify idempotency (step 3) before considering the update done.
- A new model string or un-ingested experiment is a STOP-and-ask, not a silent skip.
Troubleshooting
--health reports an unmapped model — add it to FEATURED_STACKS (with a models
list and short column name) if it should appear in the blog, or to KNOWN_NONFEATURED
if it's legacy/a serving variant. Re-run health until only the accepted gaps remain.
Blank-model count grew — the harness stopped recording model for some runs. Check
stack_metadata() is still called by every runner's provision()
(local_runner, metaharness_runner, docker_runner); a new runner or a bypass would
reintroduce the original bug. Do not paper over it in the generator.
A local language's number looks too low — confirm the curated selection in
FEATURED_STACKS still points at the tuned-config experiments, not an all-experiment
average (which includes early bad-config runs and understates the tuned reality).
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

