agentleFS
adriancoadrianco/retort/skills/update-optimal-blog/SKILL.md

update-optimal-blog

Skill212 starsChanged 7 months agoApache-2.0View on GitHub

Refresh the data tables in optimal-blog.md from master.db. Checks the data for integrity problems FIRST, then runs the generator that picks per-language winners and splices every GEN-marked table, then reconciles the surrounding prose. Use after new experiment results land, or when the optimal-blog numbers are stale.

The file

Files
SKILL.md

Update optimal-blog.md

Overview

optimal-blog.md records what to run today, per language and task size. Its data tables are not hand-written — they are generated from master.db by the retort report optimal subcommand (code in src/retort/reporting/optimal.py) and live between <!-- GEN:<key> START/END --> markers. This skill is the safe procedure for refreshing them: check the data before you trust it, regenerate, verify the round-trip, then fix any prose whose numbers moved.

The order matters. master.db does not record the full stack/config (see the health gaps below), so a blind regenerate can silently publish wrong numbers. Always run the health check first and stop if it reports anything new.

Why per-language, not aggregates

A single cross-language reliability number is misleading — it blends a stack's strong languages with its weak ones (local Qwen passes Python/Go but fails Rust; Opus 4.8 dips on Java). The generator's centrepiece is the per-language success-rate matrix; the leading-stacks routine aggregate is explicitly labelled the least-useful number. Keep it that way — do not "promote" an aggregate back into the recommendation.

Steps

1. Health-check master.db FIRST (gate)

retort report optimal --health

Compare against the known, accepted gaps (already documented in the blog's Keeping this current section):

  • ⚠️ No sampling columns / max_context_tokens unpopulated — the qualified config is curated in FEATURED_STACKS, not filtered from data.
  • ⚠️ ~250 rows have a blank model (local provenance bug) — attributed by experiment slug.
  • ⚠️ experiment-11, experiment-29 not ingested.

You MUST STOP and surface to the user if the report shows anything beyond those:

  • "Unmapped model strings" — a new model appeared that no featured/legacy entry covers. Decide whether it's a new featured stack (add to FEATURED_STACKS) or legacy (add to KNOWN_NONFEATURED) before regenerating. Publishing without deciding would drop it silently.
  • New experiment dirs not in master.db — results exist on disk but aren't ingested; the refresh would omit them. Re-ingest first, or note the omission to the user.
  • A jump in the blank-model count — the harness may have regressed; a run recording no model is invisible to the tables. (The fix landed in src/retort/playpen/runner.py stack_metadata(); if new blanks appear, that path is being bypassed.)

2. Regenerate the tables

retort report optimal --write optimal-blog.md

This splices every GEN:* block (leading stacks, per-language matrix, per-language winners, prompt method). It only touches text between markers.

3. Verify the round-trip

retort report optimal --write optimal-blog.md   # run twice
git diff --stat optimal-blog.md

A second --write MUST produce no further change — the tables are idempotent. If git diff shows table cells changed, that's the real new data; if it shows nothing, the blog was already current.

4. Reconcile the prose

The generator owns the tables, not the sentences around them. After a refresh, read the diff and fix any prose that quotes a number that moved:

  • The bullets under Leading stacks (e.g. "Opus 4.8 ~0.59 on hard").
  • The per-language recommendation table (the curated one with the prompt/testing column) and its †/‡ footnotes — reconcile its picks against the generated matrix and winner table. A cell that flipped qualified↔unqualified (e.g. a language gaining local support) changes the recommendation.
  • The decision procedure list.
  • The language split sentence if a language moved between local and cloud.

You MUST NOT edit numbers inside the GEN markers by hand — re-run the generator instead.

5. Report

Tell the user: which table cells changed, any prose you reconciled, and — if step 1 found anything — what you stopped on. If nothing changed, say the blog was already current.

Constraints Summary

  • You MUST run --health and clear it against the known gaps before --write.
  • You MUST NOT hand-edit between GEN:* markers; the generator is the source of truth.
  • You MUST NOT introduce or re-elevate a cross-language aggregate as a recommendation — per-language success rates are the point.
  • You MUST verify idempotency (step 3) before considering the update done.
  • A new model string or un-ingested experiment is a STOP-and-ask, not a silent skip.

Troubleshooting

--health reports an unmapped model — add it to FEATURED_STACKS (with a models list and short column name) if it should appear in the blog, or to KNOWN_NONFEATURED if it's legacy/a serving variant. Re-run health until only the accepted gaps remain.

Blank-model count grew — the harness stopped recording model for some runs. Check stack_metadata() is still called by every runner's provision() (local_runner, metaharness_runner, docker_runner); a new runner or a bypass would reintroduce the original bug. Do not paper over it in the generator.

A local language's number looks too low — confirm the curated selection in FEATURED_STACKS still points at the tuned-config experiments, not an all-experiment average (which includes early bad-config runs and understates the tuned reality).

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.