agentleFS
917Dhj917Dhj/dkitchen/skills/build-runner/SKILL.md

build-runner

Skill10 starsChanged 8 days agoMITView on GitHub

Build or change the experiment runner of a dkitchen kitchen in this repo — one resumable entry point for work done item by item (evaluations, batch inference, data processing), with run identity and per-item commits. Use when the user asks to set up a runner, make runs resumable, or stop writing one script per experiment.

The file

We split this skill into the rules it gives an agent

4 rules. About 0 are common, 2 are distinctive, and 2 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈2

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 2

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    Follow Installing for the prerequisites, language, rendering, updating in place, pointer blocks, and approval.

    line 12
  2. Distinctive:

    Run the checks in Installing, then:

    line 68
  3. Distinctive:

    Every later job compares before starting any item.

    line 88
  4. Very specific:

    Only an error that makes every item wrong stops the run: the task function raises StopRun, such as on an authentication error.

    line 111

≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Build the runner

Give the repo one runner for its item-by-item work, written by you in the repo's library and shaped to what the repo runs, plus docs/agents/runner.md from kitchen/runner/, its pointer line, and tests that keep the runner honest. Without a runner, every experiment carries its own loop with its parameters in code, and every fix to the loop is a copy; with one, an experiment is a spec and a task function.

You write the runner; this skill gives you a reference to read, not code to copy. reference/runner.py shows the hard parts working in one file; The core logic says what each must guarantee and where it goes wrong. The repo's runner keeps every guarantee and takes the shape the repo needs.

Follow Installing for the prerequisites, language, rendering, updating in place, pointer blocks, and approval. When /setup-dkitchen brought you here, the prerequisites and the language are already settled: skip them.

Sources

  • Template: kitchen/runner/runner.md.
  • Reference: reference/: runner.py, the toy task function toy_task.py, and test_runner.py, the behaviours every runner must keep.

1. Explore

  • Earlier skills: docs/agents/code.md and [check] in .dkitchen.toml must exist, since the runner goes in the library area and its checks run under ding check. When they are missing, stop and offer build-repo first. The runner is Python only, as build-repo's checks are.
  • Installed runner: docs/agents/runner.md, and any runner module already in the library, read in full when present.
  • The work: every script that runs items in a loop: what it reads, which parameters sit in its code, what it calls (a hosted API, a local server, only CPU), how it writes results, and whether rerunning it resumes. Name the scripts that fit item-by-item work, and the ones that do not, such as training.
  • Services: where model and service names appear, counted by file, and whether the repo already has one module that reaches them.
  • Formats: the config format the repo already uses, and its Python version.

Done when you can name, with evidence, the work the runner would carry, the services it calls, and the config format.

2. Present the findings

Summarize the scripts that fit and those that do not, how each handles resuming and failures today, and where service names appear. When nothing in the repo runs item by item, say so and recommend not installing the runner.

3. Settle each section

One section, one answer, then the next. Lead each with your recommendation.

A. Scope. The runner carries new item-by-item runs. Recommend leaving existing scripts as they are: they move to the runner when a ticket changes them.

B. The spec. Propose the fields from the work found: items, arms with a task function and its arguments, passes, concurrency, output, inputs, and what the repo's runs add, such as models and generation settings. Recommend the repo's config format; with none, TOML.

C. Services. When runs call models or services, propose one registry module in the library that holds every name, address, and client setting, and one way in for task functions, such as ctx.client(name). Recommend the service-name check below. A repo that calls no services has neither.

D. Place and command. The runner's module in the library, and its command, such as python -m <package>.runner <spec>.

E. Identity. Recommend that only concurrency and the output folder stay outside the run identity. Anything else left out is the user's choice, with the reason.

Done when every section has an answer.

4. Draft and confirm

Show the user:

  • the runner module, the registry when there is one, and their tests;
  • the full text of docs/agents/runner.md (a diff when updating), and the new pointer blocks;
  • the service-name check and its line for [check] commands, when there is a registry;
  • the line for code.md's reuse sentence naming the runner and the registry.

Done when the user approves the draft.

5. Write

Render with --templates runner, as Installing says, then fill and translate. Rewrite both pointer blocks from what render.py prints. Write the runner, the registry, the tests, and the check. Adding to [check] commands is a guard change: the commit carries Guard-change: #<ticket> <reason>.

Done when every approved file exists with its approved content.

6. Verify

Run the checks in Installing, then:

  • the runner's tests pass, each behaviour of test_runner.py included;
  • ding check --base HEAD passes;
  • a toy spec started with ding launch --progress <output>/progress.json shows its progress on ding board.

Done when every check passes, or the user has been told exactly which one fails and why.

7. Hand over

Tell the user, briefly: new runs are a spec and a task function, started with the runner's command under ding launch; rerunning the command resumes; changing the spec or an input needs a new output folder, and changing code does not; summary.json says how every run ended; and what the runner leaves out (below) is theirs to add when a run needs it.

The core logic

Each part below is one guarantee. Keep all six; shape the code to the repo.

Run identity

The identity is a digest of the spec without its operational fields, plus a content digest of every input: the item source, the spec's input files, and each task module's INPUTS. run.json holds it, written when the run starts.

  • Every later job compares before starting any item. On a mismatch, stop and name what changed; the output stays as it was.
  • Code is outside the identity. A job on different code appends a segment to run.json (commit, and a digest of the task modules for uncommitted edits), warns in the log, and goes on. Each result records its segment. This is what lets code change in place: no copies to keep old runs reproducible.
  • Pitfalls: digest content, never paths or modification times; INPUTS misses files a task reads indirectly, such as a template included by another; excluding a field from identity is a decision, not a convenience.

One writer

One job writes an output folder: an exclusive flock on a file in it, held for the whole job. The OS releases it when the process dies, even on SIGKILL, so no stale lock is left. Inside the job, only the main thread writes.

Commit on completion

Each (item, arm, pass) is one file, written atomically: a temp file in the same folder, flushed and fsynced, then os.replace. The main thread keeps at most concurrency items in flight and commits each the moment it finishes.

  • Pitfalls: committing in spec order lets one slow item hold back every commit, the progress, and the disk counts behind it; appending results to one shared file loses atomicity; a temp file in another folder makes os.replace a copy across filesystems.
  • Threads fit work that waits on services. For CPU-heavy items, use worker processes that only compute, and keep the commits in the main process.

Resume

The original command resumes. After the identity check, load every committed result, skip the succeeded ones, and redo the rest. A result that does not parse stops the job: it is evidence of a bug, kept and investigated. A leftover temp file is a write that never committed, and is removed.

Failures

A failure of one item never stops the run. Retry the same input a bounded number of times (3 in the reference), then commit the item as failed, with its error kind, message, and traceback, and go on. A failed item counts as wrong and stays in the denominator; the next job retries it.

  • Only an error that makes every item wrong stops the run: the task function raises StopRun, such as on an authentication error.
  • Pitfall: when the runner is started as python -m pkg.runner, the module runs as __main__, and a task function's from pkg.runner import StopRun gets a second class that except StopRun never catches. Keep StopRun in a module that is never __main__, or have __main__ call into the imported module, as the reference does.

Progress and summary

  • progress.json uses the progress-v1 shape ding board reads: schema, state, stage, current with done, total, and failed, rate_per_hour, eta, updated. It is display only, never read back as state.
  • summary.json is written however the job ends: SIGINT and SIGTERM become exceptions so the finally runs. It holds the state, the stop reason, counts from what is on disk, the failed items, and the segments. Exit 0 when every item succeeded, 1 when the run finished with failures, 2 when it stopped.
  • Logs go to stderr and to a log file in the output folder, appended to on every resume.

The service-name check

With a registry, write a check script in the tools area and list it in [check] commands. It reads every model and service name from the registry, takes the lines git diff <base> adds under the experiments area (ding check passes --base <commit>), and fails, with file and line, on a string that equals a registered name. Lines already in the repo are left alone, so the check holds from the first PR with no exception list.

Left out on purpose

The reference leaves out spec schema checks beyond unknown keys, the service layer, checkpoint and canary rules, a cost ledger, pausing in place to recheck a service or disk, and GPU placement. Add one when a run needs it, keeping the six guarantees.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.