Agent Guidelines for shirabe

This is a skills repository. The quality of skills is validated through evals, not just code review. These guidelines apply to any agent working on this repo.

Eval Requirement

When adding or modifying a skill:

1. Write eval scenarios in skills/<name>/evals/evals.json that cover the skill's key behaviors. Each eval has a prompt (what a real user would say), expected output (what the skill should produce), and assertions (objectively verifiable checks).
2. Pass the skill gate. A pull request's definition of done for a skill change is scripts/check-skill.sh --changed <skill-name>: shirabe validate, the template compile, the script suites the change touches, and the shape of evals/evals.json. The pull request doesn't run the evals.
3. The evals run at release. /shirabe:release runs the evals of every skill changed since the last tag and stops on a pass-rate drop (see .claude/shirabe-extensions/release.md). To try a scenario before then, run scripts/run-evals.sh <skill-name> locally (see Running Evals below) and iterate on the skill if it fails.

This applies to:

• New skills (write evals during skill creation)
• Skill modifications (update existing scenarios + add new ones for changed behavior)
• Specification changes (reference files, format specs) that affect skill behavior

Eval Structure

Evals are co-located with their skill, not in a central directory:

skills/<name>/
  SKILL.md
  evals/
    evals.json                # This skill's eval scenarios + expectations
    fixtures/                 # Optional. <eval-name>/<path> supplies the content
                              # for a path that eval declares under files:;
                              # files/<path> supplies it for the whole suite.
    workspace/                # gitignored; created by run-evals.sh
      iteration-N/
        <eval-name>/
          workspace/            # This scenario's working tree. Paths declared
                                # under files: are materialized here before the run.
          with_skill/outputs/   # Agent outputs with skill loaded, plus
                                # post_run_tree/ and post_run_tree_manifest.txt:
                                # the scenario working tree as the run left it
          without_skill/outputs/ # Baseline outputs without skill
          eval_metadata.json    # Prompt + resolved criteria for this eval
          grading.json          # Pass/fail per assertion
          timing.json           # Token count and duration
        benchmark.json          # Aggregated results for this iteration
        validation_summary.json # Machine-readable tally for this iteration

A scenario's graded criteria come from its expectations key, falling back to assertions where a suite still uses the older name. A scenario that grades zero criteria fails the run: a suite reporting green having graded nothing is the failure this harness exists to prevent. Assertions about files a run was supposed to produce are graded against post_run_tree/, not against the agent's account of what it did.

Running Evals

Two paths: automated (CLI/CI) and interactive (Claude Code session).

Automated (CLI / CI)

# List skills that have evals
scripts/run-evals.sh --list

# Run evals end-to-end (prep + execute via claude -p + validate)
scripts/run-evals.sh decision

# Run all skills
scripts/run-evals.sh --all

# Re-validate the latest iteration without re-running
scripts/run-evals.sh --validate decision

# Run one scenario five times and report how often it passed
scripts/run-evals.sh --scenario standalone-decision-simple --runs 5 decision

Exit status: 0 all graded and passing, 1 an assertion failed, 2 something graded nothing (no results, a missing grading.json, or a scenario with an empty criteria list), 3 a missing prerequisite, 4 the nested claude session stopped in plan mode or ran no command and wrote no file, so no scenario ran and the runner or the host is at fault rather than the skill.

The runner starts its nested claude -p session with --permission-mode acceptEdits --allowedTools Bash, from the repo root, with a scratch directory it creates per run as the session's TMPDIR and its one --add-dir. The host's default permission mode never reaches it. The session's transcript is saved as runner_session.jsonl in the iteration directory; the header of scripts/run-evals.sh says why the mode is this one.

Interactive (Claude Code with /skill-creator)

# Prepare workspace only
scripts/run-evals.sh --prep-only decision

After --prep-only, invoke /skill-creator in your Claude Code session and point it at the prepared workspace. This gives a tighter feedback loop: the skill-creator handles agent spawning, grading, and the eval viewer within your session. After running, validate with scripts/run-evals.sh --validate decision.

The preflight check during evals

scripts/run-evals.sh exports SHIRABE_PREFLIGHT_DISABLE=1, which turns off the prerequisite check every skill body injects at load (scripts/skill-preflight.sh). Tier-2 fixtures put shim binaries under the working directory and prepend that directory to PATH; the check refuses to probe a binary that resolves under $PWD and says so, and since these evals are transcript-graded that block would silently change the input to every scenario in the corpus. Keep the variable set for every agent the harness spawns.

The exception is an eval carrying "preflight": "live". The harness gives those a different instruction: run with the variable cleared. They exist to prove the injected line still executes, and with the check switched off they would assert nothing. skills/inflight/evals/evals.json has the pair — one fixture skill whose declaration cannot be satisfied, one that every host meets — loaded from the self-contained fixture plugin under skills/inflight/evals/fixtures/preflight-liveness/. The deterministic half of the same assertion runs on every pull request in scripts/skill-preflight_test.sh.

Shell Scripts Target bash 3.2

Every shell script under skills/ and scripts/ has to run under bash 3.2, because that is what macOS ships as /bin/bash and always will. Nothing about that is enforced by your editor and most of it is invisible on a modern Linux box: bash 5 accepts constructs bash 3.2 either rejects or, worse, accepts and handles differently.

Before pushing a change to any of them, run the suite that covers it on the floor:

# what suites exist and which scripts each one runs
scripts/check-bash-floor.sh --list

# check one
scripts/check-bash-floor.sh plan

# check every suite
scripts/check-bash-floor.sh all

On Linux this runs the suite in a bash:3.2 container with a GNU userland; on macOS it uses /bin/bash directly, with a shim first on PATH so a nested bash is 3.2 too. Same command either way, and the same one CI runs.

The container sees the checkout read-only, so a suite can't leave anything behind in it, and it works from a linked worktree. It runs on whichever docker daemon DOCKER_HOST reaches. On a rootful daemon (stock Docker, and CI's hosted runners) it runs as your user. A rootless daemon is recommended, since it also keeps the daemon's own work off the host's root. On a host where containers must be rootless, pass --require-rootless (or set SHIRABE_FLOOR_REQUIRE_ROOTLESS=1) and a rootful daemon is refused.

Every macOS CI leg reaches the floor that way, as scripts/check-bash-floor.sh --backend system <suite>. On a macOS runner a bare bash, and a script run by path through #!/usr/bin/env bash, are Homebrew's bash 5, and /bin/bash on a harness leaves its nested bash calls there too. scripts/check-macos-floor-legs.sh fails a workflow whose macOS leg runs a suite any of those ways, so a new suite belongs in the runner's registry and its workflow's macOS leg calls the runner.

Do not rely on a grep for declare -A and friends. A pattern list only catches what its author remembered - the sweep that fixed plan-to-tasks.sh found declare -A by grep and missed a bash 4.3 nameref entirely. Run the suite.

Skill Quality Standards

Skills in this repo follow the patterns documented in the skill-creator skill:

• Progressive disclosure (SKILL.md under 500 lines, reference files under 300)
• Imperative instructions that explain "why" not just "what"
• Decision blocks for all non-trivial choices (see references/decision-protocol.md)
• Non-interactive mode (--auto) support at all decision points