agentleFS
joelbarmettlerUZHjoelbarmettlerUZH/ultrainit.sh/.claude/skills/run-tests/SKILL.md

run-tests

Skill8 starsChanged 7 months agoApache-2.0View on GitHub

The Makefile targets use Docker. To run one file, use the docker command directly: Add --tap for TAP format or --verbose-run for assertion details: Integration tests always show green in CI even when failing — they run with || true in .github/workflows/test.yml. After make test-integration, read the output explicitly. Add echo "TMPDIR: $TEST_TMPDIR" >&3 inside the failing @test body (bats redirects >&3 to the terminal). Run the test in verbose mode. Look for CALL: lines. If the mock was never…

50 other projects use a version of this.

Adopted by

Most recently taken up by

Used with

Projects with it also use prepare-release, create-pr and 3 more.

Other files people use alongside this one.

Files found in the same public projects as a version of this one, more often than they turn up anywhere else. Each project counts once, large collections count for less, and this file's own repository is left out.
  • ShopifySkill

    prepare-release

    Prepare the changeset-release/main branch for release by syncing version.ts files and running pnpm install

    Also in 3 projects that use this

  • SentrySkill

    create-pr

    Create pull requests following Sentry conventions. Use when opening PRs, writing PR descriptions, or preparing changes for review. Follows Sentry's code review guidelines.

    Also in 4 projects that use this

  • SentrySkill

    commit

    Create commit messages following Sentry conventions. Use when committing code changes, writing commit messages, or formatting git history. Follows conventional commits with Sentry-specific issue references.

    Also in 4 projects that use this

  • MicrosoftSkill

    release

    Use ONLY when the user explicitly asks to cut/publish a Huabu desktop release or tag a version. Performs an outward action — bumps the version in apps/desktop/package.json, commits + annotated-tags vX.Y.Z, and pushes the branch + tag. That tag push auto-builds the macOS/Windows installers and publishes a GitHub Release. Provides `status`, `bump`, `tag-push`, and a one-command `run` (bump → tag-push).

    Also in 5 projects that use this

  • GoogleSkill

    code-review

    Adversarial systems, concurrency, memory-model, and architecture code review protocol. Use this skill whenever asked to review code, PRs, diffs, or branches in Filament.

    Also in 4 projects that use this

The file

We split this skill into the rules it gives an agent

4 rules. 0 are common, 0 are distinctive, and 4 are very specific to this project.

  • 0

    Common

    Most of the 45 edited versions we read keep these.

  • 0

    Distinctive

    This author's own choices. Worth a look.

  • 4

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    Add --tap for TAP format or --verbose-run for assertion details:

    line 59
  2. Very specific:

    Add echo "TMPDIR: $TEST_TMPDIR" >&3 inside the failing @test body (bats redirects >&3 to the terminal).

    line 79
  3. Very specific:

    Look for CALL: lines. If the mock was never called when it should have been, check if setup_mock_claude was called in setup().

    line 87
  4. Very specific:

    Only needed when Dockerfile.test changes:

    line 107

“Common” is measured: kept by at least 6 in 10 of the 45 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Before You Start

  • Makefile — all test targets and their Docker invocations
  • tests/helpers/test_helper.bash — _common_setup, make_claude_envelope
  • tests/helpers/mock_claude.bash — setup_mock_claude, dispatch mode
  • Dockerfile.test — the test image (must be built if changed)

Test Tiers

CommandWhat it testsSpeed
make test-unitIndividual lib/*.sh functions~30s
make test-scriptsscripts/validate-skill.sh, scripts/validate-subagent.sh~15s
make test-edgeBudget exhaustion, corrupt state, special chars, numeric edge cases~20s
make test-integrationPhase-level pipeline (gather, synthesize, resume)~60s
make test-allAll of the above~2min

Running Tests

Full suite

make test-all

Single tier

make test-unit        # fastest, run after every lib change
make test-scripts     # after changes to scripts/validate-*.sh
make test-edge        # after changes to budget, config, or CLI arg handling
make test-integration # after changes to gather.sh, synthesize.sh, or merge.sh

Single .bats file

The Makefile targets use Docker. To run one file, use the docker command directly:

docker run --rm \
    -v "$(pwd)":/workspace \
    -w /workspace \
    ghcr.io/joelbarmettleruzh/ultrainit-test:latest \
    bats tests/unit/agent_run.bats

Verbose output

Add --tap for TAP format or --verbose-run for assertion details:

docker run --rm -v "$(pwd)":/workspace -w /workspace \
    ghcr.io/joelbarmettleruzh/ultrainit-test:latest \
    bats --verbose-run tests/unit/agent_run.bats

Interpreting Output

  • ok N - test name = pass
  • not ok N - test name = fail, followed by assertion details
  • # (from function ... source ... line N) = exact location of failed assertion

Integration tests always show green in CI even when failing — they run with || true in .github/workflows/test.yml. After make test-integration, read the output explicitly.

Debugging a Failing Test

1. Find the tmpdir

Add echo "TMPDIR: $TEST_TMPDIR" >&3 inside the failing @test body (bats redirects >&3 to the terminal). Run the test in verbose mode.

2. Inspect mock logs

cat $TEST_TMPDIR/.ultrainit/mock_claude.log  # all mock invocations

Look for CALL: lines. If the mock was never called when it should have been, check if setup_mock_claude was called in setup().

3. Check dispatch matching

Dispatch mode matches agent name from the --json-schema argument basename (case-insensitive substring). If docs matches before docs-scanner, rename the dispatch file: longer/more-specific names first alphabetically.

4. Verify envelope wrapping

Passing raw fixture JSON where an envelope is expected causes run_agent to extract .structured_output as null, writing "null" to the findings file silently:

# Wrong — raw JSON, no envelope
cat tests/fixtures/findings/identity.json > "$MOCK_CLAUDE_RESPONSE"

# Correct — envelope-wrapped
make_claude_envelope "$(cat tests/fixtures/findings/identity.json)" "0.15" > "$MOCK_CLAUDE_RESPONSE"

Building the Test Image

Only needed when Dockerfile.test changes:

make test-image   # builds locally

The CI image is pulled from ghcr.io/joelbarmettleruzh/ultrainit-test:latest. If the image doesn't exist yet on a fresh fork, run make test-image first.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.