ultrainit.sh / tests
CLAUDE.md8 starsChanged 7 months agoApache-2.0View on GitHub
All tests run inside Docker using the pre-built ghcr.io/joelbarmettleruzh/ultrainit-test:latest image. No real Claude API calls are made in any test. Smoke tests are intentionally excluded from make test-all — they make real API calls: Every .bats file must follow this exact setup order: Do not call setup_mock_claude before _common_setup — TEST_TMPDIR must exist first. Tests for standalone scripts (tests/scripts/) do not call setup_mock_claude because validators never invoke claude. _common_setup() sources ALL 9 lib files in dependency order on every test…
- Deletes or force-pushes
In this repository22 other files it gives its agents.
- CLAUDE.mdCLAUDE.md
- .github/workflows/CLAUDE.mdCLAUDE.md
- lib/CLAUDE.mdCLAUDE.md
- prompts/CLAUDE.mdCLAUDE.md
- schemas/CLAUDE.mdCLAUDE.md
- scripts/CLAUDE.mdCLAUDE.md
- add-bats-testSkill
- add-edge-testSkill
- add-gather-agentSkill
- add-integration-testSkill
- add-synthesis-fixtureSkill
- audit-prompt-schema-syncSkill
- bundle-and-verifySkill
- check-ci-locallySkill
- debug-agent-failureSkill
- fixture-syncSkill
- pre-pr-checklistSkill
- rebuild-test-imageSkill
- release-ultrainitSkill
- run-testsSkill
- tune-synthesis-contextSkill
- validate-generated-artifactSkill
The file
We split this CLAUDE.md into the rules it gives an agent
9 rules. About 0 are common, 0 are distinctive, and 9 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈0
Distinctive · estimate
This author's own choices. Worth a look.
9
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 3
All tests run inside Docker using the pre-built ghcr.io/joelbarmettleruzh/ultrainit-test:latest image.
- Very specific: line 49
Every .bats file must follow this exact setup order:
- Very specific: line 64
Do not call setup_mock_claude before _common_setup — TEST_TMPDIR must exist first.
- Very specific: line 79
All invocations are logged to $MOCK_CLAUDE_LOG with CALL: prefix.
- Very specific: line 83
Every mock response must match the real claude -p --output-format json envelope:
- Very specific: line 88
Use make_claude_envelope from test_helper.bash:
- Very specific: line 93
Never pass raw fixture JSON where an envelope is expected — run_agent() extracts .structured_output from the envelope and will get null, …
- Very specific: line 98
Budget comparisons: always string equality against what bc produces (.20 not 0.20 for values < 1)
- Very specific: line 99
Fixture mutations: always copy with cp into $TEST_TMPDIR.
≈ “Common” is an estimate: at least 1 in 5 of the 28,029 CLAUDE.md files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
tests/ — bats-core Test Suite
All tests run inside Docker using the pre-built ghcr.io/joelbarmettleruzh/ultrainit-test:latest image. No real Claude API calls are made in any test.
Running Tests
make test-image # Build test image locally (required if Dockerfile.test changed)
make test-unit # tests/unit/ — individual lib functions
make test-scripts # tests/scripts/ — validate-skill.sh, validate-subagent.sh
make test-edge # tests/edge/ — boundary conditions, known bugs
make test-integration # tests/integration/ — phase-level pipeline
make test-all # all of the aboveSmoke tests are intentionally excluded from make test-all — they make real API calls:
bash tests/smoke/run-smoke.sh --source # ~$2 with haikuDirectory Structure
tests/
├── helpers/
│ ├── test_helper.bash # _common_setup, _common_teardown, make_claude_envelope
│ └── mock_claude.bash # PATH-based mock claude binary
├── unit/ # 11 .bats files, one per lib subsystem
├── integration/ # 3 .bats files: gather_phase, synthesize_phase, resume
├── scripts/ # 2 .bats files: validate_skill, validate_subagent
├── edge/ # 8 .bats files: budget, corrupt state, special chars, etc.
├── fixtures/
│ ├── findings/ # 8 JSON fixtures for Phase 1 agents
│ ├── synthesis/ # output-docs.json, output-tooling.json, output.json
│ ├── skills/ # valid + invalid SKILL.md fixtures
│ ├── subagents/ # valid + invalid subagent fixtures
│ ├── envelopes/ # (empty, reserved)
│ ├── hooks/ # (empty, reserved)
│ └── developer-answers.json
└── smoke/
├── run-smoke.sh
└── mini-project/ # 4-file minimal Flask API targetTest Infrastructure
Setup Pattern
Every .bats file must follow this exact setup order:
setup() {
load '../helpers/test_helper'
_common_setup # creates TEST_TMPDIR, exports env defaults, sources all 9 libs
load '../helpers/mock_claude'
setup_mock_claude # writes mock binary to TEST_TMPDIR/mock_bin/, prepends to PATH
}
teardown() {
_common_teardown # rm -rf TEST_TMPDIR
}Do not call setup_mock_claude before _common_setup — TEST_TMPDIR must exist first. Tests for standalone scripts (tests/scripts/) do not call setup_mock_claude because validators never invoke claude.
Why All Libs Are Sourced Together
_common_setup() sources ALL 9 lib files in dependency order on every test setup. This is intentional: synthesize.sh sets shopt -s nullglob globally at source time. If sourced alone, this wouldn't affect config.sh. Sourced together, the glob option bleeds across libs — exactly as in production. Tests in tests/edge/corrupt_state.bats explicitly test this interaction.
Mock Claude Binary
Function-level mocking fails across subprocess boundaries. Instead, a real executable claude binary is written to $TEST_TMPDIR/mock_bin/ and prepended to $PATH. Every child process spawned by run_agents_parallel inherits this PATH.
Three modes:
- Single mode: set
$MOCK_CLAUDE_RESPONSEto a file path; the mock returns that file's content for all calls - Dispatch mode: set
$MOCK_CLAUDE_DISPATCH_DIR; the mock matches agent name from--json-schemafilename and returns the matching file - Error mode: set
$MOCK_CLAUDE_EXIT_CODE=1; the mock exits with that code
All invocations are logged to $MOCK_CLAUDE_LOG with CALL: prefix. Stdin is logged as STDIN: N bytes. Assert on these to verify mock was called (or not called, for skip tests).
Response Envelope
Every mock response must match the real claude -p --output-format json envelope:
{"is_error": false, "total_cost_usd": 0.42, "structured_output": {...}}Use make_claude_envelope from test_helper.bash:
make_claude_envelope '{"key":"val"}' "0.42" > "$MOCK_CLAUDE_RESPONSE"Never pass raw fixture JSON where an envelope is expected — run_agent() extracts .structured_output from the envelope and will get null, writing "null" to the findings file silently.
Test Conventions
- Cost file format:
phase|agent_name|amount_usd— one file per agent in$WORK_DIR/costs/ - Budget comparisons: always string equality against what
bcproduces (.20not0.20for values < 1) - Fixture mutations: always copy with
cpinto$TEST_TMPDIR; never reference fixture files in-place ULTRAINIT_MAX_RETRIES=1: set in test environments to prevent synthesis retry delays- Color codes exported empty:
_common_setupexportsRED='' GREEN='' BOLD=''etc. to prevent ANSI codes from breakingassert_output --partialmatching - Env vars before sourcing: set all environment overrides before calling
_common_setup, not after — libs read${VAR:-default}at source time DRY_RUN=truein merge tests: asserts no files were created; useassert_file_not_existsbefore callingwrite_artifacts()
Known Test Behaviors (Bug-Documenting Tests)
tests/edge/validation_regex.bats documents that (never |don.t |do not ) requires trailing spaces — Never at end-of-line is not detected. Do not fix this without updating the test.
tests/edge/corrupt_state.bats documents that cost files without pipe delimiters are silently treated as $0 spend. Do not fix without updating the test.
tests/edge/numeric_edge_cases.bats documents that bc produces .20 not 0.20 for values < 1. Always accept both forms in assertions: [[ "$x" == ".20" || "$x" == "0.20" ]].
Fixture Maintenance
tests/fixtures/findings/
One JSON file per Phase 1 agent, named to match the agent name exactly (three-way identity: fixture filename = agent name in run_agent() = findings file in .ultrainit/findings/). Breaking any link breaks mock dispatch routing.
5 of 8 fixtures have fields that violate their corresponding schemas but tests still pass — no runtime schema validation exists in agent.sh. Known violations: patterns.json (invalid type enum), tooling.json (missing purpose), structure-scout.json (invalid role strings, extra fields).
tests/fixtures/synthesis/
output-docs.json and output-tooling.json must NOT be pre-merged — they test the merge function. output.json is independently maintained as the expected post-merge result. When updating either pass fixture, regenerate output.json:
jq -s '.[0] * .[1]' tests/fixtures/synthesis/output-docs.json \
tests/fixtures/synthesis/output-tooling.json > tests/fixtures/synthesis/output.jsondeveloper-answers.json lives one level up in tests/fixtures/ (not in findings/) because it is Phase 2 output, not Phase 1. Copy it to $WORK_DIR/developer-answers.json (not $WORK_DIR/findings/) in synthesis tests.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

