karate-bootstrap
Skill0 starsChanged 35 days agoMITView on GitHub
Bootstrap a first ground-truth Karate integration-test suite that runs green under Testcontainers for a Spring Boot, Quarkus, ASP.NET Core or Python service that has no Karate tests. Use when asked to add karate tests, bootstrap integration tests, or build a testcontainers suite for a service or repo. Not for unit tests.
In this repository2 other files it gives its agents.
- document-driftSkill
- tech-debt-scanSkill
The file
Show its rules and text
We split this skill into the rules it gives an agent
15 rules. About 0 are common, 2 are distinctive, and 13 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈2
Distinctive · estimate
This author's own choices. Worth a look.
13
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 17
Read reference/testcontainers-notes.md once before a run.
- Very specific: line 23
If any expected output file from a numbered step is missing, abort with exit 5.
- Very specific: line 48
Run every python scripts/...
- Distinctive: line 66
Every step names one command and one output.
- Very specific: line 144
Verify app.auth yourself even when discover.py wrote confirmed: true: it guesses disabled from a config key such as app.security.enabled …
- Very specific: line 154
Use --mode none when the app has no auth and --mode blocked when it cannot be switched off or pointed at a test issuer.
- Very specific: line 176
If merge reports unresolved: N above zero, re-render with the first unresolved location and dispatch again, then merge again.
- Very specific: line 183
If merge reports incomplete: no exits and no exits_none_reason (exit 2), the trace found no exit and gave no reason.
- Very specific: line 191
When a reply is both unresolved and incomplete, follow the unresolved bullet first: its --focus location is the specific line the tracer …
- Very specific: line 205
Run a --focus trace on the file:line of every disagreement: line, merging each result with --union too, before the gate.
- Distinctive: line 219
Do not edit the ledger to silence a gap.
- Very specific: line 268
Add --force only when re-scaffolding a module whose harness files you intend to refresh.
- Very specific: line 349
Read the group it prints (the largest failure signature with its evidence) and classify it with reference/failure-triage.md: infra, …
- Very specific: line 357
Add --unfixable to an infra iteration that cannot be fixed from inside <tests>.
- Very specific: line 414
No live LLM in this repo's tests: kb_check_skill.py lints these commands and tests/test_kb_dry_run.py executes the chain with canned …
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
karate-bootstrap
Takes a service repository with no Karate tests and leaves it with a first "ground truth"
suite under karate-tests/ that runs green under Testcontainers, locally and in Azure DevOps.
The suite documents the behaviour the service has today; suspected defects are quarantined
and reported, never fixed. Scripts do every deterministic step (discovery, the ledger, gates,
scaffolding, report parsing, git checkpoints); you do the judgement inside narrow subagent
tasks (tracing code paths, confirming validation rules, writing features) and the fix loop.
Read reference/testcontainers-notes.md once before a run. Read the stack cheat sheet the
ledger names (reference/stack-<stack>.md) when you dispatch the first trace. Read
reference/failure-triage.md before the first fix iteration.
No improvisation
If any expected output file from a numbered step is missing, abort with exit 5. Do not retry
steps you were not told to retry. Do not edit flow-map.yaml, rules/*.csv or
kb-runtime.json by hand: every change goes through a script. Do not compose a subagent
prompt yourself: render it with kb_prompt.py, and add nothing to the rendered text except
the one ## Gaps from the last gate section the Subagents table allows. Never edit the
application's source, its Dockerfile or its pipeline. Never push.
Invocation
/karate-bootstrap <repo-path> [--service-dir <sub>] [--migrations-image <ref>] [--app-image <tag>]
[--max-iterations 15] [--double-trace] [--no-commit]| Flag | Effect |
--service-dir <sub> | The service lives in a sub-directory of the repo (monorepo). Every script takes the repository root as its positional (or --repo) argument and resolves the service under <sub> itself, so pass --service-dir <sub> alongside <root>, never a joined path. |
--migrations-image <ref> | The db-manager image that owns the schema. Without it the scaffold looks the database up in ~/.karate-bootstrap/config.yaml and aborts with exit 4 when nothing matches. |
--app-image <tag> | Test a prebuilt image instead of building the Dockerfile: adds -Dapp.image=<tag> to every Maven run. |
--max-iterations <n> | Fix-loop cap (default 15). |
--double-trace | Trace every entry twice with independent subagents and merge both results; disagreements get a third, narrower trace. |
--no-commit | Write files only; pass --no-commit to every kb_checkpoint.py call so git is never touched. |
Conventions
- Run every
python scripts/...command from this skill's directory (skills/karate-bootstrap/) so the relative script paths resolve; scripts are direct-path invocable, no package install. Pass<root>as an absolute path. <root>is<repo-path>: the repository root, and the positional (or--repo) argument of every script.<sub>is the--service-dirvalue. A script that accepts--service-dirjoins the two itself, so never write<root>/<sub>. Omit--service-dirfrom every command below when the run has no sub-directory.<tests>is<repo-path>/<sub>/karate-tests, or<repo-path>/karate-testswhen there is no sub-directory. Every artefact the run produces lives there. Rendered prompts go to<tests>/.prompts/(ignored by the template's.gitignore).kb_checkpoint.py commitstages a path relative to<repo-path>, not to<tests>: pass--tests-dir <sub>/karate-tests, or drop the flag entirely (its default iskarate-tests) when there is no sub-directory.<skill>is this skill's absolute directory;templates/karate-tests/README.md.tmplis referenced relative to it.- Maven runs from
<tests>:mvn -B test(JDK 17 or newer and a container engine on the machine;./mvnw -B testwhen Maven is not installed). Append-Dapp.image=<tag>when--app-imagewas given. Podman users:reference/podman.md. - Every step names one command and one output. A step's postcondition is the file it must leave behind.
Exit codes
| Code | Meaning |
| 0 | Suite green (or green with quarantined defects listed in defects.md). |
| 2 | A gate or script rejected its input; the message names the gap. |
| 3 | Unsupported stack (not Spring Boot, Quarkus, ASP.NET Core or a Python web framework). |
| 4 | No schema source: no --migrations-image and no matching central config entry. |
| 5 | An expected output file is missing (the no-improvisation rule). |
| 6 | Stopped by a stop condition after committing what exists. |
| 7 | Container runtime, JDK 17+ or Maven missing on this machine. |
Subagents
Three kinds, each driven by a prompt file kb_prompt.py renders for one entry. Read the
rendered file and pass its complete text as the subagent's prompt; remove nothing. The only
permitted addition is a final section ## Gaps from the last gate holding, verbatim, the
output that sent you back: the gate's gap list, or merge's incomplete: line. Nothing else
may be added, and nothing may be reworded.
| Kind | Agent | May write | Returns |
| trace | read-only (Explore) | nothing | JSON matching the ledger entry schema |
| rules | read-only (Explore) | nothing | JSON with csv (header plus rows), rows, dropped_candidates, notes |
| generate | general-purpose | files under <tests> only (features/, stubs/, seed/) | JSON listing the files written |
Save every JSON reply to <tests>/.prompts/<kind>-<slug>.json before you feed it to a script —
one file per kind and slug, except under --double-trace, where the pair for one entry is
trace-<slug>-a.json and trace-<slug>-b.json. Dispatch trace subagents one at a time unless
--double-trace asks for pairs.
Workflow
Step 0: Preflight
- Prerequisite:
<repo-path>exists. - Commands:
python scripts/kb_checkpoint.py begin --repo <repo-path>
python scripts/detect.py <root> --service-dir <sub> --out <tests>/stack.jsonbegincreates and checks outkarate-bootstrapwhen the repo is on its default branch and leaves any other branch alone. Add--no-commitwhen the flag was given (then it does nothing).detect.pychecks the container runtime, JDK and Maven, then classifies the stack; omit--service-dirwhen there is none. Exit 7 means the machine cannot run Step 7: stop and report what is missing. Add--skip-toolchainonly when the user asked for a dry run that stops before Step 7 (Steps 0 to 6 need no toolchain).- Postcondition:
<tests>/stack.json. Exit 3 or 7 fromdetect.pyends the run with that code.
Step 1: Discover
- Prerequisite:
<tests>/stack.json. - Command:
python scripts/discover.py <root> --service-dir <sub> --stack <tests>/stack.json --out-env <tests>/env-map.json --out-ledger <tests>/flow-map.yaml- Omit
--service-dirwhen there is no sub-directory. - Postcondition:
<tests>/env-map.json(config keys with roles, port, readiness, auth mode) and<tests>/flow-map.yamlseeded with one untraced entry per entry point.
Step 2: Confirm entry points and auth
- Prerequisite: the two files from Step 1.
- Open
<tests>/flow-map.yamland the routes in the code once. For every entry point the regexes missed (a route, a listener), add it:
python scripts/flow_map.py add-entry --ledger <tests>/flow-map.yaml --id "<METHOD> <path>" --kind http --handler <file:line> --method <METHOD> --path <path>
python scripts/flow_map.py add-entry --ledger <tests>/flow-map.yaml --id "amq <destination>" --kind amq-subscribe --handler <file:line> --destination <destination> --type queue- Verify
app.authyourself even whendiscover.pywroteconfirmed: true: it guessesdisabledfrom a config key such asapp.security.enabledwithout proving the code reads it. Grep the source for that key (cheat sheet: "Auth switches"). When nothing reads it, or the mode is otherwise wrong, read the security configuration and record what you found:
python scripts/flow_map.py set-auth --ledger <tests>/flow-map.yaml --mode disabled --key <ENV_VAR> --value <off-value>
python scripts/flow_map.py set-auth --ledger <tests>/flow-map.yaml --mode jwks --issuer-keys <ISSUER_ENV>,<JWKS_ENV>Use --mode none when the app has no auth and --mode blocked when it cannot be switched
off or pointed at a test issuer.
- Postcondition:
<tests>/flow-map.yamllists every entry point andapp.auth.modeis one you verified against the code (disabledadditionally carrieskey,valueandconfirmed: true;jwkscarrieskeys;noneandblockedcarry only the mode).
Step 3: Trace every entry point
- Prerequisite:
<tests>/flow-map.yaml,<tests>/env-map.json. - Loop until
nextprints{"done": true}:
python scripts/flow_map.py next --phase traced --ledger <tests>/flow-map.yaml
python scripts/kb_prompt.py render --prompt trace --ledger <tests>/flow-map.yaml --env <tests>/env-map.json --entry "<id>" --repo <root> --service-dir <sub> --out <tests>/.prompts/trace-<slug>.md
python scripts/flow_map.py merge <tests>/.prompts/trace-<slug>.json --ledger <tests>/flow-map.yamlBetween render and merge, dispatch the trace subagent with the rendered prompt and save
its reply as <tests>/.prompts/trace-<slug>.json (merge accepts the bare object or one
wrapped in a code fence). Omit --service-dir when there is no sub-directory. next prints
the entry id, its handler and the cheat sheet path; <slug> is the id lower-cased with
non-alphanumerics collapsed to - (POST /api/deals becomes post-api-deals).
- If
mergereportsunresolved: Nabove zero, re-render with the first unresolved location and dispatch again, then merge again; repeat until the entry merges with zero unresolved:
python scripts/kb_prompt.py render --prompt trace --ledger <tests>/flow-map.yaml --env <tests>/env-map.json --entry "<id>" --repo <root> --service-dir <sub> --focus <file:line> --out <tests>/.prompts/trace-<slug>-2.md- If
mergereportsincomplete: no exits and no exits_none_reason(exit 2), the trace found no exit and gave no reason. Re-render with--focusat the entry's handler line, dispatch once more with theincomplete:line under a final## Gaps from the last gateheading in the rendered prompt, and merge again. If it comes back incomplete a second time, leave the entry untraced and let the gate name it: do not write it intodefects.md(that file is for suspected application defects found by a run, not for tracing gaps), and do not edit the ledger.validate --phase tracedalready fails ontraced: false, which is the correct end state for an entry nobody could trace. When a reply is both unresolved and incomplete, follow the unresolved bullet first: its--focuslocation is the specific line the tracer stopped at, and a re-trace from there usually resolves both. - With
--double-trace: dispatch two independent subagents from the same rendered prompt and save their replies as<tests>/.prompts/trace-<slug>-a.jsonand-b.json. Merge the first, then merge the second with--union, which keeps the union of exits, reads and responses (an exit that differs only inviais the same exit; the firstviais kept) and prints onedisagreement: <kind> <what> via <file:line>line per item only one reply saw:
python scripts/flow_map.py merge <tests>/.prompts/trace-<slug>-a.json --ledger <tests>/flow-map.yaml
python scripts/flow_map.py merge <tests>/.prompts/trace-<slug>-b.json --ledger <tests>/flow-map.yaml --unionRun a --focus trace on the file:line of every disagreement: line, merging each result
with --union too, before the gate.
One --focus round per disagreement is the cap: merge each result with --union, then run
the gate. A via still seen by only one trace after that round is a disagreement the gate's
verify-refs check adjudicates, not grounds for a third trace.
- Gate:
python scripts/flow_map.py validate --phase traced --ledger <tests>/flow-map.yaml --repo <root> --service-dir <sub> --env <tests>/env-map.jsonOmit --service-dir when there is no sub-directory. The gate also runs verify-refs: every
exit's via must sit within three lines before or after a marker token from the cheat sheet,
or the entry is reset to untraced. On any gap, go back to the loop for the entries it names.
Do not edit the ledger to silence a gap.
- Postcondition:
validate --phase tracedprintsphase traced: pass.
Step 4: Validation rules
- Prerequisite: the traced ledger.
- Command:
python scripts/kb_rules.py extract <root> --service-dir <sub> --ledger <tests>/flow-map.yaml --out-dir <tests>- Omit
--service-dirwhen there is no sub-directory. - For every entry whose
rules.sourceshas a file withscanned: false, render the rules prompt for that source, dispatch a read-only rules subagent, save thecsvfield of its JSON reply as<tests>/rules/<slug>-<n>.rows.csv(the header line is included;<n>is the same 1-based source number as the rendered prompt, so an entry with two sources leaves two rows files), then append the rows and mark the source scanned:
python scripts/kb_prompt.py render --prompt rules --ledger <tests>/flow-map.yaml --entry "<id>" --source <source-file> --repo <root> --service-dir <sub> --tests-dir <tests> --out <tests>/.prompts/rules-<slug>-<n>.md
python scripts/kb_rules.py add "<id>" <tests>/rules/<slug>-<n>.rows.csv --ledger <tests>/flow-map.yaml --out-dir <tests>
python scripts/kb_rules.py mark-scanned "<id>" <source-file> --ledger <tests>/flow-map.yamlOmit --service-dir when there is no sub-directory. add de-duplicates on field, mutation
and value and assigns rule_ids; run it once per rows file even when the file is empty of
new rows.
- An entry whose
responsescarry a validation response (rules: true) but whoserules.sourcesis empty still needs a pass, or the Step 6 gate fails with "validation responses but no rules file" and Step 8 cannot fix it: render the rules prompt with--sourceset to that entry'srequest.schema_reffile, thenaddandmark-scannedit exactly as above. - Postcondition: every entry with validation responses has
rules.fileand arules.countmatching its CSV, and every source isscanned: true. The generated gate in Step 6 checks this.
Step 5: Scaffold the module
- Prerequisite: the traced ledger and
<tests>/env-map.json. - Command:
python scripts/kb_scaffold.py <root> --service-dir <sub> --ledger <tests>/flow-map.yaml --env <tests>/env-map.json --out <tests> --migrations-image <ref>Omit --service-dir when there is no sub-directory.
Omit --migrations-image to resolve the image from ~/.karate-bootstrap/config.yaml
(--config <path> points elsewhere). Exit 4 means no schema source: stop and report it.
Add --force only when re-scaffolding a module whose harness files you intend to refresh;
generated content is never overwritten either way.
- Postcondition:
<tests>/pom.xml,<tests>/src/test/resources/kb-runtime.json, the harness classes and<tests>/defects.md. Then:
python scripts/kb_checkpoint.py commit --repo <repo-path> --tests-dir <sub>/karate-tests --phase 5 --message "scaffold the Karate module"Drop --tests-dir when there is no sub-directory; its default is karate-tests.
Step 6: Generate features
- Prerequisite: the scaffolded module and the rules files.
- Loop until
nextprints{"done": true}:
python scripts/flow_map.py next --phase generated --ledger <tests>/flow-map.yaml
python scripts/kb_prompt.py render --prompt generate --ledger <tests>/flow-map.yaml --env <tests>/env-map.json --entry "<id>" --repo <root> --service-dir <sub> --tests-dir <tests> --out <tests>/.prompts/generate-<slug>.md
python scripts/flow_map.py mark --entry "<id>" --generated --feature features/<slug>.feature --stub stubs/<downstream>/default.json --seed seed/<slug>.sql --ledger <tests>/flow-map.yamlOmit --service-dir when there is no sub-directory.
Between render and mark, dispatch the generate subagent (general-purpose, may write only
under <tests>) and save its JSON reply as <tests>/.prompts/generate-<slug>.json. Pass
every path from the reply's features, stubs and seeds lists to mark (repeat
--feature, --stub, --seed as needed; omit a flag whose list is empty). Features are
recorded relative to src/test/resources; mark trims that prefix when a reply includes it.
- Gate:
python scripts/flow_map.py validate --phase generated --ledger <tests>/flow-map.yaml --repo <root> --service-dir <sub> --tests-dir <tests>Omit --service-dir when there is no sub-directory.
Gaps name the entry and what is missing (a feature, a Db. assertion for a written table, a
Jms. assertion for a published destination, a Stubs.verify for an outbound call, a rules
count mismatch, an exclusive-state call without @parallel=false). Re-dispatch the generate
subagent for that entry with the gate's output verbatim under a final
## Gaps from the last gate heading in the rendered prompt, then mark and validate again.
- Postcondition:
validate --phase generatedprintsphase generated: pass. Then:
python scripts/kb_checkpoint.py commit --repo <repo-path> --tests-dir <sub>/karate-tests --phase 6 --message "generate features, stubs and seeds"Drop --tests-dir when there is no sub-directory.
Step 7: First run
- Prerequisite: the generated gate passed.
- Self-test first, from
<tests>:mvn -B test -Dkb.skipContainers=true. It compiles the module and runs the harness smoke feature without containers. A failure here is a scaffold or environment problem (JDK, Maven Central): fix the environment or report exit 7; never edit the Java. - Full run, from
<tests>:mvn -B test(plus-Dapp.image=<tag>when given). The first run builds the app image and can take several minutes. Then:
python scripts/kb_report.py parse --reports <tests>/target/karate-reports --out <tests>/target/report.json
python scripts/flow_map.py record-run --ledger <tests>/flow-map.yaml --report <tests>/target/report.json
python scripts/flow_map.py validate --phase green --ledger <tests>/flow-map.yaml --repo <root> --service-dir <sub> --report <tests>/target/report.json --defects <tests>/defects.md- Omit
--service-dirwhen there is no sub-directory. - A run that produced no feature reports still yields
report.json:parsewrites a single(startup)failure whoseerroris the tail oftarget/app.logortarget/db-manager.log. Carry it into Step 8, classify itinfra, and fix it inside<tests>(wait strategy, image, env) — never in the application.parseexits 5 only when<tests>/targetitself is missing, which means Maven never ran. - Postcondition:
<tests>/target/report.json. When the green gate passes, go to Step 9.
Step 8: Iterate until green or stopped
- Prerequisite:
<tests>/target/report.jsonwith failures. - Loop:
python scripts/kb_iterate.py next --report <tests>/target/report.json --tests-dir <tests>Read the group it prints (the largest failure signature with its evidence) and classify it
with reference/failure-triage.md: infra, stub-or-seed, expectation, or app-defect. Then
log the iteration before changing anything:
python scripts/kb_iterate.py log --log <tests>/.iterations.log --signature "<signature>" --hypothesis "<one sentence>" --change "<one sentence>" --classification <class>Add --unfixable to an infra iteration that cannot be fixed from inside <tests>.
- Make the one change the hypothesis names, inside
<tests>only. For an expectation change, also record it:
python scripts/flow_map.py override --ledger <tests>/flow-map.yaml --entry "<id>" --scenario "<scenario>" --field <what> --old "<expected>" --new "<observed>" --reason "<why>"For a suspected app defect, tag the scenario @known-defect and append the defects.md
entry described in failure-triage.md; no override.
- Re-run the touched feature from
<tests>:mvn -B test -Dkarate.options="classpath:features/<slug>.feature", then the fullmvn -B test, then:
python scripts/kb_report.py parse --reports <tests>/target/karate-reports --out <tests>/target/report.json
python scripts/flow_map.py record-run --ledger <tests>/flow-map.yaml --report <tests>/target/report.json
python scripts/kb_iterate.py check-stop --log <tests>/.iterations.log --report <tests>/target/report.json --max-iterations 15
python scripts/flow_map.py validate --phase green --ledger <tests>/flow-map.yaml --repo <root> --service-dir <sub> --report <tests>/target/report.json --defects <tests>/defects.mdOmit --service-dir when there is no sub-directory.
check-stop prints continue, done, or stop:<reason> with exit 6. Pass the user's
--max-iterations value. On stop: go to Step 9 and finish with exit 6. On done and a
passing green gate, go to Step 9. Otherwise loop.
- Postcondition per iteration: a new line in
<tests>/.iterations.logand a fresh<tests>/target/report.json.
Step 9: Report
- Prerequisite:
<tests>/target/report.jsonand<tests>/defects.md. - Commands:
python scripts/kb_report.py summary --ledger <tests>/flow-map.yaml --defects <tests>/defects.md --report <tests>/target/report.json --template templates/karate-tests/README.md.tmpl --out <tests>/README.md
python scripts/kb_checkpoint.py commit --repo <repo-path> --tests-dir <sub>/karate-tests --phase 9 --message "first ground-truth Karate suite"- Drop
--tests-dirwhen there is no sub-directory. - Postcondition:
<tests>/README.md. Tell the user: the branch and commit, the counts tablesummaryprinted, how many scenarios are quarantined and wheredefects.mdis, the auth and schema modes used, and how to run the suite (cd karate-tests && mvn test). Exit 0 on a passing green gate, 6 after a stop condition.
Token budget
One trace subagent per entry point (a small service has 3 to 10), one rules subagent per validation source, one generate subagent per entry point, then the fix loop. Budget about 30k output tokens per entry point for trace, rules and generate together, and 5 to 15k per fix iteration. Scripts do no LLM work.
Caveats
- Postgres only; ActiveMQ Artemis over AMQP 1.0 only; downstream HTTP is stubbed, never called.
- The generated suite documents current behaviour. A quarantined scenario is a suspected defect for a developer to judge, not a verdict.
- No live LLM in this repo's tests:
kb_check_skill.pylints these commands andtests/test_kb_dry_run.pyexecutes the chain with canned subagent replies. - Windows:
mvnneedsJAVA_HOME; the template test in this repo shows the exact value used.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

