agentleFS
emp3thyemp3thy/claude-skills/skills/karate-bootstrap/SKILL.md

karate-bootstrap

Skill0 starsChanged 35 days agoMITView on GitHub

Bootstrap a first ground-truth Karate integration-test suite that runs green under Testcontainers for a Spring Boot, Quarkus, ASP.NET Core or Python service that has no Karate tests. Use when asked to add karate tests, bootstrap integration tests, or build a testcontainers suite for a service or repo. Not for unit tests.

The file

Show its rules and text

We split this skill into the rules it gives an agent

15 rules. About 0 are common, 2 are distinctive, and 13 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈2

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 13

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    Read reference/testcontainers-notes.md once before a run.

    line 17
  2. Very specific:

    If any expected output file from a numbered step is missing, abort with exit 5.

    line 23
  3. Very specific:

    Run every python scripts/...

    line 48
  4. Distinctive:

    Every step names one command and one output.

    line 66
  5. Very specific:

    Verify app.auth yourself even when discover.py wrote confirmed: true: it guesses disabled from a config key such as app.security.enabled …

    line 144
  6. Very specific:

    Use --mode none when the app has no auth and --mode blocked when it cannot be switched off or pointed at a test issuer.

    line 154
  7. Very specific:

    If merge reports unresolved: N above zero, re-render with the first unresolved location and dispatch again, then merge again.

    line 176
  8. Very specific:

    If merge reports incomplete: no exits and no exits_none_reason (exit 2), the trace found no exit and gave no reason.

    line 183
  9. Very specific:

    When a reply is both unresolved and incomplete, follow the unresolved bullet first: its --focus location is the specific line the tracer …

    line 191
  10. Very specific:

    Run a --focus trace on the file:line of every disagreement: line, merging each result with --union too, before the gate.

    line 205
  11. Distinctive:

    Do not edit the ledger to silence a gap.

    line 219
  12. Very specific:

    Add --force only when re-scaffolding a module whose harness files you intend to refresh.

    line 268
  13. Very specific:

    Read the group it prints (the largest failure signature with its evidence) and classify it with reference/failure-triage.md: infra, …

    line 349
  14. Very specific:

    Add --unfixable to an infra iteration that cannot be fixed from inside <tests>.

    line 357
  15. Very specific:

    No live LLM in this repo's tests: kb_check_skill.py lints these commands and tests/test_kb_dry_run.py executes the chain with canned …

    line 414

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

karate-bootstrap

Takes a service repository with no Karate tests and leaves it with a first "ground truth" suite under karate-tests/ that runs green under Testcontainers, locally and in Azure DevOps. The suite documents the behaviour the service has today; suspected defects are quarantined and reported, never fixed. Scripts do every deterministic step (discovery, the ledger, gates, scaffolding, report parsing, git checkpoints); you do the judgement inside narrow subagent tasks (tracing code paths, confirming validation rules, writing features) and the fix loop.

Read reference/testcontainers-notes.md once before a run. Read the stack cheat sheet the ledger names (reference/stack-<stack>.md) when you dispatch the first trace. Read reference/failure-triage.md before the first fix iteration.

No improvisation

If any expected output file from a numbered step is missing, abort with exit 5. Do not retry steps you were not told to retry. Do not edit flow-map.yaml, rules/*.csv or kb-runtime.json by hand: every change goes through a script. Do not compose a subagent prompt yourself: render it with kb_prompt.py, and add nothing to the rendered text except the one ## Gaps from the last gate section the Subagents table allows. Never edit the application's source, its Dockerfile or its pipeline. Never push.

Invocation

/karate-bootstrap <repo-path> [--service-dir <sub>] [--migrations-image <ref>] [--app-image <tag>]
                  [--max-iterations 15] [--double-trace] [--no-commit]
FlagEffect
--service-dir <sub>The service lives in a sub-directory of the repo (monorepo). Every script takes the repository root as its positional (or --repo) argument and resolves the service under <sub> itself, so pass --service-dir <sub> alongside <root>, never a joined path.
--migrations-image <ref>The db-manager image that owns the schema. Without it the scaffold looks the database up in ~/.karate-bootstrap/config.yaml and aborts with exit 4 when nothing matches.
--app-image <tag>Test a prebuilt image instead of building the Dockerfile: adds -Dapp.image=<tag> to every Maven run.
--max-iterations <n>Fix-loop cap (default 15).
--double-traceTrace every entry twice with independent subagents and merge both results; disagreements get a third, narrower trace.
--no-commitWrite files only; pass --no-commit to every kb_checkpoint.py call so git is never touched.

Conventions

  • Run every python scripts/... command from this skill's directory (skills/karate-bootstrap/) so the relative script paths resolve; scripts are direct-path invocable, no package install. Pass <root> as an absolute path.
  • <root> is <repo-path>: the repository root, and the positional (or --repo) argument of every script. <sub> is the --service-dir value. A script that accepts --service-dir joins the two itself, so never write <root>/<sub>. Omit --service-dir from every command below when the run has no sub-directory.
  • <tests> is <repo-path>/<sub>/karate-tests, or <repo-path>/karate-tests when there is no sub-directory. Every artefact the run produces lives there. Rendered prompts go to <tests>/.prompts/ (ignored by the template's .gitignore).
  • kb_checkpoint.py commit stages a path relative to <repo-path>, not to <tests>: pass --tests-dir <sub>/karate-tests, or drop the flag entirely (its default is karate-tests) when there is no sub-directory.
  • <skill> is this skill's absolute directory; templates/karate-tests/README.md.tmpl is referenced relative to it.
  • Maven runs from <tests>: mvn -B test (JDK 17 or newer and a container engine on the machine; ./mvnw -B test when Maven is not installed). Append -Dapp.image=<tag> when --app-image was given. Podman users: reference/podman.md.
  • Every step names one command and one output. A step's postcondition is the file it must leave behind.

Exit codes

CodeMeaning
0Suite green (or green with quarantined defects listed in defects.md).
2A gate or script rejected its input; the message names the gap.
3Unsupported stack (not Spring Boot, Quarkus, ASP.NET Core or a Python web framework).
4No schema source: no --migrations-image and no matching central config entry.
5An expected output file is missing (the no-improvisation rule).
6Stopped by a stop condition after committing what exists.
7Container runtime, JDK 17+ or Maven missing on this machine.

Subagents

Three kinds, each driven by a prompt file kb_prompt.py renders for one entry. Read the rendered file and pass its complete text as the subagent's prompt; remove nothing. The only permitted addition is a final section ## Gaps from the last gate holding, verbatim, the output that sent you back: the gate's gap list, or merge's incomplete: line. Nothing else may be added, and nothing may be reworded.

KindAgentMay writeReturns
traceread-only (Explore)nothingJSON matching the ledger entry schema
rulesread-only (Explore)nothingJSON with csv (header plus rows), rows, dropped_candidates, notes
generategeneral-purposefiles under <tests> only (features/, stubs/, seed/)JSON listing the files written

Save every JSON reply to <tests>/.prompts/<kind>-<slug>.json before you feed it to a script — one file per kind and slug, except under --double-trace, where the pair for one entry is trace-<slug>-a.json and trace-<slug>-b.json. Dispatch trace subagents one at a time unless --double-trace asks for pairs.

Workflow

Step 0: Preflight

  • Prerequisite: <repo-path> exists.
  • Commands:
python scripts/kb_checkpoint.py begin --repo <repo-path>
python scripts/detect.py <root> --service-dir <sub> --out <tests>/stack.json
  • begin creates and checks out karate-bootstrap when the repo is on its default branch and leaves any other branch alone. Add --no-commit when the flag was given (then it does nothing). detect.py checks the container runtime, JDK and Maven, then classifies the stack; omit --service-dir when there is none. Exit 7 means the machine cannot run Step 7: stop and report what is missing. Add --skip-toolchain only when the user asked for a dry run that stops before Step 7 (Steps 0 to 6 need no toolchain).
  • Postcondition: <tests>/stack.json. Exit 3 or 7 from detect.py ends the run with that code.

Step 1: Discover

  • Prerequisite: <tests>/stack.json.
  • Command:
python scripts/discover.py <root> --service-dir <sub> --stack <tests>/stack.json --out-env <tests>/env-map.json --out-ledger <tests>/flow-map.yaml
  • Omit --service-dir when there is no sub-directory.
  • Postcondition: <tests>/env-map.json (config keys with roles, port, readiness, auth mode) and <tests>/flow-map.yaml seeded with one untraced entry per entry point.

Step 2: Confirm entry points and auth

  • Prerequisite: the two files from Step 1.
  • Open <tests>/flow-map.yaml and the routes in the code once. For every entry point the regexes missed (a route, a listener), add it:
python scripts/flow_map.py add-entry --ledger <tests>/flow-map.yaml --id "<METHOD> <path>" --kind http --handler <file:line> --method <METHOD> --path <path>
python scripts/flow_map.py add-entry --ledger <tests>/flow-map.yaml --id "amq <destination>" --kind amq-subscribe --handler <file:line> --destination <destination> --type queue
  • Verify app.auth yourself even when discover.py wrote confirmed: true: it guesses disabled from a config key such as app.security.enabled without proving the code reads it. Grep the source for that key (cheat sheet: "Auth switches"). When nothing reads it, or the mode is otherwise wrong, read the security configuration and record what you found:
python scripts/flow_map.py set-auth --ledger <tests>/flow-map.yaml --mode disabled --key <ENV_VAR> --value <off-value>
python scripts/flow_map.py set-auth --ledger <tests>/flow-map.yaml --mode jwks --issuer-keys <ISSUER_ENV>,<JWKS_ENV>

Use --mode none when the app has no auth and --mode blocked when it cannot be switched off or pointed at a test issuer.

  • Postcondition: <tests>/flow-map.yaml lists every entry point and app.auth.mode is one you verified against the code (disabled additionally carries key, value and confirmed: true; jwks carries keys; none and blocked carry only the mode).

Step 3: Trace every entry point

  • Prerequisite: <tests>/flow-map.yaml, <tests>/env-map.json.
  • Loop until next prints {"done": true}:
python scripts/flow_map.py next --phase traced --ledger <tests>/flow-map.yaml
python scripts/kb_prompt.py render --prompt trace --ledger <tests>/flow-map.yaml --env <tests>/env-map.json --entry "<id>" --repo <root> --service-dir <sub> --out <tests>/.prompts/trace-<slug>.md
python scripts/flow_map.py merge <tests>/.prompts/trace-<slug>.json --ledger <tests>/flow-map.yaml

Between render and merge, dispatch the trace subagent with the rendered prompt and save its reply as <tests>/.prompts/trace-<slug>.json (merge accepts the bare object or one wrapped in a code fence). Omit --service-dir when there is no sub-directory. next prints the entry id, its handler and the cheat sheet path; <slug> is the id lower-cased with non-alphanumerics collapsed to - (POST /api/deals becomes post-api-deals).

  • If merge reports unresolved: N above zero, re-render with the first unresolved location and dispatch again, then merge again; repeat until the entry merges with zero unresolved:
python scripts/kb_prompt.py render --prompt trace --ledger <tests>/flow-map.yaml --env <tests>/env-map.json --entry "<id>" --repo <root> --service-dir <sub> --focus <file:line> --out <tests>/.prompts/trace-<slug>-2.md
  • If merge reports incomplete: no exits and no exits_none_reason (exit 2), the trace found no exit and gave no reason. Re-render with --focus at the entry's handler line, dispatch once more with the incomplete: line under a final ## Gaps from the last gate heading in the rendered prompt, and merge again. If it comes back incomplete a second time, leave the entry untraced and let the gate name it: do not write it into defects.md (that file is for suspected application defects found by a run, not for tracing gaps), and do not edit the ledger. validate --phase traced already fails on traced: false, which is the correct end state for an entry nobody could trace. When a reply is both unresolved and incomplete, follow the unresolved bullet first: its --focus location is the specific line the tracer stopped at, and a re-trace from there usually resolves both.
  • With --double-trace: dispatch two independent subagents from the same rendered prompt and save their replies as <tests>/.prompts/trace-<slug>-a.json and -b.json. Merge the first, then merge the second with --union, which keeps the union of exits, reads and responses (an exit that differs only in via is the same exit; the first via is kept) and prints one disagreement: <kind> <what> via <file:line> line per item only one reply saw:
python scripts/flow_map.py merge <tests>/.prompts/trace-<slug>-a.json --ledger <tests>/flow-map.yaml
python scripts/flow_map.py merge <tests>/.prompts/trace-<slug>-b.json --ledger <tests>/flow-map.yaml --union

Run a --focus trace on the file:line of every disagreement: line, merging each result with --union too, before the gate. One --focus round per disagreement is the cap: merge each result with --union, then run the gate. A via still seen by only one trace after that round is a disagreement the gate's verify-refs check adjudicates, not grounds for a third trace.

  • Gate:
python scripts/flow_map.py validate --phase traced --ledger <tests>/flow-map.yaml --repo <root> --service-dir <sub> --env <tests>/env-map.json

Omit --service-dir when there is no sub-directory. The gate also runs verify-refs: every exit's via must sit within three lines before or after a marker token from the cheat sheet, or the entry is reset to untraced. On any gap, go back to the loop for the entries it names. Do not edit the ledger to silence a gap.

  • Postcondition: validate --phase traced prints phase traced: pass.

Step 4: Validation rules

  • Prerequisite: the traced ledger.
  • Command:
python scripts/kb_rules.py extract <root> --service-dir <sub> --ledger <tests>/flow-map.yaml --out-dir <tests>
  • Omit --service-dir when there is no sub-directory.
  • For every entry whose rules.sources has a file with scanned: false, render the rules prompt for that source, dispatch a read-only rules subagent, save the csv field of its JSON reply as <tests>/rules/<slug>-<n>.rows.csv (the header line is included; <n> is the same 1-based source number as the rendered prompt, so an entry with two sources leaves two rows files), then append the rows and mark the source scanned:
python scripts/kb_prompt.py render --prompt rules --ledger <tests>/flow-map.yaml --entry "<id>" --source <source-file> --repo <root> --service-dir <sub> --tests-dir <tests> --out <tests>/.prompts/rules-<slug>-<n>.md
python scripts/kb_rules.py add "<id>" <tests>/rules/<slug>-<n>.rows.csv --ledger <tests>/flow-map.yaml --out-dir <tests>
python scripts/kb_rules.py mark-scanned "<id>" <source-file> --ledger <tests>/flow-map.yaml

Omit --service-dir when there is no sub-directory. add de-duplicates on field, mutation and value and assigns rule_ids; run it once per rows file even when the file is empty of new rows.

  • An entry whose responses carry a validation response (rules: true) but whose rules.sources is empty still needs a pass, or the Step 6 gate fails with "validation responses but no rules file" and Step 8 cannot fix it: render the rules prompt with --source set to that entry's request.schema_ref file, then add and mark-scanned it exactly as above.
  • Postcondition: every entry with validation responses has rules.file and a rules.count matching its CSV, and every source is scanned: true. The generated gate in Step 6 checks this.

Step 5: Scaffold the module

  • Prerequisite: the traced ledger and <tests>/env-map.json.
  • Command:
python scripts/kb_scaffold.py <root> --service-dir <sub> --ledger <tests>/flow-map.yaml --env <tests>/env-map.json --out <tests> --migrations-image <ref>

Omit --service-dir when there is no sub-directory. Omit --migrations-image to resolve the image from ~/.karate-bootstrap/config.yaml (--config <path> points elsewhere). Exit 4 means no schema source: stop and report it. Add --force only when re-scaffolding a module whose harness files you intend to refresh; generated content is never overwritten either way.

  • Postcondition: <tests>/pom.xml, <tests>/src/test/resources/kb-runtime.json, the harness classes and <tests>/defects.md. Then:
python scripts/kb_checkpoint.py commit --repo <repo-path> --tests-dir <sub>/karate-tests --phase 5 --message "scaffold the Karate module"

Drop --tests-dir when there is no sub-directory; its default is karate-tests.

Step 6: Generate features

  • Prerequisite: the scaffolded module and the rules files.
  • Loop until next prints {"done": true}:
python scripts/flow_map.py next --phase generated --ledger <tests>/flow-map.yaml
python scripts/kb_prompt.py render --prompt generate --ledger <tests>/flow-map.yaml --env <tests>/env-map.json --entry "<id>" --repo <root> --service-dir <sub> --tests-dir <tests> --out <tests>/.prompts/generate-<slug>.md
python scripts/flow_map.py mark --entry "<id>" --generated --feature features/<slug>.feature --stub stubs/<downstream>/default.json --seed seed/<slug>.sql --ledger <tests>/flow-map.yaml

Omit --service-dir when there is no sub-directory. Between render and mark, dispatch the generate subagent (general-purpose, may write only under <tests>) and save its JSON reply as <tests>/.prompts/generate-<slug>.json. Pass every path from the reply's features, stubs and seeds lists to mark (repeat --feature, --stub, --seed as needed; omit a flag whose list is empty). Features are recorded relative to src/test/resources; mark trims that prefix when a reply includes it.

  • Gate:
python scripts/flow_map.py validate --phase generated --ledger <tests>/flow-map.yaml --repo <root> --service-dir <sub> --tests-dir <tests>

Omit --service-dir when there is no sub-directory. Gaps name the entry and what is missing (a feature, a Db. assertion for a written table, a Jms. assertion for a published destination, a Stubs.verify for an outbound call, a rules count mismatch, an exclusive-state call without @parallel=false). Re-dispatch the generate subagent for that entry with the gate's output verbatim under a final ## Gaps from the last gate heading in the rendered prompt, then mark and validate again.

  • Postcondition: validate --phase generated prints phase generated: pass. Then:
python scripts/kb_checkpoint.py commit --repo <repo-path> --tests-dir <sub>/karate-tests --phase 6 --message "generate features, stubs and seeds"

Drop --tests-dir when there is no sub-directory.

Step 7: First run

  • Prerequisite: the generated gate passed.
  • Self-test first, from <tests>: mvn -B test -Dkb.skipContainers=true. It compiles the module and runs the harness smoke feature without containers. A failure here is a scaffold or environment problem (JDK, Maven Central): fix the environment or report exit 7; never edit the Java.
  • Full run, from <tests>: mvn -B test (plus -Dapp.image=<tag> when given). The first run builds the app image and can take several minutes. Then:
python scripts/kb_report.py parse --reports <tests>/target/karate-reports --out <tests>/target/report.json
python scripts/flow_map.py record-run --ledger <tests>/flow-map.yaml --report <tests>/target/report.json
python scripts/flow_map.py validate --phase green --ledger <tests>/flow-map.yaml --repo <root> --service-dir <sub> --report <tests>/target/report.json --defects <tests>/defects.md
  • Omit --service-dir when there is no sub-directory.
  • A run that produced no feature reports still yields report.json: parse writes a single (startup) failure whose error is the tail of target/app.log or target/db-manager.log. Carry it into Step 8, classify it infra, and fix it inside <tests> (wait strategy, image, env) — never in the application. parse exits 5 only when <tests>/target itself is missing, which means Maven never ran.
  • Postcondition: <tests>/target/report.json. When the green gate passes, go to Step 9.

Step 8: Iterate until green or stopped

  • Prerequisite: <tests>/target/report.json with failures.
  • Loop:
python scripts/kb_iterate.py next --report <tests>/target/report.json --tests-dir <tests>

Read the group it prints (the largest failure signature with its evidence) and classify it with reference/failure-triage.md: infra, stub-or-seed, expectation, or app-defect. Then log the iteration before changing anything:

python scripts/kb_iterate.py log --log <tests>/.iterations.log --signature "<signature>" --hypothesis "<one sentence>" --change "<one sentence>" --classification <class>

Add --unfixable to an infra iteration that cannot be fixed from inside <tests>.

  • Make the one change the hypothesis names, inside <tests> only. For an expectation change, also record it:
python scripts/flow_map.py override --ledger <tests>/flow-map.yaml --entry "<id>" --scenario "<scenario>" --field <what> --old "<expected>" --new "<observed>" --reason "<why>"

For a suspected app defect, tag the scenario @known-defect and append the defects.md entry described in failure-triage.md; no override.

  • Re-run the touched feature from <tests>: mvn -B test -Dkarate.options="classpath:features/<slug>.feature", then the full mvn -B test, then:
python scripts/kb_report.py parse --reports <tests>/target/karate-reports --out <tests>/target/report.json
python scripts/flow_map.py record-run --ledger <tests>/flow-map.yaml --report <tests>/target/report.json
python scripts/kb_iterate.py check-stop --log <tests>/.iterations.log --report <tests>/target/report.json --max-iterations 15
python scripts/flow_map.py validate --phase green --ledger <tests>/flow-map.yaml --repo <root> --service-dir <sub> --report <tests>/target/report.json --defects <tests>/defects.md

Omit --service-dir when there is no sub-directory. check-stop prints continue, done, or stop:<reason> with exit 6. Pass the user's --max-iterations value. On stop: go to Step 9 and finish with exit 6. On done and a passing green gate, go to Step 9. Otherwise loop.

  • Postcondition per iteration: a new line in <tests>/.iterations.log and a fresh <tests>/target/report.json.

Step 9: Report

  • Prerequisite: <tests>/target/report.json and <tests>/defects.md.
  • Commands:
python scripts/kb_report.py summary --ledger <tests>/flow-map.yaml --defects <tests>/defects.md --report <tests>/target/report.json --template templates/karate-tests/README.md.tmpl --out <tests>/README.md
python scripts/kb_checkpoint.py commit --repo <repo-path> --tests-dir <sub>/karate-tests --phase 9 --message "first ground-truth Karate suite"
  • Drop --tests-dir when there is no sub-directory.
  • Postcondition: <tests>/README.md. Tell the user: the branch and commit, the counts table summary printed, how many scenarios are quarantined and where defects.md is, the auth and schema modes used, and how to run the suite (cd karate-tests && mvn test). Exit 0 on a passing green gate, 6 after a stop condition.

Token budget

One trace subagent per entry point (a small service has 3 to 10), one rules subagent per validation source, one generate subagent per entry point, then the fix loop. Budget about 30k output tokens per entry point for trace, rules and generate together, and 5 to 15k per fix iteration. Scripts do no LLM work.

Caveats

  • Postgres only; ActiveMQ Artemis over AMQP 1.0 only; downstream HTTP is stubbed, never called.
  • The generated suite documents current behaviour. A quarantined scenario is a suspected defect for a developer to judge, not a verdict.
  • No live LLM in this repo's tests: kb_check_skill.py lints these commands and tests/test_kb_dry_run.py executes the chain with canned subagent replies.
  • Windows: mvn needs JAVA_HOME; the template test in this repo shows the exact value used.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.