agentleFS
marksmayomarksmayo/qa-skills/qa-orchestrator-skill/SKILL.md

qa-orchestrator

Skill1 starsChanged yesterdayMITView on GitHub

Take a change through the QA gates - Ready, Designed, Built and tested, Verified in CI, Release ready, Deployed, Hypercare (configurable) - by running the right skill at each step, recording each report with its SHA-256 and its own validator's result, evaluating the gate's exit criteria, and asking the named person to approve, approve with an override, or reject before moving on. Run state lives in a temporary folder; a script keeps the state and refuses an approval without evaluated criteria, a person, or (when a criterion failed) an override reason. Never auto-approves unless the configuration says so, never overrides a step skill's own rules, and never calls QA complete.

3 other projects use a version of this.

Adopted by

Most recently taken up by

The file

We split this skill into the rules it gives an agent

4 rules. About 0 are common, 2 are distinctive, and 2 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈2

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 2

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    Step skills keep their own rules: Run each step by its skill's SKILL.md: the CI suite runner still needs its trigger confirmation, the …

    line 34
  2. Distinctive:

    On rejection: stop; say what would reopen the gate (a fresh report, a fix).

    line 62
  3. Distinctive:

    Summarise: when asked or at the end.

    line 63
  4. Very specific:

    The person approves a gate with failed criteria: ask for the reason and pass it as --override.

    line 84

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

QA orchestrator

Mission

Sequence the QA skills across a change's gates and keep a person in the loop:

GateSteps (default)Exit criteria (default)
G1 Readyticket-readiness-reviewerreview valid; verdict ready or ready with questions
G2 Designedtest-case-designer (risk-based-test-planner optional)cases valid; every design row covered
G3 Built and testedticket-to-tdd, code-security-auditorreports valid; no P0 security finding
G4 Verified in CIci-suite-runner (ci-failure-triager optional)run valid; no failed tests
G5 Release readyrelease-readiness-gategate report valid; verdict go
G6 Deployedpost-deploy-validatorreport valid; recommendation go
G7 Hypercarehypercare-anomaly-watcherwatch valid; no incident drafts

The gates, steps, criteria and approvers are in data/orchestrator.json and can be changed per team.

Non-negotiable rules

  1. A person approves each gate (who and how), unless the config turns on auto_approve for that gate; an auto-approval is recorded as such. Off by default everywhere.
  2. Criteria before approval. Evaluate first. A failed criterion needs an override reason to approve; a rejection needs a reason and stops the run until someone reopens the gate.
  3. In order. A gate starts only after the previous one is approved or skipped (with a reason).
  4. Reports are evidence: a step counts only with its report recorded (SHA-256) and its validator exiting 0; a report changed afterwards must be recorded again.
  5. Step skills keep their own rules. Run each step by its skill's SKILL.md: the CI suite runner still needs its trigger confirmation, the post-deploy validator stays read-only, the release gate never calls a release safe. A gate approval is not any of those confirmations.
  6. State in a temporary folder (<temp>/qa-orchestrator/<run id>/), not the repository; export a summary only when asked.
  7. No "QA complete" or "approved by QA": the summary lists each gate's decision, who made it, and on what evidence.

Workflow

python scripts/orchestrate.py init --subject "SHOP-310 for release 2026.12" [--config orchestrator.json]
python scripts/orchestrate.py next --state <dir>
#   run the step's skill (its SKILL.md), then:
python scripts/orchestrate.py record --state <dir> --gate G1 --step readiness --report review.json
python scripts/orchestrate.py evaluate --state <dir> --gate G1
#   ask the gate's approver; record their words:
python scripts/orchestrate.py approve --state <dir> --gate G1 --by "QA lead" --how "in chat" [--override "why"]
python scripts/orchestrate.py reject --state <dir> --gate G5 --by "Release manager" --how "release meeting" --reason "..."
python scripts/validate_run.py <dir>
python scripts/render_report.py <dir> [--markdown summary.md]
  1. Init: the subject (ticket, release, repository, commit) and the config; start later with --start-gate and --skip-reason when earlier gates were done elsewhere.
  2. For each gate: next; run each step's skill; record its report; evaluate; show the person the criteria results and the reports' headlines; record their decision exactly as given. Never decide for them, and never treat silence or a general "carry on" as an approval of a gate with failed criteria.
  3. On rejection: stop; say what would reopen the gate (a fresh report, a fix).
  4. Summarise when asked or at the end.

Rating

See RUBRIC.md: gate statuses, criteria kinds, decisions.

Output

<state>/state.json (schemas/run_state.schema.json): subject, config and its hash, gates (status, steps with report, hash and validator result, criteria results, decision, history) and the event log.

Edge cases

  • A step's report comes from CI or another person: record it the same way; its validator still runs.
  • A validator needs its skill's folder: set the step's cwd in the config.
  • Validator commands are never taken from a state file. record and validate_run.py --rerun rebuild each command from the config file (it must still have the SHA-256 recorded at init) and run it only when it is python <skill>/scripts/validate_*.py ... inside the trusted skills folder (the folder holding this skill, or --skills-root), with this Python. --rerun also needs every recorded command to equal the rebuilt one and every report unchanged; otherwise it refuses (exit 2) before running anything. A state folder someone else wrote is data: validate it, but never edit its commands to make a rerun pass.
  • The person approves a gate with failed criteria: ask for the reason and pass it as --override.
  • Reopening: reopen with who and why; the earlier decision stays in the gate's history.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.