qa
Skill0 starsChanged 8 days agoMITView on GitHub
Systematically test a web application, fix issues caused by the current change, and re-verify. Use for full QA loops, critical user flows, release checks, or test-and-fix requests.
- Reads credentials
In this repository27 other files it gives its agents.
- CLAUDE.mdCLAUDE.md
- llms.txtllms.txt
- agent-first-docsSkill
- analytics-reviewSkill
- autopilotSkill
- browseSkill
- create-prSkill
- design-uiSkill
- foremanSkill
- harness-auditSkill
- image-genSkill
- implementSkill
- investigateSkill
- marketing-videoSkill
- plan-draftSkill
- reconSkill
- release-prdSkill
- releaseSkill
- review-prSkill
- semantic-decisionSkill
- setup-projectSkill
- teamSkill
- telemetry-reviewSkill
- testSkill
- token-auditSkill
- triageSkill
- ui-auditSkill
The file
We split this skill into the rules it gives an agent
10 rules. About 0 are common, 6 are distinctive, and 4 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈6
Distinctive · estimate
This author's own choices. Worth a look.
4
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 53
Before launching anything, record the pre-existing target server and automation resources, then keep an ownership ledger for every process …
- Very specific: line 175
Run the cleanup finalizer before persisting the verdict or returning any status, including BLOCKED and tool-error paths.
- Very specific: line 195
Use the browser session name the assignment gives.
- Very specific: line 197
Write the full run report — per case: steps, observed result, evidence paths.
- Distinctive: line 225
Do not leave a process running solely for human review.
- Distinctive: line 251
Keep test data reversible.
- Distinctive: line 257
Do not report a fix until the original reproducer passes.
- Distinctive: line 262
Prefer deep checks over many vague clicks.
- Distinctive: line 268
Every QA summary names the local target or blocker, the case matrix, and at least one non-happy-path result.
- Very specific: line 286
When a ticket resolves, persist the full block (RUBRIC, EVIDENCE, and CLEANUP included) — reviews/qa.md is the report a human or …
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
qa
Exercise the application like a user and leave reproducible evidence.
Flow
- Load what to test — disk first, conversation as fallback. Resolve the
ticket (
bbs ticket resolve); when it resolves, read whichever exist:bbs ticket path requirement --read— acceptance criteriabbs ticket path plan --read— intended behavior, accepted scope, contracts/invariants,**Unknowns:**,**Design:**and**Verify:**- For changed UI, the Task spec's
## Prototypesection (Team), else the plan's design/prototype links orbbs ticket path design --read— inspect the accepted prototype and covered flows/states; presence of a path alone is not a comparison bbs ticket path handoff --skill implement --latest --read— and its## Deviations: each deviation is where the plan diverged from reality, the likeliest home of a wrong guess; also read findings/unknown outcomes and prototype-comparison evidence, checking their relevance and freshnessbbs ticket path verdict --skill review-pr --read— unresolvedFINDINGSgo into the case matrix;RISK_AREAS/FIXEDseed security and regression cases Missing docs are not a gate — fall back to conversation. Change surface:BASE=$(bbs autopilot base-branch);git diff $(git merge-base origin/"$BASE" HEAD). Target URL and login from.babysit/qa.yaml:
Target iseval "$(bbs secrets load | sed -E '/^export CLOUDFLARE_(ACCOUNT_ID|API_TOKEN)=/d')" # exports .babysit/.env values ENV=$(bbs secrets qa default-env); ENV=${ENV:-local} eval "$(bbs secrets qa probe --env "$ENV" 2>/dev/null)" # QA_ENV_URL, QA_ENV_{USERNAME,PASSWORD}_ENV, … QA_USER=$(printenv "${QA_ENV_USERNAME_ENV:-QA_USER}" 2>/dev/null || true) # standard: QA_USER / QA_PASS QA_PASS=$(printenv "${QA_ENV_PASSWORD_ENV:-QA_PASS}" 2>/dev/null || true)$QA_ENV_URL; sign in with$QA_USER/$QA_PASS. A needed credential resolving empty means the value is missing from.babysit/.env— record that as the local-run blocker rather than testing signed-out. No.babysit/qa.yamlis not a gate: derive the target from conversation or the repo and note how credentials were obtained.NEEDS_CONTEXTonly when neither docs nor conversation can name a single intended behavior to verify. - Boot or probe the local target first; hosted URLs only when local run is
impossible and the reason is recorded. QA the current checkout as-is —
normally the branch the user is already on, whose dev server is the one
running: nothing to land or compose. If the local target is down
and can't be started here, that's the recorded blocker. Server prep: when
QA_ENV_PREPAREis set (qa.yamlprepare:, idempotent install + migrate), run it before probing. Before trusting any surface, confirm it actually serves the change (probe a marker from the diff); if not, name the stale surface rather than testing blind. Fixes edit the files in the checkout under test — committing them stays the invoking workflow's job — re-verify on the updated surface. Before launching anything, record the pre-existing target server and automation resources, then keep an ownership ledger for every process this attempt starts (server handle/PID, browser session/namespace, simulator UDID, Appium or other driver). Prefer supervised process handles over detached shell jobs. Ownership, not process name, decides what cleanup may stop. Team integration mode: a Task explicitly assigned a phase's integration worktree (detached at the phase commitbbs team integratebuilt) plus the cumulative acceptance (every criterion of this phase, and the earlier ones its diff touches; the lead carries the rest) tests that exact commit read-only, and runsbbs contract check <id>there for each contract the phase changed (contracts_changedin the integrate output). Verify the worktree is clean at that commit before and after testing. Do not edit the integration worktree, commit or switch revisions while testing. Once the attempt is recorded, the session fixes every finding itself in the owning ticket's worktree per the lead's phases reference (integratetakes the fix,land, re-test what changed; a conflict that integrate lays out in the integration worktree is resolved there before its commit exists); nothing goes back to the lead as a repair, and only a fix beyond the approved scope endsBLOCKED. Persist evidence/verdict on the parent. Inside a ticket worktree, install and serve from that worktree with the ports, databases and secretsqa.yamlnames. When the run uses a resource only one ticket can hold at a time, takebbs ticket surface acquirefirst, runQA_ENV_REVERTafter, andsurface releaseon every outcome — see../references/worktrees.md. Shared refs (../references/*.md) are filesystem paths beside this skill's directory, so read them by path, not asskill://. - Code-level checks (tests, typecheck, lint) first — they gate, they don't
prove user journeys. Read test for affected-test
selection and failure investigation. Use the implementation handoff as
evidence, then check its coverage against the current change and callers.
Reuse automated results only when they cover the same tested code state
(including uncommitted changes), environment/prerequisites and required
scope, and no mandatory gate requires a fresh run. Otherwise run the
affected lanes and required checks. Never weaken or skip a failing test
to obtain green; follow
test's investigation and authorization rules. Whole-suiteauditandoptimizeare not routine QA steps. - Size the matrix per § Case count.
Trace material plan findings/unknowns to acceptance cases or supporting
evidence, including resolved discoveries that changed an assumption. Reuse
evidence only when code state, environment and coverage still match; an
implementation claim alone is not proof.
Carry relevant open questions into checks with observable expected results;
non-runtime questions use the appropriate contract/code evidence. Record a
reason for items outside this change's acceptance, without reopening every
resolved discovery. Include accepted prototype flows/states for changed UI.
Build a flow matrix from the acceptance criteria — not just the diff —
covering happy path, validation, empty/error states, failure/retry, and
responsive behavior. Three axes escaped past QA most and are mandatory
wherever the change reaches them:
- Ship surface — each case names the surface that ships (packaged
app shell, live provider/sandbox payload, prod-shaped schema, real
agent path) and runs there. A fixture, stub, browser-mode build, dev-only
DB or implementer-written payload in its place is a named gap, and work
deferred to hypercare goes into
SUMMARYas a release-checklist item. - Actor — changed UI runs signed-out and as a lower-privilege user; where session/credential state is shared, add one wrong-credential or expired-session case.
- Hard-to-reach states — every dialog, confirm or dirty-state guard
gets cancel and Escape; background/hidden-window, offline and timing
states are probed. A state the fixture cannot reach is forced by
injection or named as a gap — unit tests alone do not cover a case.
Derive the change's reach independently: for each
changed file/function, find callers and flows sharing its state or
routes, and give each an adjacent-regression case (BLAST_RADIUS is the
producer's own claim). Write the matrix before touching the app, each
case anchored to its source ("criterion 2", "review-pr finding 1",
"derived: shares session state"). Self-review: every criterion,
BLAST_RADIUS entry, unresolved review finding, and implement deviation
has a case; each axis above the change reaches has its case or a named
gap; material plan findings/unknowns have evidence or a named gap;
each touched flow has ≥1 non-happy-path case; behavior the
code walk surfaced that the requirement never mentions gets a derived
criterion case with the gap named in
SUMMARY; an uncoverable criterion is named as a gap now, never silently dropped. Save the matrix:bbs ticket path evidence --skill qa --name test-matrix.md --write. Automated coverage never replaces the executed journeys below.
- Ship surface — each case names the surface that ships (packaged
app shell, live provider/sandbox payload, prod-shaped schema, real
agent path) and runs there. A fixture, stub, browser-mode build, dev-only
DB or implementer-written payload in its place is a named gap, and work
deferred to hypercare goes into
- Execute the flows end-to-end with a real client. Whenever the harness has
a subagent tool, steps 2, 5, 7 and 8 run in one Executor mode
subagent with a fresh context — in every mode, Team integration
included; a caller's own executor assignment (autopilot § Current-session
gates) satisfies this. This session keeps only the matrix, the returned
block and the failures it triages; run inline only when no subagent tool
exists or the executor cannot reach the target, and record why. Web UI:
the
browseengine — Read only../browse/SKILL.md§ Engine before the first browser command; its session-name export and setup are mandatory. Non-UI: a real call sequence (curl, CLI, the repo's e2e suite). Never "test" a flow by reading code or unit tests alone. Prototype conformance. For changed UI, every accepted prototype state the change owns is a case — all of them, not a sample (a Team spec's## Prototypesection lists them with links). Per state: open its link, element-screenshot it, then drive the real build with data that produces that state and screenshot it at the same viewport; also run each screen with 80-character user-named strings and at its narrowest supported width, even where the prototype omits them. Compare layout, interactions, content hierarchy, overflow and empty/error behavior, and saveprototype-conformance.md: one row per state — both screenshot paths,match/fixed/deviates, one sentence on where it deviates. The prototype is the standard: fix the build and re-shoot; never edit the prototype to match the build. A state leftdeviatesneeds an accepted reason tied to a criterion; one not checked, or whose reference will not load, is a conformance gap that blocks PASS. This is acceptance verification, not pixel equality or a new design exercise. Off-script time around the changed surface — findings feed back as derived cases — a few minutes, more when § Case count makes it mandatory. - Fix regressions owned by the current branch. Use
test's scopedcreateworkflow to decide whether the fix needs a regression test; existing meaningful coverage may suffice. Rerun the original reproducer, affected flow and checks on the repaired code. - Finish with one full end-to-end pass of the primary user journey on the
final code state — any code change after it invalidates the verdict.
Screenshot this verdict-bearing pass — and each failure or fixed
reproducer — to
bbs ticket path evidence --skill qa --name <f>.png --write; list the paths inEVIDENCE:. (Ad-hocbrowsechecks stay screenshot-light; the QA verdict's screenshots are the durable proof a human audits later.) Summarize prototype conformance and material finding/unknown outcomes as confirmed, corrected or open, linked to case/evidence and acceptance impact. Surface discoveries that change the human's understanding; do not bury them in logs. Required unknowns or conformance checks still unverified are coverage gaps and prevent PASS; unrelated follow-ups may remain nonblocking concerns. Run the cleanup finalizer before persisting the verdict or returning any status, including
BLOCKEDand tool-error paths. Thebrowseskill closes and verifies its exact browser session plus any simulator it booted. Stop every server, driver, recorder, proxy, or watcher this QA attempt started, and wait for exit; leave pre-existing targets and coordinator-owned shared surfaces untouched. When this attempt took the surface lease, also run the requiredQA_ENV_REVERTand release it.Compare the final process state with the ownership ledger. Never use broad
pkill/killall,agent-browser close --all, or simulator-wide shutdown as routine cleanup. If an owned process survives a scoped stop, retry once, record the exact process/resource and cleanup command, and returnSTATUS: BLOCKED; a passing product flow does not excuse a leaked automation fleet.
Executor mode
An assignment that names this mode, a matrix path and a run number (autopilot
§ Current-session gates) runs only the mechanical part: step 2 (boot/probe the
target, ownership ledger), the check commands the matrix lists, step 5 for the
assigned cases, step 7's final primary-journey pass when assigned, and step 8.
Use the browser session name the assignment gives. Edit no code, run no git,
write no gate, verdict or set-review; a failure is reported, never fixed.
Write the full run report — per case: steps, observed result, evidence paths;
per check: argv, cwd, exit code, log path — to
bbs ticket path evidence --skill qa --name run-<n>.md --write, and the
checks as verification record JSON beside it (run-<n>.json,
verification.md). Return only:
STATUS: DONE | BLOCKED
CASES: <n passed>/<n run>; failed: <case ids or none>
FAILURES: <one line per failed case: id — observed vs expected — evidence path>
RUBRIC: <proposed grades from what was executed>
REPORT: <run-<n>.md path> <run-<n>.json path>
CLEANUP: <owned resources stopped and verified; retained ones named>BLOCKED means the run itself could not execute (target down, missing
credential, tool failure), named in FAILURES.
Case design
One case = user journey + expected observable + evidence — as a user, do
<steps> → observe <result>, not a component check. The primary
journey delivers the feature's core value end-to-end; it carries the
verdict and runs last. Tie every failure to the criterion it violates, with
reproduction steps.
Case count
5–10 cases: the full matrix from the acceptance criteria plus
adjacent-regression cases. When the diff touches auth, money or data paths,
security, a11y and compat must carry a real grade — an N/A there needs
a reason that would survive review — off-script exploration is mandatory, and
security-review runs too.
Do not leave a process running solely for human review. The screenshots and evidence are the handoff: stop a target this QA attempt started, but preserve and name a target that was already running. Always revert and release a surface lease this attempt took.
Coverage rubric
Grade every dimension A–D against the change's risk surface; a dimension the
change can't touch is N/A with a one-line reason — never a silent skip.
- flow: A = happy + every alternate/branch flow e2e · B = happy + ≥2 alternates · C = happy only, or the primary journey ran on a substitute for the ship surface · D = none e2e
- boundary: A = limits, invalid input, empty/error, failure/retry all exercised · B = ≥1 boundary + ≥1 error · C = mentioned, not executed · D = none
- regression: A = existing behavior around the change re-verified · B = adjacent flow spot-checked · C = assumed intact · D = not considered
- data: A = state correct across reload/nav/concurrent edits · B = persists across reload · C = not checked · D = loss/corruption seen
- compat: A = target browsers + responsive breakpoints · B = one extra viewport · C = default viewport only · D = broken layout seen
- security: A = permission gates + input safety (authz, injection) probed · B = auth-required paths checked · C = noted, not tested · D = access-control gap seen
- a11y: A = keyboard path, labels/roles, contrast, clear error copy · B = keyboard + visible focus · C = not checked · D = blocking defect
- perf: A = responsive under realistic data volume · B = no obvious lag on happy path · C = not observed · D = timeout/jank seen
- freshness (always applies): A = final full e2e pass on the final code state · B = e2e passed but code changed after · C = partial/stale e2e · D = unit/curl/code-read only
Grades are letters
A–DorN/A: <reason>; anything else ("ok", "pass") is a missing grade.PASS/FIXEDrequire every applicable dimension at B or better and freshness at A. Any applicable dimension at C or D, missing required coverage or failed acceptance forcesVERDICT: FAIL(→STATUS: BLOCKED).DONE_WITH_CONCERNSis only for nonblocking residuals once all required acceptance and coverage pass. Report the grade line for every dimension, includingN/A: <reason>.
Rules
- Keep test data reversible; no destructive production actions.
- Distinguish current-change bugs from pre-existing failures — and only call
one pre-existing after watching it fail at the base commit (a scratch
git worktree addkeeps this tree untouched). Arguing from how old the code looks is not evidence; "pre-existing" is the most common sentence a real regression ships behind. - Do not report a fix until the original reproducer passes.
PASS/FIXEDrequire an executed end-to-end run as the most recent evidence — tool, journey steps, observed result — and the app proven running locally (or the local-run blocker named). Happy-path-only is not a PASS.- Prefer deep checks over many vague clicks.
- Turns are the cost: every turn re-pays the whole context. Batch each case group into one script that prints one pass/fail line per case; on a failure print only its last 30 lines. Write the matrix, scripts and evidence once and reference them by path — never re-read or re-run a whole script to inspect one failure. Prefer a scoped snapshot to a full page.
- Every QA summary names the local target or blocker, the case matrix, and at least one non-happy-path result.
VERDICT: FAILalways pairs withSTATUS: BLOCKED— neverDONE*. The PR gate treats anyDONE*status as ready; a failing QA that reportsDONE_WITH_CONCERNSsilently opens the gate.- A terminal status requires completed cleanup evidence. An owned browser,
daemon, simulator, server, driver, recorder, proxy, or watcher still alive
forces
STATUS: BLOCKED, even when the exercised product behavior passed.
Output
STATUS: DONE | DONE_WITH_CONCERNS | NEEDS_CONTEXT | BLOCKED
VERDICT: PASS | FIXED(<N>) | FAIL
SUMMARY: <local target/blocker + flow matrix + material findings/unknown outcomes + prototype conformance when applicable>
SOURCES: <context read in step 1: requirement, plan, design/prototype, implement-handoff, review-pr — or conversation-only>
RUBRIC: flow=<> boundary=<> regression=<> data=<> compat=<> security=<> a11y=<> perf=<> freshness=<> (grade or N/A each)
EVIDENCE: <last e2e run: tool + journey + result; screenshot + errors/report paths under evidence/qa/>
CLEANUP: <owned resources stopped and verified; pre-existing/shared resources retained>When a ticket resolves, persist the full block (RUBRIC, EVIDENCE, and
CLEANUP included) — reviews/qa.md is the report a human or orchestrator
reads after the fact — repair and phase QA included; a report without
STATUS and VERDICT is unreadable to the gate. Record it even when full QA was impossible
(VERDICT: FAIL, STATUS: BLOCKED, with the missing required coverage and
blocker named):
bbs ticket set-review --skill qa --body "$(cat <<'EOF'
STATUS: ...
VERDICT: ...
SUMMARY: ...
SOURCES: ...
RUBRIC: flow=<> boundary=<> regression=<> data=<> compat=<> security=<> a11y=<> perf=<> freshness=<>
EVIDENCE: <last e2e run: tool + journey + result; screenshot/log paths under evidence/qa/>
CLEANUP: <owned resources stopped and verified; pre-existing/shared resources retained>
EOF
)"The gate is the caller's bbs autopilot verification begin --gate qa /
record pair around this run
(verification.md), which projects
verdicts/qa.md; a FAIL is recorded, with its blocker under limitations.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

