agentleFS
lohi-ailohi-ai/babysit/.claude/skills/qa/SKILL.md

qa

Skill0 starsChanged 8 days agoMITView on GitHub

Systematically test a web application, fix issues caused by the current change, and re-verify. Use for full QA loops, critical user flows, release checks, or test-and-fix requests.

  • Reads credentials

The file

We split this skill into the rules it gives an agent

10 rules. About 0 are common, 6 are distinctive, and 4 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈6

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 4

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Before launching anything, record the pre-existing target server and automation resources, then keep an ownership ledger for every process …

    line 53
  2. Very specific:

    Run the cleanup finalizer before persisting the verdict or returning any status, including BLOCKED and tool-error paths.

    line 175
  3. Very specific:

    Use the browser session name the assignment gives.

    line 195
  4. Very specific:

    Write the full run report — per case: steps, observed result, evidence paths.

    line 197
  5. Distinctive:

    Do not leave a process running solely for human review.

    line 225
  6. Distinctive:

    Keep test data reversible.

    line 251
  7. Distinctive:

    Do not report a fix until the original reproducer passes.

    line 257
  8. Distinctive:

    Prefer deep checks over many vague clicks.

    line 262
  9. Distinctive:

    Every QA summary names the local target or blocker, the case matrix, and at least one non-happy-path result.

    line 268
  10. Very specific:

    When a ticket resolves, persist the full block (RUBRIC, EVIDENCE, and CLEANUP included) — reviews/qa.md is the report a human or …

    line 286

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

qa

Exercise the application like a user and leave reproducible evidence.

Flow

  1. Load what to test — disk first, conversation as fallback. Resolve the ticket (bbs ticket resolve); when it resolves, read whichever exist:
    • bbs ticket path requirement --read — acceptance criteria
    • bbs ticket path plan --read — intended behavior, accepted scope, contracts/invariants, **Unknowns:**, **Design:** and **Verify:**
    • For changed UI, the Task spec's ## Prototype section (Team), else the plan's design/prototype links or bbs ticket path design --read — inspect the accepted prototype and covered flows/states; presence of a path alone is not a comparison
    • bbs ticket path handoff --skill implement --latest --read — and its ## Deviations: each deviation is where the plan diverged from reality, the likeliest home of a wrong guess; also read findings/unknown outcomes and prototype-comparison evidence, checking their relevance and freshness
    • bbs ticket path verdict --skill review-pr --read — unresolved FINDINGS go into the case matrix; RISK_AREAS/FIXED seed security and regression cases Missing docs are not a gate — fall back to conversation. Change surface: BASE=$(bbs autopilot base-branch); git diff $(git merge-base origin/"$BASE" HEAD). Target URL and login from .babysit/qa.yaml:
    eval "$(bbs secrets load | sed -E '/^export CLOUDFLARE_(ACCOUNT_ID|API_TOKEN)=/d')"                         # exports .babysit/.env values
    ENV=$(bbs secrets qa default-env); ENV=${ENV:-local}
    eval "$(bbs secrets qa probe --env "$ENV" 2>/dev/null)"  # QA_ENV_URL, QA_ENV_{USERNAME,PASSWORD}_ENV, …
    QA_USER=$(printenv "${QA_ENV_USERNAME_ENV:-QA_USER}" 2>/dev/null || true)  # standard: QA_USER / QA_PASS
    QA_PASS=$(printenv "${QA_ENV_PASSWORD_ENV:-QA_PASS}" 2>/dev/null || true)
    Target is $QA_ENV_URL; sign in with $QA_USER/$QA_PASS. A needed credential resolving empty means the value is missing from .babysit/.env — record that as the local-run blocker rather than testing signed-out. No .babysit/qa.yaml is not a gate: derive the target from conversation or the repo and note how credentials were obtained. NEEDS_CONTEXT only when neither docs nor conversation can name a single intended behavior to verify.
  2. Boot or probe the local target first; hosted URLs only when local run is impossible and the reason is recorded. QA the current checkout as-is — normally the branch the user is already on, whose dev server is the one running: nothing to land or compose. If the local target is down and can't be started here, that's the recorded blocker. Server prep: when QA_ENV_PREPARE is set (qa.yaml prepare:, idempotent install + migrate), run it before probing. Before trusting any surface, confirm it actually serves the change (probe a marker from the diff); if not, name the stale surface rather than testing blind. Fixes edit the files in the checkout under test — committing them stays the invoking workflow's job — re-verify on the updated surface. Before launching anything, record the pre-existing target server and automation resources, then keep an ownership ledger for every process this attempt starts (server handle/PID, browser session/namespace, simulator UDID, Appium or other driver). Prefer supervised process handles over detached shell jobs. Ownership, not process name, decides what cleanup may stop. Team integration mode: a Task explicitly assigned a phase's integration worktree (detached at the phase commit bbs team integrate built) plus the cumulative acceptance (every criterion of this phase, and the earlier ones its diff touches; the lead carries the rest) tests that exact commit read-only, and runs bbs contract check <id> there for each contract the phase changed (contracts_changed in the integrate output). Verify the worktree is clean at that commit before and after testing. Do not edit the integration worktree, commit or switch revisions while testing. Once the attempt is recorded, the session fixes every finding itself in the owning ticket's worktree per the lead's phases reference (integrate takes the fix, land, re-test what changed; a conflict that integrate lays out in the integration worktree is resolved there before its commit exists); nothing goes back to the lead as a repair, and only a fix beyond the approved scope ends BLOCKED. Persist evidence/verdict on the parent. Inside a ticket worktree, install and serve from that worktree with the ports, databases and secrets qa.yaml names. When the run uses a resource only one ticket can hold at a time, take bbs ticket surface acquire first, run QA_ENV_REVERT after, and surface release on every outcome — see ../references/worktrees.md. Shared refs (../references/*.md) are filesystem paths beside this skill's directory, so read them by path, not as skill://.
  3. Code-level checks (tests, typecheck, lint) first — they gate, they don't prove user journeys. Read test for affected-test selection and failure investigation. Use the implementation handoff as evidence, then check its coverage against the current change and callers. Reuse automated results only when they cover the same tested code state (including uncommitted changes), environment/prerequisites and required scope, and no mandatory gate requires a fresh run. Otherwise run the affected lanes and required checks. Never weaken or skip a failing test to obtain green; follow test's investigation and authorization rules. Whole-suite audit and optimize are not routine QA steps.
  4. Size the matrix per § Case count. Trace material plan findings/unknowns to acceptance cases or supporting evidence, including resolved discoveries that changed an assumption. Reuse evidence only when code state, environment and coverage still match; an implementation claim alone is not proof. Carry relevant open questions into checks with observable expected results; non-runtime questions use the appropriate contract/code evidence. Record a reason for items outside this change's acceptance, without reopening every resolved discovery. Include accepted prototype flows/states for changed UI. Build a flow matrix from the acceptance criteria — not just the diff — covering happy path, validation, empty/error states, failure/retry, and responsive behavior. Three axes escaped past QA most and are mandatory wherever the change reaches them:
    • Ship surface — each case names the surface that ships (packaged app shell, live provider/sandbox payload, prod-shaped schema, real agent path) and runs there. A fixture, stub, browser-mode build, dev-only DB or implementer-written payload in its place is a named gap, and work deferred to hypercare goes into SUMMARY as a release-checklist item.
    • Actor — changed UI runs signed-out and as a lower-privilege user; where session/credential state is shared, add one wrong-credential or expired-session case.
    • Hard-to-reach states — every dialog, confirm or dirty-state guard gets cancel and Escape; background/hidden-window, offline and timing states are probed. A state the fixture cannot reach is forced by injection or named as a gap — unit tests alone do not cover a case. Derive the change's reach independently: for each changed file/function, find callers and flows sharing its state or routes, and give each an adjacent-regression case (BLAST_RADIUS is the producer's own claim). Write the matrix before touching the app, each case anchored to its source ("criterion 2", "review-pr finding 1", "derived: shares session state"). Self-review: every criterion, BLAST_RADIUS entry, unresolved review finding, and implement deviation has a case; each axis above the change reaches has its case or a named gap; material plan findings/unknowns have evidence or a named gap; each touched flow has ≥1 non-happy-path case; behavior the code walk surfaced that the requirement never mentions gets a derived criterion case with the gap named in SUMMARY; an uncoverable criterion is named as a gap now, never silently dropped. Save the matrix: bbs ticket path evidence --skill qa --name test-matrix.md --write. Automated coverage never replaces the executed journeys below.
  5. Execute the flows end-to-end with a real client. Whenever the harness has a subagent tool, steps 2, 5, 7 and 8 run in one Executor mode subagent with a fresh context — in every mode, Team integration included; a caller's own executor assignment (autopilot § Current-session gates) satisfies this. This session keeps only the matrix, the returned block and the failures it triages; run inline only when no subagent tool exists or the executor cannot reach the target, and record why. Web UI: the browse engine — Read only ../browse/SKILL.md § Engine before the first browser command; its session-name export and setup are mandatory. Non-UI: a real call sequence (curl, CLI, the repo's e2e suite). Never "test" a flow by reading code or unit tests alone. Prototype conformance. For changed UI, every accepted prototype state the change owns is a case — all of them, not a sample (a Team spec's ## Prototype section lists them with links). Per state: open its link, element-screenshot it, then drive the real build with data that produces that state and screenshot it at the same viewport; also run each screen with 80-character user-named strings and at its narrowest supported width, even where the prototype omits them. Compare layout, interactions, content hierarchy, overflow and empty/error behavior, and save prototype-conformance.md: one row per state — both screenshot paths, match / fixed / deviates, one sentence on where it deviates. The prototype is the standard: fix the build and re-shoot; never edit the prototype to match the build. A state left deviates needs an accepted reason tied to a criterion; one not checked, or whose reference will not load, is a conformance gap that blocks PASS. This is acceptance verification, not pixel equality or a new design exercise. Off-script time around the changed surface — findings feed back as derived cases — a few minutes, more when § Case count makes it mandatory.
  6. Fix regressions owned by the current branch. Use test's scoped create workflow to decide whether the fix needs a regression test; existing meaningful coverage may suffice. Rerun the original reproducer, affected flow and checks on the repaired code.
  7. Finish with one full end-to-end pass of the primary user journey on the final code state — any code change after it invalidates the verdict. Screenshot this verdict-bearing pass — and each failure or fixed reproducer — to bbs ticket path evidence --skill qa --name <f>.png --write; list the paths in EVIDENCE:. (Ad-hoc browse checks stay screenshot-light; the QA verdict's screenshots are the durable proof a human audits later.) Summarize prototype conformance and material finding/unknown outcomes as confirmed, corrected or open, linked to case/evidence and acceptance impact. Surface discoveries that change the human's understanding; do not bury them in logs. Required unknowns or conformance checks still unverified are coverage gaps and prevent PASS; unrelated follow-ups may remain nonblocking concerns.
  8. Run the cleanup finalizer before persisting the verdict or returning any status, including BLOCKED and tool-error paths. The browse skill closes and verifies its exact browser session plus any simulator it booted. Stop every server, driver, recorder, proxy, or watcher this QA attempt started, and wait for exit; leave pre-existing targets and coordinator-owned shared surfaces untouched. When this attempt took the surface lease, also run the required QA_ENV_REVERT and release it.

    Compare the final process state with the ownership ledger. Never use broad pkill/killall, agent-browser close --all, or simulator-wide shutdown as routine cleanup. If an owned process survives a scoped stop, retry once, record the exact process/resource and cleanup command, and return STATUS: BLOCKED; a passing product flow does not excuse a leaked automation fleet.

Executor mode

An assignment that names this mode, a matrix path and a run number (autopilot § Current-session gates) runs only the mechanical part: step 2 (boot/probe the target, ownership ledger), the check commands the matrix lists, step 5 for the assigned cases, step 7's final primary-journey pass when assigned, and step 8. Use the browser session name the assignment gives. Edit no code, run no git, write no gate, verdict or set-review; a failure is reported, never fixed. Write the full run report — per case: steps, observed result, evidence paths; per check: argv, cwd, exit code, log path — to bbs ticket path evidence --skill qa --name run-<n>.md --write, and the checks as verification record JSON beside it (run-<n>.json, verification.md). Return only:

STATUS: DONE | BLOCKED
CASES: <n passed>/<n run>; failed: <case ids or none>
FAILURES: <one line per failed case: id — observed vs expected — evidence path>
RUBRIC: <proposed grades from what was executed>
REPORT: <run-<n>.md path> <run-<n>.json path>
CLEANUP: <owned resources stopped and verified; retained ones named>

BLOCKED means the run itself could not execute (target down, missing credential, tool failure), named in FAILURES.

Case design

One case = user journey + expected observable + evidence — as a user, do <steps> → observe <result>, not a component check. The primary journey delivers the feature's core value end-to-end; it carries the verdict and runs last. Tie every failure to the criterion it violates, with reproduction steps.

Case count

5–10 cases: the full matrix from the acceptance criteria plus adjacent-regression cases. When the diff touches auth, money or data paths, security, a11y and compat must carry a real grade — an N/A there needs a reason that would survive review — off-script exploration is mandatory, and security-review runs too.

Do not leave a process running solely for human review. The screenshots and evidence are the handoff: stop a target this QA attempt started, but preserve and name a target that was already running. Always revert and release a surface lease this attempt took.

Coverage rubric

Grade every dimension A–D against the change's risk surface; a dimension the change can't touch is N/A with a one-line reason — never a silent skip.

  • flow: A = happy + every alternate/branch flow e2e · B = happy + ≥2 alternates · C = happy only, or the primary journey ran on a substitute for the ship surface · D = none e2e
  • boundary: A = limits, invalid input, empty/error, failure/retry all exercised · B = ≥1 boundary + ≥1 error · C = mentioned, not executed · D = none
  • regression: A = existing behavior around the change re-verified · B = adjacent flow spot-checked · C = assumed intact · D = not considered
  • data: A = state correct across reload/nav/concurrent edits · B = persists across reload · C = not checked · D = loss/corruption seen
  • compat: A = target browsers + responsive breakpoints · B = one extra viewport · C = default viewport only · D = broken layout seen
  • security: A = permission gates + input safety (authz, injection) probed · B = auth-required paths checked · C = noted, not tested · D = access-control gap seen
  • a11y: A = keyboard path, labels/roles, contrast, clear error copy · B = keyboard + visible focus · C = not checked · D = blocking defect
  • perf: A = responsive under realistic data volume · B = no obvious lag on happy path · C = not observed · D = timeout/jank seen
  • freshness (always applies): A = final full e2e pass on the final code state · B = e2e passed but code changed after · C = partial/stale e2e · D = unit/curl/code-read only Grades are letters A–D or N/A: <reason>; anything else ("ok", "pass") is a missing grade. PASS/FIXED require every applicable dimension at B or better and freshness at A. Any applicable dimension at C or D, missing required coverage or failed acceptance forces VERDICT: FAIL (→ STATUS: BLOCKED). DONE_WITH_CONCERNS is only for nonblocking residuals once all required acceptance and coverage pass. Report the grade line for every dimension, including N/A: <reason>.

Rules

  • Keep test data reversible; no destructive production actions.
  • Distinguish current-change bugs from pre-existing failures — and only call one pre-existing after watching it fail at the base commit (a scratch git worktree add keeps this tree untouched). Arguing from how old the code looks is not evidence; "pre-existing" is the most common sentence a real regression ships behind.
  • Do not report a fix until the original reproducer passes.
  • PASS/FIXED require an executed end-to-end run as the most recent evidence — tool, journey steps, observed result — and the app proven running locally (or the local-run blocker named). Happy-path-only is not a PASS.
  • Prefer deep checks over many vague clicks.
  • Turns are the cost: every turn re-pays the whole context. Batch each case group into one script that prints one pass/fail line per case; on a failure print only its last 30 lines. Write the matrix, scripts and evidence once and reference them by path — never re-read or re-run a whole script to inspect one failure. Prefer a scoped snapshot to a full page.
  • Every QA summary names the local target or blocker, the case matrix, and at least one non-happy-path result.
  • VERDICT: FAIL always pairs with STATUS: BLOCKED — never DONE*. The PR gate treats any DONE* status as ready; a failing QA that reports DONE_WITH_CONCERNS silently opens the gate.
  • A terminal status requires completed cleanup evidence. An owned browser, daemon, simulator, server, driver, recorder, proxy, or watcher still alive forces STATUS: BLOCKED, even when the exercised product behavior passed.

Output

STATUS: DONE | DONE_WITH_CONCERNS | NEEDS_CONTEXT | BLOCKED
VERDICT: PASS | FIXED(<N>) | FAIL
SUMMARY: <local target/blocker + flow matrix + material findings/unknown outcomes + prototype conformance when applicable>
SOURCES: <context read in step 1: requirement, plan, design/prototype, implement-handoff, review-pr — or conversation-only>
RUBRIC: flow=<> boundary=<> regression=<> data=<> compat=<> security=<> a11y=<> perf=<> freshness=<>  (grade or N/A each)
EVIDENCE: <last e2e run: tool + journey + result; screenshot + errors/report paths under evidence/qa/>
CLEANUP: <owned resources stopped and verified; pre-existing/shared resources retained>

When a ticket resolves, persist the full block (RUBRIC, EVIDENCE, and CLEANUP included) — reviews/qa.md is the report a human or orchestrator reads after the fact — repair and phase QA included; a report without STATUS and VERDICT is unreadable to the gate. Record it even when full QA was impossible (VERDICT: FAIL, STATUS: BLOCKED, with the missing required coverage and blocker named):

bbs ticket set-review --skill qa --body "$(cat <<'EOF'
STATUS: ...
VERDICT: ...
SUMMARY: ...
SOURCES: ...
RUBRIC: flow=<> boundary=<> regression=<> data=<> compat=<> security=<> a11y=<> perf=<> freshness=<>
EVIDENCE: <last e2e run: tool + journey + result; screenshot/log paths under evidence/qa/>
CLEANUP: <owned resources stopped and verified; pre-existing/shared resources retained>
EOF
)"

The gate is the caller's bbs autopilot verification begin --gate qa / record pair around this run (verification.md), which projects verdicts/qa.md; a FAIL is recorded, with its blocker under limitations.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.