release-qa
Skill521 starsChanged 16 days agoMITView on GitHub
Run the pi-lens release-readiness QA pass — witness the feature × modality matrix against a real pi, count coverage, and issue a ship / ship-with-caveats / don't-ship / blocked line. Use before cutting a release tag, or when asked whether a build is shippable.
6 other projects use a version of this.
Adopted by
Most recently taken up by
- beekeeper-studio/beekeeper-studiotheir own version, first seen Nov 1, 2026
- Beakpoint-Insights/roosttheir own version, first seen Nov 1, 2026
- alfredoperez/speckit-companiontheir own version, first seen Oct 19, 2026
- AishwaryShrivastav/vibe-testingtheir own version, first seen Oct 17, 2026
- Harry-kp/vortixtheir own version, first seen Oct 8, 2026
- SUDARSHANCHAUDHARI/QACrafttheir own version, first seen Jul 21, 2026
Other versions6 projects use a version of this.
Word for word · 0
None found.
Edited · 0
None found.
Wrote their own version · 6
- beekeeper-studio/beekeeper-studio★ 24ksame name
- Harry-kp/vortix★ 757same name
- alfredoperez/speckit-companion★ 105same name
- Beakpoint-Insights/roost★ 32same name
- AishwaryShrivastav/vibe-testing★ 18same name
- SUDARSHANCHAUDHARI/QACraft★ 2same name
How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.
In this repository9 other files it gives its agents.
- AGENTS.mdAGENTS.md
- CLAUDE.mdCLAUDE.md
- orchestrateSkill
- retroSkill
- verify-pi-lensSkill
- pi-lens-ast-grepSkill
- pi-lens-lsp-navigationSkill
- pi-lens-write-ast-grep-ruleSkill
- pi-lens-write-tree-sitter-ruleSkill
The file
We split this skill into the rules it gives an agent
11 rules. 0 are common, 5 are distinctive, and 6 are very specific to this project.
0
Common
Most of the 6 edited versions we read keep these.
5
Distinctive
This author's own choices. Worth a look.
6
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 34
BLOCKED is not a failure and not a pass: If pi cannot boot, no ship verdict is issued at all.
- Very specific: line 63
add the row to docs/release-qa-baseline.md (and a probe in scripts/release-qa.mjs — the two lists are tied and the unit test enforces it), …
- Very specific: line 94
Use the pi version install-smoke.yml pins (bump by hand, #731) and, when a newer line exists, repeat against the newest one.
- Very specific: line 110
Do not read a 2 as a failure, and do not read it as a clean bill either: name the caveat.
- Very specific: line 151
Never run any release probe without those pins — an unpinned probe writes into the maintainer's real ~/.pi and ~/.pi-lens (#2506).
- Very specific: line 175
Read the release body too: (cd "$S" && node scripts/changelog-extract.mjs <version> --summary) must list only audience: user entries and …
- Distinctive: line 178
Every red here is a release-only calibration: fix it IN the bump PR (a trailing commit after the bump+roll commit is fine) and name it in …
- Distinctive: line 197
every non-PASS row with its cause or reason.
- Distinctive: line 206
Not a substitute for CI. Unit tests and Lint are still the per-PR gate.
- Very specific: line 207
Not a per-language tool sweep — scripts/smoke-tools.mjs does that nightly.
- Distinctive: line 208
Not a model-driven test. Every row is model-free by construction.
“Common” is measured: kept by at least 6 in 10 of the 6 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Release QA
The pass that runs before a release is cut. Sibling of the merge policy (docs/pi-lens-merge-policy.md): that policy
decides whether one PR may land, this one decides whether the accumulated result
may ship.
Why it exists: #2587. Four shipped skills were suspected of never registering for four releases (3.8.51 → 4.1.3) because no check ever asked a real pi what it loaded. The unit suite (~10.8k tests) and the nightly smokes (install, compat, tool, lifecycle, parser) are per-seam; nothing composed them into a release verdict with counted coverage, so a missing row read as absence rather than arithmetic.
Hard rules
These outrank convenience, habit, and the shape of any previous pass.
- No witness, no verdict. A row is PASS only when an artifact SHOWS the pass criterion. A witness that merely exists — a log file that was written, a process that exited 0 — is not a witness. Quote the line.
- Reachability is decided in planning, not discovered mid-run. A row that cannot be reached in this run is SKIPPED with the reason written down BEFORE the run, never a PASS that quietly measured nothing.
- Coverage is counted, never claimed. Report the
discovered / rows / untestedarithmetic the runner prints, verbatim. "All the important paths were checked" is not a coverage statement. - An expired poll is UNTESTED, never PASS. Async rows are polled to a terminal state under a stated cap. Expiry means you did not see the result.
- BLOCKED is not a failure and not a pass. If pi cannot boot, no ship verdict is issued at all. Say BLOCKED and stop.
- The repo's runbook outranks habit.
docs/release-qa-baseline.md,docs/pi-lens-release.mdandAGENTS.mdare the contract. If a step here disagrees with them, they win and this file is wrong.
Procedure
1. Read the baseline
docs/release-qa-baseline.md is the matrix — row ids, entry points, pass
criteria, witness kinds. Read it before running anything; the runner parses this
same document, so a row you cannot explain is a row the runner will execute
anyway.
2. Derive the DIFF rows
The BASE rows are the matrix. The DIFF rows are what THIS release changed, and
.changelog/ is the diff inventory:
git describe --tags --abbrev=0 # the last tag
ls .changelog/*.md # every pending entry = one changeFor each fragment, ask: does it touch a surface the matrix already covers, or a new one? A fragment that adds a modality, a new entry point, or a new shipped resource needs a row. Two outcomes are acceptable and one is not:
- add the row to
docs/release-qa-baseline.md(and a probe inscripts/release-qa.mjs— the two lists are tied and the unit test enforces it), or - record in the report that the change is covered by an existing row, naming which.
Silently leaving a change unrowed is the failure this skill exists to prevent.
A change to .github/workflows/release.yml or to package.json's
packageManager pin is already rowed: publish-toolchain-pinned drives the
publish job's own npx -y "npm@<pin>" invocation against the candidate (#2940).
Name it in the report as the covering row rather than re-deriving one — and if
it reads SKIPPED, the publish path was NOT witnessed for this candidate.
3. Run the runner
git status --porcelain # must be empty; the runner packs a commit
node scripts/release-qa.mjs --pi <path-to-pi>Useful flags:
--from npm:pi-lens@<version>— QA a PUBLISHED release instead of the working tree. This is how a regression is demonstrated against the last release.--git-ref <ref>— enable thegit-installrow against a pushed ref. Without it that row is SKIPPED, because agit:install resolves a pushed ref rather than the working tree.--poll-cap-ms <n>— the cap for polled rows. State the value you used.--keep— leave the scratch root for inspection.
Use the pi version install-smoke.yml pins (bump by hand, #731) and, when a
newer line exists, repeat against the newest one. Both readings go in the
report.
Exit codes — read them, do not infer from the log
| code | verdict | what it means |
| 0 | SHIP | every discovered row PASSED with a witness |
| 1 | DO-NOT-SHIP | a row FAILED, or the candidate would not install or activate |
| 2 | SHIP-WITH-CAVEATS | every witnessed row passed; some produced no witness |
| 3 | BLOCKED or INCONCLUSIVE | no verdict: pi did not boot, or nothing was witnessed |
| 4 | usage / self-check error | bad option, unparseable matrix, arithmetic mismatch |
2 is the EXPECTED verdict for a plain working-tree run — git-install-loads
is SKIPPED without --git-ref, and one SKIPPED row is a caveat by definition.
Do not read a 2 as a failure, and do not read it as a clean bill either: name
the caveat. A CI lane treats 2 as a warning and 1/3/4 as failures.
What the run touches, and what it must not
The runner pins HOME, USERPROFILE, PI_LENS_HOME, PILENS_DATA_DIR,
PI_LENS_INSTALL_LOG and npm_config_cache inside its own scratch root, and
passes that environment to every child — pi, the MCP server, node, and
npm. All six matter. PI_LENS_INSTALL_LOG is the one that is easy to miss:
scripts/warm-loader-cache.mjs (which prepare runs on every pack) prefers
THAT variable for its install log and falls back to PI_LENS_HOME/install.log
(or ~/.pi-lens/install.log when neither is set). The runner's first six runs
pinned the pi-lens home, passed no environment to npm, and put 41
warm_loader_cache records into the maintainer's real ~/.pi-lens/install.log
(#2619 review F1).
The pack runs in a git archive HEAD export inside the scratch root, never in
the live checkout, because npm pack fires our own prepack
(scripts/strip-dev-deps-for-pack.mjs rewrites package.json AND
package-lock.json, restored only by postpack, with no signal trap — an
interrupted pack leaves the checkout stripped) and prepare (rebuilds dist/,
downloads grammars over the network, reinstalls the git hooks). A dirty checkout
is REFUSED rather than packed as its last commit, because a report whose "QA
target" is not what the operator is looking at is the failure this runner
exists to end.
Cost and cleanup. A run installs a production dependency tree twice (the
export, then the scratch project) and builds dist/, so budget ~1.1 GB under
/tmp and about a minute of wall clock. The scratch root is removed on exit
unless --keep — but an INTERRUPTED run leaves it behind, so
ls -d /tmp/pi-lens-release-qa-* after a Ctrl-C session, and remove what you
find.
Remaining side effects on the checkout: none that persist. git archive and
git status are reads. The runner writes release-qa-report.md and
release-qa-evidence/ into --out (the cwd by default; both are gitignored at
the repo root). Everything else lands in the scratch root, which is removed
unless --keep. Verify rather than trust: git status --porcelain before and
after a run must be identical, and
wc -l ~/.pi-lens/install.log must not change.
Never run any release probe without those pins — an unpinned probe writes into
the maintainer's real ~/.pi and ~/.pi-lens (#2506).
3b. Pre-bump dry roll (before opening the bump PR)
The bump PR changes three things the unit suite reads as data: the package
version, the CHANGELOG.md release headings, and the .changelog/ fragment
population (which the roll DELETES). Tests calibrated against the pre-roll
tree go red only on the bump PR, after the gate has already said ship. The
4.1.4 bump (2026-09-07) hit two: config-deprecation-registry demanded
deprecatedSince <= the announcing release (the window said 4.2.0; the
Deprecated entries shipped in 4.1.4), and tracked-control-bytes's Markdown
floor of 80 had been calibrated while 110 fragments existed (64 files remain).
So, in a scratch export, roll for the target version and run the tests that read release state:
S=$(mktemp -d) && git archive HEAD | tar -x -C "$S"
node scripts/changelog-release.mjs <version> --root-dir "$S"
(cd "$S" && npm version <version> --no-git-tag-version >/dev/null && ln -s "$OLDPWD/node_modules" node_modules && npm run build >/dev/null && \
npx vitest run $(grep -rl "CHANGELOG.md\|\.changelog\|package.json" tests/ | grep -v "^tests/support\|fixtures" | tr '\n' ' '))Read the release body too: (cd "$S" && node scripts/changelog-extract.mjs <version> --summary)
must list only audience: user entries and end with the internal count.
Every red here is a release-only calibration: fix it IN the bump PR (a
trailing commit after the bump+roll commit is fine) and name it in the PR
body. Also read the rolled section for a "next version" literal that the
fragments assumed (4.2.0 in a Deprecated since … line when the release is
a patch) and correct it in code, docs and the rolled section together.
The bump PR title is chore(release): <version> (refs #<tracker>) — the
PR-title lint requires the conventional prefix AND an issue ref, and a title
missing either reds ci-verdict on an otherwise green workflow (the 4.2.0
bump was retitled after exactly that).
4. Report
The runner writes release-qa-report.md and release-qa-evidence/<row-id>.*.
The report you hand back carries, in this order:
- the verdict line —
ship/ship-with-caveats/don't ship/blocked/inconclusive— with its exit code; - the coverage arithmetic, quoted verbatim from the runner;
- every non-PASS row with its cause or reason;
- the pi version(s) driven and the QA target (
tree, or the published version); - the DIFF rows derived in step 2 and where each landed.
ship-with-caveats requires the caveats to be named in the release notes. A
caveat nobody wrote down is a defect that shipped.
What this is not
- Not a substitute for CI. Unit tests and Lint are still the per-PR gate.
- Not a per-language tool sweep —
scripts/smoke-tools.mjsdoes that nightly. - Not a model-driven test. Every row is model-free by construction; a row that needs a real LLM turn cannot be a release gate.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

