audit-tests
Skill4 starsChanged 2 days agoApache-2.0View on GitHub
Audit Pnyx tests for the real behavior a change promises, missing regression coverage, brittle assertions and safe isolation.
13 other projects use a version of this.
Adopted by
Most recently taken up by
- dzhng/skillstheir own version, first seen Nov 15, 2026
- nicholasjconn/skillstheir own version, first seen Nov 15, 2026
- dzhng/jevgreptheir own version, first seen Nov 11, 2026
- Buddhilive/buddhi-clitheir own version, first seen Nov 1, 2026
- dzhng/duet-agenttheir own version, first seen Oct 30, 2026
- dzhng/yaptheir own version, first seen Oct 15, 2026
Used with
Projects with it also use close-spec, design-with-images and 9 more.
Other files people use alongside this one.
Files found in the same public projects as a version of this one, more often than they turn up anywhere else. Each project counts once, large collections count for less, and this file's own repository is left out.
dzhngSkillclose-spec
When a spec under specs/<feature>/ is done shipping (a write-spec build, or any planned task that produced a spec), archive it to specs/done/ and rewrite it from a build-plan into a durable rationale — the why, the principles, the invariants — pointing back to the real code for the how. Use when implementation has landed and the plan no longer matches what shipped, or the user says a feature/spec is finished. Pairs with [write-spec](../write-spec/SKILL.md) (the plan this closes) and [review](../review/SKILL.md) (run before closing).
Also in 5 projects that use this
dzhngSkilldesign-with-images
Explore UI and visual design with image generation, then iterate the real implementation against the user-selected concept. Use when brainstorming visual directions, offering mockup variants, or implementing an approved generated design; do not stop at the mockups or the first code approximation.
Also in 5 projects that use this
transitive-bullshitSkillcompare-screenshots
Compare screenshots against the intended design, distinguishing approved references from historical baselines. Use for iterative implementation matching, before/after visual review, objective image telemetry, or checking a lone capture for flat, empty, or badly framed content.
Also in 5 projects that use this
LucasMatuszewskiSkillaudit-agents
Audit and maintain agent configuration files (.claude/agents, AGENTS.md, CLAUDE.md, agent-memory). Checks for duplicates, outdated info, naming conflicts, wrong technical facts, and files over 200 lines. Fixes all issues and commits.
Also in 5 projects that use this
wolverin0Skillaudit-decisions
Standardize remediation decisions, baseline-policy entries, and legal-review banners for accepted, deferred, or policy-blocked audit findings.
Also in 4 projects that use this
rhnfzlSkillexplore-unknowns
Walk the user through mapping the unknowns of a task before, during, and after they build it, and hand them a four-quadrant map (settled ground, open decisions, taste, and landmines) they keep. Reach for this eagerly whenever a request is ambiguous or underspecified, the codebase or domain is unfamiliar, the user will "know it when they see it", a reference implementation must be understood before porting, a build-queue or architecture-changing task is about to start, a mid-build deviation needs capturing, or a finished change needs buy-in or verified understanding before merge. Trigger even when the skill is not named: "do a blindspot pass", "find my unknown unknowns", "interview me about this change", "brainstorm the approach for a task I am about to build", "help me scope this", "what am I missing here", "plan this before I build", "I know nothing about X in this codebase". When a territory profile is present it reads memory and the graph first and feeds the map into the build pipeline; without one it degrades to a plain general map. Not for trivial or mechanical tasks with clear acceptance criteria, single-fact lookups, pure debugging of a known symptom, or pure code review, those have their own tools. Do not use it to run the build itself; the map is the deliverable and implementation remains a separate task, whether it has not started or is already underway.
Also in 5 projects that use this
- +5 more from dzhng
Other versions13 projects use a version of this.
Word for word · 0
None found.
Edited · 0
None found.
Wrote their own version · 13
- dzhng/jevgrep★ 3.2ksame name
- trycompai/comp★ 2.1ksame name
- dzhng/skills★ 1.5ksame name
- dzhng/yap★ 118same name
- dzhng/duet-agent★ 55same name
- TalonT-Org/AutoSkillit★ 5same name
- daanvandenbergh/claudekit★ 2same name
- Buddhilive/buddhi-cli★ 1same name
- dzakirinf/skills★ 1same name
- nicholasjconn/skills★ 1same name
3 more not listed
How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.
In this repository17 other files it gives its agents.
- AGENTS.mdAGENTS.md
- CLAUDE.mdCLAUDE.md
- deslopSkill
- follow-conventionsSkill
- review-codeSkill
- write-ci-workflowSkill
- write-front-endSkill
- write-pull-requestSkill
- write-skillSkill
- inscribeSkill
- pnyx-arrange-columnsSkill
- pnyx-cliSkill
- pnyx-firstmate-as-hydraSkill
- pnyx-manage-workersSkill
- pnyx-orchestrateSkill
- pnyx-report-progressSkill
- pnyx-set-upSkill
The file
Files
Audit tests
A test must pass unchanged when the implementation is renamed or rewritten with the same behavior, and go red when that behavior breaks. If a behavior-preserving change forces a test edit, the test is brittle: recommend rewriting it against visible behavior or deleting it. This audit checks that every core behavior in scope has regression coverage. Use the PR's Intent, changed behavior and any tests the branch edited as the scope unless a wider area was requested; when the requester names the behaviors they consider core, start from that list. Read docs/testing.md for the test harness.
This is the Tests part of every review, next to Bugs and Simpler in review-code; all three block the merge. Each test earns its place: it fails when the behaviour it covers breaks, and a test that does not is rewritten or deleted. Nothing user-facing is left uncovered. A test hard to follow, duplicated or hand-rolling what tests/setup/ already provides is a Simpler finding too, with the replacement named.
- Every change to behavior brings tests: each new or changed behavior has a test in the same PR, and nothing in scope is left to manual checking. A change with no test, or a test that does not cover the changed behavior, is must-fix.
- Inventory behavior as “when X happens, Y results”. Include success, refusal, invalid or empty input, failure, recovery and restart when relevant. For Pnyx, cover permissions, event persistence and replay, worker lifecycle and reply delivery where touched. If a behavior can't be phrased that way yet, keep narrowing until it can.
- Map each requested behavior to a test and its observable assertion through the feature's public surface: CLI output and exit status, persisted events, server responses, delivered replies or the rendered page. For live agent behavior, also require isolated validation with real agents. Rebuilt history or a mocked view alone does not cover live behavior. A behavior with no test is the top finding. A test that names the behavior but asserts something else is the same gap.
- Run the flip test for each core flow and each new or changed test in scope: in a throwaway copy, invert a condition, omit an event or delivery, or return the wrong result. Run the relevant test and confirm its behavioral assertion goes red, then discard the copy. A regression test must fail with the fix reverted in that copy. For each bug fixed in the area's recent history (
git log), check that such a test exists. - Reject tests that grep, regex-match or snapshot source text. Reading implementation files to assert spelling, helper names or code shape does not exercise behavior. Invoke the real function, CLI, server or page instead. Snapshots of outputs are useful only when the output itself is the contract and a behavioral break changes the assertion.
- Assert behavior from the user's point of view, through public APIs (the CLI, server, events and rendered state), not code structure. Reject tests that call private helpers or mock Pnyx's own modules.
- Keep logic out of tests: no loops, conditionals or calculations that build the expected value. Write expected values literally.
- Remove brittleness from assertions on incidental copy, log order, private helper calls or timing. Flag tests the branch edited even though the behavior they cover stayed the same. Wait for an observable condition with a bounded wait; do not use fixed sleeps or retries to hide a race.
- Register completion observers before triggering work. For absence, drain the named operation; for domain deadlines, control time and assert just before and at the boundary. Force race phases with gates instead of repeated scheduling attempts or PID order.
- Each test file runs in its own process and its tests share that file's server, so each test sets up what it needs and passes alone and in any order. Close owned resources.
- Reject unbounded waits, including
for (;;)andwhile (true)polling without a deadline or attempt limit. Bad:while (true) { if (await ready()) break }. Good:await until({ what: 'the reply', check }), where timeout fails with the last observed state. - Do not match exact event sequence numbers read in separate operations: another writer can append between reads. Bad:
expect(reply.seq).toBe(previouslyRead.seq + 1). Good: find the reply by its returned ID in one event snapshot, then assert its payload and causal link. - In tests, use IDs returned by setup or the operation under test. Bad:
taskShow({ id: 'task-1' })after creating a task. Good:taskShow({ id: created.id }). Fixed IDs belong only in synthetic input whose identity is the contract. - After mechanical import-path updates, moving code between files should leave behavioral assertions unchanged; flag assertions about implementation location or private helpers.
- Put a test in its area's
tests/<area>.test.ts, through the real server,pnyx-axiand the page's data calls; call a function directly only for a tricky edge case the server cannot reach cheaply, in the same file. Add a browser journey intests/e2e/only for what needs a real page (rendering, scrolling, live updates); check other page changes in the local preview and the pull request's screenshots. A core behavior covered only by tests over mocks that reimplement Pnyx is a gap: the test proves the mock. - Isolate every test with temporary
PNYX_HOME, random ports and synthetic fixtures. Never access real user folders, tools, services or accounts. Fake external services at their boundary; a test that needs the network is a finding, not coverage. Remove inherited worker identity in owner-CLI subprocesses. - Prefer real dependencies over fakes: use the real file system, event log, server or Postgres through a scratch home rather than mocking them. Fake only at the outer edge: third-party services, AI providers and the network.
- Reject duplicate, cosmetic and snapshot tests: a test that repeats what another already pins, that checks copy, spacing or markup nobody depends on, or that stores output to compare later. Delete them.
- A bug fix starts with a test that reproduces the bug and fails on the unfixed code; it passes only with the fix. The PR shows that each new bug-fix test fails without its change. Other new tests run the flip test above.
- Fold a new test that fits an existing test into assertions in that named test. Fold several tests, new or existing, that set up the same flow or real system (such as a server or database) into one flow test that touches everything once. Map and carry over every unique assertion before folding; keep coverage of each affected CLI, transport or restart boundary.
- Give every test in scope a verdict: keep (name the regression it catches), fold into
<named test>, or delete (name what still covers the behavior, or say “nothing does”). Keep only tests for behavior a user would notice or a named past bug; delete tests kept just in case. Flag incidental timing, internals or implementation details, duplicates of a better test, and doubtful tests that have flaked. - Run focused tests and the suites selected by
pnpm verifybefore readiness. Respect explicit test holds; record pending suites and known failures without changing unrelated tests or claiming a pass.
Report a short behavior-to-test mapping, every test’s verdict, gaps and brittle tests. Treat a core behavior with no behavior-level test, a brittle test standing in as its only coverage, and a test that touches real user data or services as must-fix; level changes and merging redundant tests are recommendations. For each gap, name the file, the behavior as “when X, Y” and the assertion that goes red when it breaks. Record each flip, the test and assertion that went red, and what else ran, passed, failed or remains pending. Once complete, tick Ran audit-tests in the PR and describe what you fixed (or say no findings). The audit can be complete while a suite is pending; pending suites still prevent a verification marker.
For an agent: inspect assertions and exercise the behavior yourself. Never edit, commit or push the audited branch; make behavior-breaking probes only in throwaway copies. Do not substitute a source-text search or a list of test filenames for the audit.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

