agentleFS
ToddGBensonToddGBenson/keel/.claude/skills/test-strategy/SKILL.md

test-strategy

Skill0 starsChanged 3 months agoApache-2.0View on GitHub

Decide what to test at which layer, and audit whether existing tests actually constrain behavior. Use when planning tests for a story, running the G4 QA gate, reviewing test quality in a PR, deciding between unit and integration coverage, or when asked whether coverage is meaningful. Includes the delete-the-implementation audit and flaky-test handling.

41 other projects use a version of this.

Adopted by

Most recently taken up by

Used with

Projects with it also use database-testing, data-pipelines and 8 more.

Other files people use alongside this one.

Files found in the same public projects as a version of this one, more often than they turn up anywhere else. Each project counts once, large collections count for less, and this file's own repository is left out.
  • petrkindlmannSkill

    database-testing

    Validate database integrity, test migrations forward and backward, verify schema constraints, manage seed data, detect migration drift, and identify query performance issues. Covers PostgreSQL, MySQL, MongoDB with Prisma, TypeORM, Drizzle, and SQLAlchemy, plus Testcontainers test databases. Use when: "database test," "migration test," "migration rollback," "rollback test," "data integrity," "SQL test," "schema validation," "seed data," "query performance," "Testcontainers." Not for: synthetic data generation/masking at scale — use test-data-management; Docker/IaC test-environment provisioning — use test-environments; SQL injection — use security-testing. Related: test-data-management, test-environments, security-testing, ci-cd-integration.

    Also in 3 projects that use this

  • YepAPISkill

    data-pipelines

    ETL patterns, cron scheduling, queue-based processing, and batch jobs with Inngest/BullMQ.

    Also in 3 projects that use this

  • ffsshhttiikkSkill

    unit-testing

    Unit testing best practices and patterns

    Also in 4 projects that use this

  • AnthropicSkill

    incident-response

    Run an incident response workflow — triage, communicate, and write postmortem. Trigger with "we have an incident", "production is down", an alert that needs severity assessment, a status update mid-incident, or when writing a blameless postmortem after resolution.

    Also in 7 projects that use this

  • OpenAISkill

    observability

    Vercel Observability expert guidance — Drains (logs, traces, speed insights, web analytics), Web Analytics, Speed Insights, runtime logs, custom events, OpenTelemetry integration, and monitoring dashboards. Use when instrumenting, debugging, or optimizing application performance and user experience on Vercel.

    Also in 7 projects that use this

  • Microsoft AzureSkill

    release-readiness

    Complete ODIN release documentation and validation checks before declaring user-visible work complete, preparing a release, or opening the Release-to-main pull request.

    Also in 4 projects that use this

The file

We split this skill into the rules it gives an agent

3 rules. 0 are common, 3 are distinctive, and 0 are very specific to this project.

  • 0

    Common

    Most of the 36 edited versions we read keep these.

  • 3

    Distinctive

    This author's own choices. Worth a look.

  • 0

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Only the second is SA-11 security evidence.

    line 52
  2. Distinctive:

    If a plausible bug would leave a test green, it is decorative — and worse than absent, because it inflates the coverage number and buys …

    line 61
  3. Distinctive:

    Never retry it away. The retry hides a real race condition, and the race is the finding — it is telling you something true about the system …

    line 95

“Common” is measured: kept by at least 6 in 10 of the 36 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Test strategy

Consumers: qa-engineer, /qa-gate, developer, /review, process/gates/g4-verified.md.

Shape

        ╱╲          Exploratory — charter-driven, time-boxed. Finds what nobody imagined.
       ╱  ╲
      ╱────╲
     ╱  E2E ╲       Few. Critical journeys only. Slow, brittle — and irreplaceable for
    ╱────────╲      proving the whole thing hangs together.
   ╱Integration╲    Real collaborators: DB, queue, HTTP boundary, authz. This layer catches
  ╱────────────╲    what unit tests mock away, which is most real defects.
 ╱     Unit     ╲   Many, fast, deterministic. Pure logic, branches, boundaries.
╱────────────────╲

Integration is the layer most teams under-invest in. Unit-only suites produce systems where every part passes and the whole fails. E2E-heavy suites produce slow, flaky suites nobody trusts and everyone re-runs until green — which is worse than no suite, because it still shows green.

Placement

ConcernLayerWhy
Business logic, branches, boundariesUnitFast, exhaustive on the interesting cases
Authorization decisionsIntegration — alwaysUnit-mocked authz proves the mock, not the authorizer
Persistence, transactions, migrationsIntegration, real engineAn in-memory substitute has different semantics exactly where it matters
Contract with an external serviceContract test + recorded-fixture integration
Critical user journeyE2E — one per journey, no more
Input validationProperty-based where the space is large
Performance NFRLoad test, threshold as the assertion, representative data volumePerformance on an empty database is not verified
AccessibilityAutomated axe scan + manual keyboard and screen-reader passThe scan catches maybe half
A security control from the threat modelIntegration, negative caseSee below

The negative-case rule

For every control, the test that matters is the one proving it denies.

Positive:  owner reads their draft → 200      ← the feature works
Negative:  non-owner reads it      → 403      ← the CONTROL works

Only the second is SA-11 security evidence. A story whose controls have only positive tests fails G4 regardless of coverage percentage.

The test-quality audit

Judge tests by whether they constrain behavior. The check:

Mentally delete the implementation. Which tests fail?

If a plausible bug would leave a test green, it is decorative — and worse than absent, because it inflates the coverage number and buys false confidence.

Watch for:

  • Asserting implementation detail rather than observable behavior — passes with the logic removed, fails on harmless refactors. Exactly backwards.
  • Assertions too loose to fail — assert result is not None, assert len(x) > 0.
  • Setup so elaborate the test's meaning is unrecoverable; nobody will maintain it and everybody will skip it.
  • Shared mutable state between tests — passes in isolation, fails in CI, or vice versa.
  • Wall-clock or network dependence.
  • Tests that mirror the implementation's structure — they were written by reading the code, so they encode the same misunderstanding.

Report the audit as a finding, specifically:

"AC-5's test asserts internal eviction ordering rather than observable behavior. It passes with the eviction logic removed entirely. Not valid coverage."

Coverage

A floor, not a target. Coverage measures reach, not quality — 100% coverage of assertions that cannot fail is 0% verification.

Two rules worth enforcing in CI:

  • Absolute threshold met
  • No drop versus the base branch — otherwise a large untested change passes by diluting an already-high number

Flaky tests

A flaky test is a defect. File it as one.

Never retry it away. The retry hides a real race condition, and the race is the finding — it is telling you something true about the system that you are choosing not to hear. A suite with retries configured is a suite that has stopped being evidence.

Traceability

Build AC → test → result. An AC with no test fails G4. A test with no AC is either dead weight or an undocumented requirement — resolve which; do not leave it ambiguous.

| AC | Test | Type | Result | Evidence |
|----|------|------|--------|----------|
| AC-1 draft offered on re-auth | test_draft_offered_after_expiry | Integration | ✅ | run #4821 |
| NFR authz | test_other_user_cannot_read_draft | Integration (negative) | ✅ | run #4821 |
| NFR p95 < 200 ms | load_draft_restore | Performance | ⚠️ 240 ms | run #4823 |

Non-functional verification

  • Performance — the NFR number is the assertion, under representative load and data volume
  • Reliability — exercise dependency failure deliberately. Kill the DB mid-transaction and watch. Make the dependency slow rather than down; that path is usually untested and usually worse.
  • Recovery — roll back for real, then roll forward. An untested rollback is a hypothesis.
  • Observability — assert the required log, metric, or trace actually appears

Reporting

Report results exactly as observed. If unverified, say unverified — never "passed" for something you did not run. A QA sign-off that overstates coverage is the most damaging artifact in this process, because everything downstream trusts it.

Reject specifically, naming the criterion and the observation. "Needs more testing" is not a rejection.

Controls

SA-11 · SA-11(1) · SA-11(8) · CA-2 · CM-4 · AU-2 (audit event verification).

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.