agentleFS
l-gevityl-gevity/l-gevity-skills/.claude/skills/test-strategy/SKILL.md

test-strategy

Skill88 starsChanged 3 days agoMITView on GitHub

Designs and audits risk-driven test strategies that map grounded requirements, acceptance criteria, quality risks, architecture, and escaped defects to the smallest sufficient evidence portfolio. Use when deciding what to test, which design technique or test fidelity to use, when to use real dependencies versus fakes, stubs, or mocks, how to manage test data and environments, or whether unit, component, integration, contract, E2E, exploratory, property, fuzz, performance, resilience, or recovery testing is justified. Also use to audit hollow assertions, redundant coverage, flaky tests, and residual risk. Do not use for framework-specific test implementation, pipeline placement or gating, or trace evidence state; hand those to stack skills, defect-shift-left, ci-cd-reliability-architecture, and requirements-traceability.

41 other projects use a version of this.

Adopted by

Most recently taken up by

Used with

Projects with it also use database-testing, exploratory-testing and 10 more.

Other files people use alongside this one.

Files found in the same public projects as a version of this one, more often than they turn up anywhere else. Each project counts once, large collections count for less, and this file's own repository is left out.
  • petrkindlmannSkill

    database-testing

    Validate database integrity, test migrations forward and backward, verify schema constraints, manage seed data, detect migration drift, and identify query performance issues. Covers PostgreSQL, MySQL, MongoDB with Prisma, TypeORM, Drizzle, and SQLAlchemy, plus Testcontainers test databases. Use when: "database test," "migration test," "migration rollback," "rollback test," "data integrity," "SQL test," "schema validation," "seed data," "query performance," "Testcontainers." Not for: synthetic data generation/masking at scale — use test-data-management; Docker/IaC test-environment provisioning — use test-environments; SQL injection — use security-testing. Related: test-data-management, test-environments, security-testing, ci-cd-integration.

    Also in 3 projects that use this

  • testified-ossSkill

    exploratory-testing

    Guide exploratory testing sessions including charter creation, session-based testing, heuristics application, and finding documentation. Use when planning exploratory testing, conducting test sessions, or documenting discoveries.

    Also in 4 projects that use this

  • YepAPISkill

    data-pipelines

    ETL patterns, cron scheduling, queue-based processing, and batch jobs with Inngest/BullMQ.

    Also in 3 projects that use this

  • ffsshhttiikkSkill

    unit-testing

    Unit testing best practices and patterns

    Also in 4 projects that use this

  • AnthropicSkill

    incident-response

    Run an incident response workflow — triage, communicate, and write postmortem. Trigger with "we have an incident", "production is down", an alert that needs severity assessment, a status update mid-incident, or when writing a blameless postmortem after resolution.

    Also in 7 projects that use this

  • OpenAISkill

    observability

    Vercel Observability expert guidance — Drains (logs, traces, speed insights, web analytics), Web Analytics, Speed Insights, runtime logs, custom events, OpenTelemetry integration, and monitoring dashboards. Use when instrumenting, debugging, or optimizing application performance and user experience on Vercel.

    Also in 7 projects that use this

The file

Show its rules and text

We split this skill into the rules it gives an agent

29 rules. 0 are common, 26 are distinctive, and 3 are very specific to this project.

  • 0

    Common

    Most of the 36 edited versions we read keep these.

  • 26

    Distinctive

    This author's own choices. Worth a look.

  • 3

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Risk before test type: Start from a requirement, failure mode, quality risk, incident, or contract.

    line 26
  2. Distinctive:

    Minimum sufficient fidelity: Use the smallest scope that can faithfully observe the named failure.

    line 30
  3. Distinctive:

    Use this skill after a requirement or remediation increment has usable completion conditions, or retrospectively when an existing behavior, …

    line 44
  4. Distinctive:

    Use a Combined pass only for Audit mode against a stable architecture, or when the relevant architecture is already accepted and will not …

    line 57
  5. Very specific:

    If requirement meaning, actors, permissions, or completion conditions are unclear, return to requirements-grounding or …

    line 73
  6. Distinctive:

    Use this Alchemy hand-off when both passes apply:

    line 93
  7. Distinctive:

    Do not restart the Obligation pass after architecture unless the architecture changes requirement meaning, a failure mode, or an oracle.

    line 107
  8. Distinctive:

    Keep impact, exposure, and evidence confidence separate.

    line 144
  9. Very specific:

    If no credible oracle exists, emit DEFER for that obligation and name the missing requirement, contract, signal, or human judgment.

    line 159
  10. Distinctive:

    Only then apply labels such as unit, component, integration, contract, system, E2E, or production probe.

    line 200
  11. Distinctive:

    Prefer the smallest candidate whose fidelity can observe the failure.

    line 208
  12. Distinctive:

    Read the same reference for test doubles, production-compatible dependencies, test data, environment isolation, quarantine policy, adequacy …

    line 214
  13. Distinctive:

    Use real, deterministic, in-process owned collaborators.

    line 220
  14. Distinctive:

    Use a mock only when the interaction itself is required behavior.

    line 223
  15. Distinctive:

    Check the applicable surface without inventing requirements:

    line 233
  16. Distinctive:

    When a quality area is material but lacks an authoritative requirement or specialist policy, record a blocker or route to the matching …

    line 246
  17. Distinctive:

    Do not manufacture thresholds.

    line 248
  18. Distinctive:

    Treat every escaped defect as evidence that the portfolio missed a risk, technique, oracle, fidelity, or placement decision.

    line 258
  19. Distinctive:

    Treat flakes, order dependence, uncontrolled time, shared state, and environment leakage as test defects.

    line 260
  20. Distinctive:

    Use coverage as a gap locator.

    line 262
  21. Distinctive:

    Remove a redundant broader test only after the smaller replacement proves the same scope.

    line 264
  22. Very specific:

    Do not emit ADEQUATE from test counts, pyramid shape, coverage percentage, green CI, or one successful release.

    line 297
  23. Distinctive:

    Do not invent requirement meaning, thresholds, users, traffic, or risk acceptance.

    line 327
  24. Distinctive:

    Do not use a test framework preference as a strategy.

    line 329
  25. Distinctive:

    Do not require every technique for every system.

    line 330
  26. Distinctive:

    Do not equate mocks with isolation or real dependencies with confidence.

    line 331
  27. Distinctive:

    Do not automate subjective evaluation merely to make it countable.

    line 332
  28. Distinctive:

    Do not retain expensive tests because they were expensive to create.

    line 333
  29. Distinctive:

    Do not claim an ownership boundary listed under Boundary.

    line 334

“Common” is measured: kept by at least 6 in 10 of the 36 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Test Strategy

Convert accepted behavior and quality risks into the smallest reliable evidence portfolio. Choose tests by the failure they must detect, not by a universal pyramid ratio, coverage target, or preferred tool.

Core Directives

  1. Risk before test type. Start from a requirement, failure mode, quality risk, incident, or contract. Never start from “we need more unit tests.”
  2. Oracle before harness. State how correct and incorrect behavior can be distinguished before selecting a test level, framework, or environment.
  3. Minimum sufficient fidelity. Use the smallest scope that can faithfully observe the named failure. Add broader tests only for distinct confidence.
  4. Evidence over percentages. Coverage locates unexecuted code; it does not prove useful assertions, satisfied requirements, or acceptable residual risk.
  5. Real boundaries where semantics matter. A fast double cannot prove database, wire, framework, migration, or provider compatibility.
  6. Reliability is part of validity. A flaky or order-dependent result is not trustworthy verification.
  7. Keep unknowns visible. Record untested risks, manual obligations, missing environments, and accepted residual risk instead of silently marking them covered.

Boundary

Use this skill after a requirement or remediation increment has usable completion conditions, or retrospectively when an existing behavior, incident, or test portfolio provides a bounded subject. In Design mode, use two passes when architecture can change boundaries, dependency semantics, deployment topology, or observability:

  1. Obligation pass — after readiness, before A. Define risks, failure modes, oracles, and required confidence without freezing architecture- dependent scope or fidelity.
  2. Portfolio pass — after final A/L/C and E when applicable, before H. Consume the accepted architecture and finalize technique, scope, fidelity, dependencies, data, environment, and stimulus.

Use a Combined pass only for Audit mode against a stable architecture, or when the relevant architecture is already accepted and will not change.

Consume:

  • canonical requirement and acceptance-criterion IDs when available;
  • actor workflows, contracts, current or accepted architecture boundaries, and data ownership;
  • functional and quality constraints;
  • the readiness aspect matrix, at capability level, for the Obligation pass;
  • the subsystem-level governed set of each aspect — from the accepted placement decision or the architecture-as-code registry — for the Portfolio pass;
  • historical defects, incidents, change hotspots, and production signals;
  • delivery, environment, privacy, compliance, and cost constraints.

If requirement meaning, actors, permissions, or completion conditions are unclear, return to requirements-grounding or implementation-readiness. Existing tests and code are evidence, not product intent.

This skill owns:

  • the risk-to-evidence mapping;
  • the oracle, technique, scope, fidelity, dependency, data, and environment decisions for each test obligation;
  • portfolio adequacy, duplication, and residual-risk analysis.

This skill does not own:

  • requirement meaning or readiness;
  • implementation recipes for a specific test framework;
  • the earliest pipeline stage, job structure, blocking action, or evidence freshness;
  • the distinction between implemented and executed/verified evidence;
  • specialist security, accessibility, safety, or domain policy.

Use this Alchemy hand-off when both passes apply:

Implementation Readiness
→ Test Strategy — Obligation pass: risks, failure modes, oracles, confidence
→ A → L/C → E, as justified: accepted architecture and enforcement
→ Test Strategy — Portfolio pass: technique, scope, fidelity, dependencies,
  data, environment, stimulus
→ Defect Shift-Left: earliest capable stage
→ CI/CD Reliability: execution trigger, gate, artifact, freshness, failure action
→ Requirements Traceability: implementation and executed-evidence state
→ System Optimization: suite flow, cost, duplication, and bottlenecks

Do not restart the Obligation pass after architecture unless the architecture changes requirement meaning, a failure mode, or an oracle. If accepted architecture changes after the Portfolio pass, rerun only the affected portfolio rows before handing them to H.

test-strategy is a task-matched Alchemy companion. It is not a new A.L.C.H.E.M.Y. letter, qualification stage, or gate.

Workflow

1. Qualify the subject

Select a mode:

  • Design — derive evidence obligations for admitted or clearly bounded work.
  • Audit — assess an existing portfolio against current requirements, risks, incidents, and architecture.

In Design mode, also select the strategy pass defined under Boundary: Obligation, Portfolio, or Combined.

Define the subject as a requirement increment, subsystem, contract, workflow, change, release risk, or bounded system. Do not strategize for “the whole application” when no risk or decision boundary is stated.

2. Build the risk ledger

For each material risk, record:

Risk / criterion: <canonical ID or named risk>
Failure mode:     <observable wrong outcome>
Impact:           <user, business, security, operational, or compliance harm>
Exposure:         <where and how plausibly it occurs>
Risk evidence:    <requirement, incident, telemetry, contract, or assumption>

Keep impact, exposure, and evidence confidence separate. Do not collapse them into one score. Unknown exposure is not low exposure. External legal, contractual, accessibility, safety, or security obligations remain required even when usage is low.

3. Define the oracle

For every required obligation, state:

  • the source of truth;
  • the observable pass and fail condition;
  • tolerances, time windows, ordering, or eventual-consistency rules;
  • false-pass and false-fail risks;
  • the data and instrumentation needed to observe the result.

If no credible oracle exists, emit DEFER for that obligation and name the missing requirement, contract, signal, or human judgment. A test that merely runs code or asserts “not null” is not evidence unless that is the actual criterion.

4. Select the technique

Read references/technique-selection.md when the technique is not obvious, the subject has several failure modes, or property, model-based, contract, fuzz, exploratory, performance, resilience, or recovery testing may apply.

Select the technique that attacks the risk directly:

  • examples for named business cases;
  • partitions and boundaries for classified input ranges;
  • decision tables or combinations for interacting rules;
  • state-transition or model-based tests for workflows;
  • property, metamorphic, or differential tests for invariants and broad input;
  • contract or real-boundary integration tests for compatibility;
  • a few E2E tests for irreducible critical journeys;
  • exploratory or user evaluation for unknown, subjective, or human-centered risks.

Record why each rejected technique would add less confidence or more cost. In an Obligation pass, stop at the architecture-independent technique family; the Portfolio or Combined pass fixes the concrete harness and scope.

5. Select scope and fidelity

Describe every proposed test by its:

System under test | exercised dependencies | environment | stimulus | oracle

stimulus is the input, actor action, event, state transition, elapsed time, or injected fault exercised by the test. It is not the CI/CD execution trigger (push, pull request, schedule, manual dispatch, deployment, or production event), which belongs to ci-cd-reliability-architecture.

Only then apply labels such as unit, component, integration, contract, system, E2E, or production probe. Labels vary across teams; the five dimensions above must not.

Compare candidate scopes on speed, maintainability, utilization, reliability, and fidelity with the scorecard in references/portfolio-governance.md.

Prefer the smallest candidate whose fidelity can observe the failure. Keep a broader test only when it covers composition, environment, or behavior the smaller test cannot.

6. Decide dependencies, data, and environment

Read the same reference for test doubles, production-compatible dependencies, test data, environment isolation, quarantine policy, adequacy signals, and suite budgets.

Apply these defaults:

  • Use real, deterministic, in-process owned collaborators.
  • Stub slow, nondeterministic, destructive, or unavailable side effects when testing local decisions.
  • Use a mock only when the interaction itself is required behavior.
  • Exercise the real dependency when compatibility or provider semantics are the risk.
  • Protect every double that represents an external boundary with a shared schema, executable contract, or real-boundary backstop.
  • Make data self-contained, resettable, privacy-safe, versioned where needed, and provisionable in parallel.

7. Complete the quality scan

Check the applicable surface without inventing requirements:

  • functional correctness and actor outcomes;
  • compatibility, concurrency, version skew, and migrations;
  • performance, capacity, and resource use;
  • reliability, resilience, recovery, and data integrity;
  • security, privacy, compliance, and auditability;
  • usability, accessibility, and supported environments;
  • operability, configuration, deployment, and production verification;
  • aspects: for every aspect that holds across the subject, one oracle per governed subsystem or one generative check over the governed set — a hand-written test per cell is the n-copies smell.

When a quality area is material but lacks an authoritative requirement or specialist policy, record a blocker or route to the matching companion skill. Do not manufacture thresholds.

8. Audit the portfolio

In Audit mode:

  1. Map current tests to risks, criteria, boundaries, and observed incidents.
  2. Find missing obligations, hollow oracles, stale tests, unverified doubles, excessive E2E coverage, and same-scope duplicates. A criterion tag whose assertion does not exercise the criterion is a hollow oracle.
  3. Treat every escaped defect as evidence that the portfolio missed a risk, technique, oracle, fidelity, or placement decision.
  4. Treat flakes, order dependence, uncontrolled time, shared state, and environment leakage as test defects.
  5. Use coverage as a gap locator. Use mutation testing selectively on changed or high-risk deterministic logic when assertion strength is uncertain.
  6. Remove a redundant broader test only after the smaller replacement proves the same scope; keep distinct broader backstops.

A quarantine must name an owner, reason, expiry, and retained signal. A quarantined test cannot count as verified evidence.

9. Hand off

  • Hand selected checks to defect-shift-left only after a Portfolio or Combined pass finalizes their architecture-dependent fields.
  • Send pipeline execution triggers, schedules, blocking behavior, artifacts, freshness, and failure actions to ci-cd-reliability-architecture.
  • Send criterion mappings and executed results to requirements-traceability.
  • Send suite latency, resource cost, flakiness, queues, and duplication to system-optimization.
  • Send a gate aggregator that could count skipped, retried, or quarantined results as verified to gate-lifecycle.
  • Send security, accessibility, safety, UX, data, or domain-specific obligations to the applicable companion skill without weakening them.

Decision Rules

ConditionDecision
Every material risk has a credible oracle and sufficient evidence pathADEQUATE
The critical path is covered but named residual gaps remainPARTIAL
A material risk has no credible evidence path, or the portfolio gives false confidenceNOT-ADEQUATE
Meaning, oracle, environment, specialist policy, or owner is unresolvedDEFER

An Obligation pass is provisional by design and cannot emit ADEQUATE. Reserve ADEQUATE for a Portfolio or Combined pass against accepted architecture.

Do not emit ADEQUATE from test counts, pyramid shape, coverage percentage, green CI, or one successful release.

Output Contract

Emit one row per material obligation:

Risk / criterionFailure modeOracleTechniqueScope / fidelityDependenciesData / environmentEvidenceResidual blind spot

Then emit:

Subject:              <requirement, increment, subsystem, change, or system>
Mode:                 Design | Audit
Strategy pass:        Obligation | Portfolio | Combined
Decision:             ADEQUATE | PARTIAL | NOT-ADEQUATE | DEFER
Test basis:           <requirements, criteria, contracts, incidents, signals>
Architecture basis:   <not yet fixed | accepted decision refs | stable existing>
Quality risks:        <ranked risks or none found>
Manual evaluation:    <exploratory, usability, accessibility, acceptance, none>
Adequacy signals:     <trace coverage, gaps, mutation, escaped defects, flakes>
Handoffs:             <shift-left, CI/CD, traceability, optimization, companions>
Residual risk:        <accepted, blocked, deferred, unknown, or none identified>
Next action:          <one concrete evidence, decision, or removal step>
Verification:         <portfolio/test review, commands, results, or Not run + reason>

Guardrails

  • Do not invent requirement meaning, thresholds, users, traffic, or risk acceptance.
  • Do not use a test framework preference as a strategy.
  • Do not require every technique for every system.
  • Do not equate mocks with isolation or real dependencies with confidence.
  • Do not automate subjective evaluation merely to make it countable.
  • Do not retain expensive tests because they were expensive to create.
  • Do not claim an ownership boundary listed under Boundary; hand it off.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.