agentleFS
ekirtonekirton/Poule/.claude/skills/writing-tests/SKILL.md

writing-tests

Skill0 starsChanged 8 months agoApache-2.0View on GitHub

Test writing standards — formula-derived bounds, tests must fail to reveal, mock discipline, contract tests. Use when creating or editing test/ files.

42 other projects use a version of this.

Adopted by

Most recently taken up by

Used with

Projects with it also use reviewing-code, commits and 5 more.

Other files people use alongside this one.

Files found in the same public projects as a version of this one, more often than they turn up anywhere else. Each project counts once, large collections count for less, and this file's own repository is left out.
  • bostonaholicSkill

    reviewing-code

    Code review methodology for evaluating implementation changes. Use when reviewing code changes for quality, design, correctness, and maintainability. Focuses on changes made during implementation using Conventional Comments for clear, actionable feedback.

    Also in 4 projects that use this

  • JetBrainsSkill

    commits

    Use this skill whenever the user asks to commit changes, write or fix a commit message, amend or rename a commit, or do a workflow that includes committing in the IntelliJ repository. This is a thin repo-specific overlay: use IntelliJ commit format, write full commit messages by default, and keep requested suffixes such as IJ-MR trailers in a final separate paragraph.

    Also in 3 projects that use this

  • OpenAISkill

    running-tests

    Guide for running tests in the openai-dotnet repository. Use this when asked to run, debug, or validate tests, or when writing new tests. Explains test modes (Playback, Record, Live), how to identify recorded vs non-recorded tests, environment variable configuration, and what to do when recordings are missing or stale.

    Also in 3 projects that use this

  • BlockSkill

    debugging

    Debugging guidelines and best practices

    Also in 5 projects that use this

  • VercelSkill

    technical-writing

    Write, edit, review, or audit user-facing documentation for the eve repository. Use for changes under docs/, documentation tied to eve APIs or CLI behavior, docs work based on Slack or support feedback, and requests to make eve docs clearer, more natural, or less AI-patterned while verifying claims against current source, tests, CLI help, public releases, and repository conventions.

    Also in 3 projects that use this

  • SentrySkill

    security-review

    Finds exploitable application security vulnerabilities in code changes. Use for Warden security scans, appsec review, OWASP-style checks, authentication or authorization bugs, injection, XSS, SSRF, path traversal, secrets, unsafe crypto, webhook verification, open redirects, or sensitive data exposure.

    Also in 3 projects that use this

The file

We split this skill into the rules it gives an agent

6 rules. 0 are common, 3 are distinctive, and 3 are very specific to this project.

  • 0

    Common

    Most of the 42 edited versions we read keep these.

  • 3

    Distinctive

    This author's own choices. Worth a look.

  • 3

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    When a specification defines a formula, all test bounds and expected values must be computed from that formula -- never estimated by …

    line 10
  2. Distinctive:

    No conditional verification: If a contract requires fields to be populated, assert they are populated unconditionally.

    line 33
  3. Distinctive:

    Prefer a failing test over a passing one: When an implementation bug causes a test to pass for the wrong reason (e.g., empty output …

    line 34
  4. Very specific:

    Every Mock() or patch() requires a corresponding contract test that exercises the real implementation against the same interface.

    line 39
  5. Very specific:

    Before declaring a task phase complete, verify every Mock()/patch() has a corresponding contract test.

    line 59
  6. Very specific:

    When a test appears to conflict with its specification, file feedback in test/feedback/<test-file-name>.md describing the discrepancy.

    line 67

“Common” is measured: kept by at least 6 in 10 of the 42 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Test Writing Standards

Numeric Bounds Must Be Formula-Derived

When a specification defines a formula, all test bounds and expected values must be computed from that formula -- never estimated by intuition.

  • Compute the expected value by substituting the test input into the spec formula before choosing an assertion bound.
  • Show the derivation in a comment next to the assertion so reviewers can verify it.
  • Do not use "round number" bounds (e.g., < 1.01) unless the formula confirms they hold at the chosen input.

Example -- wrong:

# "Should be very close to 1.0 for large freq"
assert symbol_weight(1_000_000) < 1.01  # intuition, not derived

Example -- correct:

# 1.0 + 2.0 / log2(1_000_001) ~ 1.1003
assert symbol_weight(1_000_000) < 1.2

Tests Must Fail to Reveal, Not Pass to Reassure

A test that cannot fail is worthless. Every assertion must be capable of catching a real deficiency in the implementation. A passing test should mean the implementation satisfies the spec -- not that the test avoided checking.

  • No vacuous assertions. assert isinstance(result, list) followed by if result: assert ... verifies nothing when the implementation returns [] due to a bug. If the spec says the list should be populated, assert len(result) > 0.
  • No conditional verification. If a contract requires fields to be populated, assert they are populated unconditionally. A test that only checks field values "if they exist" will pass when the implementation silently returns empty data.
  • Prefer a failing test over a passing one. When an implementation bug causes a test to pass for the wrong reason (e.g., empty output triggers the correct error path by accident), rewrite the test so it fails until the bug is fixed.
  • Contract tests must exercise the real interface. A contract test that gets no data from the real backend (due to a transport bug, timeout, etc.) and passes anyway is not a contract test -- it is dead code with a green checkmark.

Mock Discipline

Every Mock() or patch() requires a corresponding contract test that exercises the real implementation against the same interface. Skipping via pytest marker (e.g., @pytest.mark.requires_coq) is acceptable when external tools are needed; omitting the test is not.

# Good: consumer test mocks the backend
def test_pipeline_calls_backend():
    backend = Mock()
    backend.list_declarations.return_value = [("A", "Lemma", {})]
    ...

# Good: contract test verifies real backend satisfies the same interface
@pytest.mark.requires_coq
def test_coq_lsp_backend_list_declarations():
    backend = CoqLspBackend(...)
    decls = backend.list_declarations(Path("test_fixture.vo"))
    assert isinstance(decls, list)
    assert all(len(d) == 3 for d in decls)

# Bad: consumer test exists but no contract test for the real implementation

Before declaring a task phase complete, verify every Mock()/patch() has a corresponding contract test. If not, the phase is incomplete.

Mock Return Values Must Use Real Types

Mock return_value must use the actual type the real implementation returns (e.g., dataclass, not dict). This exercises serialization and attribute-access paths in the consumer.

Test File Feedback

When a test appears to conflict with its specification, file feedback in test/feedback/<test-file-name>.md describing the discrepancy. Do not silently adjust the test or the implementation.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.