ABOUTME: Test quality assessment using Farley's 8 Properties of Good Tests

ABOUTME: Detects tautological tests, mock theatre, and structural test weaknesses

Test Design Reviewer

Process

Step 1: Collect test files

Identify all test files in scope. Use language-appropriate patterns:

• Go: *_test.go
• Python: test_*.py, *_test.py
• Ruby: *_spec.rb
• JS/TS: *.test.ts, *.spec.ts

Step 2: Score against Farley's 8 Properties

Rate each property 0-10 across the test suite. Provide evidence.

Scoring methodology:

• Static scoring: compute 0-10 per property using sigmoid normalization on signal densities (negative signals / test methods, positive signals / test methods). Use lib/cli_calculator.py for deterministic math (JSON in, JSON out).
• LLM scoring: assess holistically per property, focusing on semantic aspects static analysis misses (naming quality, assertion appropriateness, tautology theatre)
• Blend: final_property_score = 0.60 * static_score + 0.40 * llm_score per property
• Conservative default: when no signals detected for a property, default to 5.0 (unknown quality, not good quality)

Per-property scoring rubrics: anchor each 0-10 score to the full 8-properties x 5-bands matrix in references/bands.md. Rough guide: 9-10 exemplary, 7-8 strong, 5-6 mixed, 3-4 weak, 1-2 harmful.

Aggregation methodology:

• Per-test-method: collect signals at individual method level
• Per-test-file: mean for positive signals, P90 for negative signals (worst offenders must surface)
• Per-test-suite: LOC-weighted mean across files

Sampling for large suites:

• Under 50 test files: analyze all
• Over 50: SHA-256 deterministic selection (30%) plus all files exceeding 100 test methods

Weighted Farley Index = (U*1.5 + M*1.5 + R*1.25 + A*1.0 + N*1.0 + G*1.0 + F*0.75 + T*1.0) / 9.0

Divisor is 9.0 (sum of weights), not 8 (number of properties). U/M weighted highest (readability, coupling); F weighted lowest (speed is contextual).

Step 3: Tautology Theatre Detection

The critical question: "Would this test still pass if all production code were deleted?"

Scan for these 4 patterns:

Also scan for mock interaction anti-patterns (affect Maintainable score):

For each tautology or anti-pattern found: report the file, line, pattern type, and why it's problematic.

In Go, a mutant that survives gremlins unleash on the package under test answers the critical question computationally: the production code changed and the tests still passed. Cite the surviving mutant (file, line, mutation type) as the evidence for the tautology claim instead of arguing it in prose.

Step 4: Report

## Test Design Review

### Farley Index: X.X / 10.0 (Rating)

| Property | Static | LLM | Blended | Weight | Weighted | Key Evidence |
|----------|--------|-----|---------|--------|----------|--------------|
| Understandable | X.X | X.X | X.X | 1.50x | X.XX | ... |
| Maintainable | X.X | X.X | X.X | 1.50x | X.XX | ... |
| Repeatable | X.X | X.X | X.X | 1.25x | X.XX | ... |
| Atomic | X.X | X.X | X.X | 1.00x | X.XX | ... |
| Necessary | X.X | X.X | X.X | 1.00x | X.XX | ... |
| Granular | X.X | X.X | X.X | 1.00x | X.XX | ... |
| Fast | X.X | X.X | X.X | 0.75x | X.XX | ... |
| First (TDD) | X.X | X.X | X.X | 1.00x | X.XX | ... |

### Tautology Theatre Analysis

Each subsection always present; use "None detected." when empty.

#### Mock Tautologies
| Test Method | Line | Mock Setup | Assertion |
#### Mock-Only Tests
| Test Method | Line | Evidence |
#### Trivial Tautologies
| Test Method | Line | Assertion |
#### Framework Tests
| Test Method | Line | Assertion | What It Actually Tests |

**Summary**: {total} instances across {affected}/{total_methods} test methods.

### Top 3 Improvements
1. [Highest-impact fix targeting weakest high-weight property]
2. [Second priority]
3. [Third priority]

### Methodology Notes
- Static/LLM blend: 60/40
- Files analyzed: {count} ({sampling note})
- Language: {lang}, Framework: {framework}

Integration with Review Pipeline

This skill is invoked by the orchestrator when test files are in scope (see the orchestrator skill, review routing step). Can also be invoked directly via /test-design-reviewer.

Deterministic Scoring Calculator

See references/scoring.md.

Common Issues

See references/scoring.md.