flaky-test-finder

You are detecting flaky Apex tests by re-running them and looking for inconsistent results. A test is flaky if it passes some runs and fails others without code changes between them.

Read Project Config First

source "${CLAUDE_PLUGIN_ROOT}/hooks/lib/config.sh"
source "${CLAUDE_PLUGIN_ROOT}/hooks/lib/sf-cli.sh"
sf_cli_check || exit 2
ORG="$(sf_config_get '.platform.defaultTargetOrg' "$ENV")"
APEX_SRC="$(sf_config_get '.paths.apexSource' "$ENV")"
TEST_SUFFIX="$(sf_config_get '.naming.apex.testSuffix' "$ENV")"

Input

$ARGUMENTS: required.

• <TestClass> — class to scan (e.g., OrderServiceTest)
• <ProductionClass> — production class; the skill infers <ProductionClass><testSuffix>
• --runs <n> — how many times to run (default 5)
• --methods <m1,m2,...> — only run specific test methods
• --target-org <alias> / --env <name> — standard overrides
• --ci — emit findings instead of a Markdown report

Steps

1. Resolve the test class

• If <arg> ends in <testSuffix>, treat as test class directly
• Else, treat as production class and append <testSuffix>
• Verify the file exists in ${APEX_SRC}

2. Discover test methods (if --methods not specified)

Grep the class for @isTest static void <name> patterns:

grep -E "@isTest\s+(public|private)?\s*static\s+void\s+(\w+)" "${APEX_SRC}/${TEST_CLASS}.cls" | sed -E 's/.*void (\w+).*/\1/'

3. Run each method N times

for i in $(seq 1 "$RUNS"); do
  sf apex run test --target-org "$ORG" --tests "${TEST_CLASS}.${METHOD}" --result-format json --synchronous --wait 10 > "/tmp/flaky-${TEST_CLASS}-${METHOD}-${i}.json"
done

--synchronous is important here — async runs queue and may interfere with each other.

4. Aggregate per method

For each method, count passes vs. fails across runs. Capture the failure messages so the user can see whether they vary.

5. Output

Markdown report:

# Flaky Test Report: <TestClass>

Org: <ORG> | Runs per method: 5

| Method                    | Pass | Fail | Verdict |
|---------------------------|------|------|---------|
| createOrder_returnsId     | 5/5  | 0/5  | ✅ stable |
| persist_handlesBulkUpdate | 3/5  | 2/5  | ⚠️ flaky |
| ...

## Flaky details

### persist_handlesBulkUpdate
Failed 2 of 5 runs. Failure messages:
- Run 2: "System.QueryException: Record not found"
- Run 4: "System.AssertException: Expected: 200, Actual: 199"

Common flakiness causes for this signature:
- Async work not flushed before assertion (missing Test.stopTest())
- Time-dependent assertions (Date.today() at midnight rollover)
- Shared state across test methods (static caches not reset)
- Order-dependent SOQL (no ORDER BY on lists asserted by index)

CI mode: emit one finding per flaky method:

• ruleId: "FLAKY-TEST", severity: "warning", file = test class file, line = method start, message includes pass/fail counts and a hint at likely cause

6. Exit codes

• 0 — all stable (or all failed deterministically — that's a different problem)
• 1 — at least one method is flaky (mixed pass/fail across runs)
• 2 — invocation error

Rules

• Run synchronously. --synchronous keeps runs from interfering. Async batches queue and can deliver out of order
• Don't auto-fix. This skill detects; the user fixes
• Don't mute or skip flaky tests. Muting hides bugs. The fix is always to make the test deterministic (mock time, isolate state, use Test.startTest/Test.stopTest correctly)
• Be quick about it. Default 5 runs is a balance — enough to catch most flakiness, fast enough to iterate
• Persist history. Append the per-method pass/fail rates to ${CLAUDE_PLUGIN_DATA}/argo/flaky/<project>/<TestClass>.jsonl so trends are visible across runs

Likely causes catalog

When you flag a test as flaky, prepend the most likely cause to the report based on heuristics from the test source: