Flaky test triage

Investigate intermittent test failures using empirical CI history and controlled reproductions. Fix the underlying root cause with the smallest behavior-preserving change while maintaining complete test assertions and coverage.

Triggers

• A test exhibits inconsistent outcomes (intermittent pass and fail) across recent CI runs or controlled local reproductions on the same revision.
• A test failure is sensitive to timing, execution ordering, concurrency, shared state, platform differences, or transient external dependencies.
• A maintenance task or CI report flags a specific test as flaky and requests investigation and remediation.

Non-triggers

• Deterministic regressions or consistently failing tests caused by recent intentional code or interface changes (treat these as product defects or necessary test updates).
• Broad requests to redesign the test suite, migrate test frameworks, or rewrite large test files.
• Masking flakiness by deleting tests, skipping tests (@skip, test.skip), weakening assertions, inserting arbitrary sleeps (sleep()), or increasing test-runner retry counts without fixing the defect.
• Modifying live shared services, CI runner infrastructure, or production environments (treat live systems and configuration as read-only).

Process

1. Establish the evidence. Identify the test identifier, commit/revision, job name, and environment. Inspect recent runs of the same revision: pass/fail outcomes, timestamps, retries, worker/shard IDs, platform details, and failure logs. Compare against nearby revisions only to identify relevant code changes. Do not diagnose flakiness from a single failure or retry.
2. Reproduce narrowly. Run the specific test repeatedly using standard project commands. Vary one suspected environmental factor at a time (execution order, concurrency/parallelism, system clock, platform, or dependency availability). Record exact commands and outcomes. Avoid altering production configurations or shared services; treat live systems as read-only and use isolated local resources or approved test doubles. If reproduction fails, state the limitation rather than inventing a cause.
3. Form and test a root-cause hypothesis. Look for unbounded waits, shared mutable state, leaked fixtures between tests, race conditions, clock/time-zone assumptions, non-deterministic iteration order, or unmanaged filesystem/network dependencies. Use logs, traces, and targeted minimal assertions to discriminate hypotheses. Separate observed facts from inferences and calibrate confidence to the evidence.
4. Choose the smallest behavior-preserving fix. Prefer deterministic synchronization (explicit condition polling or wait-for-predicate with timeout instead of static sleeps), explicit cleanup and isolation between tests, controlled virtual clocks or seeds, or scoped test doubles. Preserve all meaningful assertions and test coverage. Do not perform broad rewrites. Change production code only when evidence demonstrates a genuine product defect.
5. Prepare a reviewable diff. Formulate the change as a minimal, reviewable diff or draft PR limited to the root-cause fix and necessary fixtures. Ensure live configuration and systems remain read-only. Include rollback considerations if production code was modified.

Evidence and Reporting

Provide a structured report containing:

• Test and Evidence: Test identifier, CI run history and revisions examined, reproduction commands, repetition counts, and pass/fail distribution.
• Root Cause: Hypothesized mechanism supported by log/trace observations, with confidence calibrated as High, Medium, or Low with stated reasoning.
• Proposed Change: Minimal fix diff explaining why it resolves the nondeterminism while preserving assertions, or justification if no change was warranted.
• Uncertainty & Risks: Unresolved alternative hypotheses, missing CI logs, platform-specific unknowns, or residual risks.

Verification

• Run the targeted test repeatedly (e.g., 20–50 consecutive passes under stress or concurrency) to demonstrate stability.
• Execute the full enclosing test module and required project test suites across all supported platform matrices.
• Verify that the test still fails when the underlying defect is reintroduced, confirming assertion integrity.
• Document exact execution commands, environments, repetition counts, runtimes, and outcomes; do not claim stability from a single passing run.

Example

Trigger: CI job reports intermittent failures in tests/services/test_report_generator.py::test_concurrent_export across pull request runs on the same commit revision.

Investigation & Evidence:

• CI log analysis reveals the test passes reliably in single-threaded runs but fails intermittently in parallel test shards (pytest -n auto).
• Controlled reproduction: 30 consecutive single-worker runs pass; running under pytest -n 4 reproduces 6 failures out of 30 runs (hypothetical illustrative rate).
• Root-cause diagnosis: The test writes output to a fixed shared file path /tmp/export_result.json instead of a worker-isolated path, causing file-access collisions between concurrent parallel test runners.

Minimal Fix & Verification:

• Replace the fixed path with an isolated temporary directory fixture (tmp_path / "export_result.json"). All assertions validating exported payload structure and record counts are strictly preserved.
• Verification: 50 consecutive runs under pytest -n 4 pass with zero failures. Reintroducing the shared path reproduces the failure, verifying assertion integrity.