agentleFS
KushpawanKushpawan/kmp-mobile-skills/skills/kmp-flaky-test-triage/SKILL.md

kmp-flaky-test-triage

Skill15 starsChanged 25 days agoApache-2.0View on GitHub

Use when a Kotlin Multiplatform test is flaky — passes and fails without code changes, fails only on CI, only on iOS or Kotlin/Native, only with the whole suite or in another order; runTest timeouts, Turbine "No value produced in", "Module with the Main dispatcher had failed to initialize", missed SharedFlow or StateFlow emissions, "A Koin Application has already been started", or results that depend on time zone, locale or the clock.

The file

We split this skill into the rules it gives an agent

6 rules. About 0 are common, 2 are distinctive, and 4 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈2

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 4

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  • could be automatic: a setting could enforce it
  1. Very specific:

    CI.: Test retry plugins (org.gradle.test-retry) or retry wrappers that mask flakiness, maxParallelForks, build cache use, the machine type …

    line 49
  2. Very specific:

    Follow the project's patterns for any fix: if it already has a DispatcherProvider or a MainDispatcherRule, use it rather than adding a …

    line 52
  3. Distinctive:

    Run the single test many times in fresh processes, then in the same process, then with the

    line 67
  4. Very specific:

    Start collecting a SharedFlow before the action that emits (Turbine test {} or backgroundScope.launch), and never assert on a transient …

    line 133
  5. Very specific:

    Never: add delay/Thread.sleep, raise a timeout to make it pass, add a retry, or mark it ignored.

    line 138
  6. Distinctive:

    If production code has a real race, fix the production code — the flaky test found a bug.

    line 139

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

KMP flaky test triage

A flaky test depends on something it does not control: real time, a thread it did not choose, state left by another test, the order of events, the machine's clock or locale. Find that input and put it under the test's control. Retries, longer timeouts, delay and @Ignore hide the problem and usually hide a real bug in production code too.

When to use

  • A test that passes and fails on the same code; "fails about one run in five on CI".
  • A test that fails only on CI, only on the iOS simulator, only in the full suite, or only after another test.
  • runTest timeouts, Turbine timeouts, Dispatchers.Main errors, missed or extra flow emissions.

When not to use

  • The test fails every time → it is a real failure; fix it with mobile-bug-fix.
  • The code under test crashes the app → kmp-crash-triage.
  • The test task does not compile or link → kmp-build-failure-triage.
  • Setting up test infrastructure from scratch → the test setup guidance in android/skills testing/testing-setup for the Android side; this skill is about making existing tests deterministic.

Step 0 — Read the project first

  1. Test stack. From gradle/libs.versions.toml and test sources: kotlin.test, JUnit 4 or 5 on the Android host, Kotest, kotlinx-coroutines-test (version), Turbine, MockK or hand-written fakes, Ktor MockEngine.
  2. Task names per target. ./gradlew :shared:tasks --group verification — e.g. testDebugUnitTest (androidTarget + com.android.library), testAndroidHostTest (KMP Android library plugin), jvmTest, iosSimulatorArm64Test, allTests.
  3. Dispatchers. How production code gets dispatchers: an injected DispatcherProvider or CoroutineDispatcher parameter, or hard-coded Dispatchers.IO / Dispatchers.Default. How tests set Dispatchers.Main: a base class, a helper, a JUnit rule (Android host only).
  4. Shared state. object singletons, top-level vals with caches, Koin started in tests (startKoin, KoinTest), databases or files used by tests, NSUserDefaults / preferences.
  5. Time. Does production code call Clock.System.now(), TimeZone.currentSystemDefault(), Random, UUID/Uuid.random() directly, or receive them injected?
  6. CI. Test retry plugins (org.gradle.test-retry) or retry wrappers that mask flakiness, maxParallelForks, build cache use, the machine type for iOS tests.

Follow the project's patterns for any fix: if it already has a DispatcherProvider or a MainDispatcherRule, use it rather than adding a second mechanism.

Workflow

1. Capture the failure

Record: the exact test (class and method), the target (Android host / JVM / iOS simulator), local or CI, the assertion message and stack, how often it fails, and what changed recently. Pull CI's JUnit XML from build/test-results/<testTask>/ — it holds the full message and stdout.

Done when you know which target fails and have the failing assertion text.

2. Reproduce by repetition

Run the single test many times in fresh processes, then in the same process, then with the whole class / suite. Each mode exposes a different cause.

# Android host / JVM: one test, 50 runs, each in a fresh test JVM. Stop at the first failure.
# --rerun forces the task to execute even when Gradle considers it up to date.
for i in $(seq 1 50); do
  ./gradlew :shared:testDebugUnitTest --tests 'com.example.app.notes.NotesViewModelTest' --rerun -q \
    || { echo "FAILED on run $i"; break; }
done

# iOS simulator through Gradle — same loop, iOS task (macOS only)
for i in $(seq 1 20); do
  ./gradlew :shared:iosSimulatorArm64Test --tests 'com.example.app.notes.NotesViewModelTest' --rerun -q \
    || { echo "FAILED on run $i"; break; }
done

--rerun (Gradle 7.6+) re-executes just that task; on older Gradle use --rerun-tasks (reruns every task) or run cleanTest<Task> first. If the output says FROM-CACHE, add --no-build-cache.

Repeat inside one process on Kotlin/Native — this surfaces state leaking between runs:

./gradlew :shared:linkDebugTestIosSimulatorArm64          # -> shared/build/bin/iosSimulatorArm64/debugTest/test.kexe
xcrun simctl boot "<an installed iPhone simulator>"        # once; list with: xcrun simctl list devices available
BIN=shared/build/bin/iosSimulatorArm64/debugTest/test.kexe
xcrun simctl spawn booted "$BIN" --ktest_list_tests | grep NotesViewModelTest   # exact test names
xcrun simctl spawn booted "$BIN" --ktest_filter='com.example.app.notes.NotesViewModelTest.*' --ktest_repeat=50

Then run the whole class and the whole module suite in the same loops. Passing alone but failing in the suite means order dependence or shared state.

To check time-zone dependence quickly, run once with an extreme zone: TZ=Pacific/Kiritimati ./gradlew --no-daemon … for JVM tests, or SIMCTL_CHILD_TZ=Pacific/Kiritimati xcrun simctl spawn booted "$BIN" … for the iOS binary (simctl passes SIMCTL_CHILD_-prefixed variables to the child). Run the loop while the machine is busy (a second build) to approximate a loaded CI runner.

Done when you have a failure rate, e.g. "7 / 50 alone, 0 / 50 in-process, 10 / 10 after OrdersRepositoryTest".

3. Isolate

Shrink to the smallest failing setup: one test alone; the test plus the one test that runs before it; one target. To find a polluting test, bisect the list of tests that ran before the failure (run the first half + the flaky test, then the second half). Note the target: iOS-only flakes are usually threading, main-queue or GC-timing issues; JVM-only ones are often Dispatchers.Main, Looper or file-system issues.

4. Classify

Match the evidence against the causes table. Write the cause as one sentence: "The test asserts before the repository coroutine on Dispatchers.IO finishes, so it passes only when the IO thread wins the race." If no row fits, add temporary logging (thread name, timestamps) through the project's logger and rerun the loop.

5. Fix the cause

Change the test, the production code, or both, so the uncontrolled input becomes controlled. Patterns with code: references/fix-patterns.md. Common ones:

  • Inject dispatchers into production code; in tests, build every dispatcher from the test's testScheduler so runTest controls all of them.
  • Set Dispatchers.Main before each test and reset it after.
  • Start collecting a SharedFlow before the action that emits (Turbine test {} or backgroundScope.launch), and never assert on a transient StateFlow value you did not control.
  • Give each test fresh instances; stop Koin after each test or avoid the global context.
  • Inject Clock, TimeZone and Random; use fixed values in tests.

Never: add delay/Thread.sleep, raise a timeout to make it pass, add a retry, or mark it ignored. If production code has a real race, fix the production code — the flaky test found a bug.

6. Prove it is stable

Rerun the same loops from step 2 on both platforms and record the numbers:

Before   NotesViewModelTest: 7/50 failed (Android host, fresh JVM), 3/50 (iOS, --ktest_repeat)
After    0/100 Android host, 0/100 iOS in-process, full module suite 10/10 on both
Cause    <one sentence>      Fix  <file:line, what changed>

A fix proven only on the platform where it flaked is not done: the same test runs on every target.

Quick reference — causes

SymptomCauseFix
Passes locally, fails on slower CIReal time: delay, Thread.sleep, withTimeout on a real dispatcherVirtual time via runTest and injected dispatchers
Module with the Main dispatcher had failed to initialize or …is missing (JVM), or a hang on iOSDispatchers.Main used without Dispatchers.setMainsetMain in @BeforeTest
Next test behaves oddly after a failureDispatchers.resetMain() not calledresetMain in @AfterTest
Assertion sees old state, sometimesProduction code hard-codes Dispatchers.IO / Default; work races the assertionInject the dispatcher; use the test scheduler
State never updates in the testStandardTestDispatcher queued the work; nothing advanced itadvanceUntilIdle() / runCurrent(), or UnconfinedTestDispatcher where eager is intended
Two schedulers: delays not skipped, odd orderingA second TestCoroutineScheduler (new dispatcher not built from testScheduler)Build every test dispatcher from testScheduler
One-shot event "sometimes missing"SharedFlow (replay 0) emitted before collection startedCollect first: Turbine test {} or backgroundScope collector
Loading state "sometimes missing"StateFlow conflates; fast updates skip intermediate valuesControl the dispatcher and step with runCurrent(), or assert only the final state
Turbine No value produced in 3s on CIEmission blocked by a dispatcher problem (Turbine's timeout is wall-clock, default 3 s)Fix the dispatcher; do not just raise the timeout
runTest fails after 60 sA coroutine never completes (infinite collector in the test scope)Launch long-running collectors in backgroundScope
A Koin Application has already been startedGlobal Koin from a previous test still runningstopKoin() in @AfterTest, or an isolated koinApplication {}
Passes alone, fails in the suiteShared mutable state: object caches, fakes reused across tests, DB filesFresh instances per test; reset or avoid globals
Fails near midnight, month end, or on a UTC CIClock.System.now() / system time zone used directlyInject Clock and TimeZone; fixed instants in tests
Fails on machines with another languageLocale-dependent formatting or parsingExplicit locale/format in production code; fixed in tests
Different result per run with random dataUnseeded Random, random UUIDs in assertionsInject a seeded Random / ID generator
Fails on one target only, collections involvedHashMap / HashSet iteration order is unspecified and can differ by platformAssert on sorted lists or set equality
Fails only on iOS, involves deinit / weak refsSwift/ObjC objects held by Kotlin are freed only after a Kotlin GCDo not assert release timing; test the cleanup call instead
Fails only on iOS on the second runFiles or NSUserDefaults persisted in the simulator between runsUnique temp directories; clean up in @AfterTest
Fails when tests run in parallelShared database file, port or temp path across test JVMsIn-memory DB or unique names per test
Network-dependent resultReal HTTP in a unit testKtor MockEngine or a fake data source
"Fixed" but the test did not runGradle up-to-date or build-cache hit--rerun, --no-build-cache; check the test count in the report

Verify

Report each check as PASS, FAIL or NOT RUN (reason), with run counts. Never report a check as passed that you did not run.

# Single test, fresh processes, both platforms (counts in the report)
for i in $(seq 1 100); do ./gradlew :shared:testDebugUnitTest --tests '<Test>' --rerun -q || break; done
for i in $(seq 1 30);  do ./gradlew :shared:iosSimulatorArm64Test --tests '<Test>' --rerun -q || break; done
# Same process, iOS
xcrun simctl spawn booted shared/build/bin/iosSimulatorArm64/debugTest/test.kexe --ktest_filter='<Class>.*' --ktest_repeat=100
# Whole suite, every target, several times
for i in $(seq 1 5); do ./gradlew :shared:allTests --rerun-tasks -q || break; done

Also confirm in the test report that the test actually executed (count > 0), and that no retry plugin reran it.

Common mistakes

MistakeWhy it bitesFix
Adding delay(100) before the assertionWins the race on your laptop, loses on CIControl the dispatcher; advance virtual time
Raising the Turbine or runTest timeoutThe event still never arrives on slow runnersFind why it is not emitted
Enabling a test retry pluginFlakiness — and the production race — ship anywayFix the cause; keep retries off for unit tests
Test dispatchers that do not share one scheduler (setMain inside the test body, a fresh StandardTestDispatcher() for an injected dispatcher)Main, injected dispatchers and runTest run on different virtual clockssetMain in @BeforeTest (then runTest adopts its scheduler); build injected dispatchers from testScheduler
Collecting a SharedFlow after calling the actionThe emission is already goneStart the collector first
Asserting every StateFlow value in orderConflation drops values depending on timingAssert what the UI needs: the settled state
Proving the fix only on the JVMThe same commonTest also runs on Kotlin/NativeLoop the iOS task and the in-process repeat too
Fixing only the test when production code racesUsers hit the race the test foundFix production code; keep the test

Related skills

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.