kmp-flaky-test-triage
Skill15 starsChanged 25 days agoApache-2.0View on GitHub
Use when a Kotlin Multiplatform test is flaky — passes and fails without code changes, fails only on CI, only on iOS or Kotlin/Native, only with the whole suite or in another order; runTest timeouts, Turbine "No value produced in", "Module with the Main dispatcher had failed to initialize", missed SharedFlow or StateFlow emissions, "A Koin Application has already been started", or results that depend on time zone, locale or the clock.
In this repository34 other files it gives its agents.
- AGENTS.mdAGENTS.md
- android-to-kmp-migrationSkill
- compose-design-system-componentSkill
- compose-multiplatform-localizationSkill
- compose-multiplatform-navigationSkill
- compose-multiplatform-screenSkill
- figma-to-mobile-uiSkill
- kmp-api-endpointSkill
- kmp-build-checkSkill
- kmp-build-failure-triageSkill
- kmp-ci-github-actionsSkill
- kmp-code-reviewSkill
- kmp-crash-triageSkill
- kmp-data-layerSkill
- kmp-dev-machine-cleanupSkill
- kmp-expect-actualSkill
- kmp-feature-moduleSkill
- kmp-gradle-version-catalogSkill
- kmp-koin-diSkill
- kmp-module-healthSkill
- kmp-room-databaseSkill
- kmp-secure-storageSkill
- kmp-swiftui-interopSkill
- kmp-test-coverage-backfillSkill
- kmp-testing-patternsSkill
- kmp-xcframework-distributionSkill
- mobile-accessibility-test-idsSkill
- mobile-bug-fixSkill
- mobile-feature-implementSkill
- mobile-feature-planSkill
- mobile-performance-fixSkill
- mobile-platform-paritySkill
- mobile-releaseSkill
- mobile-security-reviewSkill
The file
We split this skill into the rules it gives an agent
6 rules. About 0 are common, 2 are distinctive, and 4 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈2
Distinctive · estimate
This author's own choices. Worth a look.
4
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- could be automatic: a setting could enforce it
- Very specific: line 49
CI.: Test retry plugins (org.gradle.test-retry) or retry wrappers that mask flakiness, maxParallelForks, build cache use, the machine type …
- Very specific: line 52
Follow the project's patterns for any fix: if it already has a DispatcherProvider or a MainDispatcherRule, use it rather than adding a …
- Distinctive: line 67
Run the single test many times in fresh processes, then in the same process, then with the
- Very specific: line 133
Start collecting a SharedFlow before the action that emits (Turbine test {} or backgroundScope.launch), and never assert on a transient …
- Very specific: line 138
Never: add delay/Thread.sleep, raise a timeout to make it pass, add a retry, or mark it ignored.
- Distinctive: line 139
If production code has a real race, fix the production code — the flaky test found a bug.
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
KMP flaky test triage
A flaky test depends on something it does not control: real time, a thread it did not choose,
state left by another test, the order of events, the machine's clock or locale. Find that input and
put it under the test's control. Retries, longer timeouts, delay and @Ignore hide the problem
and usually hide a real bug in production code too.
When to use
- A test that passes and fails on the same code; "fails about one run in five on CI".
- A test that fails only on CI, only on the iOS simulator, only in the full suite, or only after another test.
runTesttimeouts, Turbine timeouts,Dispatchers.Mainerrors, missed or extra flow emissions.
When not to use
- The test fails every time → it is a real failure; fix it with
mobile-bug-fix. - The code under test crashes the app →
kmp-crash-triage. - The test task does not compile or link →
kmp-build-failure-triage. - Setting up test infrastructure from scratch → the test setup guidance in
android/skills
testing/testing-setupfor the Android side; this skill is about making existing tests deterministic.
Step 0 — Read the project first
- Test stack. From
gradle/libs.versions.tomland test sources:kotlin.test, JUnit 4 or 5 on the Android host, Kotest,kotlinx-coroutines-test(version), Turbine, MockK or hand-written fakes, KtorMockEngine. - Task names per target.
./gradlew :shared:tasks --group verification— e.g.testDebugUnitTest(androidTarget +com.android.library),testAndroidHostTest(KMP Android library plugin),jvmTest,iosSimulatorArm64Test,allTests. - Dispatchers. How production code gets dispatchers: an injected
DispatcherProviderorCoroutineDispatcherparameter, or hard-codedDispatchers.IO/Dispatchers.Default. How tests setDispatchers.Main: a base class, a helper, a JUnit rule (Android host only). - Shared state.
objectsingletons, top-levelvals with caches, Koin started in tests (startKoin,KoinTest), databases or files used by tests,NSUserDefaults/ preferences. - Time. Does production code call
Clock.System.now(),TimeZone.currentSystemDefault(),Random,UUID/Uuid.random()directly, or receive them injected? - CI. Test retry plugins (
org.gradle.test-retry) orretrywrappers that mask flakiness,maxParallelForks, build cache use, the machine type for iOS tests.
Follow the project's patterns for any fix: if it already has a DispatcherProvider or a
MainDispatcherRule, use it rather than adding a second mechanism.
Workflow
1. Capture the failure
Record: the exact test (class and method), the target (Android host / JVM / iOS simulator), local
or CI, the assertion message and stack, how often it fails, and what changed recently. Pull CI's
JUnit XML from build/test-results/<testTask>/ — it holds the full message and stdout.
Done when you know which target fails and have the failing assertion text.
2. Reproduce by repetition
Run the single test many times in fresh processes, then in the same process, then with the whole class / suite. Each mode exposes a different cause.
# Android host / JVM: one test, 50 runs, each in a fresh test JVM. Stop at the first failure.
# --rerun forces the task to execute even when Gradle considers it up to date.
for i in $(seq 1 50); do
./gradlew :shared:testDebugUnitTest --tests 'com.example.app.notes.NotesViewModelTest' --rerun -q \
|| { echo "FAILED on run $i"; break; }
done
# iOS simulator through Gradle — same loop, iOS task (macOS only)
for i in $(seq 1 20); do
./gradlew :shared:iosSimulatorArm64Test --tests 'com.example.app.notes.NotesViewModelTest' --rerun -q \
|| { echo "FAILED on run $i"; break; }
done--rerun (Gradle 7.6+) re-executes just that task; on older Gradle use --rerun-tasks (reruns
every task) or run cleanTest<Task> first. If the output says FROM-CACHE, add
--no-build-cache.
Repeat inside one process on Kotlin/Native — this surfaces state leaking between runs:
./gradlew :shared:linkDebugTestIosSimulatorArm64 # -> shared/build/bin/iosSimulatorArm64/debugTest/test.kexe
xcrun simctl boot "<an installed iPhone simulator>" # once; list with: xcrun simctl list devices available
BIN=shared/build/bin/iosSimulatorArm64/debugTest/test.kexe
xcrun simctl spawn booted "$BIN" --ktest_list_tests | grep NotesViewModelTest # exact test names
xcrun simctl spawn booted "$BIN" --ktest_filter='com.example.app.notes.NotesViewModelTest.*' --ktest_repeat=50Then run the whole class and the whole module suite in the same loops. Passing alone but failing in the suite means order dependence or shared state.
To check time-zone dependence quickly, run once with an extreme zone: TZ=Pacific/Kiritimati
./gradlew --no-daemon … for JVM tests, or SIMCTL_CHILD_TZ=Pacific/Kiritimati xcrun simctl spawn
booted "$BIN" … for the iOS binary (simctl passes SIMCTL_CHILD_-prefixed variables to the
child). Run the loop while the machine is busy (a second build) to approximate a loaded CI runner.
Done when you have a failure rate, e.g. "7 / 50 alone, 0 / 50 in-process, 10 / 10 after
OrdersRepositoryTest".
3. Isolate
Shrink to the smallest failing setup: one test alone; the test plus the one test that runs before
it; one target. To find a polluting test, bisect the list of tests that ran before the failure
(run the first half + the flaky test, then the second half). Note the target: iOS-only flakes are
usually threading, main-queue or GC-timing issues; JVM-only ones are often Dispatchers.Main,
Looper or file-system issues.
4. Classify
Match the evidence against the causes table. Write the cause as one
sentence: "The test asserts before the repository coroutine on Dispatchers.IO finishes, so it
passes only when the IO thread wins the race." If no row fits, add temporary logging (thread name,
timestamps) through the project's logger and rerun the loop.
5. Fix the cause
Change the test, the production code, or both, so the uncontrolled input becomes controlled. Patterns with code: references/fix-patterns.md. Common ones:
- Inject dispatchers into production code; in tests, build every dispatcher from the test's
testSchedulersorunTestcontrols all of them. - Set
Dispatchers.Mainbefore each test and reset it after. - Start collecting a
SharedFlowbefore the action that emits (Turbinetest {}orbackgroundScope.launch), and never assert on a transientStateFlowvalue you did not control. - Give each test fresh instances; stop Koin after each test or avoid the global context.
- Inject
Clock,TimeZoneandRandom; use fixed values in tests.
Never: add delay/Thread.sleep, raise a timeout to make it pass, add a retry, or mark it ignored.
If production code has a real race, fix the production code — the flaky test found a bug.
6. Prove it is stable
Rerun the same loops from step 2 on both platforms and record the numbers:
Before NotesViewModelTest: 7/50 failed (Android host, fresh JVM), 3/50 (iOS, --ktest_repeat)
After 0/100 Android host, 0/100 iOS in-process, full module suite 10/10 on both
Cause <one sentence> Fix <file:line, what changed>A fix proven only on the platform where it flaked is not done: the same test runs on every target.
Quick reference — causes
| Symptom | Cause | Fix |
| Passes locally, fails on slower CI | Real time: delay, Thread.sleep, withTimeout on a real dispatcher | Virtual time via runTest and injected dispatchers |
Module with the Main dispatcher had failed to initialize or …is missing (JVM), or a hang on iOS | Dispatchers.Main used without Dispatchers.setMain | setMain in @BeforeTest |
| Next test behaves oddly after a failure | Dispatchers.resetMain() not called | resetMain in @AfterTest |
| Assertion sees old state, sometimes | Production code hard-codes Dispatchers.IO / Default; work races the assertion | Inject the dispatcher; use the test scheduler |
| State never updates in the test | StandardTestDispatcher queued the work; nothing advanced it | advanceUntilIdle() / runCurrent(), or UnconfinedTestDispatcher where eager is intended |
| Two schedulers: delays not skipped, odd ordering | A second TestCoroutineScheduler (new dispatcher not built from testScheduler) | Build every test dispatcher from testScheduler |
| One-shot event "sometimes missing" | SharedFlow (replay 0) emitted before collection started | Collect first: Turbine test {} or backgroundScope collector |
Loading state "sometimes missing" | StateFlow conflates; fast updates skip intermediate values | Control the dispatcher and step with runCurrent(), or assert only the final state |
Turbine No value produced in 3s on CI | Emission blocked by a dispatcher problem (Turbine's timeout is wall-clock, default 3 s) | Fix the dispatcher; do not just raise the timeout |
runTest fails after 60 s | A coroutine never completes (infinite collector in the test scope) | Launch long-running collectors in backgroundScope |
A Koin Application has already been started | Global Koin from a previous test still running | stopKoin() in @AfterTest, or an isolated koinApplication {} |
| Passes alone, fails in the suite | Shared mutable state: object caches, fakes reused across tests, DB files | Fresh instances per test; reset or avoid globals |
| Fails near midnight, month end, or on a UTC CI | Clock.System.now() / system time zone used directly | Inject Clock and TimeZone; fixed instants in tests |
| Fails on machines with another language | Locale-dependent formatting or parsing | Explicit locale/format in production code; fixed in tests |
| Different result per run with random data | Unseeded Random, random UUIDs in assertions | Inject a seeded Random / ID generator |
| Fails on one target only, collections involved | HashMap / HashSet iteration order is unspecified and can differ by platform | Assert on sorted lists or set equality |
Fails only on iOS, involves deinit / weak refs | Swift/ObjC objects held by Kotlin are freed only after a Kotlin GC | Do not assert release timing; test the cleanup call instead |
| Fails only on iOS on the second run | Files or NSUserDefaults persisted in the simulator between runs | Unique temp directories; clean up in @AfterTest |
| Fails when tests run in parallel | Shared database file, port or temp path across test JVMs | In-memory DB or unique names per test |
| Network-dependent result | Real HTTP in a unit test | Ktor MockEngine or a fake data source |
| "Fixed" but the test did not run | Gradle up-to-date or build-cache hit | --rerun, --no-build-cache; check the test count in the report |
Verify
Report each check as PASS, FAIL or NOT RUN (reason), with run counts. Never report a check as passed that you did not run.
# Single test, fresh processes, both platforms (counts in the report)
for i in $(seq 1 100); do ./gradlew :shared:testDebugUnitTest --tests '<Test>' --rerun -q || break; done
for i in $(seq 1 30); do ./gradlew :shared:iosSimulatorArm64Test --tests '<Test>' --rerun -q || break; done
# Same process, iOS
xcrun simctl spawn booted shared/build/bin/iosSimulatorArm64/debugTest/test.kexe --ktest_filter='<Class>.*' --ktest_repeat=100
# Whole suite, every target, several times
for i in $(seq 1 5); do ./gradlew :shared:allTests --rerun-tasks -q || break; doneAlso confirm in the test report that the test actually executed (count > 0), and that no retry plugin reran it.
Common mistakes
| Mistake | Why it bites | Fix |
Adding delay(100) before the assertion | Wins the race on your laptop, loses on CI | Control the dispatcher; advance virtual time |
Raising the Turbine or runTest timeout | The event still never arrives on slow runners | Find why it is not emitted |
| Enabling a test retry plugin | Flakiness — and the production race — ship anyway | Fix the cause; keep retries off for unit tests |
Test dispatchers that do not share one scheduler (setMain inside the test body, a fresh StandardTestDispatcher() for an injected dispatcher) | Main, injected dispatchers and runTest run on different virtual clocks | setMain in @BeforeTest (then runTest adopts its scheduler); build injected dispatchers from testScheduler |
Collecting a SharedFlow after calling the action | The emission is already gone | Start the collector first |
Asserting every StateFlow value in order | Conflation drops values depending on timing | Assert what the UI needs: the settled state |
| Proving the fix only on the JVM | The same commonTest also runs on Kotlin/Native | Loop the iOS task and the in-process repeat too |
| Fixing only the test when production code races | Users hit the race the test found | Fix production code; keep the test |
Related skills
mobile-bug-fix— when the "flaky" test turns out to fail deterministically.kmp-crash-triage— when the code under test crashes on one platform.kmp-ci-github-actions— CI setup for running both platforms' tests.- cashapp/turbine — flow testing; read its notes on hot flows and timeouts.
- kotlinx-coroutines-test —
runTest, test dispatchers,setMain. - CharlesWiltgen/Axiom
axiom-analyze-test-failures— flaky XCTest / Swift-side tests. - android/skills
testing/testing-setup— Android test infrastructure.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

