smix
goliajp/smix/llms-full.txt
Generated by scripts/dev/gen-llms.py. The evergreen AI-facing guides, concatenated for a single-file read. Consumer-correspondence notes are deliberately excluded. Goal: from a fresh repo, get a passing YAML run in under 10 minutes on both iOS and Android. Building from source works too, and is what you want when changing smix itself: The three lines above install to three different places — npm's global prefix, cargo's ~/.cargo/bin, and a build tree — and none of them is "where smix lives". A consumer's…
llms.txt1 starsChanged 3 days ago
- Installs packages
# smix — evergreen guides (full text)
Generated by `scripts/dev/gen-llms.py`. The evergreen AI-facing guides, concatenated for a single-file read. Consumer-correspondence notes are deliberately excluded.
<!-- ===== docs/ai-guide/01-quickstart.md ===== -->
# 01 — Quickstart
> Goal: from a fresh repo, get a passing YAML run in under 10 minutes on both iOS and Android.
## Prerequisites
| Need | iOS | Android |
|---|---|---|
| OS | macOS (Xcode required) | macOS or Linux (with Android SDK) |
| Tools | Xcode 16+, `xcrun simctl` | `adb`, `emulator`, SDK cmdline-tools |
| Sim/emulator | iOS Simulator (installed with Xcode) | AVD (any name; refer to it by that name) |
| Rust | only to build from source | same |
| Bun | 1.x (only if running the web UI; not for YAML flows) | n/a |
## Get the binaries
```bash
npm install -g @goliapkg/smix-cli # or: cargo install smix-cli --locked
smix --version
```
Building from source works too, and is what you want when changing smix
itself:
```bash
cd /path/to/smix
cargo build --release # ~30s cold, <5s warm
ls target/release/smix
```
### If you are scripting smix, resolve it from PATH
The three lines above install to three different places — npm's global
prefix, cargo's `~/.cargo/bin`, and a build tree — and none of them is
"where smix lives". A consumer's runner hard-coded `~/.local/bin/smix`,
the machine had the npm one, and the whole line failed with `smix not
found` before a single flow ran.
```bash
SMIX_BIN="${SMIX_BIN:-$(command -v smix)}"
[ -n "$SMIX_BIN" ] || { echo "smix is not on PATH" >&2; exit 1; }
```
Keep the `SMIX_BIN` override: it is how you point a script at a build
tree without reinstalling, and it is what smix's own gates use.
**A gate in your own tree should pass its build, not whatever is on
PATH.** The two are different questions and the difference is a released
version: running a gate bare here once tested the globally installed
6.8.0 while the change under test sat in `target/release`. The gate said
green about a binary nobody was changing.
One binary, one product: `smix`. All subcommands (boot, runner, run a flow, low-level probes) live under it.
## Start here: a dedicated device for your project
The one command to reach for first is `smix init` — it registers a device
**for this project**, derives an alias from the project directory, and records
it as the project's default, so `smix run` here needs no `--device`:
```bash
smix init --device <UDID> --app ./YourApp.app # register this project's device + install
smix run examples/hello.yaml # no --device: resolves the project's default
```
Everything below is the same steps by hand, when you want the low-level control
`smix init` wraps.
## iOS path: boot → run → teardown
```bash
# 1. Discover or pick a simulator UDID
smix sim list # UDID / name / state / runtime for every sim
# 2. Boot it (alias or UDID both work)
smix sim boot <device>
# → booted: 5D087114-ECB3-443C-...
# 3. Bring up the smix runner (XCUITest server)
smix runner up 5D087114-ECB3-443C-... --bundle com.example.YourApp
# → runner up: http://localhost:22087/health = 200
# 4. Install your app under test (build it with your usual toolchain first)
smix sim exec <device> install /path/to/YourApp.app
# 5. Run a YAML flow
smix run \
--platform ios \
--device 5D087114-ECB3-443C-... \
--runner-port 22087 \
--no-launch \
examples/hello.yaml
# 6. Teardown
smix runner down
```
If XCUITest processes remain after teardown, kill any residual `xctrunner` / `xcodebuild` processes before starting a new run.
## Android path: boot → run → teardown
```bash
# 1. Start the emulator
emulator @<avd-name> -no-audio -no-snapshot \
-no-boot-anim -port 5554 \
-skin 1080x2340 &
# Wait for boot
until adb -s emulator-5554 shell getprop sys.boot_completed 2>/dev/null | grep -q 1; do sleep 5; done
# 2. Build + install the app under test
adb -s emulator-5554 install -r /path/to/app-debug.apk
# 3. Build + start the smix runner (installs, forwards, instruments,
# and blocks until /health answers)
(cd android-runner && ./gradlew :app:assembleDebugAndroidTest)
smix runner up emulator-5554 --platform android
# Only if you use the WebView eval bridge:
adb -s emulator-5554 forward tcp:28081 tcp:28081
# 4. Launch the app under test
adb -s emulator-5554 shell am start -n com.example.app/.MainActivity
# 5. Run a YAML flow
smix run \
--platform android \
--apps-config apps.yaml \
--device emulator-5554 \
--runner-port 28080 \
--no-launch \
examples/hello.yaml
# 6. Teardown
smix runner down --platform android --device emulator-5554
adb -s emulator-5554 shell am force-stop com.example.app
adb -s emulator-5554 emu kill
```
## Common first-run errors
| Symptom | Cause | Fix |
|---|---|---|
| `simctl io ... is retired` | You called bare `xcrun simctl`. The sim safety hook is active. | Use `smix sim ...` or `smix sim exec <udid> ...` instead. |
| `runner /health` 502 / connection refused | Runner did not finish coming up. | Wait 3–5s after `smix runner up`, re-curl /health, or run `smix runner list` — it names every runner on this machine, its port, and whether the ledgers know about it. |
| `tap_by_id: element not found` | Element off-screen (LazyRow / horizontal-scroll tab bar) | Scroll first (`adb shell input swipe`) or use a different selector. See [03-selectors.md](03-selectors.md). |
| iOS build fails with `Cannot find 'FooScreen' in scope` | You added a new file to your Xcode project but the `.pbxproj` wasn't updated. | Add PBXBuildFile + PBXFileReference + PBXGroup + PBXSourcesBuildPhase entries. |
| `webview_eval: runner webview-bridge unreachable (501)` | WebView shim not started | Navigate to the WebView-hosting screen at least once before exercising `webview_eval`, and ensure `adb forward tcp:28081` ran. |
## Where to go next
- Want to **write** a YAML? → [02-yaml-reference.md](02-yaml-reference.md)
- Need to **find an element** by some property other than testTag? → [03-selectors.md](03-selectors.md)
- Want a reference for how to organize testTags in your test app? → [06-fixtures.md](06-fixtures.md)
---
Driving a physical iPhone or Android device works the same way once the
device is registered — see [05-cli.md — Physical
devices](./05-cli.md#physical-devices) for registration, signing, and
what a phone cannot do.
<!-- ===== docs/ai-guide/02-yaml-reference.md ===== -->
# 02 — YAML reference
> Every YAML verb smix supports, with a copy-pasteable example. smix accepts maestro-compatible YAML; if a flow works under `maestro test`, it should also work under `smix run` modulo platform-specific extensions.
## File layout
A smix YAML file has two parts: **header** (optional) and **flow** (a YAML list of steps).
```yaml
appId: com.example.app # iOS bundle id OR Android package
# OR for cross-platform:
app: myapp # logical key resolved via --apps-config apps.yaml
--- # YAML doc separator (header ends, flow begins)
- launchApp:
clearState: true
- assertVisible: "Hello"
- tapOn: "Submit"
```
The `app:` form (cross-platform) needs `--apps-config <path>` flag. The `appId:` form picks a literal id (use this when you want a single platform).
## The verb set (organized by purpose)
### App lifecycle
```yaml
- launchApp # no args: foreground default app
- launchApp:
appId: com.acme.app # override yaml-level appId
clearState: true # wipe NSUserDefaults / shared prefs
clearKeychain: true # iOS keychain wipe
arguments: ["--debug", "--env=prod"] # launch args
- killApp # force-quit current app
- killApp: com.acme.other # force-quit specific
- stopApp # graceful background
- clearState # wipe state without restart
- clearKeychain # wipe keychain (iOS only)
# v1.0.27 — delete specific NSUserDefaults keys (iOS only). Surgical
# alternative to clearState when one persisted value poisons the next
# run (canonical case: expo-dev-launcher re-delivers its stored deep
# link after every JS bundle load — delete its key between stopApp
# and the next launch to neutralize the replay at the source).
# Contract: "ensure keys absent" — already-absent keys succeed.
# Terminate the app first; running processes cache defaults in-memory.
- clearUserDefaults:
keys:
- "expo.devlauncher.pendingDeepLink"
bundleId: com.acme.app # optional; default = flow appId
```
### Assertions (read-only checks; do not change app state)
```yaml
- assertVisible: "Welcome" # text literal
- assertVisible:
id: "home-counter-label" # accessibility identifier
- assertVisible:
text: "Error"
optional: true # if absent, step is "skipped" not "failed"
- assertNotVisible: "Error" # element must NOT be on screen
- assertNotVisible:
id: "modal-overlay"
- assertTrue: ${1 > 0} # ==, !=, <, <=, >, >=, &&, ||, !, .contains()
- assertScreenshot: "home.png" # visual regression — full-frame 64-bit dhash
- assertScreenshot:
path: "home.png" # baseline path, relative to the flow file
threshold: 5 # max hamming distance (default 5)
mask: # left out of the comparison (0..1 shares)
- { x: 0.0, y: 0.0, width: 1.0, height: 0.4 }
- assertScreenshot:
path: "card.png"
cropOn: { id: "summary-card" } # compare only this element's region
thresholdPercentage: 95 # maestro's measure: % of pixels that match
- rememberBounds: # keep where an element is, by name
id: "btn-ptz-stop"
as: "stop"
- assertBoundsUnchanged: # …and that it has not moved since
id: "btn-ptz-stop"
was: "stop"
within: 1 # device-independent pixels (default 0)
- neverVisible: # at no moment while these steps run
id: "loading-overlay"
during:
- tapOn: { id: "row-alert-1" }
- extendedWaitUntil:
visible: { id: "player-surface" }
timeout: 5000
```
`rememberBounds` / `assertBoundsUnchanged` are smix's own — maestro has
no way to say "nothing moved between these steps". Asserting `visible` in
each state passes whether or not the layout jumped 8 pixels between them;
this pair compares the element's box itself. Boxes are compared in
device-independent pixels (points on iOS; pixels divided by the display
density on Android), so `within: 1` means the same on every phone, and
every edge is compared — a control that grew without moving its corner
has changed. A failure prints both boxes and how far each edge moved.
The box compared is the one the element occupies, not the part that
shows: scrolling a row half out of view changes what shows without
moving anything.
`neverVisible` is smix's own too. `assertNotVisible` answers about one
instant, and a loading state that flashed for 200 ms between two steps is
gone by any instant after them. `neverVisible` runs the steps under
`during` as they would run anywhere else and, beside them, keeps asking
whether the element is on screen — the same question `assertNotVisible`
asks, as fast as the device answers, with no sleep between looks. It
looks at least once after every inner step before the next one starts.
One sighting fails it: the failure says how long after the span began the
element was seen, which inner step was running, and what was on screen
just after. A pass is reported with how many times it looked and the
longest stretch nobody was looking — including before the first look and
after the last — because "never seen" only means something if the looks
were close together. A watch whose every look failed is a failure, not a
pass: nothing is known about a screen nobody read. A failing inner step
is reported as itself. `optional: true` turns a sighting into a skip, as
it does on a `runFlow` block; `label:` is the element's accessibility
label here, as on every verb with a selector.
`assertScreenshot` auto-records the baseline on the first run and diffs
against it afterwards. It compares one of two ways, and they measure
different things:
- `threshold` (the default, 5): a 64-bit perceptual hash of each frame;
passes when at most that many bits differ. Tolerant of anti-aliasing
and small shifts; blind to a change too small to move a brightness
gradient.
- `thresholdPercentage`: maestro's comparison — the share of pixels
whose colour is within 10% of the baseline's, passing when it is at
least this many percent. A screenshot of a different size fails. It
may be written as a number, a string, or `${…}`; a value that is not a
number from 0 to 100 is refused.
Writing both is an error rather than a choice between them. Unlike
maestro, whose only comparison is pixels with a default of 95, a flow
that writes neither compares by hash as it always has.
`cropOn:` takes a selector, waits for it like any element verb, and
compares only that element's region; the baseline recorded on the first
run is the cropped image, and `takeScreenshot` takes `cropOn:` too, to
write one. An element with no area on screen is `NOT_VISIBLE`.
`mask:` regions — shares (0..1) of the image being compared, as
`{ x, y, width, height }`, the cropped image when there is a `cropOn` —
are left out of the comparison: a masked pixel counts neither as a match
nor as a difference. Masks that cover everything are refused: a
comparison of nothing would pass whatever was on the screen. maestro's
`label` and `optional` are refused by name rather than ignored.
`assertTrue` reads `${…}` with a small expression engine, not a JS
runtime — comparison, boolean operators, parentheses and `.contains()`,
and nothing else. It can read `output.*`, which is written by
`extractWithAI` and by `runFlow: {as: name}`; both store **strings**, so
`${output.total > "100"}` compares text, not numbers. Comparing a
string to a number is refused rather than converted, because an
assertion that answers on a reading you did not intend is worse than
one that stops.
### Tap / touch actions
```yaml
- tapOn: "Submit" # text literal
- tapOn:
id: "home-increment-btn"
- tapOn:
text: "Item.*" # regex pattern (anchored)
nth: 2 # 0-indexed within matches
- tapOn:
text: "Continue"
optional: true # skip if not visible (no error)
- tapOn:
point: "50%,80%" # fraction of the viewport, NOT pixels
# `"0.5,0.8"` is the same point; `%` is optional
# pixels are refused — a flow written in them
# runs on one screen size
# Several taps on one element, spaced by a number you state.
#
# `repeat` around `tapOn` sends a request per tap, and each one costs a
# synthesised event round trip (~400 ms on iOS 26.5) — so the interval
# is whatever that cost, and a gesture gated on a short inter-tap window
# cannot be driven. This sends them in one request, and the runner keeps
# the interval. On iOS one touch takes about 280 ms to deliver, so an
# interval shorter than that arrives as ~280 ms; Android keeps it.
- repeatTap:
id: "hidden-trigger"
times: 10
intervalMs: 80 # optional; runner default
holdMs: 50 # optional; how long each touch stays down
# maestro's own spelling of the same thing: the target is found once and
# tapped `repeat` times, `delay` ms apart (100 when not given). `optional`,
# `dispatch` and `point` do not go with `repeat` and are refused by name.
# Either way the target is waited for to hold still, and the first touch
# is judged like a `tapOn`: delivered to something else, it fails TAP_MISSED.
- tapOn:
id: "hidden-trigger"
repeat: 10
delay: 200
- doubleTapOn: "Reset"
- longPressOn:
id: "list-row-3"
duration: 1500 # ms
captureDuring: true # optional; write PNGs of the held state
# needs duration >= 800 (see below)
```
### Text input
```yaml
- inputText: "alice@example.com" # types into focused field, appending
- inputText:
id: "form-email-input" # names a field, so it replaces
text: "alice@example.com"
- eraseText: 5 # backspace N chars (default 50)
- copyTextFrom:
id: "form-result-label" # copies result to clipboard env
- pasteText # paste clipboard into focused field
- setClipboard: "hello" # set clipboard programmatically
- hideKeyboard # dismiss soft keyboard
```
### Scrolling
```yaml
- scroll # scroll viewport down (default direction)
- scroll:
direction: UP # UP / DOWN / LEFT / RIGHT
- scrollUntilVisible:
element:
text: "Row #5000"
direction: DOWN # UP / DOWN / LEFT / RIGHT
timeout: 30000 # ms; default 20000
visibilityPercentage: 100 # 1-100; default 100 = wholly on screen
centerElement: false # stop once it is near the middle
optional: false # not reached -> skip, not failure
- swipe:
direction: LEFT
duration: 400
- swipe:
start: "10%,50%"
end: "90%,50%"
duration: 400
# `over:` — swipe INSIDE one element, by shares of ITS box.
#
# The two forms above take shares of the SCREEN, which is what you reach
# for when there is nothing nameable to swipe between. When there is
# something nameable, `over:` names it and the numbers stop meaning
# anything about the display:
- swipe:
over: { id: "view-timeline" }
from: 0.3 # three tenths DOWN, halfway across
to: 0.8 # four fifths down
- swipe:
over: { id: "view-timeline" }
from: { x: 0.1, y: 0.5 } # both axes, when the drag is not along one
to: { x: 0.9, y: 0.5 }
# `over:` takes any selector form, `fallback:` included — which is what a
# flow driving both platforms needs when the same element is named
# differently on each:
- swipe:
over:
fallback:
- id: "view-timeline" # iOS accessibilityIdentifier
- id: "view_timeline" # Android resource id
from: { x: 0.3, y: 0.5 }
to: { x: 0.8, y: 0.5 }
duration: 600
```
A bare number is the share along **y** — down the element — with `x`
centred: `from: 0.85, to: 0.15` drags upward through the middle of the
box, which is what scrolling a list looks like. A drag along a strip has
one interesting axis, and guessing the other as `0` would run along its
edge.
**A horizontal drag needs the `{x, y}` form.** `from: 0.1, to: 0.9`
would drag *down* the middle, not across — the same two numbers with a
different meaning, which is the kind of mistake that produces a swipe
that ran and did the wrong thing.
**Percent strings are refused here.** `"30%"` of an element and `"30%"`
of the screen are the two things this form exists to keep apart, and one
spelling meaning both is how a flow ends up measuring the display again.
**It does not fall back to the screen.** If the element is not there, the
step fails and says so — swiping across the middle of whatever is showing
would do something the flow did not ask for and report it as done.
Reported by a consumer who had a nameable timeline and still had to
measure it: 45.3–50.5% of screen height on Android, 47.7–53.2% on iOS,
49% taken as the overlap, and a note that a device of a different shape
would need measuring again.
### Keyboard / hardware keys
```yaml
- pressKey: ENTER # ENTER / TAB / SPACE / DELETE / ESCAPE
- pressKey: HOME # iOS home button
- pressKey: VOLUME_UP # Android; iOS fails the step by name
- pressKey: Back # the same as `- back`
- back # navigation back
```
Key names are read when the flow is read, and one table reads them for
the flow, `smix press-key` and the MCP tool alike: maestro's spellings
(`Enter`, `Backspace`, `Volume Up`), the wire names (`return`,
`volumeUp`), and shorthands (`esc`, `up`). Case, spaces, `_` and `-` do
not matter. maestro's TV remote keys are refused by name — smix drives
no TV — and `Power` points to `LOCK`.
### System / device controls
```yaml
- setOrientation: LANDSCAPE_LEFT # PORTRAIT / LANDSCAPE_LEFT / LANDSCAPE_RIGHT / PORTRAIT_UPSIDE_DOWN
- setLocation:
latitude: 35.6812
longitude: 139.7671
- travel:
points:
- { latitude: 35.0, longitude: 139.0 }
- { latitude: 36.0, longitude: 140.0 }
speedMps: 50
- clearLocation # stop simulating; the way back from the two above
- setPermissions: # iOS / Android permission grants
camera: allow
location: allow
notifications: allow
storage: allow # Android's WRITE_EXTERNAL_STORAGE; a no-op on iOS
```
A simulated location outlives the flow that set it, and on a phone it
outlives the cable — `clearLocation` is how smix puts it back. On an
Android emulator it stops a `travel` in progress and leaves the device
where it stands: the emulator console has no inverse of a position fix,
and there is no real position to return to.
`setPermissions` and `launchApp.permissions` name permissions the same
way, and the list is neither platform's — a name with no counterpart on
the device in front of you is a no-op there, not an error.
### Deep links
```yaml
- openLink: "myapp://home/details/42"
- openLink:
link: "https://example.com"
```
### Recording (video output)
```yaml
- startRecording: "trace.mp4"
- stopRecording # writes path passed to start
- takeScreenshot: "step5.png"
- takeScreenshot:
path: "card.png"
cropOn: { id: "summary-card" } # only this element — a cropOn baseline
```
### Control flow
```yaml
# Repeat fixed count
- repeat:
times: 5
commands:
- tapOn: "Increment"
# Repeat while condition
- repeat:
while:
visible: "Loading..."
commands:
- waitForAnimationToEnd: 500
# Repeat while NOT visible (poll for appearance)
- repeat:
while:
notVisible:
id: "home-result-label"
commands:
- waitForAnimationToEnd: 200
# `repeat.while` takes the same conditions as `runFlow.when` (below),
# and a block can be labelled and made optional
- repeat:
while:
platform: Android
visible: "Load more"
label: "drain the list"
optional: true
commands:
- tapOn: "Load more"
# Both together: another pass needs the condition to hold AND the count
# to be unspent. Give neither and it is a parse error — nothing would
# say when the loop ends.
- repeat:
times: 5
while:
visible: "Load more"
commands:
- tapOn: "Load more"
# Retry on failure
- retry:
maxRetries: 3
commands:
- tapOn: "Flaky Button"
- assertVisible: "Confirmed"
# Run a sub-flow (file include)
- runFlow: "../subflows/launch-fresh.yaml"
# Run a sub-flow conditionally (same `runFlow` verb + `when:` gate)
- runFlow:
when:
visible: "Need Login" # enter only when this is visible
file: "../subflows/login.yaml"
# Inverse gate (v1.0.24) — enter only when the selector is NOT visible.
# Idempotency pattern: skip a ceremony whose end state is already reached.
- runFlow:
when:
notVisible: { id: "qa-bubble" }
file: "../subflows/enter-qa-mode.yaml"
# Inline body instead of a file (`commands:`); `when:` gates identically.
- runFlow:
when:
visible: "Open in"
commands:
- tapOn: "Open"
- waitForAnimationToEnd
# Conditions combine: every one present must hold (AND), checked in
# maestro's order — platform, true, visible, notVisible — and the first
# that fails skips the block without checking the rest.
- runFlow:
when:
platform: Android # Android | iOS | Web, any case
visible: "Allow"
commands:
- tapOn: "Allow"
# `true:` is a template, expanded when the gate is checked. It is false
# when the result is blank, `false` (any case), `undefined`, `null` or a
# number equal to zero, and true otherwise.
- runFlow:
when:
true: ${output.skipOnboarding == 'yes'}
label: "onboarding already done" # names the skip in the log
commands:
- tapOn: "Skip"
# `env:` names values for the block only; each value is expanded in the
# caller's scope on entry. `optional: true` turns a failure inside the
# block into a skip with the failure's own words, and the flow goes on.
- runFlow:
env:
GREETING: "hello"
label: "type the greeting"
optional: true
commands:
- tapOn: "Message"
- inputText: ${GREETING}
# Every key of `runFlow:`, `repeat:`, `when:` and `while:` is one smix
# acts on. A key it does not know — a typo, or `when.optional`, which
# maestro accepts and ignores — is a parse error naming the key.
# The gate selector supports the full selector table, including
# `fallback:` chains with `ocrText` (OCR fires for the check).
# Run inline JS (evaluated in the maestro JS context)
- evalScript: |
output.userId = output.someResponse.id
- runScript: "../scripts/setup.js"
# `runScript` also takes maestro's mapping form, `when:` included — so a
# script meant for one platform is skipped on the other rather than
# failing there. smix has no JS runtime, so one whose condition holds
# fails saying exactly that; what `when:` changes is whether it is
# reached at all.
- runScript:
file: "../scripts/seed-android.js"
when:
platform: Android
label: "seed the database"
```
### Wait / sync
```yaml
# Fixed sleep — NOT an XCTest quiescence wait. smix no-ops XCTest's
# internal idle-wait for performance (SmixQuiescenceSwizzle); this verb
# has always been a bounded pause at the runtime layer.
#
# On Android a run zeroes the animation scales before it starts, so
# this verb has little left to wait for; `--animations` restores the
# device's own settings and with them the original reason for it.
#
# On iOS nothing is quietened — no host-side lever exists — so this
# verb keeps its full value there.
- waitForAnimationToEnd # bare form: ceiling 400 ms (maestro-compat)
- waitForAnimationToEnd: 500 # integer: ceiling 500 ms
- waitForAnimationToEnd:
timeout: 5000 # map form: ceiling 5000 ms (maestro-canonical)
# The number is a CEILING in all three forms, never a sleep. The verb
# samples the screen and returns as soon as two frames match, so a still
# screen costs two captures and not the number written here. It used to
# sleep, which charged for an animation whether or not one ran; this note
# said "sleep N ms" for longer than that was true, and a consumer wrote
# `waitForAnimationToEnd: 500` meaning "pause here" on the strength of it.
#
# There is no bare sleep verb, deliberately. To wait for something, name
# it: `extendedWaitUntil` for an element. The keyboard is an element —
# `extendedWaitUntil: { visible: { role: "keyboard" } }` — which is the
# wait people most often reach for a pause instead of, because the keys
# are not in the tree and `describe` leaves the keyboard out of its
# summary. See "role: keyboard" in the selectors guide.
#
# If the thing you are waiting for really is not in the tree, that is a
# gap worth reporting rather than routing around with a pause.
#
# A screen that never settles is a warning, not a failure — so is a window
# in which the device refused every capture (see CAPTURE_BACKPRESSURE in
# the errors guide). The verb waits refusals out inside its own ceiling.
- extendedWaitUntil:
visible: "Loading complete"
timeout: 30000
- extendedWaitUntil:
notVisible: "Spinner"
timeout: 10000
# On timeout, extendedWaitUntil auto-captures a screenshot + tree JSON
# to `.smix/timeouts/` (or `--debug-output` dir when set) and appends
# the written paths to the failure hint — no flag needed.
# Inside a flow, this is the verb. Driving the runner from OUTSIDE one —
# a shell harness deciding whether the app is up before it starts —
# reach for `smix wait-for` instead (see 05-cli):
#
# smix wait-for --device <d> --port <p> --timeout 30 id:tab-device
#
# Same question, no yaml. A consumer wrote a one-step `ready.yaml` and
# ran `smix run` on it before finding `wait-for` in `--help`; the two
# belong beside each other because the choice is "am I in a flow or
# around one", not "which is better".
```
### Media (file upload simulators)
```yaml
- addMedia: # iOS Photos / Android Gallery seed
- "/path/to/test-photo.jpg"
```
### WebView JS bridge
```yaml
- webViewEval: |
document.getElementById('user-input').value = 'alice';
submitForm();
document.getElementById('form-result').textContent
# Returns the JS expression result (last-expression value). Async support via
# evaluating a Promise-returning expression — handled by the smix runner.
```
## Selector forms inside steps
Wherever a step takes a selector (tapOn, assertVisible, etc.), the selector may be:
- **String** literal → text match
`tapOn: "Submit"`
- **Object** → typed selector with modifiers
`tapOn: { id: "tab-home" }`
`tapOn: { text: "Sub.*", nth: 1 }`
`tapOn: { ocrText: "Done" }` (Vision/ML Kit)
`tapOn: { anchor: "Header", below: "Title" }` (spatial)
`tapOn: { point: "50%,50%" }` (coords)
`tapOn: { fallback: [ {id: "foo"}, {text: "Foo"} ] }` (first hit wins)
Full selector taxonomy is in [03-selectors.md](03-selectors.md).
## Conditional + optional patterns
- `optional: true` on tapOn/assertVisible: step never fails the flow; it is skipped silently if the element is absent (used for cross-platform yamls where a step is iOS-only or Android-only).
- `runFlow` with `when: { visible: ... }`: include a sub-flow only when a condition holds at runtime (e.g., "if Need Login is shown, run the login subflow").
- `runFlow` with `when: { notVisible: ... }` (v1.0.24): inverse gate — enter only when the selector is NOT visible ("run the ceremony unless its end state is already on screen").
- `runFlow` with `when: { platform: Android }`: a block for one platform in a flow written for both. The platform is checked first, so a block for the other platform never looks at the screen.
- `when:` conditions combine with AND; `visible` and `notVisible` may appear together.
- `optional: true` on a `runFlow` or `repeat`: a failure on what the block checked (element not found, assertion failed, timeout, …) is reported as skipped with its own message. A failure of the machinery — the device or runner gone, a malformed flow — still fails the step.
- Skipped conditionals emit `STEP N: … → SKIPPED: <reason>` to stderr with the checked selector + evaluation outcome, so batch logs show WHY a gate short-circuited.
- Step lines are printed as each step starts, so the last one in the log
is the step that was in flight. A failure also names itself: the error
reads `step N (verb): …`, and a `STEP N: verb → FAILED` line joins the
skipped one. A subflow's inner step is named once — the `runFlow` that
contains it does not claim the failure.
## OCR behavior in verbs
`ocrText:` parses anywhere a selector does, standalone or inside a
`fallback:` chain. Which verbs actually fire Vision (iOS) / ML Kit
(Android) for it is one table, in
[03-selectors](03-selectors.md#11-fallback-chain) — generated from the
one the code decides by. A verb that does not read it says so and says
what to write instead; none of them quietly matches nothing.
The verbs that fire OCR, and what they do with it:
- `extendedWaitUntil.visible` — polls tree + OCR per iteration until timeout.
- `tapOn` with `fallback: [..., ocrText]` — polls the whole chain within `SMIX_TAP_OCR_POLL_MS` (default 3000 ms) before failing, closing tap-vs-mount races.
- `scrollUntilVisible` — every look reads the tree first, then each `ocrText` the selector names, in the chain's order (off-screen items dropped from a degraded a11y tree are still found by OCR once scrolled into view).
- `runFlow.when.visible` / `when.notVisible` — the gate check fires OCR.
Cost note: one OCR call is ~500 ms on-sim. Order `fallback:` chains cheapest-first (`id` → `text` → `ocrText`) so tree hits pre-empt Vision cost.
### `SMIX_AUTO_OCR_FALLBACK=1` (env opt-in)
Auto-lifts every bare-string selector `visible: 'X'` to `fallback: [text: 'X', ocrText: 'X']` at parse time. Strings containing top-level `|` (regex OR) split per alternative for the OCR tiers: `'A|B'` → `fallback: [text: /A|B/i, ocrText: 'A', ocrText: 'B']`. Accepted truthy values: `1`, `true`, `TRUE`, `yes`.
## Output variables (between steps)
- `evalScript` and `runScript` write to `output.*` namespace.
- Subsequent steps read via `${output.foo}` interpolation.
- Useful for chaining assertions on dynamic content (id from API, name from form input, etc).
## Cross-platform YAML conventions
When writing a YAML that should run on both iOS + Android via `--platform ios|android`:
1. Use `app: <logical-key>` (cross-platform) instead of a literal `appId:` (single-platform).
2. Add `--apps-config apps.yaml` (or your own resolver file).
3. Mark platform-only steps `optional: true`.
4. Use `id:` selectors (mirror via testTag) over `text:` (i18n drift) over `ocrText:` (slow).
## Exit codes (from `smix run`)
| Code | Meaning |
|---|---|
| 0 | success |
| 2 | YAML parse error |
| 3 | runtime SDK failure (sim/app problem mid-flow) |
| 4 | unknown key / unknown direction (bad verb / arg) |
| 5 | runFlow cycle / file IO |
| 6 | runner unreachable |
A non-zero exit code is paired with a structured JSON error written to stdout (see [07-errors.md](07-errors.md)).
<!-- ===== docs/ai-guide/03-selectors.md ===== -->
# 03 — Selectors
> How to target an element on screen. smix supports 12 base selector forms + spatial / index modifiers. Pick by **specificity**: testid > role > text > spatial > coord.
## Specificity order (use the first one that works for your case)
```
strongest (production-ready, stable across rebuild / i18n / screen size)
1. Id — accessibility identifier (testTag) — PREFERRED
2. Role — semantic role + optional name pattern
3. Label — accessibility label (often = display text)
weaker (display text — can break under i18n / rename)
4. Text — literal or regex
5. LocalizedText — per-locale text table
6. OcrText — Vision (iOS) / ML Kit (Android) — also handles non-accessibility text (PDFs, video labels, hardware tests)
spatial (when target lacks own id but a sibling does)
7. Anchor — anchor-only ("the button below Header")
8. AnchorRelative — anchor + (dx, dy) normalized offset
worst (escape hatch — fragile, breaks on layout change)
9. Focused — current focus only
10. Point — direct viewport coordinate
11. Fallback — chain of any above; first hit wins
```
**Rule of thumb**: a yaml that uses `Id:` is durable; one that uses `Point:` is brittle. Reach for spatial / OCR only when the upstream component genuinely lacks an accessibility identifier you can fix.
## The same selector in three places
A selector is one idea written three ways, and the three are easy to
confuse when you move between them in one session:
| where | how it is written |
|---|---|
| flow yaml | `tapOn:`<br>` id: "submit-button"` |
| CLI | `smix tap "id:submit-button"` |
| MCP tool | `{"id": "submit-button"}` |
The CLI's form is `field:value`, one colon, no spaces — `id=` is not
accepted, and the error says so with the right form in it. Every field
name is the same across all three; only the punctuation differs.
## Naming two things at once
A selector map may carry more than one base key. It means **and** — the
one element that satisfies both:
```yaml
# the element whose id is home-counter-label AND whose text is "1"
- assertVisible:
id: "home-counter-label"
text: "1"
```
The more specific key becomes the form (`id` > `label` > `text`, the
same order as the specificity list above); the rest narrow it. Any verb
that takes a selector reads it the same way.
**What this is not.** The spatial keys (`near`, `below`, `inside`, …)
constrain a candidate by *another* node's geometry, and `ancestor`
constrains it by its parent chain. This constrains the candidate
itself.
`role` + `name` is one form spelled with two keys, not a conjunction —
`name` has only ever been role's optional qualifier.
> Before 3.0 the second key was read by nobody: `{ id: X, text: Y }`
> parsed to `Text { Y }` and the id was dropped, so it matched any
> element reading Y. The form was already written in these guides, and
> in flows; what it meant is what it says now.
## The 12 selector forms
### 1. Id (preferred, stable)
```yaml
- tapOn:
id: "home-increment-btn"
```
- Matches `accessibilityIdentifier` (iOS). On Android it matches the
**resource id**: `android:id` on a View, or a Compose `testTag` exposed
through `testTagsAsResourceId`.
- **It never matches `contentDescription`.** A hand-written View layer
that names its elements with `contentDescription` and no `android:id`
is reachable by `text:` or `label:` and not by `id:` — which reads as
"the element is not there" rather than "you asked by the wrong name".
A consumer lost an afternoon to this, and the line above named only
the Compose half, so nothing they read said otherwise.
- Strict equality on id string.
- Use this whenever you can add a testid to the app under test. Where you
cannot, `fallback: [{ id: … }, { text: … }]` covers both shapes in one
flow — the first form that resolves wins.
### 2. Text (literal or regex)
```yaml
# Literal
- tapOn:
text: "Submit"
# Regex — say so. A plain string is matched literally unless it
# contains `|`, which is the one character that promotes it.
- tapOn:
text: { regex: "Row #[0-9]+" }
# Common pattern: anchored ^...$
- tapOn:
text: { regex: "^Help$" } # exclude "Help Center" etc
```
- Matches displayed text (label, attributed string, button title, accessible label).
- Regex syntax = NSRegularExpression on iOS, java.util.regex.Pattern on Android.
- `flags` defaults to `"i"`; matching is case-insensitive either way.
- A bare string is **not** scanned for metacharacters. `Delete?` and
`3.5` are ordinary labels, and treating them as patterns would
silently widen what they match — the failure that does not announce
itself. The one exception is `|`, which has meant alternation here
since before the explicit form existed.
### 3. Label (accessibility label)
```yaml
- tapOn:
label: "Settings"
```
- Strictly matches `accessibilityLabel` (iOS) or `contentDescription` (Android).
- Differs from Text: an icon button often has `text:""` but `label:"Settings"`.
### 4. Role (semantic)
```yaml
- tapOn:
role: button
name: "OK" # optional name pattern (text or regex)
- assertVisible:
role: heading
name: "Welcome"
```
- Supported roles (docs-friendly lowercase and camelCase both accepted; wire is camelCase): button, link, textField, secureTextField, searchField, switch, toggle, checkBox, radio, image, staticText (accepts `heading` as an alias), tab, tabBar, navigationBar, cell, alert, dialog, slider, progressBar, picker, menu, menuItem, scrollView, segmentedControl, table, collectionView, webView, keyboard.
- iOS: derived from XCUIElement type + traits. Android: from AccessibilityNodeInfo class name + roleDescription.
- `role:` and its optional `name:` work anywhere a selector map does — `tapOn:`, `assertVisible:`, `extendedWaitUntil.visible:`, `scrollUntilVisible:`, etc.
#### `role: keyboard` — waiting for the software keyboard
The last entry in that list is the one people ask for without finding.
The software keyboard is an element, so it can be waited for like any
other — which is what you want when a field is focused by a tap and the
keyboard slides in over the button the next step means to press:
```yaml
- assertNotVisible: { role: "keyboard" }
- tapOn: { id: "email" }
- extendedWaitUntil:
visible: { role: "keyboard" }
timeout: 5000
- hideKeyboard
- extendedWaitUntil:
notVisible: { role: "keyboard" }
timeout: 5000
```
Both platforms answer, and it is asked of both on every release —
`keyboard-comes-and-goes.yaml` in the portable corpus and A12 in the
Android behaviour gate. On Android it was unanswerable until 7.1: the
runner knew (an input-method window is the keyboard, which is what
`hideKeyboard` has always decided on) and the tree did not carry it, so
the same flow that passed on iOS timed out there.
Two things about it are worth knowing, because both have sent people
looking for a pause instead:
- **The keys are not in the tree, and the keyboard is.** Do not wait for
`text: "A"`; wait for the keyboard.
- **`smix describe` leaves the keyboard out of its summary**, so it is
invisible in exactly the place you would look for it. `smix find
"role:keyboard"` answers regardless, and so does `/tree`.
### 5. Focused
```yaml
- inputText: "hello" # the scalar form types into the focused field, appending
- pressKey: ENTER # often used after focused: true to submit
```
- Targets whichever element currently has keyboard focus.
- No modifiers (no `nth`, no spatial — there's only one focused element).
### 6. Anchor (spatial — anchor-only)
```yaml
- tapOn:
text: "Edit"
below: { text: "Settings" } # the Edit below the Settings row
# Pixel-offset form — anchor plus a dx/dy nudge, all three required:
- tapOn:
anchored:
anchor: { text: "Settings" }
dx: 0.0
dy: 40.0
```
- `anchor` finds a stable reference element by text or id.
- Then a spatial key selects what to tap relative to it.
- Spatial keys: `near` / `below` / `above` / `leftOf` / `rightOf` / `inside` / `ancestor`.
### 7. AnchorRelative (anchor + offset)
```yaml
- tapOn:
anchorRelative:
anchor: "Header bar"
dx: 0.4 # +0.4 viewport width to the right
dy: -0.05 # -0.05 viewport height above
```
- Useful for elements that are visually adjacent but lack any accessibility relationship.
- dx/dy are viewport-normalized (so layout-tolerant).
### 8. LocalizedText (per-locale tables)
```yaml
- tapOn:
localizedText:
en: "Submit"
ja: "送信"
es: "Enviar"
```
- Detects current locale at runtime (iOS Locale.current / Android LocalConfiguration).
- Picks the matching string from the table.
- Fails if current locale has no entry.
### 9. OcrText (Vision / ML Kit)
```yaml
- tapOn:
ocrText: "Done"
- tapOn:
ocrText: "Confirm"
locales: ["en", "ja"] # optional recognition language list
recognition_level: accurate # iOS Vision (fast | accurate); ignored on Android
```
- iOS: Apple Vision VNRecognizeTextRequest.
- Android: ML Kit Latin TextRecognition.
- Slow (~150-300ms per call) — use only when target has no testid AND text-based selectors miss (e.g., text rendered as image, PDF viewer, video frame label).
### 10. Point (direct coordinate)
```yaml
- tapOn:
point: "50%,80%" # X%,Y% of viewport
```
- nx/ny are normalized [0, 1] of viewport.
- Brittle: fails any time layout shifts.
- Use only when the target has no other identifier AND you control the layout (e.g., dev menu coord targets, OCR-derived coord).
### 11. Fallback (chain)
```yaml
- tapOn:
fallback:
- id: "home-cta-btn" # try first
- text: "Get Started" # if id absent
- ocrText: "Get Started" # if text absent (rendered as image)
- point: "50%,75%" # last resort
```
- An entry is any selector — `label`, `role` with `name`, a modifier
such as `below:`, two keys at once — plus `point`, which only makes
sense as a last resort. A chain inside a chain is refused: it says
nothing a flat chain does not.
- Iterates entries in order; uses the first that resolves. Order is a
promise, not a tiebreak: `[id, text]` prefers the id even when both
match, so a chain keeps picking the same element as an app's copy
changes.
- Works in any selector position, in every verb that takes a selector —
`assertVisible`, `tapOn`, `extendedWaitUntil`, `fill`, and the rest —
and on both platforms.
- Where a verb returns every match rather than one, a chain gives you
the first layer that matched, not the union of the layers.
- A layer whose pattern cannot compile is skipped; the layers after it
still get their turn.
- Handy for production code where the same logical button might have testid in dev builds but not release builds.
- `ocrText`, `localizedText` and `anchorRelative` describe something the
accessibility tree cannot show, so each verb reads them above the
resolver or refuses them by name. Which verb does which is the table
below — generated from the one the code decides by, so the two cannot
drift apart. A refusal always says what to write instead.
<!-- BEGIN selector-matrix (generated by scripts/dev/gen-selector-matrix.py) -->
| verb | `ocrText:` | `localizedText:` | `anchored:` |
|---|---|---|---|
| `tapOn` | reads it | reads it | reads it |
| `doubleTapOn` | reads it | reads it | reads it |
| `longPressOn` | reads it | reads it | reads it |
| `repeatTap` | refuses, and says why | reads it | refuses, and says why |
| `inputText: { <target>, text }` | reads it | reads it | reads it |
| `copyTextFrom` | refuses, and says why | reads it | refuses, and says why |
| `assertVisible` | reads it | reads it | refuses, and says why |
| `assertNotVisible` | refuses, and says why | reads it | refuses, and says why |
| `extendedWaitUntil.visible` | reads it | reads it | refuses, and says why |
| `extendedWaitUntil.notVisible` | refuses, and says why | reads it | refuses, and says why |
| `scrollUntilVisible` | reads it | reads it | refuses, and says why |
| `runFlow.when.visible` / `.notVisible` | reads it | reads it | refuses, and says why |
| `takeScreenshot` annotation `at:` | refuses, and says why | refuses, and says why | refuses, and says why |
<!-- END selector-matrix -->
### 12. (Implicit shortcut) String
```yaml
- tapOn: "Submit" # equivalent to: tapOn: { text: "Submit" }
```
- Convenience shorthand for `text:` only.
## One control, two phones
A control with no words on it — an icon button, an icon-only tab — has
one name, given for accessibility: `contentDescription` on Android,
`accessibilityLabel` on iOS. smix reads both into `label`, and `text:`
checks `label` among the fields it matches, so either form reaches it.
Measured on both platforms with the fixture's icon-only controls, each
pressed once per row and the press read from the app's own count:
| platform | control | name lands in | `text:` finds / presses | `label:` finds / presses | `fallback: [text, label]` presses |
|---|---|---|---|---|---|
| Android | Compose, `semantics { contentDescription }` | `label` (both readers) | yes / yes | yes / yes | yes |
| Android | `ImageButton` hosted by `AndroidView`, `contentDescription` | `label` (both readers) | yes / yes | yes / yes | yes |
| iOS | SwiftUI `Button` with an image, `.accessibilityLabel` | `label` | yes / yes | yes / yes | yes |
So for an icon-only control, `label:` is the precise form and `text:`
also works. Write a chain when the two phones name the control
differently — an Android build describing it "Pause", an iOS build
labelling it "Pause recording":
```yaml
- tapOn:
fallback:
- label: "Pause"
- label: "Pause recording"
```
A control hosted by `AndroidView` inside Compose is visible to flows
since 11.0 — before it, the semantics probe did not walk into it, and a
flow on an app carrying the probe could not see it under any name.
## Modifiers
### Spatial (works on Text / Id / Label / Role / Anchor selectors)
```yaml
- tapOn:
text: "Cancel"
near: "Login" # closest visible "Cancel" to "Login" anchor
- tapOn:
role: button
below: "Header" # button below the Header element
- tapOn:
role: textfield
rightOf: "Email" # textfield to the right of "Email" label
- tapOn:
id: "list-row-content"
inside: "modal-list" # row that is descendant of modal-list
- tapOn:
role: button
ancestor: "modal-overlay" # button anywhere in the modal subtree
```
### Index (nth / first / last)
```yaml
- tapOn:
text: "Item.*"
nth: 2 # 0-indexed: 3rd match
- tapOn:
text: "Card.*"
first: true # equivalent to nth: 0
- tapOn:
text: "Page.*"
last: true # last in document order
```
### Combining modifiers
Spatial + index = picky:
```yaml
- tapOn:
text: "Edit"
inside: "modal"
nth: 1 # 2nd "Edit" within modal subtree
```
## Selector resolution diagnostics
If a selector matches zero elements, the failure message includes:
- The selector you used (full canonicalized form)
- `on screen:` the windows the screen held, whose they are, which holds the focus
- `visible elements (N of M, …):` the first N of M elements a reader could name, the focused app's first
- `suggestions:` text strings of nearby elements that *could* have been what you meant
Read the suggestions block carefully — it's tuned to surface i18n drift and missing testid issues.
```
ELEMENT_NOT_FOUND: tap_by_id: element not found — id="home-incremnt-btn"
on screen: com.example.app (application, focused)
visible elements (3 of 3, the focused app's first):
- id="home-counter-label" text="0"
- id="home-increment-btn" text="+1" ← typo in your selector!
- id="home-reset-btn" text="Reset"
suggestions:
- "home-increment-btn" (id, exact match to a sibling)
```
## Common pitfalls
- **Compose modal hosts in separate window.** `testTagsAsResourceId` on a root screen does NOT propagate into BottomSheet / AlertDialog. Each modal content + dialog button must add its own `.semantics { testTagsAsResourceId = true }`. See [08-cookbook.md](08-cookbook.md) §modal-testtags.
- **AlertDialog buttons need PER-BUTTON testTagsAsResourceId**, not just on content level.
- **LazyRow / LazyColumn lazy-render**: off-screen items don't generate AccessibilityNodeInfo. Scroll first (`scrollUntilVisible` or `adb shell input swipe`) then select. Tree dump only shows currently-visible items.
- **iOS Map (MapKit) and Camera (AVPreviewView) sub-widgets** are heavy native views — XCUI sees them as opaque `Other` elements with no children. Use accessibilityIdentifier on the wrapping View, not internal sub-widgets.
- **Vision OCR returns frame in (nx, ny, w, h) tuple**, not (nx_center, ny_center) — adjust if you compute tap point from OCR.
## See also
- [02-yaml-reference.md](02-yaml-reference.md) §selectors inside steps
- [06-fixtures.md](06-fixtures.md) — a testTag layout example you can mirror in your own app
<!-- ===== docs/ai-guide/04-actions.md ===== -->
# 04 — Actions
> Every action verb (tap / fill / swipe / scroll / press-key / etc.), the route it takes under the hood, and when a tap needs a different one.
## Action mental model
```
yaml verb (tapOn) → smix-adapter (translation) → Driver trait method
├─ IosDriver → host resolve → /tap-at-norm-coord
│ dispatch: xcui → /tap-by-id
│ dispatch: daemonProxy → /tap
└─ AndroidDriver → host resolve → /tap-at-norm-coord
/tap-by-id (view-id anchored)
```
The YAML is platform-agnostic. The driver picks the right strategy for the target platform.
## Tap
### Default tap
```yaml
- tapOn:
id: "home-increment-btn"
```
**iOS dispatch**: the host fetches the a11y tree, resolves the selector
against it, and sends the element's centre to `/tap-at-norm-coord`,
which taps via `coordinate(withNormalizedOffset:).tap()` — the Apple
native UI event chain. This is the only route a `tapOn` without
`dispatch:` takes; there is no fallback to another one, and having an
`accessibilityIdentifier` does not change it.
**Android dispatch**: the same host-side resolve, then the Kotlin
runner's `/tap-at-norm-coord`. A view-id anchored tap
(`findAccessibilityNodeInfosByViewId` → centre → native synthesize) is
what `dispatch: xcui` reaches, mirroring the iOS route of that name.
The two other routes exist because two specific runtimes need them, and
both are opt-in — see the next section.
**A `tapOn` waits for its target to stop moving.** The point to touch is
worked out from a reading of the screen, and on a screen still coming in
that reading is out of date by the time the touch lands. So after
finding the element, smix reads it again every 100 ms until two readings
in a row put it in the same place (every edge within one point, or one
dp on Android), for up to 3 seconds — the same wait maestro's `tapOn`
does. A target still moving after 3 seconds fails the step with
`TIMEOUT`, naming its last two positions; it is not tapped where it
used to be. `doubleTapOn` and `longPressOn` aim the same way. On a
still screen this costs one extra reading.
**What a successful `tapOn` means.** The runner reports every named
element containing the point it aimed at, and the step fails with
`TAP_MISSED` if the element you named is not among them. So success
means **the aim was inside your target**, judged against the
accessibility snapshot — a stronger statement than "a touch was
synthesised somewhere", and a weaker one than "your element was
touched".
The two runners report slightly different lists. The sentence above is
the iOS one — iOS lists the *named* elements under the point. Android lists *every* element under the
point, read from the window on top before the touch goes in — so on
Android an element with no id and no description is judged by its
frame, and a touch that reaches no window at all is a miss. The same
applies to `doubleTapOn` and `longPressOn` on Android. A runner that
reports nothing about where a touch went fails the step and says so
(until 11.0, every Android tap reported nothing and was printed as
`not verified` — and passed).
The gap between those two is not hypothetical. The comparison happens
entirely in the coordinate space the snapshot describes; the
synthesised event is read in whatever space it is stamped with, and
on a landscape screen those are not the same space —
every tap reports the button it aimed at and the screen does not
change. `smix` refuses the tap outright when it can see the two
disagree, but the general shape of the limit stands: this line is
evidence about aim.
It does **not** mean the target received the touch. Something drawn
over your element contains the same point, and the a11y snapshot
carries no z-order, so smix cannot tell which one is on top. This is a
limit of the platform, not an omission: the snapshot is a dead frame
with no frontmost or occlusion field on it, and the only z-order-aware
signal XCUITest offers is `isHittable`, which reports false under
floating overlays that are genuinely visible and assertable. If a tap
reports success and nothing happens, see `07-errors.md` →
"tap returns `ok: true` but state doesn't change".
**A control that goes away on its own.** `tapOn` waits for its target
itself — it keeps reading the screen until the element resolves (up to
5 s), then taps. So a control that shows for a few seconds and hides
(a player's buttons after the picture is touched) is tapped by writing
`tapOn` directly; there is no need to wait for it first. Measured on the
repository's fixture, from the moment the control appeared to the moment
the app counted the press, on the app's own clock: about 0.8 s on an
Android emulator and 0.35 s on an iOS simulator with `tapOn` alone, and
about 0.05–0.08 s more with an `extendedWaitUntil` in front of it. Most
of that time is the previous step finishing — the tap that revealed the
control — not the tap on it. A screen whose accessibility tree is much
larger than the fixture's takes longer to read, so measure on your own
screen before assuming a margin.
### Tap with explicit dispatch (v1.0.26)
The default tap path (host-resolve → IOHID native-event synthesize) fires SwiftUI `onTap` and RN Pressable `onPress` reliably. Two runtime-specific cases need a different mechanism — declare it per tap:
```yaml
# SwiftUI .sheet / .alert / .confirmationDialog / .fullScreenCover
# dismiss BINDINGS don't flip from coord-based taps on iOS 17+; the
# XCUIElement-anchored path is the one that fires them. Requires `id:`.
- tapOn:
id: "modal-sheet-dismiss-btn"
dispatch: xcui
# XCTRunnerDaemonSession synthesize — bypasses the XCUIElement gesture-
# recognizer chain so RN RCTTouchHandler receives the raw touch. Use
# when a Pressable swallows the default path on an older RN.
- tapOn:
id: "btn-login"
dispatch: daemonProxy
```
- Omit `dispatch:` for the default routing — it is right for almost every tap.
- `dispatch: xcui` requires an `id:` selector (resolution is by accessibilityIdentifier). Cross-platform: on Android it routes through the runner's id-anchored tap (`/tap-by-id`), same semantics.
- `dispatch: daemonProxy` is an iOS-runner mechanism; on Android it errors with an explicit unsupported message (native synthesize is already the Android default — the override is never needed there).
- On Android, `dispatch: xcui` first asks the view for its accessibility click (`ACTION_CLICK`), and touches the view only when, for 1.5 seconds, it offers no such action or refuses it. The runner's answer says which: `"path": "a11y"` or `"path": "touch"`. **`path: a11y` means the view accepted the accessibility action, not that the screen changed.** A view that advertises a click but reacts only in `onTouchEvent` accepts the action and does nothing, and the step still passes. So follow a `dispatch: xcui` tap with the assertion that proves it worked (`assertVisible` on what the tap should bring up), and if a view does nothing on `xcui` but works on a plain `tapOn`, route its click through `performClick()` in the app.
### How many times a tap reads the screen
Each read of the screen is a round trip to the runner and a walk of the
app's accessibility tree, so on a busy machine they are what a step's
time goes on. Counted from the runner's log on an iOS simulator, one
step each:
| step | tree reads | then |
|---|---|---|
| `tapOn: { id: … }` (default) | 2 — one to find the target, one to see it has stopped moving | one `/tap-at-norm-coord`; where it landed is judged by the runner inside that request, with no further read |
| `tapOn: { id: …, dispatch: xcui }` | 0 | one `/tap-by-id`; the runner finds the element itself |
| `inputText: { id: …, text: … }` | 2 | one tap to focus, then one `/fill` |
A target still coming in takes one more read for every 100 ms it keeps
moving (see above). Android runs the same host-side steps, and each of
its tree reads is the probe's semantics tree and the accessibility tree
together, so a read costs more there, and more again on a loaded
emulator. A control that disappears on its own a few seconds after it
appears is best pressed with `dispatch: xcui`, which reads nothing
before it acts.
### Tap by coord
```yaml
- tapOn:
point: "50%,80%" # fraction of the viewport, not pixels ("0.5,0.8" is the same)
```
- nx, ny in viewport normalized [0, 1].
- Bypasses element resolution — direct touch dispatch at coord.
- Brittle; only use when no element identifier is reliable.
### Tap with OCR fallback
```yaml
- tapOn:
fallback:
- id: "btn-skip" # tier 1 — a11y identifier (cheap)
- text: "Skip" # tier 2 — a11y text
- ocrText: "Skip" # tier 3 — Vision (iOS) / ML Kit (Android)
```
- Each tier is probed in order; first hit taps. OCR taps at the recognized text's bounding-box center via native event synthesize.
- When the chain contains any `ocrText`, the whole chain is **polled** for up to `SMIX_TAP_OCR_POLL_MS` (default 3000 ms, 250 ms cadence) before failing — this closes the race where the tap fires while the target is still mounting and a one-shot OCR would snapshot too early. Chains without OCR keep single-pass semantics (no perf change).
- On full miss the failure hint carries a per-layer trace (`L1 …: MISS; L2 …: MISS; L3 …: MISS`) naming exactly which tiers were probed.
- One OCR call is ~500 ms on-sim; keep at least one cheap tree tier ahead of `ocrText`.
### Double tap
```yaml
- doubleTapOn:
id: "photo-1"
```
- Two touches, 80ms apart on iOS, ~150ms on Android — inside every
platform's double-tap window.
- Resolved on the host and judged like `tapOn`: the step fails with
`TAP_MISSED` when the touch went to something other than the element
(a dialog over it, a row that moved), instead of passing because a
touch was sent.
- iOS: one synthesised event carrying both touches. Android: the
runner's coordinate double tap.
### Long press
```yaml
- longPressOn:
id: "list-row-3"
duration: 1500 # ms
```
- iOS: a synthesised touch held on the element's centre for `duration`.
`XCUIElement.press(forDuration:)` is not used — it was measured taking
a constant ~2.6s for every hold from 500ms to 6000ms on iOS 26.5, so
the duration it performed was not the one it was given.
- Android: the runner's coordinate long press, held for `duration`.
- Judged like `tapOn` on both platforms: a press delivered to something
other than the element fails with `TAP_MISSED`.
#### Capturing the held state
Verifying a pressed appearance — a highlight, a scale-down, a ghost
background — means seeing the screen *while* the touch is down.
```yaml
- longPressOn:
id: "hdr-back-btn"
duration: 1500
captureDuring: true
```
Frames are written to `--debug-output` when given, else `.smix/press/`,
one PNG per capture, named for where it sits relative to the press:
| suffix | meaning |
|---|---|
| `-during` | the touch was provably down for the whole capture |
| `-unplaced` | the capture straddles a boundary; its pixels could be from either side |
| `-outside` | the touch was provably not down for part of it |
The step **fails** when no frame can be placed inside the press, rather
than handing back frames that might be of the resting screen. That
failure is the one this exists to prevent: screenshotting alongside a
press and reading a resting screen as "the animation never fired".
`duration` must be at least **800ms**. The gesture call carries
290-342ms of overhead and a capture takes 190-350ms, so a shorter hold
leaves no stretch a frame provably sits inside. 800ms yields one placed
frame; 1500ms yields three.
Android does not support `captureDuring`: `UiDevice.swipe` reports
nothing about when the touch was down, so no frame could be placed.
## Text input
### Fill
```yaml
- inputText:
id: "form-email-input"
text: "alice@example.com" # replaces what the field holds
- inputText: "alice@example.com" # types into the focused field, appending
```
**You can only replace a field you named.** The mapping form above
empties the field first, so returning to a screen and filling it again
leaves the second value rather than both concatenated — which is
invisible in a password field and surfaces as a login that fails with
the right password typed. The scalar form has no field to empty: it
types wherever focus already is, and appends, which is also what
maestro's `inputText` and `pasteText` do.
To append to a named field, clear nothing and type into the focus you
already have: `tapOn` it, then the scalar form.
- iOS: XCUI `typeText` after explicit field tap (focus + type).
- Android: Kotlin `/input-text` → `am instrument` shell input (UiAutomation.executeShellCommand wraps `input text`).
- Android, no field named: nothing is tapped. The text goes to the field
that holds input focus, read back afterwards; with no such field the
step fails as nothing to type into. `tapOn` the field first.
- Android read-back: the runner checks that the characters arrived
before answering. A field that masks its contents cannot be asked
that question — its accessibility node reports one bullet per
character and never the characters — so a masked field is judged by
how much longer it got instead. The masked branch is keyed on the
node reporting itself as a password, never on the text looking like
one, so a plaintext field holding `aaaa` is still checked by content.
- Android targeting: a named fill tells the runner where the field was
tapped, and the runner waits for focus to reach *that* field rather
than for any field to have focus. Without it, a fill naming one field
could clear and type into whichever field had focus a moment earlier.
A scalar `inputText` names no field and still means "wherever focus
is".
- Special chars: spaces, unicode are handled; backslash-escaping is internal.
- Important Android quirk: UiAutomation.executeShellCommand does NOT use `sh -c`, so quote-wrapping the text causes literal quotes. The runner handles this; do not pre-quote yourself.
### Clear
```yaml
- eraseText: 5 # backspace 5 times (default 50 if omitted)
- eraseText # default depth
```
- iOS: XCUI `typeText(XCUIKeyboardKeyDelete)` × N.
- Android: dispatched as N "BACK" key events (or N input shell calls if BACK not available).
### Hide keyboard
```yaml
- hideKeyboard
```
- iOS: several strategies in order — dismiss/return keys, a tap outside
any text field, and a tap just above the keyboard's own frame — until
the keyboard is gone.
- Android: Kotlin runner `/hide-keyboard` → `IME.hide()`.
**It answers about the keyboard, not about itself.** No keyboard present
is success (there is nothing to hide). A keyboard still up after every
strategy ran is `keyboard_did_not_close`, and a runner that raised while
looking is `keyboard_state_unknown` — see
[07 — Error codes](07-errors.md). Until 9.0.0 all four cases answered
`ok: true`.
## Scroll
### One-shot scroll
```yaml
- scroll
- scroll:
direction: UP
```
- Default: scroll viewport DOWN by one screen height (or by component's pageSize for paginated content).
- Directions: UP / DOWN / LEFT / RIGHT.
- iOS: `element.swipeUp()` / `coord.scroll(byDeltaX:deltaY:)`.
- Android: `/swipe-once` with hardcoded distance ratios.
### Scroll until visible
```yaml
- scrollUntilVisible:
element:
text: "Row #5000"
direction: DOWN
timeout: 30000 # ms; default 20000
# Stop earlier, or stop with the element near the middle of the screen.
- scrollUntilVisible:
element:
id: "row-line_crossing"
visibilityPercentage: 60 # 1-100; default 100, the whole element
centerElement: true # default false
# OCR tier — for virtualized lists whose off-screen items are dropped
# from the a11y tree (RN Fabric LazyColumn/LazyRow etc.). Every look
# reads the tree first, then each ocrText in the chain's order.
- scrollUntilVisible:
element:
fallback:
- id: "btn-target"
- ocrText: "Target label"
direction: DOWN
```
- Swipes until the element is reached, then stops.
- **Reached means wholly on screen** (`visibilityPercentage: 100`, maestro's
default). An element that merely overlaps the screen is not reached: its
middle can be past the edge, and the `tapOn` after it would aim there.
- `centerElement: true` stops once the element's middle has come past the
centre of the screen, from the side the content is arriving from. It gives
up centring after four swipes — the last row of a list cannot be centred —
and the visibility rule decides from there.
- The scroll also waits for the content to stop: a list is still gliding
after a swipe, and a tap sent into the glide is spent stopping it.
- `timeout` (default 20 s) is the only limit. There is no swipe count.
- Failure says how much of the element the last look saw, and what was asked
for — or that it never appeared.
- `speed` and `waitToSettleTimeoutMs` (maestro) are **refused by name**: one
swipe is a fixed gesture on both runners, so there is no duration to set,
and the settling above is not a number to tune.
- `label` and `optional` work as they do on other steps: `optional: true`
turns "not reached" into a skip and the flow goes on.
### Custom swipe (explicit start/end)
```yaml
- swipe:
start: "10%,50%"
end: "90%,50%"
duration: 400
```
- nx, ny normalized 0–1.
- Useful for horizontal tab bars, carousels, gesture-driven menus.
## Press hardware key
```yaml
- pressKey: ENTER
- pressKey: HOME
- pressKey: VOLUME_UP
- pressKey: LOCK
```
- Available keys: ENTER / TAB / SPACE / DELETE / ESCAPE / HOME / LOCK / VOLUME_UP / VOLUME_DOWN / ARROW_UP / ARROW_DOWN / ARROW_LEFT / ARROW_RIGHT / BACK.
- maestro's spellings (`Enter`, `Backspace`, `Volume Up`) read too; case, spaces, `_` and `-` do not matter; a name that is no key fails when the flow is read.
- **Back.** `- back`, `pressKey: BACK` and `smix press-key back` are one
thing: on Android the system back key, on iOS the navigation bar's
back button, and the step fails when nothing went back. It is not a
keystroke that nobody checks — an earlier alias mapped `BACK` to
Delete, which turned every back step into a silent backspace that
reported success.
- **Under gesture navigation there is no back button to tap.**
`tapOn: { id: back }` finds the three-button navigation bar's button,
which gesture navigation — the default on current phones — does not
have. Use `- back`: it closes a system sheet (share, permission) the
same way in either mode.
- **`LOCK` / `VOLUME_UP` / `VOLUME_DOWN` are pressed on Android** —
`KEYCODE_POWER` and the two volume keys, through the runner like any
other key. `LOCK` turns the display off; the step does not unlock it
again, so a flow that goes on driving the app has to wake the device
first.
- **On iOS the three fail the step, naming why** (`no_such_button`):
XCUIDevice has no lock button on any iOS target, the simulator has no
volume buttons, and on a physical iPhone this runner has never been
measured pressing them. A flow that runs on both platforms presses
them inside `runFlow` with `when: { platform: Android }`. Earlier
releases reported the three as skipped on every platform, Android
included, and a flow passed with a step that did nothing.
- There is no `POWER` key; `Power` reads as `LOCK`.
- iOS: maps to XCUIRemote / device interaction methods.
- Android: maps to KeyEvent constants via runner `/press-key`.
## App lifecycle
```yaml
- launchApp:
appId: com.example.app
clearState: true
clearKeychain: true
arguments: ["--debug-mode"]
- killApp # current
- killApp: com.acme.other # named
- stopApp # graceful
```
- iOS: `simctl terminate / launch` with clearState wiping `~/Library/Caches` and defaults.
- Android: `am force-stop` + `pm clear` (clearState).
## Orientation
```yaml
- setOrientation: LANDSCAPE_LEFT
- setOrientation: PORTRAIT
```
- iOS: `simctl io <udid> orientation <name>`.
- Android: `adb shell content insert --uri content://settings/system --bind name:s:user_rotation`.
- Some apps lock orientation in Info.plist / manifest → setOrientation no-op; assert with `assertVisible` against orientation-dependent label to verify.
## System permissions
```yaml
- setPermissions:
camera: allow
location: allow
notifications: allow
photos: deny
```
- iOS: `simctl privacy grant/deny`.
- Android: `adb shell pm grant <pkg> android.permission.<NAME>` or revoke.
- Available keys: camera / microphone / location / contacts / photos / calendar / notifications / motion / faceid / siri / health.
## Deep links / openLink
```yaml
- openLink: "myapp://home/details/42"
- openLink:
link: "https://example.com"
```
- iOS: `xcrun simctl openurl`.
- Android: `adb shell am start -W -a android.intent.action.VIEW -d '<url>' <pkg>`.
## Recording
```yaml
- startRecording: "trace.mp4"
- stopRecording
- takeScreenshot: "step5.png"
```
- iOS: `simctl io recordVideo` (background process).
- Android: `screenrecord` shell command (background process).
## Output destinations
By default, screenshots/videos save to `./.smix/trace/<run-id>/`. Override via `--trace-dir` flag on `smix run`.
## See also
- [02-yaml-reference.md](02-yaml-reference.md) — full step grammar
- [03-selectors.md](03-selectors.md) — how to identify the element to act on
- [05-cli.md](05-cli.md) — when you want to invoke actions one-shot via CLI (not YAML)
<!-- ===== docs/ai-guide/05-cli.md ===== -->
# 05 — CLI reference
> Every subcommand + flag of the `smix` binary. Reference, not tutorial — for tutorial see [01-quickstart.md](01-quickstart.md).
## `smix run` — execute a flow YAML
### `smix run <YAML_FILE>` — run a flow
```bash
smix run [FLAGS] <FLOW.yaml>
```
| Flag | Env | Default | Description |
|---|---|---|---|
| `--device <DEVICE>` | `SMIX_UDID` | (none, required) | Target simulator UDID or Android device id (e.g. `emulator-5554`) |
| `--bundle-id <ID>` | `SMIX_BUNDLE_ID` | (unset) | Default app under test (overrideable by `appId:` in YAML) |
| `--runner-port <PORT>` | `SMIX_RUNNER_PORT` | `22087` | Runner HTTP port. iOS = 22087, Android = 28080 by convention |
| `--platform <ios\|android>` | `SMIX_PLATFORM` | inferred from `--device` | Optional. Read from the named device's registered kind (emulator / physical-android → android, simulator / physical-ios → ios). An explicit value wins and is **required** when naming a device that is not registered — an unregistered bare name with no platform is an error, not a silent iOS default |
| `--apps-config <PATH>` | `SMIX_APPS_CONFIG` | (unset) | Path to `apps.yaml` for cross-platform `app:` logical key resolution |
| `--no-launch` | — | (off) | Skip initial `foreground()` call. Use when you launched the app via other means |
| `--trace-dir <PATH>` | — | `./.smix/trace/<runid>` | Where screenshots / video / JSON traces go |
| `--dry-run` (alias `--check`) | — | (off) | Parse-only gate: validates every listed yaml (+ `runFlow:` includes) and reports `parse OK/FAIL` per file with step counts. No runner, no simulator. Exit 0 on clean parse, 2 on any error |
| `--retry <N>` | — | `1` | Per-flow attempt count; attempts recorded in `~/.local/share/smix/flow-attempts.json` for `smix diagnostic dump` attribution |
| `--debug-output <DIR>` | — | (unset) | Per-step JSON + on-fail screenshot/tree artifacts |
| `--nodes <PATH>` | — | (unset) | Distributed run across machines; see below |
Environment variables consumed by `smix run` (beyond the flag-bound ones above):
| Env | Default | Description |
|---|---|---|
| `SMIX_WEBVIEW_BRIDGE_PORT` | `28080` | Port of the in-app `SmixWebViewBridge` that `webviewEval` posts to on iOS. Infrastructure port like `SMIX_RUNNER_PORT`, not one of the behaviour switches below — it has no `.smix/config.yaml` equivalent |
| `SMIX_AUTO_OCR_FALLBACK` | (off) | `1`/`true`/`yes` — auto-lift bare-string selectors to `fallback: [text, ocrText]` at parse time; `A\|B` regex-OR strings split per alternative on the OCR tiers |
| `SMIX_TAP_OCR_POLL_MS` | `3000` | Poll budget for `tapOn` fallback chains that contain `ocrText` (250 ms cadence) |
### Exit codes
| Code | Meaning |
|---|---|
| 0 | success |
| 2 | YAML parse error (`RunError::Parse`) |
| 3 | runtime SDK failure (`RunError::Sdk`) — sim crashed, app died, etc. |
| 4 | unknown direction (`RunError::UnknownDirection`); an unknown key name is a parse error (2) |
| 5 | runFlow cycle / file IO (`RunError::RunFlowCycle` / `RunError::Io`) |
| 6 | runner unreachable (runner not up / port wrong) |
### Output formats
- stdout has step-by-step progress + final summary JSON
- stderr has DEBUG / WARN lines (when `RUST_LOG=info` or higher)
- Final line is `summary: N steps, X warnings, Y skipped, Z expanded subflows` on success
### Distributed runs across machines (`--nodes`)
`smix run <flows...> --nodes <roster.yaml>` shards the listed flows
round-robin across every device of every node in a roster, runs each
shard remotely over ssh, and merges the results into one JSON document
on stdout. Nodes list simulators/emulators only — the simulator-only
invariant holds across machines.
Roster shape (conventionally `.smix/nodes.yaml`):
```yaml
nodes:
- name: studio
host: localhost
repo: /Users/me/workspace/smix
devices:
- { device: sim-smix-02, platform: ios }
runnerPort: 22097 # optional; forwarded as --runner-port
- name: mini
host: mini
repo: /Users/me/workspace/smix
devices:
- { device: sim-smix-001, platform: ios }
- { device: sim-smix-android-01, platform: android }
```
Each device carries its `platform` (`ios` or `android`), and the run on
that node is given it as `--platform`. A node's smix reads a device's
platform from its own registry and refuses a device it has not
registered rather than guess; the roster is where that is declared, so a
device need not be registered on the node. A device without a platform
is a parse error that names it — including the bare-string form
(`devices: [sim-smix-02]`) that rosters used before 11.0.
Node preparation is the operator's job, not the CLI's. Before a run,
each remote node needs two steps (the scheduler repo is the authority):
```bash
rsync -a --exclude target/ --exclude .git/ ./ <host>:<repo>/
ssh <host> 'cd <repo> && cargo build --release -p smix-cli && touch target/.smix-fed-stamp'
```
The CLI runs a per-node readiness gate first (binary present, stamp
present, no source newer than the stamp) and fails fast if any node is
stale or unreachable — the gate only judges, it never rebuilds. Flow
paths are repo-relative and must exist at the same path on every node;
a flow missing on the scheduler fails before any ssh is dialed.
```bash
smix run smoke.yaml checkout.yaml --nodes .smix/nodes.yaml --debug-output ./artifacts
```
- **Output**: one merged JSON document on stdout, shaped
`{"nodes":[{"node":"studio","exit":0,"flows":[…]}],"aggregateExit":0}` —
the flow leaves are each node's `--format json` report lines verbatim.
- **Exit**: worst of nodes. `255` means an ssh transport failure on at
least one node (never produced by smix itself, so it cannot be masked).
- **`--debug-output <dir>`**: each node stages its artifacts under
`.smix/fed-artifacts` in its repo (overwritten in place, never
pre-cleaned), and the CLI rsync-pulls them back into `<dir>/<node>/`
after the run. A failed pull fails the whole run.
- **Mutually exclusive** with `--device`, `--also-device` and
`--parallel`: device placement belongs to the roster. Note that an
exported `SMIX_UDID` counts as `--device` being present and triggers
the conflict — unset it when using `--nodes`.
- **Not consulted** in this lane: `--format` (the merged report is
always the JSON document; remote leaves always run `--format json`)
and `--runner-port` (ports are per-node — set `runnerPort` in the
roster instead).
## `smix` — environment + low-level
### Sim management (`smix sim …`)
```bash
smix sim list # simulators and Android devices (JSON: --json); an attached phone nobody registered is listed from adb's line and not asked anything ("registered": false)
smix sim resolve <ALIAS> # alias → UDID
smix sim boot <ALIAS|UDID> # boot; an emulator whose registered port is taken starts on a free one, in a process group of its own so it outlives the terminal that started it
smix sim boot <ALIAS> --cold # an Android emulator boots from scratch instead of its Quick Boot snapshot, which keeps whatever state the device stopped in (a system UI that died stays dead); refused on anything else, and on an emulator already running
smix sim shutdown <ALIAS|UDID> # shutdown; for an emulator, returns once adb no longer lists it
smix sim erase <ALIAS|UDID> # wipe (reset content)
smix sim screenshot <ALIAS|UDID> <out.png> # simulator → simctl, Android → adb, physical iPhone → devicectl (Xcode 27) or the runner (Xcode <= 26, must be up)
smix sim launch <ALIAS|UDID> <bundle-id> # every kind; --child-env is a simulator facility and is refused elsewhere
smix sim terminate <ALIAS|UDID> <bundle-id> # every kind but a physical iPhone (devicectl stops by pid, not by bundle id)
smix sim install <ALIAS|UDID> <path/to.app|.apk> # every kind; reinstalling stops the running copy of the app, and the command says so
smix sim uninstall <ALIAS|UDID> <bundle-id|package> # every kind; on a physical one, after allow-destructive
smix sim openurl <ALIAS|UDID> <url> # every kind; deeplink, app-level, no device lease needed
smix sim appearance <ALIAS|UDID> light|dark # simulator only
smix sim keychain-reset <ALIAS|UDID> # simulator only
smix sim allow-destructive <ALIAS|UDID> # physical devices only; once, not per command
smix sim exec <ALIAS|UDID> <verb> [args...]
```
Which devices a verb reaches is not a list kept here: it is derived from the
platform table in `smix-sdk` (`ACTION_PLATFORMS`), which is reconciled against
the implementations themselves. A verb refused on a device says why and what to
do instead.
**A reinstall ends the running copy of the app it replaces.** That is `adb
install -r` and `simctl install` alike, and it does *not* touch the runner —
`smix sim install` names the app it stopped so the two are not confused:
```
installed: app-debug.apk on emulator-5554
dev.smix.fixture was running and this reinstall stopped it — bring it back with `smix sim launch <device> dev.smix.fixture`
```
### Physical devices
A physical device must be **registered before it can be addressed** — smix
never reaches "whatever happens to be plugged in":
```bash
smix sim register <ALIAS> --udid <UDID-or-SERIAL> --kind physical-ios --name "my phone"
smix sim register <ALIAS> --udid <SERIAL> --kind physical-android
```
The identifier is taken as given for a physical device — a UDID for iOS, an
adb serial for Android — and is not checked against a catalogue, because
there is no catalogue of phones to check against. Registration is the
deliberate act; that is why it is required.
A virtual device is checked against the catalogue its own platform keeps:
`--kind simulator` (the default) against `simctl list devices`, `--kind
emulator` against `adb devices`. Each kind also has its own identifier shape —
a CoreSimulator UDID for a simulator, `emulator-<port>` for an emulator — and
registering one under the other kind is refused, naming the shape that kind
actually uses.
One MCP caveat: `smix_use` cannot *start* a runner on a physical iPhone —
building one needs the registry and a signing team, which are the CLI's to
resolve. Run `smix runner up <alias> --bundle <id>` first; every MCP tool then
drives the phone through that runner exactly as it would a simulator.
The clipboard verbs — `setClipboard`, `copyTextFrom`, and `pasteText` with no
text — work on a registered iPhone from Xcode 27 on, through `devicectl device
pasteboard copy` and `paste`. What comes back is checked against the byte count
`devicectl` reports, and bytes that are not UTF-8 are an error rather than
replacement characters. Under Xcode <= 26 `devicectl` has no such verb and the
error is its own. Mind whose clipboard it is: a phone's general pasteboard
travels by Universal Clipboard to every device on the same Apple ID, so a flow
that sets it also sets the one on the Mac beside it.
`setLocation` and `travel` work on a registered iPhone from Xcode 27 on too,
through `devicectl device simulate location coordinate` and `route`. `travel`
returns at once and the phone keeps moving, as on a simulator; without a speed
it goes at 20 m/s. **A simulated location stays on the phone until it is
cleared, and no flow verb clears it** — every map and weather app on that phone
is wrong until you run:
```bash
xcrun devicectl device simulate location clear --device <UDID>
```
`devicectl` has no verb that reads a phone's current simulated location back, so
what smix checks is `devicectl`'s own account of what it set against what was
sent; a disagreement is an error, not a success.
### Arranging a device, and asking what it is doing
A run needs the screen on, the screen to stay on, permissions in a known
state, and — when something goes wrong — an answer to "what was in front" and
"did it crash". All five take the device explicitly.
```bash
smix sim wake <DEVICE> # turn the screen on
smix sim stay-awake <DEVICE> on # and keep it on while charging
smix sim permission <DEVICE> <BUNDLE> camera grant # grant / revoke / reset one permission
smix sim frontmost <DEVICE> # which app is in front
smix sim crashes <DEVICE> --app <BUNDLE> # what the device recorded
```
Four things worth knowing before you rely on them:
- **`wake` turns the screen on. It does not unlock.** A device with a passcode
keeps its keyguard, and nothing here gets past it.
- **`stay-awake` writes a device setting**, so it outlives the run that set it.
`off` is how it goes back.
- **`crashes` exits 0 whether or not it finds anything.** A read that found
nothing is not a failure; under `set -e` the other choice would kill a healthy
run. With `--app` the last line says how many of the reports named that
process, so "none of them was yours" reads differently from "the device
recorded none".
- **`frontmost` reads the resumed activity**, not whichever window holds focus —
a dialog on top does not change the answer. Nothing resumed (a lock screen, a
device still booting) prints that and exits 0.
These are Android verbs. `wake`, `stay-awake`, `frontmost` and `crashes` refuse
on an iPhone or a simulator and say what to do instead: neither `simctl` nor
`devicectl` has a verb for any of them, a simulator's screen never sleeps, and a
simulator's crash reports land in this machine's own `~/Library/Logs/
DiagnosticReports`, which is not divided by device. `permission` is the
exception — a simulator answers it through `simctl privacy`, and only a physical
iPhone refuses, because those grants are the owner's.
### Letting a device reach a server on this machine
An app under test often has to talk to something running on your machine — a
stub API, an asset server, a mock gateway. An emulator can reach it at the
special address `10.0.2.2`; a phone on the cable has no address for it at all.
```bash
smix sim reverse <DEVICE> 8080 # device dials 127.0.0.1:8080 → this machine's 8080
smix sim reverse <DEVICE> 8080 --to 3000 # → this machine's 3000 instead
smix sim reverse <DEVICE> 8080 --remove # close it
```
The app dials `127.0.0.1:<port>` on the device and lands here. It works the
same on an emulator as on a registered phone, so a flow never has to know which
one it is driving — that is the reason to use it on an emulator too rather than
teaching the app about `10.0.2.2`.
**What it opens stays open.** It outlives the command, the runner and the flow,
and goes only when `--remove` names it, when the device is unplugged or shut
down, or when a teardown closes the record smix keeps of it. That record is the
device's ledger, the same place a runner and a recording are written down, so a
route left behind by a session that died is closed by the next `smix lease
reconcile` rather than lingering until someone notices.
It is Android-only, and the two Apple kinds say why rather than doing nothing:
a simulator already shares this machine's loopback, so `127.0.0.1:<port>` on it
is already this machine's port; a physical iPhone has no such channel in that
direction, so the service has to be on an address the phone can reach over the
network.
Apple identifiers are normalised to upper case, because `devicectl` will not
match a lower-case spelling of a UDID it accepts in upper case. adb serials are
stored and returned verbatim, because `adb` matches them byte for byte.
Destructive actions (`erase`, `uninstall`, `keychain-reset`) are **refused on a
physical device** until you allow them once:
```bash
smix sim allow-destructive <ALIAS>
```
Recorded in the registry, not confirmed per command — a confirmation that has to
be typed every time ends up pasted into a script, which is the same as not
having one. Simulators are never gated: they can be erased and rebuilt in a
minute, and a phone in somebody's pocket cannot.
**Sim safety hook**: bare `xcrun simctl <verb>` is BLOCKED for mutating verbs. Read-only `simctl list` is allowed. To pass-through unmapped subcommands use `smix sim exec <ALIAS|UDID> <verb> ...` (this is accepted by the hook because the device id is explicit).
`<ALIAS>` resolves against this machine's device registry, under
`$XDG_DATA_HOME/smix/devices` (or `~/.local/share/smix/devices`), created and
populated by `smix sim register <alias> --udid <UDID>`. A simulator is an
operating-system object: its UDID, its runtime version and whether destruction
has been allowed on it do not change when you `cd`, so they are recorded once
for the machine rather than once per checkout.
A `.smix/` registry inside a checkout still resolves, as a read-only fallback,
and a device only a checkout knows about is named as such — no other tree can
see it. `smix sim migrate` folds those in; it adds and never removes, so it is
safe to run twice. `smix sim list --registered` shows every recorded device and
which of the two it came from.
A raw identifier works without a registry **only if the platform itself claims
it**: a UDID `simctl` lists, or an adb serial of the form `emulator-NNNN`.
Anything else — an unregistered phone's UDID, an unregistered adb serial — is
refused before the command runs:
```
$ smix sim erase D51116A4-B2AD-5432-8A75-6FBB13F17B58
error: D51116A4-… is not a device smix may address: it is not registered here,
and neither simctl nor adb calls it one of theirs.
If it is a phone or tablet, say so once and it becomes addressable:
smix sim register <name> --udid D51116A4-… --kind physical-ios|physical-android
```
This is the enforcement of "registered before it can be addressed". It is a
separate question from whether a destructive action is allowed: registering a
phone makes it *reachable*, and `allow-destructive` is still needed before
anything may be taken away from it. Two gates, asked in that order.
### Runner management
`smix runner down` stops the runner **recorded on that port** (`--runner-port`,
else 22087) in this machine's device ledger, and nothing else: with two
simulators each running one, taking one down leaves the other and its record
untouched, whichever checkout started them. If the port is held by a runner
the ledger has no record of, it says so and stops rather than ending it —
that runner may belong to another session:
```
$ smix runner down
error: port 22087 is held by a runner the device ledger has no record of
(pid 27155), and it is still running.
It may belong to another session — check before ending it:
ps -o lstart=,command= -p 27155
If it should go, say so:
smix runner down --include-unrecorded
```
`smix runner up` refuses the same port for the same reason. The two commands
agree deliberately: ending somebody else's runner used to be one keystroke
away, and silent.
On Android the unit is the **device**, not the port. The instrumentation is
one package and the device-side port is fixed, so every host port forwards
onto the same server: a device has exactly one runner, and `runner down`
ends it whoever started it. Both verbs read the machine's lease ledger
first and stop when the device is being driven by a live process that is
not this one, printing its command line:
```
$ smix runner down --device emulator-5554 --platform android
error: emulator-5554 is being driven by another process, so `runner down`
would end a run that is not ours.
pid 928: smix run --device emulator-5554 --platform android
--runner-port 28080 ~/their-project/qa/playback.yaml
An Android device has one runner -- one instrumentation package, one
device-side port -- so there is no version of this that touches only ours.
Wait for it, or say `--take-over` to replace it deliberately.
```
`--take-over` is not `--force` and not `--include-unrecorded`. `--force`
cycles a runner of yours whose session has stopped working; the runner here
is working fine and is not yours. `--include-unrecorded` is for a runner the
ledger does not know about; this one it knows and says is somebody else's.
A holder whose process has exited is not a holder, so the ordinary teardown
after a flow is unaffected.
`runner up` also checks that the runner already answering is one this smix
built. Another install's runner answers `/health` perfectly, and `up` used
to report success while driving it — a 10.0.0 CLI drove a 9.0.0 runner and
the routes added in between came back `not_implemented`. It now compares
`runnerVersion` with its own and replaces on a mismatch. A runner too old to
report a version is left alone.
They take the same `--runner-port` too. They did not until 2.4.0 —
`down` read `SMIX_RUNNER_PORT` and rejected the flag — so a teardown
written as the obvious mirror of the bring-up failed its argument parse
and left the runner running.
Every command that talks to a runner takes `--runner-port` (env
`SMIX_RUNNER_PORT`) and lists it in its `--help`. Until 11.0 eight of them
read the variable without offering the flag — `runner cycle`,
`runner list-sessions`, `capsule up` / `down`, `diagnostic dump`,
`sim screenshot` (a phone photographed through the runner), `down` and
`doctor` — so a script could not see from the help that they would reach
whichever runner the environment named.
The runner is the on-device server smix drives. `runner up` blocks until
its `/health` answers **and its session answers**, so when the command
returns you can run a flow.
Those are two questions, and it is worth knowing which is which.
`/health` says the runner's HTTP server is answering: it is a handler
over a boot date, and it never touches the app. So a runner whose app was
reinstalled underneath it — `simctl install` over a running build is the
usual way — answers `/health` 200 and fails every real call. Until 4.3
`runner up` read only the first of those and returned "already up", which
made the one command that could recover the runner the one refusing to.
Now it says so, and names the fix:
```
$ smix runner up <UDID> --bundle com.example.app
error: port 22087 answers /health, but its session is not usable: not-running
the runner said:
The app is not running — it was terminated, or reinstalled out from under
the runner. Launch it again, or `smix runner cycle` to rebind, then retry.
Recover it in place (seconds, without restarting xcodebuild):
smix runner cycle
or have this command do it for you:
smix runner up <UDID> --bundle com.example.app --force
```
`--force` runs that cycle for you. It is not a way past the ownership
check: a runner recorded for another device, or one the store has no
record of, is refused with the flag exactly as without it —
`runner down --include-unrecorded` stays the sanctioned way through, and
stays a separate decision.
**If the bring-up runs out of time, it tries once more the other way
round.** The first attempt asks the runner to launch the app; when that
never finishes, the app is often launchable by other means, so smix
foregrounds it with `simctl launch` and attaches to it instead. There is
no flag for this — it is what the command does when it runs out of time.
Two consequences worth knowing:
- **The worst case is twice `--runner-port`'s timeout**, not once: the
second attempt starts a fresh clock. Giving it the remains of the first
would be designing the fallback to fail. The startup banner says so.
- **It happens at most once.** A second timeout stops and says both ways
were tried.
It does not happen on a physical device (no `simctl launch` to
foreground with — launch the app yourself and pass `--no-launch`), when
no `--bundle` was given (there is nothing to foreground), or when the
attempt already was the attaching one (`--no-launch` asks the same
question twice). Each of those says which it is rather than quietly
skipping the retry.
**iOS** — drives `xcodebuild` against the XCUITest runner:
```bash
smix runner up <ALIAS|UDID> --bundle com.example.app
smix runner up <ALIAS|UDID> --bundle com.example.app --supervise
smix runner list
smix runner down
```
`runner list` is the one to reach for before touching anything: it reads
this machine's ledgers *and* the listening sockets, and where only one of
them has something it says which. A runner nobody wrote down is exactly
the one you cannot decide about from the ledger alone.
**A physical iPhone or iPad** takes the same command, once the device is
registered with `--kind physical-ios`:
```bash
smix runner up <ALIAS> --bundle com.example.app
smix runner up <ALIAS> --bundle com.example.app --team <TEAM_ID>
```
Two things happen that do not on a simulator, both without asking you to
configure anything:
- **The build is signed.** The team is discovered from this machine's
signing identities; `--team` is only needed when several could sign,
which is a question smix refuses to answer for you.
- **A port forward opens first.** The runner listens on the *device's*
loopback, so smix runs a forwarder that makes `127.0.0.1:<port>` reach
it. It is a separate process (`smix runner forward`, visible in `ps`)
because it has to outlive the command that started it, and it is
recorded in the device ledger so a later teardown can find it.
`runner down` closes both, in that order — the runner's last requests
still travel through the pipe.
`--bundle` is required: it binds the runner's `XCUIApplication`, and
without it every `/tree` reads the wrong app. `--supervise` attaches a
sidecar that re-cycles the runner if it dies; `runner down` cascades to
it. The sidecar is `smix runner supervise --runner-port <port>`: it finds
the runner it watches by port, so two supervised runners in one checkout
each watch their own. Port comes from `--runner-port`, else the alias's `runnerPort` in
the registry, else 22087.
**Android** — extracts the runner project if it is not already
installed, builds and installs the instrumentation APK, forwards the
port, and `am instrument`s the Kotlin runner:
```bash
smix runner up emulator-5554 --platform android
smix runner down --platform android --device emulator-5554
```
Nothing to fetch or build first: the runner project ships inside smix,
the same way the Swift one does. The first `up` of a smix build on a
machine extracts it to
`~/.local/share/smix/runner-sources/android/<version>-<digest>/` and runs a
gradle build there, which takes a build's worth of time; later runs find
the APK already built. Each set of runner sources has a directory of its
own, so two smix builds on one machine (an installed release beside one a
checkout builds) never replace each other's tree; the two most recently
used other trees are kept, and any used in the last hour. An Android SDK is required, the way the iOS side requires Xcode.
The device is an adb serial or a registered alias. An emulator's alias
resolves to the AVD it was registered as, wherever that AVD is running now —
a serial is a port, and an AVD can come up on a different one — and a
checkout's book that gives the alias to another device stops the command
(see *Which book answered*). Default port is 28080; `--runner-port` sets the host side, which
adb forwards to the runner's own port inside the device. `runner up` is
idempotent: if `/health` already answers on that port it says so and
returns rather than stacking a second instrumentation onto it.
**The app under test, and the screen in front of it.** `--bundle` means
the same here as on iOS: `runner up` finishes with that app in front —
restarted on a fresh bring-up, only brought forward with `--no-launch`
or when the runner was already up. That is what an `adb install` needs
afterwards: installing an app stops it, the launcher comes to the front,
and without `--bundle` the next flow's first step found the launcher.
```bash
smix runner up emulator-5554 --platform android --bundle com.example.app
[runner] brought com.example.app forward (com.android.launcher3 was in front)
```
Before it answers, `runner up` also looks at what is covering the
screen. With the notification shade pulled down, the only windows are
system UI, and system UI holds the focus; `runner up` puts the shade
away, says so in one line, and asks again. Each step is read back from
the device — the activity manager for what is resumed, the runner's own
window list for what it can read — and nothing waits a fixed time:
a step ends when the device says it is done, or after 20 seconds.
- **Nothing nobody named is ever brought forward.** Without `--bundle`,
whatever app is in front stays there. Whose device this is was
already settled by the lease before any of this runs; which app
belongs in front is the caller's to say.
- **A package that is not installed is refused by name**, with the
command that installs it.
- **Only system UI after the shade is put away** is refused, and the
refusal names both things it can be: a lock screen (smix does not
unlock a device) or an instrumentation that crashed and was restarted
(`--force` cycles it). Before this, a pulled-down shade was reported
as the runner's accessibility connection having fallen behind, with
`--force` as the fix — which cycled a working runner and left the
shade where it was.
`runner down --platform android` requires `--device`, because an adb
command without a serial acts on whichever device happens to be
attached — and a developer's own phone is often plugged in next to the
emulator. Every adb call smix makes names its device explicitly for the
same reason; note that `gradlew install*` does **not**, so prefer these
commands over gradle install tasks.
It stops the instrumentation and closes the forward — the one adb
actually has, read back from `adb forward --list` rather than assumed
from the port passed in. The package stays installed so the next `up`
does not re-push 50 MB. When you do want it gone — and on a device that
is not yours, you do:
```bash
smix runner uninstall --platform android --device <SERIAL>
```
It asks whether the package is there before removing it. Android answers a
missing package with `DELETE_FAILED_INTERNAL_ERROR`, which is also what a
device policy refusing the removal returns — reading idempotence out of that
string would swallow the refusal. Absent is reported as absent; a refusal
stays an error.
Bringing the Android runner up by hand, if you need to:
```bash
adb -s emulator-5554 install -r -t \
~/.local/share/smix/runner-sources/android/<version>-<digest>/app/build/outputs/apk/androidTest/debug/app-debug-androidTest.apk
adb -s emulator-5554 forward tcp:28080 tcp:28080
adb -s emulator-5554 shell am instrument -w \
-e class dev.smix.runner.RunnerTest#runServerForever \
dev.smix.runner.test/androidx.test.runner.AndroidJUnitRunner
```
A typo in any of those three coordinates produces `OK (0 tests)` — a
silent no-op that reads as success.
### First run (`smix init`)
`smix init` is the bootstrap: it registers a **dedicated device for this
project** under an alias, and records that alias as the project's default — so
later commands here need no `--device` at all. The alias, omitted, is derived
from the project directory's name (two projects no longer both silently become
`dev`). It creates the `.smix` workspace, and given an `.app` it boots the
device, installs the app, and reads the bundle id out of the bundle, so the
command it prints next is runnable as it stands.
```bash
smix init --device <UDID> # alias derived from the project dir; recorded as this project's default
smix init --device <UDID> --app ./MyApp.app # …and install the app on it
smix init --device <UDID> --alias staging # a name of your own instead of the derived one
```
Afterwards, `smix run <flow>` in this project needs no `--device` — it resolves
the project's default device (the pointer lives in the machine store, keyed by
the project path; only the pointer is per-project — the device's facts stay in
the machine registry). An explicit `--device` still wins.
It does not choose between devices: with several simulators available and no
`--device`, it lists them and exits non-zero. It also never repoints an alias
that already exists — an alias is what every later command resolves through, so
silently moving one would take over every flow written against it.
### Doctor (housekeeping)
`smix doctor` answers whether this machine can drive anything yet and, when it
cannot, prints the command to run next. Checks stop at the first blocked one:
being told to run `smix init` with no Xcode command-line tools installed would
send you to a command that cannot succeed. `--json` gives the same verdict as
`{ready, checks[], next}` for scripts.
```bash
smix doctor # verdict + the next command to run
smix doctor --json # same, machine-readable
smix down # close what is no longer held, then sweep for residue
```
`down` works from the device ledgers first: it closes each recorded resource by
name and prints what it closed — the supervisor before the runner it watches,
because a supervisor exists to restart a runner it finds dead. Pattern matching
on process names runs afterwards, as a backstop for things no ledger covers, and
anything it finds is reported rather than silently swept.
A ledger whose holder is still running, and is not this process, is named and
left alone. The ledgers describe the machine, so the directory holds other
people's sessions as well as yours; "which directory it is in" was never the
question, and "is its holder still there" is one that can be answered.
`down` differs from `smix lease reconcile` in a way worth knowing: `reconcile`
settles what a *dead* holder left behind and refuses to touch a live session,
while `down` is you saying "close what I started" — a running runner is exactly
what it will close.
If XCUITest / xcodebuild processes remain after teardown, start with `smix
runner list`: it prints every runner on this machine with its port and its
device, and says whether the ledgers know about it. `pkill xctrunner` reaches
every runner on the machine including other people's sessions, which is the
failure this ledger exists to prevent — end the one you meant by its pid.
### Screen recording (`smix record …`)
```bash
smix record start <DEVICE> --output run.mov # begin recording
smix record status <DEVICE> # is it recording, and where is it writing
smix record stop <DEVICE> # stop, letting the file finish properly
```
A recording is written into the device ledger, which is what makes it
survive the command that started it. `simctl io recordVideo` writes an
mp4's trailer when it receives SIGINT and at no other time, so a recording
whose only handle is one process's memory becomes an unplayable file the
moment that process is killed. The ledger row means `smix record stop`
works from a different shell than `smix record start`, and that a
recording left behind by a killed session is closed properly by
`smix lease reconcile` rather than left writing into nothing.
`status` reports the path, not just yes-or-no — the question people
actually have is where the footage is going. A row whose process is gone
is reported as left behind, with the command that closes it, rather than
as recording.
### Device ledger (`smix lease …`)
`smix runner up` records what it opened on a device in
`~/.local/share/smix/leases/<UDID>.json`. On the machine, not in the checkout: a
lease says who holds a device, what they opened on it and on which port, and
every one of those is a fact about this machine. Stored per checkout, a runner
could hold a port while the tree asking about it saw an empty ledger directory.
When the process holding a session dies without tearing it down — Ctrl-C on
`xcodebuild`, an IDE restart, a CI timeout — that ledger is what lets the next
command find the runner it left behind and stop it the way the dying process
never got to: SIGINT first, so `testmanagerd` ends the XCUITest session cleanly
instead of the runner dying by SIGABRT and macOS raising a crash-report dialog.
```bash
smix lease list # every device with a ledger, and whether it is in use
smix lease status <DEVICE> # the holder, what is open, and what is owed
smix lease status <DEVICE> --json # {device, path, lease, heldBy}: the ledger as stored, the file it lives in, and who holds the device now (null when free)
smix lease owner <DEVICE> # who answers for it — exit 0 yes, 3 no record, 1 cannot ask
smix lease claim <DEVICE> # answer for one this machine did not boot
smix lease release <DEVICE> # give that up again
smix lease reconcile <DEVICE> # close what an abandoned session left open
smix lease prune # drop ledgers that no longer describe anything
smix lease prune --dry-run # ...or say what it would drop, and drop nothing
smix lease prune --device <DEVICE> # ...looking at that one ledger and no other
smix lease history # devices that left while a ledger still described them (--json)
smix lease migrate --from <DIR> # fold a checkout's old ledgers into this machine's
smix runner list # every runner here: port, device, and who knows about it
```
`reconcile` never preempts a live session — it reports the holder and stops.
A session counts as live while anything it started is still running, even
after the command that started it has exited, which is the normal state
once `smix runner up` returns.
Two things it will not do, both deliberate:
- **It never signals a pid it cannot re-verify.** Each row records the process
start time alongside the pid, because a pid alone gets reissued to unrelated
processes. A row whose pid now belongs to something else is reported and left
untouched.
- **It only shuts down a device it booted.** Finding a device already up and
turning it off would take away someone else's session as the price of
cleaning up after yours.
### A device that left
A ledger describes a device as present for as long as it exists. When the
device goes without smix being told — an emulator that exits or crashes, a
simulator shut down from outside, a phone unplugged — the ledger goes on
describing it. The next `run`, `runner`, `sim` or `lease` command notices,
keeps the departure as a fact, and says so once on stderr:
```
note: emulator-5554 (AVD sim-smix-android-02) left without smix hearing about it.
Last heard from at 2026-09-24T00:31:03Z, held by pid 73835 (`smix sim boot
sim-smix-android-02`, no longer running); smix booted it — recorded;
`smix lease history` has it
```
`smix lease history` reads them back: which device and AVD, when it was
noticed, when its ledger last heard from it, who held it and whether that
process was still running, whether smix booted it, what AVD answers on its
port now — and, for an emulator smix started, the last twenty lines it
printed. smix keeps an emulator's console under
`~/.local/share/smix/emulator-console/`, one file per start, because a
restart would otherwise overwrite the run that died.
Two limits, said plainly. An emulator smix did not start has no console
here to read. And an emulator's exit status cannot be recovered afterwards:
it is started detached, so by the time it exits nothing that smix runs is
its parent. What it printed is what there is.
A departure is noticed when a smix command next runs, so `noticed` is not
when it happened; `last heard from` is the other end of that window.
### Claiming a device nothing here booted
Gates and scripts ask `lease owner` before they drive anything, and until
7.0 the only answer that let them through was "smix booted it". A machine's
own dedicated emulator, started by hand and sitting idle, was therefore
drivable by nobody — and the way past it was an environment variable that
means "this one is mine" to a single command and is gone when that command
exits. The next run had to decide again, and nobody could read what was
decided last time.
```bash
smix lease claim emulator-5554 # yours to drive; still not yours to switch off
smix lease owner emulator-5554 # exit 0, and says "claimed", not "booted"
smix lease release emulator-5554 # or let the device going off end it
```
A claim grants exactly one of the two things a boot row grants. `lease
reconcile` and every teardown path still refuse to switch off a device this
machine did not boot, because the claim says in as many words that it did
not. It is refused while somebody else's live session holds the device:
claiming is a statement about a device nobody is using, not a way past one
that is.
### Perf regression (`smix bench`)
```bash
smix bench # measure the in-process corpus, compare to the committed baseline, fail on >5% drift
smix bench --update-baseline # overwrite the baseline with this run (do this on the machine that will gate)
```
| Flag | Default | Meaning |
|---|---|---|
| `--update-baseline` | (off) | Write this run's measurement as the new baseline instead of comparing |
| `--baseline-file <PATH>` | `crates/smix-cli/bench/baseline.json` | Baseline JSON to compare against |
| `--current-file <PATH>` | (unset) | Read the "current" measurement from a file instead of measuring — for tests / CI reproduction |
The absolute `perf_gate` budgets catch a spike; this catches slow drift under them. The baseline holds absolute times from the machine that captured it, so run `--update-baseline` on the machine that will gate. Do not compare against a baseline captured on another machine: CPUs differ by more than the 5% tolerance, so every metric would read as a regression.
### Low-level probes (use a running runner)
These act on the runner's currently-active sim. Used for ad-hoc debugging without writing YAML.
The selector is a positional argument in `<kind>:<value>` shorthand —
`id:` / `text:` / `label:` / `role:` / `ocrText:` / `point:`.
`role:` takes the word a person writes, the same vocabulary yaml and MCP
read: `button`, `Button`, `text_field`, `heading`, `tab`.
`--ocr-locale zh-Hans` (repeatable, best first) says which languages to read an
`ocrText:` selector in. The value cannot ride in the token because `text:` may
legally contain any character. Left out, the recogniser decides for itself.
`ocrText:` reads the screen with Apple Vision instead of the accessibility
tree — for a canvas, an image of text, anything the tree cannot see. It
costs a vision pass, so reach for it after the others. `tap`, `find` and
`wait-for` take one; `fill` and `scroll` refuse it, because an OCR hit is
where text was one look ago, not a field or an anchor.
`point:` is the last resort — `point:50%,80%`, or the same point written
`point:0.5,0.8`. A fraction of the viewport, never pixels, because a flow
written in pixels runs on one screen size. Only `smix tap` takes one:
there is nothing at a coordinate to find, fill, scroll to or wait for,
and those four say so rather than resolving something that cannot
resolve.
```bash
smix tap id:home-increment-btn
smix tap "text:Submit"
smix tap id:play-btn --then-screenshot /tmp/bar.png # tap and frame, one command
smix find id:home-counter-label # prints exists=<bool>, exits 0 either way
smix wait-for id:loading-spinner --timeout 5 # seconds, not ms
smix wait-for id:loading-spinner --absent # wait until it is GONE
smix fill id:form-email-input --text alice@example.com
smix press-key return # positional key name, read as a flow's pressKey reads it
smix press-key back # navigation back — fails when nothing went back
smix scroll "text:Row #5000" --direction down # stops when it is wholly in view
smix scroll "ocrText:Row #5000" --direction down # looks with OCR too
smix swipe down # one gesture; `down` reveals what is below
smix hide-keyboard
smix tree --json | jq . # full a11y tree
smix tree # human-readable outline
smix tree --keyboard # …including the keyboard's keys
smix describe # visible interactive elements
smix system-popups # list active popups
smix system-popup-action <popup-id> <button-id>
```
### What `bounds` is measured in, and why the two platforms differ
Every node in `smix tree --json` carries `bounds: {x, y, w, h}`. **The
unit is not the same on both platforms**, and nothing about the number
says which one you have:
| | unit | on a device |
|---|---|---|
| iOS | **points** | iPhone 17 Pro sim: root `402×874`, screenshot `1206×2622` px (3×) |
| Android | **pixels** | Pixel-class AVD: root `1080×2340`, which is the physical size |
So a size rule written against one does not port to the other by
changing the constant:
* **iOS** — Apple's 44pt minimum compares **directly** against `w`/`h`.
* **Android** — Material's 48dp does not. Divide by the density first:
`dp = px / (densityDpi / 160)`. On a 440dpi device that is 2.75×, so
48dp is **132px**, and comparing `w >= 48` would pass a target a third
of the required size.
The failure is silent in both directions — a checker ported without the
conversion either passes everything or fails everything, and both look
like a working checker. Read the density from the device
(`adb shell wm density`) rather than assuming a number.
**`--then-screenshot` is for UI that does not wait.** A control bar that
hides itself after a few seconds outlives neither a second command nor
the turn between two tool calls, and the usual answer is to change the
app so the thing stays up long enough to photograph. This takes the
frame from the same process that tapped, in the same call, on both
platforms: measured at about 88 ms after the tap returns, against
roughly 325 ms going out to device tooling. (Before 10.2 the Android
runner served no such route and this answered `501 not_implemented`; a
runner older than that now says so and asks to be brought up again,
rather than handing back a frame taken by something else.) The line it prints says which route took the frame and
how many milliseconds late it is, so the number is reported rather than
promised.
A tap that fails writes nothing. A frame taken after a tap that did not
land is a picture of the screen nothing happened on, and it looks
exactly like evidence.
It needs a selector the tree can resolve — `point:` and `ocrText:` are
both dispatched without resolving a target, so there would be nothing to
say about where the touch landed. Use `smix tap` and `smix sim
screenshot` separately if that is what you want.
**`find` prints, `wait-for` asserts.** `smix find` writes
`exists=<bool>` and exits 0 whichever it is, so a shell script has to
read its output to branch. `smix wait-for` polls until the element is
there and exits non-zero when the timeout passes, which is what lets it
stand alone in a `&&` chain. `--absent` is the same command waiting for
the opposite: it returns as soon as the element is gone, and fails if it
is still there when time runs out. Use it for a spinner or a modal you
need off the screen before the next step.
**`swipe` is one gesture, `scroll` stops at something.** `smix scroll`
takes a selector and keeps going until that element is **wholly** on
screen and has stopped moving — the same rule and the same loop
`scrollUntilVisible` runs, with its defaults (100% visible, 20 s). An
`ocrText:` selector works: every look reads the tree first, then OCR.
`smix swipe` does a single swipe and returns. In both, the direction
names what you want to see — `down` reveals what is below — not which
way a finger travels.
**The software keyboard's keys are collapsed.** A key per letter plus
`Next keyboard`, `Dictate`, shift and delete is around sixty nodes that
are the same sixty on every screen of every app, and this output is
read by an AI paying for each one. The keyboard node itself always
shows — a keyboard covering the element you wanted is the explanation
for a failure — and the outline says how many keys were left out.
`smix tree --keyboard` includes them; `smix describe` never enumerates
them, because it is the summary view. To press a key, `smix press-key`
names them directly.
Every one of these takes `--port` and `--device`, and resolves the port
by the precedence at the bottom of this page. `--device` is narrower
here than on `smix run`: it names a UDID or a registry alias purely so
the port that device is registered on can be looked up. It does not
change which simulator the call reaches — the port already does that,
because a runner is a process listening on one.
```bash
smix sim register jp --udid <UDID> --runner-port 22088
smix tap id:home-tab --device jp # dials 22088, not 22087
```
### Run-script driver (sequential YAML of `smix` subcommands)
```yaml
# script.yaml
- name: boot
cmd: sim boot
args: [<device>]
- name: runner
cmd: runner up
args: [{from: outputs.boot.udid}, --bundle, com.example.app]
- name: tap-home
cmd: tap
args: ["id:tab-home"]
```
```bash
smix run-script script.yaml
```
Use for matrix runs, repeated probes, anything that needs sequential CLI calls. Not for app UI flows — use `smix run` for that.
### Device registry
```bash
smix sim list # find the UDID
smix sim register dev --udid <UDID> # record it under an alias
smix sim register jp --udid <UDID> --locale ja-JP --runner-port 22088
```
`register` writes to this machine's registry, creating it when absent
(`$XDG_DATA_HOME/smix/devices`, or `~/.local/share/smix/devices`;
`SMIX_MACHINE_DIR` moves the whole of smix's machine data, and
`SMIX_SIMS_JSON` names one registry and only that one). Device name,
runtime, and device type are read from `simctl`, so only the UDID and
alias are yours to choose. After this, every command accepts the alias
where it accepts a UDID — from any directory on the machine, not just
the tree you registered it from.
`smix sim unregister <alias>` is the other half: it forgets a name, not
a device, so another alias for the same device keeps working.
What is recorded, per device:
```
ALIAS UDID KIND SCOPE
dev 47ACEAE5-36BA-4C62-811B-F09B397910D7 simulator machine
```
`smix sim list --registered` prints it. Each row carries the device name,
runtime, device type, and — for a physical device — whether destructive
actions have been allowed on it. `--locale` and `--runner-port` are optional
and set at registration.
These live in a store on this machine rather than in a file you edit.
`smix sim register` and `smix sim unregister` are the way in and out.
### Which book answered
A checkout may still carry a `.smix/sims.json` from before 2.1. smix reads it
and never writes it — not even to import it — and it counts for less than this
machine's registry:
- An alias only that file holds is answered from it, with a note that
`smix sim migrate` records it on this machine.
- An alias this machine's registry holds is answered from the machine.
- An alias the two give to **different devices** is refused, naming both files
and both identifiers. Nothing is driven: a checkout's book can stop a
decision, and it does not get to make one.
`smix sim resolve` says which of these answered, on stderr — the identifier
alone goes to stdout, so `$(smix sim resolve phone)` keeps working:
```text
$ smix sim resolve phone
resolved `phone` from this machine's registry (~/.local/share/smix/devices)
47ACEAE5-36BA-4C62-811B-F09B397910D7
```
**A harness that drives smix from outside should ask smix, not read a file.**
`smix sim resolve <alias> --json` gives one object with `ref`, `id`, `alias`,
`source` (`kind` is `machine`, `checkout`, `named` or `literal`, with its
`path`), `deviceKind`, and for an emulator its `avd`. Reading the legacy
`.smix/sims.json` directly answers from a book that `smix sim register` does
not add to: a device registered a minute ago is missing from it, which reads
exactly like a broken harness.
## The probe — reading the toolkit's own tree
smix perceives through the accessibility tree. On iOS that is what SwiftUI
publishes, and it is faithful. On Android with Jetpack Compose it is not:
the whole UI is one `AndroidComposeView` and the nodes are synthesised from
semantics, so a `testTag` arrives only when the app opted in with
`testTagsAsResourceId` — and that opt-in is a property of the subtree it is
written on. A dialog composes into its own, so its controls arrive as
unnamed views and `smix find id:…` answers `exists=false` about a button
plainly on screen.
An app can hand smix the semantics tree instead. One line, debug only:
```kotlin
dependencies {
debugImplementation("jp.golia.smix:smix-probe:11.0.0")
}
```
No app code — a content provider arms it before `Application.onCreate`, which
is what puts it ahead of the first `setContent`. It is never in a release
build.
**Without it everything works exactly as before.** What changes is that smix
says which tree answered instead of quietly giving the lesser one:
```bash
smix tree --device <d> --port <p> --json | jq -r .source # a11y | semantics
```
The human outline prints the same thing as a header when the answer came from
the accessibility reader, along with what that reader cannot see.
### What the probe does and does not do
It widens what smix can **see**, never what it can **reach**.
- It reads the app, and only the app. A `semantics` tree is the whole
screen all the same: the app's windows come from the probe, and every
window that is not the app's — the keyboard, the system bars, another
app's dialog on top — from the accessibility reader, each saying whose
it is. `role: keyboard` answers on an app with the probe as it does
without one.
- With a modal open, controls behind it are not addressable through the
probe. It can see them; a user cannot touch them.
- Its action surface is only what brings a node within reach of a real
touch. A Compose semantics `OnClick` fires the composable's lambda with no
hit-testing — measured firing through a dialog's scrim onto a button no
touch could reach — so smix refuses it and names what to use instead.
### What it reports about where things are
Two things changed in 10.2, and an app that upgrades the probe will see
both.
**A View that Compose hosts is in the tree.** `AndroidView` puts a real
View inside a composition, and the probe used to walk semantics only — so
adding the probe to an app took its hosted controls *away*, while the
accessibility reader still saw them. They are now reported with their own
resource id, so `id: btn_player_fullscreen` addresses one, and with the
same role the accessibility reader gives it (`role: button`).
**A node's rectangle is the part of it on screen**, not the part the
layout drew. A row half-scrolled out of a list reports the half that
shows; one scrolled out entirely reports an empty rectangle and does not
count as visible. A node the toolkit composed but never placed — the
state a lazy list leaves a prefetched row in — is **not reported at
all**, because the only position it has is one it never had.
If a flow asserted on something in one of those last two states, it was
passing on a node that was not on screen; it will now fail. That is the
intended change.
### Whether a touch would land
`smix find` answers two questions now, because they are two facts:
```
exists=true
reachable=false it is on screen and something is on top of it; a tap here would be refused
```
`exists` is about the tree; `reachable` is about hit-testing. A `tap` at
something reported unreachable fails and names what is covering it, rather
than dispatching an event the presentation swallows and reporting success.
A runner that predates this says nothing about reachability, and silence is
not a refusal — taps proceed as they always did.
## Migration and authoring
### `smix migrate` — maestro yaml to smix
```bash
smix migrate flow.yaml # rewritten yaml to stdout
smix migrate flows/ # every .yaml under a directory
```
Renames verbs to their smix spellings (`tapOn` → `tap`,
`extendedWaitUntil` → `expect` + `timeoutMs`, `retry.max` →
`retry.maxRetries`) and drops deprecated argument forms. A verb it does
not recognise is passed through unchanged with a `WARN:` line on
stderr — migrate never silently drops a step it cannot translate.
It also warns about selector keys v2 refuses (`enabled:` is one), since
those fail at parse time and it is better to hear it here than on the
next run.
### `smix authoring` — compose against a live sim
Suggest selectors matching a partial spec, and capture or diff
accessibility-tree baselines for visual gates. Requires a runner.
**Record → generate.** A recording of a live session becomes a flow. The
capture leg is platform-specific (iOS `RecordingApp`, Android accessibility
events, web Playwright injection) but every leg emits the same
`smix-authoring-ir::IRAction` stream, so the generator produces a
byte-identical maestro or rust flow from any of them.
```bash
# live Android session -> maestro flow (records --duration seconds while you drive):
smix authoring tap-record --device emulator-5554 --format maestro -o flow.yaml --duration 15
# an IRAction JSON file (from any capture leg) -> flow, device-free:
smix authoring generate events.json --format maestro -o flow.yaml
smix authoring generate events.json --format rust -o flow.rs --test-fn-name my_test
```
`--format maestro` writes a maestro-compatible yaml flow; `--format rust`
writes an XCUITest-style rust test. `tap-record` is Android-only today (the
Android runner emits IRAction directly); web capture goes through the
`@goliapkg/smix-web-record` Playwright bridge, which writes an IRAction JSON
file that `generate` then consumes. Web recordings generate native flows —
there is no in-browser replay.
`smix authoring propose` closes the loop on a failed run: it reads the
flow plus the on-disk bundle a failed run left behind, asks the local
`claude` CLI to propose edits, applies them, and writes the amended
flow. It is device-free — producing the bundle is the caller's step:
```bash
# 1) produce the bundle by running the (failing) flow on a device:
smix run --device <SERIAL> --platform android --debug-output ./bundle --format json corrupt.yaml > ./bundle/failure.json
# 2) device-free: propose + amend from the on-disk bundle via local claude:
smix authoring propose corrupt.yaml --bundle ./bundle -o amended.yaml
```
The proposal step is non-deterministic (a model is in the loop) and
fenced like `smix-ai-tier`: deletable, opt-in, never on the sense/act
path.
### `smix annotate` — draw on a screenshot
Circle, arrow, text, box and line primitives over a PNG, for failure
reports a human or an agent has to read.
### `smix capsule` — one-command bring-up and teardown
Headless boot, capture and runner start together (`up`), and the
reverse (`down`). The guard rejects a windowed session by default;
`--soft` accepts the soft-capsule fallback.
iOS Simulators only, and it says so rather than trying: all three of its
legs — the simulator-window guard (Simulator.app on Xcode <= 26; Device Hub on
Xcode 27 does not react to a boot and never trips it), `simctl boot`, the
`/live` capture — are simulator machinery with no counterpart on an emulator or a physical
device. Those are brought up with `smix runner up` instead
(`--platform android` for an emulator, `--physical` for a phone).
### `smix diagnostic store` — read the persisted state
```bash
smix diagnostic store # this workspace's .smix (flows and traces; devices and leases are the machine's)
smix diagnostic store --root PATH # another store
```
Prints everything smix has persisted, as JSON: the device registry,
runner handles, capsule records and the diagnostic buffers. State used
to be JSON files you could `cat`; this is what replaces that. A value
that is not valid JSON is shown as hex rather than stopping the dump,
because this is what you run when something is already wrong.
### `smix session state` — which runner this session would dial
```bash
smix session state # the resolved runner port + device, and the runner's open sessions if reachable
smix session state --port 22088 # ask about a specific port
smix session state --device sim-smix-02
```
Read-only. The CLI counterpart of the `smix_session_state` MCP tool —
the one query answerable before anything is bound. An unreachable runner
is reported as a state, not an error.
## Common command recipes
### Smoke test the running sim quickly
```bash
smix find "text:Welcome" || echo "not on welcome screen"
smix tree | head -50 # see what's on screen right now
smix system-popups # check for blocking system alerts
```
### Reset to known state between tests
```bash
smix sim exec <device> terminate com.example.app
smix sim exec <device> launch com.example.app
sleep 2
smix find id:home-container # confirm fresh launch
```
### Capture per-step screenshot (debug aid)
```bash
smix sim exec <device> io screenshot /tmp/s.png && open /tmp/s.png
```
### iOS + Android in one terminal
```bash
# iOS port 22087 (default)
smix run --device <iosudid> --runner-port 22087 --no-launch flow.yaml
# Android port 28080
smix run --device emulator-5554 --platform android \
--apps-config apps.yaml \
--runner-port 28080 --no-launch flow.yaml
```
## Environment-variable precedence
```
1. Explicit CLI flag (highest)
2. ENV var (e.g. SMIX_UDID)
3. this machine's device registry, then a checkout's, for aliases
4. Hardcoded default in code (lowest)
```
<!-- ===== docs/ai-guide/06-fixtures.md ===== -->
# 06 — Test fixture layout (example)
> A reference layout for the test app you drive with smix — how to organize screens, testTags, and shared IDs so YAML flows stay maintainable across iOS and Android. Mirror this pattern in your own app; the exact IDs are yours to choose.
## What is a "fixture"?
A dedicated app (or a build variant of your production app) that exposes deterministic UI for driving via smix:
- **iOS**: SwiftUI (or UIKit) app, own bundle id (e.g. `com.example.app`).
- **Android**: Jetpack Compose (or Views) app, own package (e.g. `com.example.app`).
The key property: **testTag (Compose) ≡ accessibilityIdentifier (SwiftUI)**. The same string identifies the same role on both platforms, so a single YAML can drive both.
## testTag naming convention
Adopt a hierarchical, kebab-case pattern:
```
<screen>-<element>-<kind>
home-counter-label
home-increment-btn
home-reset-btn
form-email-input
form-submit-btn
modal-sheet-dismiss-btn
list-row-<N>
list-row-<N>-title
```
- `<screen>` — the screen area (home, form, list, modal, …)
- `<element>` — the semantic element within the screen
- `<kind>` — role suffix (`-btn`, `-label`, `-input`, `-screen`, `-row`, `-check`, …)
Consistency is worth more than terseness. A YAML that says `tapOn: { id: "form-submit-btn" }` reads unambiguously; `tapOn: { id: "fs" }` does not.
## Example screen layout
```
RootScreen (tab bar)
├── Home ← counter + increment + reset + alert
├── Form ← 3 text fields + submit
├── List ← N rows alternating bg
├── Modal ← multiple modal types (sheet / alert / action / full)
├── Perm ← permission requests
├── Loc ← location pull
├── Orient ← orientation read
├── Kbd ← keyboard focus / submit / hide
├── Clip ← clipboard copy/paste
├── Deeplink ← deep-link URL handler
├── Push ← local notification trigger
├── Localized ← untagged buttons with locale-dependent labels
├── OCR ← untagged labels at varied sizes
├── Anchor ← icon w/ clickable + last-tapped state
├── WebView ← WebView w/ inline HTML form
├── DeepNav ← N-level nested nav stack
├── Stacked ← modal stack (sheet ⊃ alert ⊃ sheet)
├── Heavy ← virtualized list (10k+ rows)
├── Map ← MapKit / osmdroid
├── Cam ← CameraX / AVCaptureSession
└── Wiz ← multi-step wizard
```
## Tab navigation testTags
Every tab should be reachable via:
```yaml
- tapOn: { id: "tab-<area>" }
# Where <area> is the lowercase area name (home, form, list, modal, ...)
```
## Per-screen testTag tables (representative)
### HomeScreen
| testTag | Description |
|---|---|
| `screen-home` | Container |
| `home-counter-label` | Counter value display |
| `home-increment-btn` | +1 button |
| `home-reset-btn` | Reset button |
| `home-show-alert-btn` | Open alert |
| `home-alert-ok-btn` | Alert OK button (in dialog) |
| `home-alert-cancel-btn` | Alert cancel |
### FormScreen
| testTag | Description |
|---|---|
| `screen-form` | Container |
| `form-name-input` | Name field |
| `form-email-input` | Email field |
| `form-password-input` | Password field |
| `form-submit-btn` | Submit |
| `form-submitted-label` | Result display after submit |
### ListScreen
| testTag | Description |
|---|---|
| `screen-list` | Container |
| `list-row-<N>` | Each row (N = 0..N-1) |
### ModalScreen
| testTag | Description |
|---|---|
| `screen-modal` | Container |
| `modal-open-sheet-btn` | Open BottomSheet / .sheet |
| `modal-open-alert-btn` | Open AlertDialog / .alert |
| `modal-open-actionsheet-btn` | Open action sheet |
| `modal-open-fullscreen-btn` | Open fullScreenCover |
| `modal-sheet-dismiss-btn` | Inside sheet — dismiss |
| `modal-alert-ok-btn` | Inside alert — OK |
| `modal-action-a-btn` | Inside action sheet — option A |
| `modal-action-cancel-btn` | Inside action sheet — cancel |
| `modal-fullscreen-dismiss-btn` | Inside full screen — dismiss |
### DeepNavScreen (multi-level nav)
| testTag | Description |
|---|---|
| `screen-deepnav` | Container |
| `deepnav-l1-screen` | Level 1 (Categories) |
| `deepnav-l1-<category>-btn` | Per-category button |
| `deepnav-l2-screen` | L2 (Subcategories) |
| `deepnav-l2-<sub>-btn` | Per-subcategory button |
| `deepnav-l3-screen` | L3 (Items) |
| `deepnav-l3-item<N>-btn` | Item N |
| `deepnav-l4-screen` | L4 (Detail) |
| `deepnav-l4-edit-btn` | Edit |
| `deepnav-l5-screen` | L5 (Edit form) |
| `deepnav-l5-text-input` | Text field |
| `deepnav-l5-save-btn` | Save → pops back with state |
### Heavy list (virtualized rows)
| testTag | Description |
|---|---|
| `screen-heavylist` | Container |
| `heavylist-list` | LazyColumn / List |
| `heavylist-first-visible-label` | "first visible: N" |
| `heavylist-last-tapped-label` | "last tapped: row#N" |
| `heavylist-jump-input` | Number input |
| `heavylist-jump-btn` | Scroll-to-row |
| `heavylist-row-<N>` | Row container (lazy — only visible rows enumerate) |
| `heavylist-row-<N>-title` | Title |
| `heavylist-row-<N>-btn` | Tap button |
| `heavylist-row-<N>-check` | Checkbox |
### WebView
| testTag | Description |
|---|---|
| `screen-webview` | Container |
| `webview-title-label` | Title text |
| `webview-container` | WebView host |
| (HTML inside: `<button data-testid="webview-submit-btn">` and `<input id="user-input">`) |
For flows using `webViewEval`, see [04-actions.md](04-actions.md) §WebView JS bridge.
### Wizard (multi-step onboarding)
| testTag | Description |
|---|---|
| `screen-wizard` | Container |
| `wizard-step-label` | "step 2 / 3" |
| `wizard-progress-bar` | LinearProgressIndicator |
| `wizard-step1` / `step2` / `step3` | Per-step subcontainer |
| `wizard-radio-<option>` | Step 1 radios |
| `wizard-name-input` | Step 2 |
| `wizard-company-input` | Step 2 (conditional) |
| `wizard-email-input` | Step 3 |
| `wizard-summary-label` | Step 3 summary |
| `wizard-back-btn` | Back |
| `wizard-next-btn` | Next |
| `wizard-submit-btn` | Submit |
| `wizard-validation-label` | Error message |
| `wizard-submitted-label` | Success confirmation |
## How to use this layout in your flows
```yaml
# Drive the home counter
- tapOn: { id: "tab-home" }
- assertVisible: { id: "home-counter-label" }
- tapOn: { id: "home-increment-btn" }
- tapOn: { id: "home-increment-btn" }
- assertVisible:
id: "home-counter-label"
text: "2" # value-aware (text + id combined)
# Drill deep nav
- tapOn: { id: "tab-deepnav" }
- tapOn: { id: "deepnav-l1-movies-btn" }
- tapOn: { id: "deepnav-l2-action-btn" }
- tapOn: { id: "deepnav-l3-item3-btn" }
- tapOn: { id: "deepnav-l4-edit-btn" }
- tapOn: { id: "deepnav-l5-save-btn" }
- assertVisible: { id: "deepnav-l1-saved-label" } # popped back
# Exercise the wizard
- tapOn: { id: "tab-wizard" }
- tapOn: { id: "wizard-radio-business" }
- tapOn: { id: "wizard-next-btn" }
- assertVisible: { id: "wizard-company-input" } # conditional field
```
## The `- fixture:` verb — host-app contract
The `- fixture: <id>` yaml verb drives a QA overlay in your app-under-test ("fire a state-priming chip, await its completion signal"). Using it requires your app to implement a small contract:
1. **Toggle element** — an always-reachable element with a11y id **`qa-bubble-toggle`** that opens/closes your QA overlay. The runtime taps it before and after firing a chip (idempotent open-then-close, even on failure).
2. **Chip elements** — each fixture chip exposes the `testID` you register (see below).
3. **Completion signal** — each chip logs a distinctive line the metro/log tail can match (regex), so the verb blocks until the primed state is actually written, not just until the tap landed.
Registry (`fixturesRegistry` in `.smix/config.json`, JSON or lightweight TS):
```jsonc
{
"version": 1,
"fixtures": {
"prime-search-history": {
"testID": "chip-prime-search-history",
"signal": { "regex": "\\[qa\\] search-history primed count=5" },
"timeoutMs": 8000
}
}
}
```
Flow usage:
```yaml
- fixture: prime-search-history # tap toggle → tap chip → await signal → close toggle
```
The verb needs a metro/log tail configured (`.smix/config.json` `metroLog` or `--metro-log-url`) — without one it errors with an actionable hint rather than silently skipping the signal await.
> Chip-fire ≠ behavior verified. The fixture verb proves the primed state was WRITTEN; follow it with assertions on the screen where that state is CONSUMED (navigate there and assert the primed content is visible).
## See also
- [03-selectors.md](03-selectors.md) — selector forms including `id:`
- [08-cookbook.md](08-cookbook.md) — recipes that use these testTag conventions
<!-- ===== docs/ai-guide/07-errors.md ===== -->
# 07 — Error codes + remediation
> Every error smix can surface, what triggered it, and exactly what to do. Optimized for "I just saw error X, what's next?"
## Failure shape
smix errors come back as structured JSON:
```json
{
"ok": false,
"code": "ELEMENT_NOT_FOUND",
"message": "tap_by_id: element not found — id=\"home-incremnt-btn\"",
"hint": "runner XCUIQuery returned no match; check id spelling or wait for screen to settle",
"step": "tapOn",
"step_index": 3,
"windows": [
{ "package": "com.example.app", "kind": "application", "focused": true },
{ "package": "com.android.systemui", "kind": "system" },
{ "package": "com.android.systemui", "kind": "system" }
],
"visibleTotal": 57,
"visibleElements": [
{ "id": "home-counter-label", "text": "0" },
{ "id": "home-increment-btn", "text": "+1" },
{ "id": "home-reset-btn", "text": "Reset" }
],
"suggestions": [
"home-increment-btn (id, exact match to a sibling — likely typo in your selector)"
]
}
```
Read **`code`** first to triage. Read **`hint`** + **`suggestions`** to know the fix.
**`visibleElements` is a sample, not the screen.** It holds the first ten of
`visibleTotal`, the focused app's window first, then any other app's, then the
keyboard, then system chrome (status bar, navigation bar). To ask "is the app on
screen at all" or "did the device stop serving it", read **`windows`** — every
window the screen held, whose it is, and which holds the focus — and
**`unreadableWindows`**, the number of windows the reader could not read. A
screen whose app window is present and one whose app window could not be read
are different failures, and only one of them is the app's. The rendered form
says the same in one line:
```
on screen: com.example.app (application, focused) · com.android.systemui (system) ×2
visible elements (10 of 57, the focused app's first):
```
On iOS the tree is the app you named, so `windows` holds that one app; the
status bar belongs to SpringBoard and is not in it. All three fields are
omitted when there is nothing to say, so an older reader sees the shape it
always did.
## A verdict, or smix unable to look
Codes fall into two kinds, and `smix run --format json` says which in
`failure.judgesTheScreen`:
| judges the screen (`true`) | smix could not look (`false`) |
|---|---|
| `ELEMENT_NOT_FOUND` `NOT_VISIBLE` `NOT_ENABLED` `AMBIGUOUS` `TIMEOUT` `ASSERTION_FAILED` `TAP_MISSED` `COORDINATE_SPACE_MISMATCH` | `DRIVER_ERROR` `APP_NOT_RUNNING` `SIMULATOR_NOT_BOOTED` `CAPTURE_BACKPRESSURE` |
The first kind is an answer about the app; the second is about the
machinery between you and it. A flow treats them differently:
- **`optional: true`** tolerates only the first kind. A block that failed
because the runner went away still fails the step.
- **`when: { visible: … }` / `notVisible:`** — a look that failed on the
second kind fails the step under its own code. It used to read as
"not visible", so a runner answering garbage quietly skipped the block
and the run went green.
- **A wait** (`extendedWaitUntil`, a fallback chain with `ocrText`) keeps
waiting through `CAPTURE_BACKPRESSURE` — it means *not now* — and fails
at once on any other failure of the second kind, instead of spending
its whole budget "missing" and then reporting a `TIMEOUT`.
- **A live on-screen check** (the one that guards against stale tree
frames on iOS) that could not be asked is a `DRIVER_ERROR`, not a
confirmation. Only a runner too old to have the route leaves the tree's
answer standing.
If you judge a run by its exit status in a script, judge it by this
field (or the `FAIL [CODE]` line) too: a non-zero exit with `DRIVER_ERROR`
is not a verdict on whatever your script was checking.
## Error codes
### ELEMENT_NOT_FOUND
**Trigger**: a selector matched zero elements after implicit-wait timeout.
**Common causes**:
1. Typo in the testid/text
2. Element off-screen (LazyRow / LazyColumn / scrollable tab bar)
3. Element in a separate window (Compose modal / iOS sheet) without testTagsAsResourceId properly enabled
4. Screen has not finished rendering (need `waitForAnimationToEnd` before)
5. Wrong screen — you forgot to navigate first
**Fix**:
- Check `suggestions` block — often shows the typo or close match
- For off-screen: prepend `scrollUntilVisible` or `swipe` step
- For modal: see [03-selectors.md](03-selectors.md) Compose modal pitfalls
- For timing: add `waitForAnimationToEnd` or `extendedWaitUntil`
- For wrong screen: dump `smix tree | head -30` to see what's actually showing
### NOT_VISIBLE
**Trigger**: element exists in tree but is occluded / off-screen / has 0 width-height / `isHidden=true`.
**Fix**:
- Element may be covered by a system popup → `smix system-popups` to check
- May be below fold → `scrollUntilVisible`
- May be hidden by overlay → check `assertNotVisible: { id: "modal-overlay" }` first
### NOT_ENABLED
**Trigger**: element matched but `isEnabled=false` (button disabled, text field readonly, etc.).
**Fix**:
- The app has a guard. Satisfy the precondition first (e.g., fill the form before clicking Submit).
- Check app state — `smix describe` shows enabled state of visible interactives.
### AMBIGUOUS
**Trigger**: a selector requiring uniqueness (e.g., `tapOn: { text: "Edit" }`) matched multiple elements.
**Fix**:
- Use a more specific selector: add `id:`, `nth:`, or spatial modifier (`inside:`, `near:`).
- Example: `tapOn: { text: "Edit", inside: "modal" }`.
### TIMEOUT
**Trigger**: an `extendedWaitUntil` or implicit-wait exceeded its budget.
**Fix**:
- Increase the timeout if the operation is genuinely slow (cold app launch, network call).
- If something's stuck, dump `smix tree` to see what's actually happening.
- If a system popup is blocking, `smix system-popups` + `smix system-popup-action` to dismiss.
### ASSERTION_FAILED
**Trigger**: `assertTrue` JS expression evaluated to falsy.
**Fix**:
- Print the eval'd context: in your `evalScript`, log `output.*` values to see what was actually computed.
- Often: an earlier `runScript` set a different field than you expected.
### APP_NOT_RUNNING
**Trigger**: target app process exited or never launched.
**Fix**:
- Check app's crash log: `smix sim exec <udid> spawn ~/Library/Logs/CoreSimulator/<udid>/system.log | tail -50`.
- For iOS, also `xcrun simctl spawn <udid> log show --last 5m --predicate 'eventMessage contains "<bundle-id>"'`.
- Verify `launchApp` step ran (or that the app is installed: `smix sim list-apps <udid>`).
### SIMULATOR_NOT_BOOTED
**Trigger**: simulator UDID exists but is shutdown.
**Fix**: `smix sim boot <udid>` then retry.
### TAP_MISSED
**Trigger**: the touch was synthesised, and the point it landed on was
not inside the element the selector matched.
`tapOn` resolves a selector against the a11y tree, takes the matched
element's centre, and synthesises a touch there. Between those two
steps the screen can move — a list settles, a banner appears, a sheet
finishes presenting — and the coordinate then belongs to something
else. The tap still happened; it happened somewhere you did not mean.
The message names both sides: what was aimed at, and what the point
turned out to be inside. On Android it can also say the touch was
**delivered outside every window the runner can read** — the point
fell where no readable window reaches, which is what a tap aimed at a
dialog's button looked like when the tap was computed against the wrong
rectangle (fixed in 11.0; the dialog was dismissed and the step used to
pass).
A different failure, `DRIVER_ERROR` saying *the runner reported nothing
about what it was delivered to*, is not a miss: it is a runner older than
the field that carries that list. `smix runner up --force` rebuilds it.
**Fixes**:
- Wait for the screen before tapping (`extendedWaitUntil`, or
`waitForAnimationToEnd` if you are running with `--animations`)
- If it reproduces on a still screen, the element's frame is wrong
rather than stale — capture `smix tree --json` and check the frame
**Escape hatch**: `SMIX_TAP_HIT_MISMATCH=warn` downgrades this to a
warning for a whole run. It exists so an existing suite can be moved
over gradually; a run under it reports success for taps that missed.
**What this does NOT catch**: an element covered by something
transparent to the a11y tree. A scrim over your button contains the
tapped point too, so the check passes and the touch still may not reach
the button. Nothing recovers this, public or private: a scrim that is
transparent to accessibility leaves no trace in the tree, so every
signal derived from that tree — snapshot fields and live hit-tests
alike — is blind to it in the same way. See "tap returns `ok: true`
but state doesn't change".
### COORDINATE_SPACE_MISMATCH
**Trigger**: the screen is described in one coordinate space and a touch
would be delivered in another, so smix refused to tap rather than aim
into the gap.
A point is worked out against the app's own frame — the same frame
`/tree` reports — and the synthesised event carries an interface
orientation that decides which space those numbers are read in. When an
app rotates its own layout without the device rotating (a
`.fullScreen` controller declaring `landscapeRight`, for instance), the
app is laid out 874×402 while the event is still stamped portrait, and
`x = 437` is read against a 402-wide screen.
**What it looks like without this check**: every tap reports the element
it aimed at and nothing on screen moves. The element resolves, the aim
is inside it, the app is fine — three signals all pointing at your
selector.
**Fixes**:
- Drive the screen in portrait. It is not a workaround for one
selector, it is the axis that works.
- There is no coordinate that gets around it. Whatever `point:` you
pass is recomputed against the app's frame before the stamp is
applied, so flipping or rotating the numbers moves the point inside
the same wrong space. Six mappings were tried before this was
diagnosed; none of them could have worked.
**Not a `TAP_MISSED`**: a miss means the element was there and the touch
went elsewhere, which invites a better point. Here there is no better
point, which is why it has its own code.
### DRIVER_ERROR
**Trigger**: catch-all for runner-side / IO / unexpected failures. Read the `message` for specifics.
**Common subtypes**:
- `runner unreachable`: runner crashed or wrong port — re-run `smix runner up`
- `socket timeout`: runner alive but slow (XCUITest hung) — `smix down` then re-up
- `webview-bridge unreachable`: WebView-hosting screen never visited; navigate first OR `adb forward tcp:28081`
- `JSON decode failed`: runner version mismatch — rebuild runner
### A fill with nothing focused
`no_focused_field` means the characters had nowhere to go: `input text`
types into whatever holds focus, and nothing did. The usual cause is
not the field — it is something else on top of it.
So the failure names it. The message ends with the foreign window
holding the focus (`Above <your app>, and holding the focus:
com.android.permissioncontroller (an app window)`), and says when your
app has no window in the stack at all. The whole stack comes back
beside it as `windows`.
A consumer lost six seconds and a screenshot to the version that said
only "no editable field had focus": their keyboard had opened its own
promotion dialog over the field. `smix system-popups` lists such a
window with its buttons, and `smix system-popup-action` presses one.
### A refused `back`
`back` fails when the screen it was on is still the screen it is on. The
runner presses the key, then watches what is on screen for up to two
seconds, and the message names which reading decided:
| `settledBy` | what it means |
|---|---|
| `screenChanged` | what is on screen is not what was on screen — the ordinary success, and what leaving the app looks like too |
| `couldNotSee` | nothing on screen could be read before the key, so there is nothing to compare against |
| `gaveUp` | the key went in and nothing changed for two seconds |
| `notInjected` | the key event never went in |
`gaveUp` usually means the app consumed the key: a screen with its own
back handling, a modal that ignores it, a root screen in a task the app
does not leave. The reply's `saw` carries the readings behind the
verdict, and `injected` says whether the key itself was delivered —
"the key never went in" and "it went in and nothing moved" are
different problems and used to arrive as the same sentence.
What the runner compares is **which nodes are on screen**, not what they
say. A clock, a spinner or a countdown changes text on every frame
without anything having gone back, and taking that for navigation is
the error the old answer made.
### CAPTURE_BACKPRESSURE
**Trigger**: the simulator's capture path is under load and is refusing
frames for a stated window. `xcrun simctl io … screenshot` at high
frequency can crash `SimRenderServer`, so smix paces those calls; when
recent captures run long, or one fails, the pacer opens a circuit for a
few seconds and refuses rather than pushing a degraded render server
further.
**Not a `DRIVER_ERROR`**: nothing is broken. It means *not now, shortly* —
the `hint` carries the window. Before 7.1 it arrived as a driver error and
a consumer's release gate could not tell the two apart.
**What to do**:
- **A verb that waits already handles it.** `waitForAnimationToEnd` keeps
waiting inside its own ceiling and only gives up when the ceiling does,
reporting that it could not observe the screen — not the pacer's state.
`extendedWaitUntil` and a fallback chain's `ocrText` layers do the same.
- **Anywhere else**: retry after the window in the hint. A run that meets
this repeatedly is telling you about the simulator, not about the flow —
the usual cause is a long-lived simulator under accumulated load, and a
clean restart clears it.
- It cannot be turned off, and that is deliberate: the alternative it
replaced was a crashed simulator mid-flow.
## Device access errors
These fire before any flow step runs — they are about whether smix may
talk to the device at all.
### "… is not a device smix may address"
The identifier is neither registered nor claimed by its platform (a
simulator `simctl` lists, or an `emulator-<port>` serial). For a phone
this is the intended first contact: registration is the deliberate act
that makes it addressable.
```
smix sim register <name> --udid <UDID> --kind physical-ios
```
### "… is a physical device and destructive actions are not allowed on it"
Erase, uninstall and keychain-reset are refused on a phone until allowed
once — recorded per device, not confirmed per command:
```
smix sim allow-destructive <alias>
```
### "no runner is answering on port …, and this device has no other way to be seen"
A phone's screenshot has two routes, and the phone says which. Under
Xcode 27, `devicectl` has `device capture screenshot` and a connected
iPhone lists the capability — smix asks it directly and no runner is
needed. Under Xcode <= 26, or for a phone `devicectl` cannot reach, the
picture comes from the runner (`XCUIScreen`), and this is what you see
when it is not up. Bring it up first: `smix runner up <alias> --bundle <id>`.
### "device … is in use by pid …"
Another *live* smix process holds the device — two concurrent runs are
contention, and the refusal names the process so you can decide. A
runner left behind by a **finished** `runner up` does not produce this:
the next command adopts that lease and drives through the runner, which
is the normal `runner up` → `run` pairing.
### "port … answers /health, but its session is not usable"
The runner's HTTP server is alive and the session behind it is not. The
usual cause is that the app was reinstalled or terminated out from under
the runner, which leaves its `XCUIApplication` bound to something that no
longer exists — `/health` cannot see that, because it never touches the
app.
**Fix**: `smix runner cycle` rebinds in place, in seconds, without
restarting `xcodebuild`. `smix runner up … --force` does the same thing
from the bring-up command. Neither reaches a runner recorded for another
device, or one the device ledger has no record of; those still need
`runner down --include-unrecorded`.
`smix_use` reports the same condition, with the same fix — the MCP server
has no `--force` because it is not a flag surface.
### "the runner's server came up … and its session never did"
The bring-up ran out of time with `/health` answering the whole way. That
is a different failure from "the runner never started", and it says so:
the HTTP server was up and the app binding never became drivable.
smix has already retried by this point — once, with the app foregrounded
and the runner attaching to it rather than launching it. The message says
which ways were tried. If it says the retry did not happen, it names the
reason: a physical device (no `simctl launch`), no `--bundle`, or an
attempt that was already `--no-launch`.
**Fix**: the log tail is in the message; start there. A cold rebuild
after a version bump can genuinely need more than the default 300 s —
`SMIX_RUNNER_UP_TIMEOUT_SECS` raises it.
### "nothing is listening for … any more"
The runner answered earlier in this run and then stopped answering. That
is a different failure from "the runner never started", and until 9.0.0
both arrived as the same bare connection error, so the message told a
reader to start a runner they had already started.
On Android the usual cause is the system killing the instrumentation
under memory pressure — `adb logcat` shows `binderDied` around the time
the flow stopped, and the emulator's RAM is the thing to raise.
**Fix**: `smix runner up`. Every step after this one reports the same
thing until you do, so the first occurrence is the one to read.
### "runner … refused: keyboard_did_not_close"
The keyboard was on screen, every dismiss strategy ran, and it was still
there afterwards. The `saw` field names what was tried and which element
still holds focus.
Before 9.0.0 this answered `ok: true`, because the handler returned a
`Bool` that meant "I ran" rather than "it closed".
**Fix**: tapping the next control often works when a field will not give
focus up. Do not guard the call — `hideKeyboard` is already a no-op when
no keyboard is present, which is a *different* answer from this one.
### "runner … refused: keyboard_state_unknown"
The runner raised while looking at the keyboard, so nothing was
established either way. **This is not evidence the keyboard is still
up.**
**Fix**: retry the step. If it repeats, the runner is the thing to look
at rather than the screen.
## Common failure patterns + fixes
### "Cannot find 'FooScreen' in scope" (iOS build)
You added a Swift file to your Xcode project's sources folder but did not add it to `<YourApp>.xcodeproj/project.pbxproj`.
**Fix**: add 4 entries (use canonical alpha-sort order):
```text
1. PBXBuildFile section: <UUID> /* FooScreen.swift in Sources */ = ...
2. PBXFileReference: <UUID> /* FooScreen.swift */ = ...
3. PBXGroup: <UUID> /* FooScreen.swift */,
4. PBXSourcesBuildPhase: <UUID> /* FooScreen.swift in Sources */,
```
Generate UUIDs via `openssl rand -hex 12 | tr a-f A-F`.
### tap returns `ok: true` but state doesn't change
First check whether you got `TAP_MISSED` — since v2.0.0 a tap that
lands outside the element it aimed at says so. If the tap is reported
as landing correctly and the app still did nothing, the cause is one of
these:
**The screen and the touch are in different coordinate spaces.** This
one is not in your app. An app that rotates its own layout while the
device stays put is laid out in a space the synthesised touch is not
read in, and every tap reports success while nothing moves. smix now
refuses these with `COORDINATE_SPACE_MISMATCH` rather than reporting
them as landed — if you are on a build old enough not to, check
whether the screen is landscape while `xcrun simctl io <udid>
screenshot` still comes out portrait-shaped.
**The element is covered.** smix cannot see this: the a11y snapshot
carries no z-order, so a scrim over your button contains the tapped
point exactly as the button does. `smix tree --json` shows both; the
one drawn later wins and the tree does not say which that is.
**The element is not a touch responder.** An image or a label inside a
button is in the tree and takes no touches. Aim at the ancestor that
handles the gesture.
**Compose interop.** Some Compose `Button` `onClick` lambdas don't fire
reliably when a heavy `AndroidView` interop component (MapView,
PreviewView) is in the same composition. The engine dispatches but the
lambda is not invoked.
**Workaround**: extract heavy `AndroidView` into a separate Composable holder.
### WebView eval flakes on first invocation
Known flake when the WebView shim hasn't been initialized yet.
**Workaround**: navigate to the WebView-hosting screen at least once before exercising `webViewEval` to warm up the bridge. Re-running the flow usually passes the second time.
### Tap on tab bar misses (Android LazyRow)
Tab is in a `LazyRow` that lazy-renders off-screen items. The tap-by-id dispatcher won't find an off-screen testid.
**Fix**:
```yaml
# Swipe the tab bar before tapping the far-right tab
- swipe:
start: "95%,8%" # right edge of tab bar
end: "5%,8%"
duration: 200
- tapOn: { id: "tab-wizard" } # now visible
```
### Android emulator dropping after long-running test
The emulator can go offline (`adb: device offline`) under load. Re-check:
```bash
adb -s emulator-5554 reconnect
adb -s emulator-5554 shell getprop sys.boot_completed
```
### YAML works in maestro but fails in `smix run`
Either:
1. Verb not supported (uncommon — only `assertScreenshot` is currently deferred).
2. Selector form not yet implemented (rare — file an issue).
3. Default direction differs — explicitly set `direction: DOWN` if you relied on maestro's default.
### "permission denied" mid-flow
Permission dialog appeared and was not dismissed. smix has `setPermissions` for declarative grant; also see `system-popups` + `system-popup-action` for one-shot dismiss.
## Debug recipes
### Capture full trace for a failing step
```bash
RUST_LOG=info smix run ... 2>&1 | tee /tmp/trace.log
```
### Snapshot the screen at point of failure
```bash
# In a separate terminal during the run, OR right after failure:
smix sim exec <udid> io screenshot /tmp/fail.png && open /tmp/fail.png
```
### Get full a11y tree at failure point
```bash
smix tree --json > /tmp/tree.json
jq . /tmp/tree.json | less
```
### See what the runner has been doing
```bash
# iOS runner stdout/stderr goes to:
ls -la ~/Library/Logs/smix-runner/<udid>/
# Android runner log (from instrumentation start):
cat /tmp/smix-runner-instrument.log
```
### Sanity check the testid you expect exists
```bash
smix find id:home-increment-btn && echo "yes" || echo "no"
```
### Verify the YAML schema independently
```bash
smix run --dry-run flow.yaml # if supported (else parse with yamllint)
```
## See also
- [01-quickstart.md](01-quickstart.md) §common first-run errors — table of install/boot issues
- [08-cookbook.md](08-cookbook.md) — patterns that avoid common errors
<!-- ===== docs/ai-guide/08-cookbook.md ===== -->
# 08 — Cookbook (recipes)
> Copy-paste patterns for things you'll write many times. Each recipe is a self-contained YAML fragment + commentary.
## Cross-platform YAML (iOS + Android, single file)
```yaml
# Header — use `app:` logical key (not `appId:`)
app: myapp
---
# 1. Any platform-only setup step — mark optional so cross-platform runs skip it silently
- tapOn:
id: "some-ios-only-btn"
optional: true
# 2. The actual test
- tapOn: { id: "tab-home" }
- assertVisible: { id: "home-counter-label" }
- tapOn: { id: "home-increment-btn" }
- assertVisible:
id: "home-counter-label"
text: "1"
```
```bash
# Run on iOS
smix run --device <iosudid> --platform ios --apps-config apps.yaml \
--runner-port 22087 --no-launch flow.yaml
# Run on Android
smix run --device emulator-5554 --platform android --apps-config apps.yaml \
--runner-port 28080 --no-launch flow.yaml
```
`apps.yaml`:
```yaml
apps:
myapp:
ios: { bundleId: com.example.app }
android: { package: com.example.app, activity: .MainActivity }
```
`activity` is an override, and most apps do not need it: omitted, smix
asks the device's package manager which activity the launcher starts.
Set it when an app has more than one entry point and a flow wants a
particular one.
## Login flow (text input + submit)
```yaml
appId: com.example.app
---
- tapOn: { id: "tab-form" }
- tapOn: { id: "form-name-input" } # focus the field
- inputText: "Alice"
- tapOn: { id: "form-email-input" }
- inputText: "alice@example.com"
- tapOn: { id: "form-password-input" }
- inputText: "secret123"
- hideKeyboard # important: keyboard may cover submit
- tapOn: { id: "form-submit-btn" }
- assertVisible: { id: "form-submitted-label" }
```
**Why hideKeyboard before submit**: soft keyboard covers the bottom 40% of the screen. Submit button often gets pushed below the keyboard. `hideKeyboard` retracts the keyboard so the button is reachable.
## Modal interaction (Compose / SwiftUI sheet)
```yaml
- tapOn: { id: "tab-modal" }
- tapOn: { id: "modal-open-sheet-btn" }
# At this point a BottomSheet / .sheet is open
- waitForAnimationToEnd # let the sheet finish animating in
- assertVisible: { id: "modal-sheet-dismiss-btn" }
- tapOn: { id: "modal-sheet-dismiss-btn" }
- waitForAnimationToEnd # let it dismiss
- assertNotVisible: { id: "modal-sheet-dismiss-btn" }
```
**Compose gotcha**: BottomSheet content + AlertDialog buttons need their own `.semantics { testTagsAsResourceId = true }`. If you write a new screen, add this on each modal content root + per AlertDialog button. See [03-selectors.md](03-selectors.md) common pitfalls.
## Stacked modals (3-level)
```yaml
- tapOn: { id: "tab-stacked" }
- tapOn: { id: "stacked-open-l1-btn" }
- waitForAnimationToEnd
- tapOn: { id: "stacked-l1-open-alert-btn" }
- waitForAnimationToEnd
- tapOn: { id: "stacked-l2-continue-btn" }
- waitForAnimationToEnd
- tapOn: { id: "stacked-l3-pay-btn" }
# After all 3 dismiss, trail label should read "completed"
- waitForAnimationToEnd
- assertVisible:
id: "stacked-trail-label"
text: ".*completed.*"
```
## Deep navigation (multi-level nav stack)
```yaml
- tapOn: { id: "tab-deepnav" }
# Drill down
- tapOn: { id: "deepnav-l1-movies-btn" }
- tapOn: { id: "deepnav-l2-action-btn" }
- tapOn: { id: "deepnav-l3-item3-btn" }
- tapOn: { id: "deepnav-l4-edit-btn" }
# At L5, edit + save
- tapOn: { id: "deepnav-l5-text-input" }
- eraseText
- inputText: "MyEdit"
- tapOn: { id: "deepnav-l5-save-btn" }
# Save pops back to L1, state propagated
- waitForAnimationToEnd
- assertVisible: { id: "deepnav-l1-saved-label", text: "saved: MyEdit" }
```
## Long list — scroll to specific row
```yaml
- tapOn: { id: "tab-heavylist" }
# Approach 1: jump-to-row via numeric input (fastest)
- tapOn: { id: "heavylist-jump-input" }
- inputText: "5000"
- hideKeyboard
- tapOn: { id: "heavylist-jump-btn" }
- waitForAnimationToEnd
- assertVisible: { id: "heavylist-row-5000-title" }
# Approach 2: scrollUntilVisible (slower but no helper required)
- scrollUntilVisible:
element: { id: "heavylist-row-9999-title" }
direction: DOWN
timeout: 60000
- tapOn: { id: "heavylist-row-9999-btn" }
```
## OCR fallback for unlabeled buttons
```yaml
- tapOn: { id: "tab-ocr" }
# Vision OCR finds "Submit" via on-screen text rendering
- tapOn: { ocrText: "Submit", recognition_level: accurate }
- assertVisible: { id: "ocr-last-tapped", text: "tapped: Submit" }
```
## Permission grant (declarative)
```yaml
appId: com.example.app
---
- setPermissions:
camera: allow
photos: allow
notifications: allow
location: allow
- launchApp:
clearState: true
- tapOn: { id: "tab-perm" }
- tapOn: { id: "perm-camera-btn" }
# Should NOT see a permission dialog since we pre-granted
- assertNotVisible: { text: "Allow.*to access your Camera" }
- assertVisible: { id: "perm-status-label", text: ".*granted.*" }
```
## WebView eval
```yaml
- tapOn: { id: "tab-webview" }
- waitForAnimationToEnd # let WebView load inline HTML
# Set input + submit via JS
- webViewEval: |
document.getElementById('user-input').value = 'alice';
submitForm();
# Read back the result
- webViewEval: |
document.getElementById('form-result').textContent
# Returns "submitted=alice"
```
## Cross-locale text (no testid)
```yaml
- tapOn: { id: "tab-localized" }
# Tap "Submit" (en), "送信" (ja), "Enviar" (es) — whichever is current
- tapOn:
localizedText:
en: "Submit"
ja: "送信"
es: "Enviar"
```
## Anchor / relative selection
```yaml
# Tap the button immediately to the right of "Email" label
- tapOn:
role: button
rightOf: "Email"
# Tap the gear icon offset slightly from a known anchor
- tapOn:
anchorRelative:
anchor: "Header"
dx: 0.45
dy: 0
```
## Conditional flow (login if needed, else skip)
```yaml
- runFlow:
when:
visible: "Log in" # enter only when login is shown
file: "../subflows/login.yaml"
# Inverse gate — run a setup ceremony only if its end state isn't
# already on screen (idempotent across a batch):
- runFlow:
when:
notVisible: { id: "qa-bubble" }
file: "../subflows/enter-qa-mode.yaml"
```
## Retry flaky step
```yaml
- retry:
maxRetries: 3
commands:
- tapOn: { id: "home-show-alert-btn" }
- assertVisible: { id: "home-alert-ok-btn" }
```
## Sub-flow include (DRY)
`subflows/launch-fresh.yaml`:
```yaml
appId: com.example.app
---
- launchApp:
clearState: true
clearKeychain: true
```
main YAML:
```yaml
- runFlow: ./subflows/launch-fresh.yaml
- tapOn: { id: "tab-home" }
```
## iOS-only / Android-only sections
For YAMLs that primarily run cross-platform but have a few platform-specific bits:
```yaml
# Both
- tapOn: { id: "tab-deeplink" }
# iOS-only — openLink to myapp://...
- openLink:
link: "myapp://home/details/42"
- assertVisible: { id: "deeplink-target-label", text: ".*home/details/42.*" }
# Android-only — back navigation (its own verb, not a key press)
- back
```
## Photograph something that hides itself
A control bar that appears on tap and disappears a few seconds later
outlives neither a second command nor the turn between two tool calls.
The usual workaround is to change the app so it stays up long enough to
photograph, which means the thing you photographed is not the thing that
ships.
Take the frame in the same call as the tap:
```bash
smix tap id:player-surface --then-screenshot /tmp/controls.png
# tapped: id:player-surface — frame via runner 91 ms later, 84213 bytes to /tmp/controls.png
```
Over MCP, the same thing is `smix_tap_then_screenshot`, which answers
with the delay and then the PNG.
What this buys is not wire speed. A tap is about 336 ms and a frame from
the runner about 88 ms — both together fit inside a UI that lives three
seconds. What it removes is the round trip between two calls, which is
where the time actually went.
Two things to know:
- **A tap that fails writes nothing.** A frame taken after a tap that
did not land is a picture of the screen nothing happened on, and it
looks exactly like evidence.
- **It needs a selector the tree can resolve.** `point:` and `ocrText:`
are dispatched without resolving a target, so there is nothing to say
about where the touch landed. A `fallback:` chain is fine — the first
layer that is on screen is the one tapped.
## Performance: skip launchApp for fast iteration
YAMLs that don't need a fresh app state can avoid the `launchApp` cost (3-5s for cold start):
```bash
smix run --no-launch ... # skip launchApp step entirely
```
Use when:
- Running many small YAMLs back-to-back against the same app
- Debugging a single YAML (rerun without restart)
## Performance: pre-warm the runner
The first `smix run` on a freshly-up runner pays ~2s of XCUITest cold start. Subsequent runs reuse. Don't tear down + bring up between YAMLs in a smoke loop:
```bash
# WRONG (slow):
for f in *.yaml; do
smix runner up <udid> --bundle com.example.app
smix run "$f"
smix runner down
done
# RIGHT (fast):
smix runner up <udid> --bundle com.example.app
for f in *.yaml; do smix run "$f"; done
smix runner down
```
## testid naming for a new screen
If you add a screen to the app under test, stick to `<screen>-<element>-<kind>`:
- `foo-screen` — container
- `foo-title-label` — text
- `foo-submit-btn` — button
- `foo-input-name` — text input
## Expo dev-client: deep-link replay after JS reloads
If your app-under-test is an Expo dev-client build and your flows drive
state through custom-scheme deep links (`myapp://…`), you will
eventually see a link you sent earlier get **re-delivered after a later
JS bundle reload** — re-opening a panel or re-firing an action over the
screen your next step asserts.
**Mechanism** (expo-dev-launcher source): any custom-scheme URL that
arrives while the React host is not running — including the window
between a dev-client relaunch's two boots (embedded file bundle, then
the metro bundle) — is stashed in `EXDevLauncherPendingDeepLinkRegistry`,
an **in-memory** registry inside the app process. The NEXT React host
start consumes it via `getLaunchOptions` and injects the URL as the
router's initial URL. It is your own in-flight URL re-emerging one boot
later, not stale persisted state.
**What does NOT work**:
- `clearUserDefaults` — the registry is in-memory; there is no persisted
key to delete.
- A hypothetical runner-side "drain the queued URL" — the registry is
private in-process state; nothing outside the app (XCUITest, simctl)
can reach it.
- Flow-side "neutralizer" URLs before/after the relaunch — the queued
URL is always delivered after the boot, so it always lands after
anything you send; the ordering race is structural.
**What works** (pick per cost):
1. **Process-level relaunch** — `stopApp` then `launchApp` instead of a
JS-level reload. Terminating the process destroys the in-memory
registry. Costs the dev-launcher ceremony on the next launch (can be
15–30 s on a dev-client); right when your ceremony budget allows it.
2. **App-side replay gate** — dev-mode code that tags flow-sent links
(e.g. a nonce query param) and drops any second delivery of the same
nonce. Zero runtime cost, needs app cooperation; the most durable
option for a QA-instrumented app.
3. **Overlay-tolerant assertions** — accept that the replayed action may
fire and make the subsequent asserts robust to it (e.g. a terminal
`close-panel` + text tiers that don't require exclusive screen
ownership). Works today with zero code, at the cost of flow-author
vigilance.
## See also
- [02-yaml-reference.md](02-yaml-reference.md) — full grammar
- [03-selectors.md](03-selectors.md) — all selector forms
- [06-fixtures.md](06-fixtures.md) — a testTag layout example
- [07-errors.md](07-errors.md) — when something goes wrong
<!-- ===== docs/ai-guide/09-sessions.md ===== -->
# 09 — Session lifecycle
Sessions solve the "activation storm" problem on long-running flows: without a session, every request that sets `App-Activate: true` triggers an `.activate()` call on the runner side, and hundreds of activations across a multi-minute gate can exhaust XCTest's process arbitration on iOS 26.5+ and crash `test_runForever()`.
With a session:
- The runner activates once at open time (or not at all — the client picks).
- Every subsequent request from the same client carries `Session-Id: <id>` and hits the session's cached `XCUIApplication` binding directly.
- No per-request activation regardless of what `App-Activate` says.
- The runner exposes an explicit `POST /session/renew-activation` escape hatch for drift recovery, rate-limited to at most one activation per 2 s per session.
Sessions are mandatory on iOS in v2 — the legacy per-request rebind path is gone. Driving without a session is a named error (`no session id on the client`), never a silent fall-back to the path the session model exists to remove. Android has no session concept and drives sessionless.
## When to use
- **`smix run`** — the CLI opens a session for you automatically at start, closes on exit. Zero yaml changes.
- **Rust SDK** — call `App::open_session(bundle_id, activate)` at the top of your flow, use `session.app()` for all subsequent calls, `session.close().await` at the end.
- **TypeScript SDK** — `Session.open(runner, bundleId, { activate: true })`, pair with `try / finally { await session.close() }`.
- **Swift SDK** — `Session.open(driver, bundleId:)` on a `SmixDriver`. Pair with `defer { Task { try? await session.close() } }`.
- **Kotlin SDK** — no public session handle; `Smix.launchApp` opens one and owns it. Pair with `try { ... } finally { app.terminate() }`.
## Rust SDK
```rust
use smix_sdk::{App, text};
let udid = std::env::var("SMIX_UDID").expect("SMIX_UDID env var required");
let app = App::connect_to_runner(22087, Some(&udid)).await?
.with_bundle_id("com.example.app");
let mut session = app.open_session("com.example.app", true).await?;
// Every call below carries `Session-Id` — no activation storm.
session.app_mut().tap(&text("Sign In")).await?;
session.app_mut().fill(&text("Email"), "user@example.com").await?;
session.app_mut().assert_visible(&text("Dashboard")).await?;
// Explicit drift recovery (optional; rate-limited to 1 / 2 s).
if some_drift_condition {
let _activated = session.renew_activation().await?;
}
// Release. Sends `POST /session/close`, clears the client-side header.
let _app = session.close().await?;
```
If you skip `close()`, `Drop` clears the client-side `Session-Id` header on a best-effort basis. It cannot `await` a network call, so the runner-side entry is only cleaned up on runner restart. Prefer explicit `close()`.
## Android
The Kotlin runner serves the same `/session/*` surface (open / close /
close-all / list / launch-app / terminate-app / relaunch-app /
renew-activation), backed by an in-memory table and `am` commands.
`smix run` still drives Android sessionlessly — the session surface is
there for the SDKs that open one explicitly.
## TypeScript SDK
Session lifecycle is fully wired in the TypeScript package; driving
(`Smix.launchApp` and the `App` act/sense methods) is not — those throw
`SmixNotImplementedError` until the native transport lands. What works:
```ts
import { Session, HttpSimRuntime } from '@goliapkg/smix'
const runtime = new HttpSimRuntime('http://127.0.0.1:22087')
const session = await Session.open(runtime, 'com.example.app', { activate: true })
try {
await session.relaunchApp()
} finally {
await session.close()
}
```
The `HttpSimRuntime` picks up the `Session-Id` header via `setSessionId` inside `Session.open` — the wiring is invisible to callers.
## Swift SDK
```swift
import SmixSDK
// Smix.launchApp opens the session for you; open one directly only
// when you want the handle before launching.
let driver = SmixDriver(port: Smix.defaultRunnerPort)
let session = try await Session.open(driver, bundleId: "com.example.app")
do {
let app = try await Smix.launchApp(.bundleId("com.example.app"))
try await app.tap(.text("Sign In"))
try await session.close()
} catch {
try? await session.close()
throw error
}
```
`SmixDriver` comes from the UniFFI-generated bindings and carries every sense / act call over the runner's HTTP surface. The Swift SDK holds no simulator I/O of its own.
## Kotlin SDK
```kotlin
import dev.smix.sdk.*
// Kotlin has no public session handle: Driver's implementation and
// App's session field are both `internal`. Smix.launchApp opens a
// session and owns it for the App's lifetime.
try {
val app = Smix.launchApp(AppTarget.BundleId("com.example.app"))
app.tap(Selector.Id("btn-login"))
} finally {
app.terminate()
}
```
Kotlin's driver wraps the same UniFFI-generated core the Swift SDK uses, so both languages speak one Rust wire client rather than a per-language HTTP client. It is `internal` on purpose — `Smix.launchApp` is the supported entry.
## Session state + relaunch-app
Sessions expose a `state` classification driven by the runner's `X-Sim-Health` response header, and a `relaunch_app()` primitive for in-place app-crash recovery.
### State
`SessionState` values:
- `healthy` — all watched signals inside envelope
- `degraded` — screenshot slow-path, `/health` age between stale and dead thresholds, or `/system-popups` throttled
- `cycling` — runner supervisor is mid auto-restart; wait for `healthy` before retrying failed calls
- `dead` — SimRenderServer or xcodebuild is gone; bail out
Consumers subscribe:
```rust
// Rust
match session.state() {
SessionState::Healthy => { /* proceed */ }
SessionState::Degraded => { /* pause the gate loop */ }
SessionState::Cycling => { /* wait for healthy */ }
SessionState::Dead => { /* abort */ }
}
```
```ts
// TypeScript
session.on('state', (state) => {
if (state === 'degraded') pauseGate()
if (state === 'dead') abortGate()
if (state === 'cycling') waitFor('healthy')
})
```
State observation exists in Rust (`session.state()`) and TypeScript
(`session.on('state', …)`, fed by `HttpSimRuntime.attachSessionState`).
The `SmixDriver`-backed Swift and Kotlin sessions carry no state
stream — poll `GET /health`, or watch the supervisor, for the same
signal there.
### Playbook — what to do on each transition
The state is a raw signal; the response is not one-size. The comments in the
snippets above compress to this runbook. "Gate loop" = the driving loop that
issues flow steps.
| State | What it means | Do | Why |
|---|---|---|---|
| `healthy` | every watched signal in envelope | proceed | nothing to recover from |
| `degraded` | one signal slow — screenshot slow-path, `/health` aging, popups throttled — but the runner is alive | **pause the gate loop, do not fail**; back off and re-check state before the next step | a step issued during a slowdown times out on latency, not on a real defect; failing it attributes the wrong cause |
| `cycling` | supervisor is mid auto-restart | **wait for `healthy`**, then re-issue the *last* step; do not count the interruption as a failed attempt | the runner is briefly absent by design; a call now fails transport, not the app |
| `dead` | SimRenderServer / xcodebuild is gone | **bail** — stop the gate loop and surface it; a fresh `runner up` (or supervisor cycle) is required before any step can succeed | nothing the gate does recovers a dead host; retrying only produces a wall of transport errors |
Two transitions deserve care beyond the table:
- **`healthy → degraded → healthy`** is normal flapping under load. Do not
treat a single `degraded` as terminal; only escalate if it persists past your
back-off budget.
- **`cycling → dead`** means the auto-restart itself failed. This is the one
case to escalate immediately rather than keep waiting — the supervisor has
given up, and so should the gate.
For an app crash *without* a host problem (state stays `healthy`, but the target
app is gone), the response is neither pause nor bail — it is
[`relaunch_app()`](#relaunch-app) below, which recovers in place without a
cycle.
### Relaunch app
When the target app crashes but the runner is still healthy, `relaunch_app()` does an in-place `terminate() + launch()` on the session's cached binding — session id + XCUITest binding preserved, no runner cycle needed.
```rust
// Rust
let wall_ms = session.relaunch_app().await?;
```
```ts
// TypeScript
const wallMs = await session.relaunchApp()
```
```swift
// Swift
let wallMs = try await session.relaunchApp()
```
```kotlin
// Kotlin
val wallMs = session.relaunchApp()
```
## CLI
`smix run` opens a session automatically. No new flags — the session is just there under the hood:
```bash
smix run visual.yaml --device ios-17 --activate --bundle-id com.example.app
```
Behind the scenes:
1. CLI POSTs `/session/open` with `{bundleId, activate}`, gets a session id.
2. Every subsequent request in the flow carries `Session-Id`.
3. On exit, CLI POSTs `/session/close`.
If `/session/open` returns a non-2xx — a runner too old to open one — `smix run` prints the reason and exits 6. There is nothing to fall through to: v2 drives iOS through a session or not at all. Re-extract the runner this CLI ships with (`smix runner install --force`) and retry.
## Wire
- `POST /session/open {bundleId, activate}` → `200 {sessionId, activatedOnce, serverTimeMs}`
- `POST /session/close {sessionId}` → `200 {ok}` (idempotent)
- `POST /session/renew-activation {sessionId}` → `200 {ok, activated}` (activated=false when rate-limited); `404 not_found` when the session id is unknown.
Client header on subsequent requests: `Session-Id: <sessionId>`.
## Semantics
- **One session per bundle-id per client.** Opening a second session for the same bundle from the same client just gives you a new id — the runner tracks both bindings independently but they alias the same `XCUIApplication`.
- **Session is per-connection, not per-flow.** If you close and reopen your `HttpRunnerClient`, you need a fresh session; the runner has no way to associate two connections.
- **Runner restart drops all sessions.** A subsequent request with a stale `Session-Id` falls through to the legacy per-request rebind path (rate-limited).
- **Renew is rate-limited.** At most one `.activate()` per session per 2 s. If you call `renew_activation()` inside that window, `activated` comes back `false`; the session is still healthy.
<!-- ===== docs/ai-guide/10-ai-assertions.md ===== -->
# 10 — AI assertions
> Ask a model whether the screen looks right, when there is nothing on it worth
> selecting. Opt-in, non-deterministic, and deliberately kept out of the
> selector path.
## When to reach for this
Almost never, and that is the point.
`assertVisible: { id: "cart-total" }` is a measurement: the element is there or
it isn't, the same way every time. `assertCondition: "the cart total looks
right"` is a judgement — a model reads a screenshot and forms an opinion. Two
runs against the same screen may not agree.
So use the deterministic verbs whenever the screen gives you something to hold
onto — an id, a label, text, an OCR string. Reach for AI assertions when it
genuinely doesn't:
- **A visual claim with no element behind it.** "The chart is not clipped."
"The avatar loaded rather than showing a broken image."
- **Porting a maestro flow** that already uses `assertWithAI` / `extractTextWithAI`,
when you want it running before you rewrite it properly.
- **Reading a value off a screen** that renders it into an image or canvas,
where OCR alone can't tell you which number is the total.
If you find yourself using it because a selector was hard to write, fix the
selector instead. A flaky judgement is worse than an honest failure.
## Turning it on
Off by default. A verdict costs a model call and isn't reproducible, so a flow
cannot reach for one by accident:
```bash
SMIX_ENABLE_AI_ASSERTIONS=1 smix run --device <udid> flow.yaml
```
Without it, a flow containing these verbs fails **at parse time** — before the
device is touched — rather than halfway through a run.
The judge is your local `claude` CLI, invoked as a subprocess. There is no
provider setting: smix does not ship a model abstraction, and it will not.
## `assertCondition`
```yaml
- assertCondition: "a red error toast is visible"
```
smix screenshots the device, hands the image and your condition to the CLI, and
reads back a verdict. When the condition doesn't hold, the step fails with the
judge's own reasoning:
```
FAIL [ASSERTION_FAILED]: [AI · non-deterministic] assertCondition did not hold:
a red error toast is visible — the judge saw: the screen shows a green success
banner, no error toast
hint: this verdict is a judgement rather than a measurement; another run may
answer differently
```
The `[AI · non-deterministic]` tag is on every AI-sourced failure, so a reader
skimming a CI log can tell a measurement from an opinion at a glance.
## `extractWithAI`
Reads named values off the screen into the output store, where the expression
engine can use them:
```yaml
- extractWithAI:
into: order
fields: ["total", "currency"]
- assertTrue: '${output["order.total"] != ""}'
```
**`into` is a key prefix, not a nested object.** The output store is flat, so
the fields above land at `order.total` and `order.currency`, and you read them
back with the bracket form. `${output.order.total}` does not parse.
`extractWithAI` is the only verb that writes the output store, which also makes
it the only way a `repeat.while` condition can change between iterations.
## When the judge doesn't answer
A missing CLI, a timeout, a non-zero exit, or a reply that isn't a verdict are
all **errors**, not `pass: false`:
```
FAIL [DRIVER_ERROR]: ai-tier: could not run the claude CLI at `claude`:
No such file or directory
hint: install the claude CLI, or point claude_bin at it (currently `claude`)
```
This distinction matters more than it looks. Collapsing "the judge never ran"
into "the condition is false" would report a broken app when the truth is a
broken toolchain, and you would go debug the wrong thing.
## What this tier is not
It is not part of how smix sees the screen. Selectors resolve through the
accessibility tree and Vision OCR, and nothing in that path can reach this
crate — a check in CI asserts the sense path's dependency tree never touches it.
Delete the AI tier and every selector still resolves exactly as before.
That fence is why "smix uses AI" doesn't mean "smix is non-deterministic". The
deterministic core is the product; this is a tool sitting next to it.
## See also
- [02-yaml-reference.md](02-yaml-reference.md) — full grammar
- [03-selectors.md](03-selectors.md) — the deterministic way to find things
- [07-errors.md](07-errors.md) — reading a failure
<!-- ===== docs/ai-guide/11-mcp.md ===== -->
# 11 — MCP
> Give an agent the simulator. Launch, look, tap, type, assert — over MCP,
> with no yaml in between.
## When this is the right tool
YAML flows are for tests you run again. MCP is for the loop *before* that
exists: an agent building a feature, reproducing a bug, or finding out what a
screen actually contains.
The rule of thumb: if you'd write the flow down and keep it, write yaml. If
you're exploring, drive it over MCP.
## Setup
Two pieces: the runner drives the simulator, the MCP server talks to the
runner. Bring the runner up first — the server does not start it.
`smix-mcp` is its own binary — `cargo install smix-cli` does **not**
include it:
```bash
cargo install smix-mcp
smix runner up <udid> --bundle com.example.app
```
Then point your MCP client at `smix-mcp`:
```json
{
"mcpServers": {
"smix": {
"command": "smix-mcp",
"env": {
"SMIX_UDID": "<booted-simulator-udid>",
"SMIX_RUNNER_PORT": "22087"
}
}
}
}
```
One server process binds one simulator. Two simulators means two entries with
different `SMIX_UDID`s.
## A session
What driving actually looks like:
```
smix_launch_app { "bundleId": "com.example.app" }
smix_describe → what's on screen
smix_tap { "id": "tab-form" }
smix_tap { "id": "form-email-input" }
smix_fill { "target": { "id": "form-email-input" }, "text": "alice@example.com" }
smix_press_key { "key": "return" }
smix_assert_visible { "id": "form-submitted-label" }
```
`smix_describe` first, most of the time. It returns the visible elements and
their ids, which is what you need to write the next call — guessing an id and
getting `ELEMENT_NOT_FOUND` is the slow path.
## Naming an element
Every element-facing tool takes the same shape. Exactly one of:
| | |
|---|---|
| `{ "id": "form-submit-btn" }` | The app's testID. **Prefer this.** |
| `{ "text": "Submit" }` | Visible text, case-insensitive. |
| `{ "label": "Close" }` | Accessibility label, exact. |
| `{ "role": "button", "name": "Reload" }` | Kind, optionally narrowed. |
| `{ "ocrText": "Submit" }` | Read off the pixels. Slower than the tree. |
| `{ "ocrText": "Zulassen", "locales": ["de"] }` | Which languages to read it in. |
| `{ "fallback": [ … ] }` | Ways to name the same thing, tried in order, first hit wins. |
| `{ "point": "50%,80%" }` | A place, not a thing. Only `smix_tap` takes one. |
`locales` is worth naming whenever the text is not English. Left out, the
recogniser works out the language itself; told the wrong one it does not fail,
it misreads, and what you then see is "no matching text" about a dialog the
word is plainly on. Android refuses a
script it cannot read rather than reporting the screen: its recogniser ships
the Latin package only.
`fallback` is the one to reach for when a name might not hold: each entry is a
whole selector, so `[{"id":"submit"},{"text":"Send"},{"point":"50%,90%"}]`
survives a missing testID and a copy edit without you looking first and
choosing. `point` may only be last — a coordinate always hits, so anything
after it would never be tried, and a chain with a dead tail reads as a plan
and is not one.
`point` is a fraction of the viewport, never pixels: `"50%,80%"`, or the same
place written `"0.5,0.8"`. Only tapping takes one — nothing is named at a
coordinate, so `smix_find`, the asserts, `smix_fill` and `smix_scroll` refuse
it and say so rather than answering about somewhere you did not ask.
Ids survive copy edits and translation. Text does not — a flow written against
`{"text": "Submit"}` breaks when someone changes the button to "Send", and
again in every other locale.
Naming nothing, or naming two, is an error. smix will not pick for you.
`smix_fill` and `smix_scroll` need a selector *and* something else, so theirs
goes under `target`:
```json
{ "target": { "id": "form-email-input" }, "text": "alice@example.com" }
{ "target": { "text": "Sign out" }, "direction": "down" }
```
## Read the failures
They are written to be read back:
```
FAIL [ELEMENT_NOT_FOUND]: no element matched id=form-submit-btn
suggestions:
- form-submit-button
on screen: com.example.app (application, focused)
visible elements (10 of 23, the focused app's first):
- button "Submit" id=form-submit-button
- textField id=form-email-input
```
The suggestion is the answer: the id has `-button`, not `-btn`. The failure
carries the near-misses and the elements that *were* there, so the next call
can be right rather than another guess.
One failure comes from `smix_use` rather than from the screen:
```
the runner on port 22087 answers /health, but its session is not usable:
not-running. That happens when the app is reinstalled or terminated out
from under the runner.
Recover it in place, then use this tool again:
smix runner cycle
```
`smix_use` answers `already driving …` only when the session actually
works. `/health` on its own does not establish that: it says the runner's
HTTP server is answering, and it never touches the app binding — so a
reinstall leaves a runner that answers 200 and drives nothing. Reporting
that as "already driving" hands you a device you cannot drive, which is
what it did before 4.3.
## Tools
**Look** — `smix_describe` · `smix_tree` · `smix_screenshot` · `smix_find`
**Act** — `smix_tap` · `smix_tap_then_screenshot` · `smix_fill` · `smix_press_key` · `smix_swipe` · `smix_scroll`
**Lifecycle** — `smix_launch_app` · `smix_stop_app`
**Assert** — `smix_assert_visible` · `smix_assert_not_visible`
**Diagnose** — `smix_session_state` · `smix_diagnostic_dump`
`smix_find` returns true/false; `smix_assert_visible` fails. Use `find` to
decide what to do next, `assert` when absence is a problem.
`smix_scroll` beats a loop of `smix_swipe` — it knows when to stop.
`smix_tap_then_screenshot` is for something that will not still be there
by the next call. What it saves is not wire time: a tap is about 336 ms
and a frame from the runner about 88 ms, so both together are well
inside a UI that lives three seconds. What it saves is **the turn
between two tool calls** — the model's own round trip, which is where
the seconds actually go. It answers with a line naming the route and the
delay, then the PNG as base64. A tap that fails returns no frame.
## Directions
`up` / `down` / `left` / `right` name **what you want to see**, not which way
the finger moves. `down` reveals what is below. Both platforms follow the same
convention, so the same call behaves the same on iOS and Android.
## What is deliberately absent
**The AI-assertion tier.** Whatever calls these tools is already a model, and
it can call `smix_screenshot` and look at the screen itself. Asking a tool to
ask another model about a screen the caller can already see is a detour. The
`assertCondition` verb exists for *yaml* flows, where nothing is watching.
**The rest of the SDK.** `smix-sdk` has dozens of methods. This is a driving
surface, not a mirror.
## See also
- [03-selectors.md](03-selectors.md) — the selector taxonomy in depth
- [07-errors.md](07-errors.md) — the failure format
- [10-ai-assertions.md](10-ai-assertions.md) — the fenced AI tier, for yaml
<!-- ===== docs/ai-guide/12-authoring.md ===== -->
# 12 — Authoring flows from an LLM
This guide is for a language model writing and debugging smix flows — the
process, not the syntax. The syntax lives in [03-selectors](03-selectors.md) and
[04-actions](04-actions.md); the recipes for common screens live in
[08-cookbook](08-cookbook.md). What follows is how to *approach* the task so the
loop converges instead of guessing.
## The one idea
smix is deterministic sense → act, and every failure is written to be read by
you, not a human. A step does not just pass or fail — a failure carries the
`visibleElements` it saw, a `code`, and `suggestions`. **Authoring is a loop:
run, read what it saw, adjust the selector, run again.** You are not expected to
know the tree in advance; you are expected to read it back.
## Before running: `--dry-run`
Parse-only, no simulator, no runner. It validates every step (and every
`runFlow:` include) and reports `parse OK/FAIL` per file. Run it first — a
parse error costs a second here instead of a device round trip.
```bash
smix run flow.yaml --device <DEVICE> --dry-run
```
## Picking a selector
Order of preference, most to least stable:
1. **`id`** — an accessibility identifier. Stable across copy changes and
locales; nothing else is. Prefer it whenever the element has one.
2. **`label`** — the accessibility label. Stable-ish, but localised.
3. **`text`** — visible text. Fine for buttons and headings; breaks when the
copy or language changes.
4. **`role` + `name`** — when several elements share text but differ in kind.
5. **`ocrText`** — the pixels, read by OCR. The escape hatch for elements the
accessibility tree does not expose at all (a React Native Fabric tree that
dropped its identifiers, a canvas-drawn control).
When the accessibility tree is unreliable — iOS 26.5 + RN Fabric is the case
this project keeps hitting — combine tiers in a `fallback` so the tap tries the
tree first and OCR second:
```yaml
- tapOn:
fallback:
- id: sign-in-btn
- text: Sign In
- ocrText: Sign In
```
## Reading a failure
A timeout or a not-found does not mean "the app is broken". It means smix
looked and tells you what it saw. Read `visibleElements` — the elements it
*did* find are the menu you pick your next selector from. If your `text: Login`
missed but `visibleElements` shows `staticText name="Log in"`, the copy differs
by a space and a case; fix the selector, not the app.
`suggestions` names the likely fix. `code` sorts the failure:
- `ELEMENT_NOT_FOUND` — nothing matched. Widen: try `fallback`, or read
`visibleElements` and pick what is actually there.
- `NOT_VISIBLE` — matched but off-screen or occluded. Scroll to it first.
- `AMBIGUOUS` — several matched. Add `role`, `nth`, or a spatial modifier
(`below:`, `rightOf:`) to disambiguate.
- `TAP_MISSED` — the tap landed outside the element it aimed at (the screen
moved between the tree fetch and the tap). Wait for the screen to settle
(`extendedWaitUntil`) before tapping.
Full code list: [07-errors](07-errors.md).
## Reading the tree directly
When you cannot infer a selector from a failure, dump the tree and pick from it:
```bash
smix tree --device <DEVICE>
```
Every node carries its `identifier`, `label`, and `role`. A node with an empty
identifier *and* empty label whose `elementTypeRaw` is not 1 is a React Native
accessibility-bridge drop — that element is unreachable by the tree, so reach it
by `ocrText` or by a nearby labelled anchor (`below:`, `rightOf:`).
## Waiting instead of sleeping
There is no bare `sleep`. Wait on the thing you actually need:
```yaml
- extendedWaitUntil:
visible: Welcome
timeout: 10000
```
For an element that appears only after an animation or a network round trip,
`extendedWaitUntil` with a `fallback` selector polls the tree and OCR each
iteration rather than guessing a fixed delay.
## Confirming a tap did something
A green `tapOn` means "the point aimed at was inside the element named" — its
success output says so in those terms (`aimed inside: …`). It does **not** mean
the app reacted, and it does not mean the app is where an unchanged screen came
from: this line is judged in the snapshot's coordinate space, and a mismatch
between that space and the one the touch is delivered in produces exactly this
output with nothing moving. Assert the *consequence*, not the tap:
```yaml
- tapOn: { id: open-menu }
- assertVisible: Settings
```
## When the tree cannot decide — the AI-assertion tier
For a check the deterministic layer cannot make ("does this look like an error
state"), the fenced AI tier takes a screenshot to a local `claude` CLI and
returns a structured verdict. It is opt-in and marked non-deterministic, and it
sits beside the resolver, never inside it — see [10-ai-assertions](10-ai-assertions.md).
Reach for it only when a selector genuinely cannot express the check; a
deterministic assertion is always preferable.
## The loop, in one place
1. `--dry-run` to catch parse errors with no device.
2. Run. On failure, read `visibleElements` + `suggestions` + `code`.
3. Adjust the selector from what was actually seen — do not re-guess blindly.
4. Stuck? `smix tree` and pick from the real nodes.
5. Assert consequences, not actions.
6. Only reach for the AI tier when no selector can express the check.
<!-- ===== docs/ai-guide/wire-format.md ===== -->
# Wire format — smix v1.0.0
> The wire format between the smix client (`smix-runner-client`) and
> the smix runner (`SmixRunnerServer`) is frozen at v1.0. All shapes
> below are semver-major (breaking change = v2.0). Adding new
> optional fields is allowed within v1.x; renaming or removing
> existing fields is not.
## HTTP transport
Base URL: `http://127.0.0.1:<port>` — port default `22087`, overridable
via the registry's `runnerPort` or the `--runner-port` flag.
All requests use a JSON body; all responses return JSON.
A client may send a request again only when it cannot have arrived (the
connection was never made), or when it only reads. A `POST` taps, types or
presses, and a runner that took one and did not answer in time may already
have carried it out; sending it again is a second tap, or the text typed
twice. The Rust client fails such a request with `SentWithoutAnswer`
instead of repeating it; a `GET` is asked again.
## Request-context headers
Every route accepts these OPTIONAL headers; absent = default behavior
(runner-boot target, no activate, a11y-anchored dispatch):
| Header | Semantics |
|---|---|
| `App-Bundle-Id: <bundle>` | Per-request `XCUIApplication` rebind target. smix also uses it to ask a question: `GET /tree` with this header answers 200 when that app can be snapshotted and 500 `snapshot_unavailable` when it cannot, which is how `runner up` tells "the runner is answering" from "the app it was asked about is drivable". |
| `App-Activate: true` | Runner calls `.activate()` on the resolved target before the operation (main-actor hop) |
| `Input-Dispatch-Mode: a11y \| key-events \| auto` | How `/fill` puts text in. `a11y` (default) resolves the field and taps it to take focus; `key-events` skips resolution and types into whatever holds focus, for fields the tree cannot address. An unknown value degrades to `auto` rather than failing the step. |
## Routes
### `POST /tap`
Body: `{ selector: { text | id | label: string }, mode?: "resolve" | "resolveAndTap" | "daemonProxySynthesize" }`
Exactly one selector key. `text` matches label or identifier (the historical
behaviour); `id` matches identifier only; `label` matches label only. Regex
patterns, roles and spatial/index modifiers are not accepted here — they need
the full tree and resolve host-side, which is what the default tap path does.
(scope via `?include=` query, same mechanism as `GET /tree`; `mode` defaults to `resolveAndTap`)
Response on success (`TapResult` in `smix-runner-wire`; all fields beyond `ok` optional):
```json
{
"ok": true,
"matchedLabel": "Sign In",
"frame": { "x": 20.0, "y": 118.5, "w": 353.0, "h": 44.0 },
"appFrame": { "x": 0.0, "y": 0.0, "w": 393.0, "h": 852.0 },
"stages": { "resolveMs": 12.5, "tapCallMs": 870.0, "totalMs": 882.5 }
}
```
`frame`/`appFrame` are returned by `mode: "resolve"` so the caller can
inject the tap at the resolved coordinate; `resolveAndTap` performs the
tap in-process and omits them. Miss → `404 { ok: false, error: "not_found", selector, visible: [] }`.
### `GET /tree?include=<scope>&hollow=<package>`
Response: `A11yNode` tree.
`hollow` (Android): every application window of that package is reported
as the window itself — its bounds, its place in the stack, whether it has
the focus — with no children. The host asks for it when the app carries
the semantics probe, whose view of the app replaces that window anyway;
the windows the probe cannot see (the keyboard, the system bars, another
app's dialog) are walked as always. A runner that predates the parameter
ignores it and walks everything.
Node fields of note:
- `rawType` — element type name. iOS: camelCase `XCUIElement.ElementType` name (`"button"`, `"staticText"`, …); Android: full a11y class name (`"android.widget.Button"`, …).
- `elementTypeRaw` (v1.0.22+) — the numeric `XCUIElement.ElementType.rawValue`. **iOS-only signal**; Android payloads omit it and deserialize to the default `1` (`.other`). Triage rule on iOS: `elementTypeRaw != 1 && identifier == "" && label == ""` ⇒ the OS typed the element but the app's a11y bridge dropped its name (app-side issue, not smix).
- `role` — curated semantic role. Android emits it directly (derived from class name); iOS consumers derive it client-side from `rawType`.
- `window` (Android, on each child of the root) — whose window the subtree is: `{ kind, package?, focused, layer, touchable }`, `kind` one of `application` / `inputMethod` / `system` / `other`. `layer` is the window's place in the stack, higher drawn over lower; the children are not listed in screen order. `touchable` (`{x, y, w, h}`, screen pixels) is where the window takes touches, which is not its bounds: the keyboard's window spans the screen below the status bar and takes touches only on the keys. A host aims a touch at the part of an element that no higher window takes touches over — an app drawn edge to edge has content under the status bar, and a touch there goes to the bar.
Response headers (metadata only — body shape unchanged; all additive):
| Header | Since | Meaning |
|---|---|---|
| `X-Tree-Size-Bytes` | v1.2 | Serialized payload size |
| `X-Tree-Node-Count` | v1.2 | Total node count |
| `X-Tree-Snapshot-Refresh-Count` | v1.0.23 (iOS) / v1.0.26 (Android) | Monotonic count of successful `/tree` serves since runner boot — a flat value across polls indicates a stalled snapshot pipeline |
| `X-Tree-Snapshot-Wall-Ms` | v1.0.23 (iOS) / v1.0.26 (Android) | Wall time of this snapshot walk — trending upward across a batch indicates the OS a11y pipeline is bogging down |
### `POST /find`
Body: `{ selector: { text: string }, requireOnScreen?: bool }`
**Text only.** This is the fast path for a simple text lookup; the
runner decodes `selector.text` and nothing else, and refuses any other
selector with `400 bad_request` / `missingText`. Send id, label, role or
any compound selector to `POST /tree` and match there — that is what the
first-party SDKs do. `include` is not read here either.
Response: `{ ok: true, found: bool }`. On `found:false` the runner may
add `diagnostics: { appState, candidates, rebound }` — advisory only,
and absent on the happy path so a client parsing the two-field shape
never meets a new field when the query worked.
`exists` is a **historical alias that no current runner emits**. The
first-party client still accepts it on input for old runners; do not
read it from a response, and do not OR-merge the two — a response has
`found` and only `found`.
`requireOnScreen: true` (v1.0.27) — `found` additionally requires the LIVE element frame to intersect the app frame. Snapshot frames drift on iOS 26.5 + RN Fabric for below-the-fold elements; the live query re-resolves current layout. The driver's visibility-semantic paths (`wait_for` / `find` / scroll probes) use this so `extendedWaitUntil` / `scrollUntilVisible` / `tapOn` agree on "visible". Deliberately checks frame∩viewport rather than `isHittable` — hittability is false under floating overlays, which are genuinely visible and assertable. `isHittable` is the only z-order-aware signal XCUITest offers, and that false-under-overlay behaviour is exactly why it stays rejected here.
### `POST /fill`
Body: `{ selector: Selector | "_focused_", text: string, clearFirst?: bool, include?: IncludeScope }`
Header: `Input-Dispatch-Mode`
Response: `{ ok: bool, focusMs?: u64, daemonSendMs?: u64 }`
`clearFirst` empties the field before typing, and is **true when
absent** — typing appends, so a route named `fill` that did not clear
concatenated old and new values. A runner too old to know the field
appends, which is what it did before, so the field is additive on the
wire. The host driver chunks long text into one POST per character;
`clearFirst` rides the first chunk alone.
`key-events` is for RN apps whose hidden `<TextInput>` defeats
a11y-focus lookup. On iOS the header reaches the runner, which skips
its focus-tap; on Android there is no header to send — `/input-text`
already types into the focused field — so the host driver honours the
mode by not resolving. Different mechanics, same guarantee.
### `POST /clear`
Body: `{ selector: Selector | "_focused_", include?: IncludeScope }`
Response: `{ ok: bool }`
iOS only. The Android runner has `/clear-text` instead, because the
work is different: there is no selector to resolve runner-side, and
the host has already tapped the field to focus it.
### `GET /windows` (Android)
Response: `{ status: "ok", count: int, windows: [ { index, type, layer,
active, focused, rootReadable, package } ] }`
`/tree` answers what is on screen; this answers why something is not.
A window whose root cannot be read is skipped by the tree walk, and a
window that was never attached is skipped too — different problems,
identical symptom, and until this route there was no way to look. The
tree's root object also carries `unreadableWindows`, a count of the
first kind.
```bash
curl -s http://localhost:28080/windows | jq '.windows[] | {package, active, rootReadable}'
```
If the app you are driving has no row at all, its window is not
attached for accessibility. If it has a row with `rootReadable: false`,
it is attached and smix cannot read it. The two want different fixes.
### `POST /clear-text` (Android)
Body: `{ focusRect?: [nx, ny, nw, nh] }` — the focused field is the target
(the one inside `focusRect` when given); the host focuses it first.
Response: `{ ok: bool, status: "ok" | "field_not_empty" | "no_focused_field",
method: "set-text" | "key-events", deletes: int, held: int }`
`ok` is whether the field is empty when the route looks again. `held` is
how many characters it still holds, and -1 when no focused field could be
found to ask — `no_focused_field`, which is not the same answer as a field
with something left in it. The whole wait for that second look is bounded
by the route's own budget (about 8 s when nothing takes focus).
`method` is not decoration. `set-text` empties the field through the
focused node's `ACTION_SET_TEXT` and is exact at any length.
`key-events` is the fallback for a field the accessibility tree cannot
address: it sends a bounded number of deletes, so a longer field
survives it partly filled, and a caller that cannot tell the two apart
cannot know which answer it got.
This replaced fifty `/press-key delete` posts from the host — fifty
sequential round trips over the adb forward, on every fill once fill
began clearing first, and still wrong past fifty characters.
### `POST /input-text` (Android)
Body: `{ text: string, focusRect?: [nx, ny, nw, nh], budgetMs?: number }`
Response: `{ ok: bool, status: "ok" | "text_did_not_land", text, before, held, chunks, retyped }`
— for a masked field `beforeLength` and `heldLength` in place of `before`
and `held`.
Types into the focused field (the one inside `focusRect` when given) and
reads it back. `ok` means the field holds what it held before with `text`
put in once, at the caret. A field holding characters this request did
not type is not `ok`, and neither is one missing some. A masked field
cannot be read, so its length is the whole of the judgement, and its
content never goes on the wire.
The text goes in 32 characters at a time, each chunk read back before
the next: one long `input text` drops characters under load. A chunk
whose end did not arrive has only its missing part typed again, up to
three times; `chunks` is how many there were and `retyped` how many of
those repairs it took. A chunk missing characters from its middle, or
holding one nobody typed, is not repaired — the failure names the
chunk and what the field held.
Typing costs time per character, so the host waits longer for a longer
text and tells the runner how long in `budgetMs`. The runner checks it
before each `input text` it sends; once it is spent it types nothing
more and answers `{ error: "text_budget_spent", message }`, the message
naming the chunk it stopped before and how many characters the field
holds. Without `budgetMs` the typing is not bounded.
`input text` goes wherever focus is, not to a node, so the field is read
back as the node that was typed into, and a repair is typed only while
that node still has focus. A field can leave the screen as it fills — a
code field that submits itself when full. When that happens after the
last chunk was sent, the answer is `{ ok: true, status: "field_left",
readBack: "unread", text, before, lastRead, chunks }`: the app took the
text and moved on, and there is nothing left to read. When it happens
with text still to send, or a short chunk's field has lost focus, the
runner types nothing more and answers `{ error: "field_left" }` or
`{ error: "focus_left" }`, the message naming the chunk and what the
field held when last read.
### `POST /press-key`
Body: `{ key: KeyName }` — the wire names: `return`, `delete`, `tab`,
`space`, `escape`, `arrowUp`, `arrowDown`, `arrowLeft`, `arrowRight`,
`home`, `lock`, `volumeUp`, `volumeDown`.
Response: `{ ok: bool }`
A key the device has no button for is refused by name:
`{ ok: false, error: "no_such_button", saw: "<why, and what to do instead>" }`,
with status 200. The iOS runner answers this for `lock`, `volumeUp` and
`volumeDown`; the Android runner presses all three.
`back` is a `KeyName` too, and never reaches this route: a client sends
it to `POST /back`, which answers whether anything went back. A key
press answers only that the event went in.
### `POST /tap-at-norm-coord`
Body: `{ nx, ny, times?, intervalMs?, holdMs?, doubleTap?, aimedBy? }` —
a point as shares of the app frame, and optionally a burst (`times`
touches `intervalMs` apart) or a held touch (`holdMs`). A tap, a double
tap (`times: 2, doubleTap: true`) and a long press (`holdMs: duration`)
are all this route. On iOS the touches of a burst are delivered one at
a time on their schedule, no closer than one delivery (about 280 ms);
`doubleTap` sends both in one event instead, close enough together to be
one gesture. The Android runner does not read `doubleTap` (its double
tap is `/double-tap-at-norm-coord`).
`aimedBy` is the tree the point was aimed from, `accessibility` or
`semantics`; absent means `accessibility`.
Response: `{ ok, chain: [{ identifier, label, frame }], complete?,
reader?, readerError?, x?, y?, latestDownOffsetMs?, earliestUpOffsetMs?,
handlerWallMs? }`. `chain` is what was under the point, innermost first,
read from the tree `aimedBy` named — `reader` says which. A runner that
cannot read that tree answers with no chain and `readerError`, and never
with the other tree's chain in its place: the host judges the tap
against the tree it aimed from, and a chain from another is a verdict on
some other aim. A runner with one tree (iOS) sends no `reader`, and its
chain is its accessibility tree's. `x`/`y` are the point in the
display's pixels. The three offsets come with a single touch and bound
when it was down, measured from the handler's entry — bounds, not
instants.
The Android runner reads `times`, `intervalMs` and `holdMs` on this route
as the iOS runner does: the point is resolved once and touched `times`
times. The Android runner's `/double-tap-at-norm-coord` and
`/long-press-at-norm-coord` take `aimedBy` and answer `reader` /
`readerError` the same way.
### `POST /swipe`
Body: `{ direction: SwipeDirection }` — variants: `up`, `down`, `left`, `right`.
Response: `{ ok: bool }`
### `POST /scroll` — retired
Gone as of 10.2. Scrolling to an element is one loop on the host: it
reads the tree, decides whether enough of the target can be seen and
whether it has stopped moving, and sends `/swipe-once` until it has.
The runner-side loop this route drove was a second implementation of
the same thing, with its own idea of "visible", and nothing had called
it since the host's three loops were unified — a route that exists and
that nobody walks reads, to anyone writing a client, like a path.
### `POST /hide-keyboard`
Body: `{ budgetMs?: number }`
Response: `{ ok: bool, error?: string, saw?: string }`
`budgetMs` is how long the host will wait. The iOS runner starts no
dismissal strategy once it is spent and answers with what it has; the
host sends 20000 and waits that long plus a margin. Without it the
dismissal is not bounded. `ok:false` carries `error`:
`keyboard_did_not_close` (the keyboard was there and is still there —
`saw` lists the strategies tried, and says so when the budget ran out
first) or `keyboard_state_unknown` (looking failed; not evidence the
keyboard is up). The Android runner ignores `budgetMs`.
### `POST /back`
Body: `{}` — empty.
Response: `{ ok: bool, settledBy?: string, saw?: string }`
`ok:false` here means the runner tapped, gestured, and did not observe
the screen change within its budget. It is a statement about what was
observed, not a proof that nothing moved — treat it as a refusal to
confirm rather than as evidence of a stuck screen.
`settledBy` names which branch decided: `titleChanged` (the navigation
bar's identifier moved), `sustainedAbsence` (no bar for long enough to
mean the destination has none), `noIdentity` (there was no title to
watch, so a fixed settle was used and nothing was verified), or
`gaveUp`. `saw` accompanies a refusal and carries the readings behind
it — whether a back button was there, the title before, the last title
read, and how many consecutive frames had no bar. Both fields are
diagnostic: an implementation may ignore them, and neither changes `ok`.
### `GET /screenshot`
Response: raw PNG bytes (`Content-Type: image/png`). A 503 means
`XCUIScreen` produced nothing, which is not the same as a blank screen —
it carries `error` and `reason`, and callers are expected to say so
rather than hand back an empty image.
Both platforms serve it, and `tap --then-screenshot` takes its frame
from it on both: the picture comes from the process that just acted.
The Android runner served no such route until 10.2, so a runner older
than that answers `501 not_implemented` — which callers are expected to
report as "this runner predates the route, bring it up again" rather
than quietly photographing the screen with something else.
### `POST /foreground`
Body: `{ bundle: string }` — sends to system `XCUIApplication(bundleId).activate()`.
Response: `{ ok: bool }`
### `GET /system-popups`
Response: `{ popups: [{ id, type, source, title, body, buttons: [{ id,
label, role, dangerous, outcomeHint? }] }] }`. `source` is who owns the
window — a bundle id on iOS, a package name on Android.
A popup is a window that **belongs to somebody other than the app under
test, took the focus, and carries something a caller can press**. All
three matter, and each was learned from a window that broke one of
them:
- The navigation bar carries four clickable, named buttons (Back, Home,
Overview, Switch input method) and is over every app at all times. It
never takes the focus, and it is not a popup.
- The keyboard is never a popup whatever is drawn on it. Its verb is
`hideKeyboard`.
- An app's own dialog is answered by `/tree`, where the flow's own
selectors can name its buttons.
Which app is "the app under test" comes from the `App-Bundle-Id` header
every smix client sends, or `?app=<pkg>`. **A caller who sends neither
gets every focused window carrying buttons, their own dialogs
included** — the alternative was guessing, and the guess that was tried
first ("the focused application window") took the permission dialog for
the app and reported nothing at all.
### `GET /health`
Response: `{ ok: true, version: string, uptime_ms: u64 }`
## Error envelope
Failures return HTTP 4xx/5xx with:
```json
{
"ok": false,
"error": "<code>",
"message": "<human-readable>"
}
```
Codes: `not_found`, `snapshot_unavailable`, `app_unavailable`,
`invalid_request`, `runner_error`.
`no_focused_field` carries two more fields, from one walk of the window
stack: the `message` names any foreign window holding the focus, and
`windows` is the whole stack as `[{ package, type, kind, layer }]`. A
fill that finds no field is most often a fill with something else on
top, and the sentence that used to come back described only the focus.
When the app under test has no window in the stack at all, the message
says that too.
## Wire selector schema
```json
// text-family
{ "text": "Sign In", "modifiers": {...}? }
// id-family
{ "id": "qa-submit", "modifiers": {...}? }
// label / role / anchor / focused
{ "label": "..." }
{ "role": "button", "name": "..." }
{ "anchor": { "text": "..." }, "below": {...} }
{ "focused": true }
```
## Modifier schema (extended selectors)
`{ near, below, above, leftOf, rightOf, inside, ancestor, nth,
first, last }` — see the `smix-selector` crate.
## Include scope
Optional query param `?include=all-windows` extends element resolution
to see-through modal overlays.
## Compatibility promise
- v1.0 client + v1.x runner: compatible
- v1.x client + v1.0 runner: compatible
- v1.0 client + v2.0 runner: not guaranteed
Any breaking change bumps the major version.
<!-- ===== docs/ai-guide/abi-stability.md ===== -->
# ABI stability — smix v1.0.0
> The following crates expose smix's public, business-agnostic
> primitives. Their public API is frozen at v1.0.0; breaking
> changes require a major bump.
## Frozen crates
| crate | primary types | wire? | Notes |
|---|---|---|---|
| `smix-error` | `ExpectationFailure`, `FailureCode`, `FailureInit` | yes (RunReport JSON) | AI-readable failure surface |
| `smix-selector` | `Selector`, `Modifiers`, `Anchor` | yes (all routes) | Central selector algebra |
| `smix-screen` | `A11yNode`, `Rect`, `Bounds`, `Role`, `ElementSummary` | yes (`/tree` route) | A11y tree wire shape |
| `smix-runner-wire` | HTTP schemas | yes | Wire schema types |
| `smix-input` | `SwipeDirection`, `KeyName` | yes (input routes) | Input primitives |
| `smix-verbs` | `VerbEntry`, `VerbCategory`, `ArgShape`, `VERB_TABLE` | yes (indirectly via parser + migrate) | Canonical verb table |
| `smix-metro-log` | `MetroLogTail`, `SignalMatcher`, `Window`, `AwaitError` | no | Signal await surface |
| `smix-fixture` | `FixtureRegistry`, `FixtureDecl`, `SignalMatcher` | no | Fixture registry |
| `smix-annotate` | `Annotator`, `Annotation`, `Color`, `Position`, `Compression` | no | Annotation primitives |
| `smix-migrate` | `Migrator`, `MigrateReport`, `MigrateError`, `Rename` | no | YAML codemod |
## Non-frozen crates
- `smix-driver` — internal driver dispatch; can add methods
- `smix-sdk` — SDK app builder; can add builder methods
- `smix-adapter-maestro` — adapter runtime; can add `Step` variants
- `smix-cli` — command surface; can add subcommands + flags
- `smix-simctl` — `simctl` wrapper; can add commands
## ABI compatibility rules
The following changes are **compatible** within v1.x:
- Adding new methods to public traits (with default impls)
- Adding new variants to `#[non_exhaustive]` enums
- Adding new fields to `#[non_exhaustive]` structs
- Adding new items to modules
- Widening trait bounds on generic methods
- Adding new re-exports
The following are **breaking** (require v2.0):
- Removing or renaming any public item
- Adding new required trait methods
- Adding new required struct fields
- Narrowing trait bounds
- Changing existing method signatures
## Semver verification
Use `cargo-semver-checks` to gate CI on the frozen crates' ABI:
```bash
for c in smix-error smix-selector smix-screen smix-runner-wire \
smix-input smix-verbs smix-metro-log smix-fixture \
smix-annotate smix-migrate; do
cargo semver-checks check-release --package $c
done
```
Any breaking change surfaced in CI blocks a v1.x release; it must bump to v2.0.
## `#[non_exhaustive]` enums
All public enums are marked `#[non_exhaustive]` at the v1.0 freeze:
```rust
#[non_exhaustive]
pub enum ExpectationFailureCode { ... }
#[non_exhaustive]
pub enum Selector { ... }
// ...
```
This allows adding new variants in v1.x without breaking downstream
consumers who exhaustively match.
<!-- ===== docs/ai-guide/verb-parity.md ===== -->
# smix verb parity — cross-platform + tier
> What each smix YAML verb does on the iOS Simulator and on the Android
> emulator. `smix_verbs::VERB_TABLE` lists the verbs; this page says what
> they do, and each row was checked against the code that runs it.
## Tier legend
- ✅ — supported
- ⚠️ — supported, with the caveat stated on the row
- ❌ — not supported; the row says what to do instead
A verb marked ❌ on a platform **returns an error there**. None of them
succeed quietly: a flow that clears the keychain does it so the next step
meets a signed-out app, and reporting success without doing it hands that
step a signed-in one and blames the step.
### Where the two platforms differ underneath
Selectors resolve in different places. On iOS the runner resolves them and
acts in one call (`/find`, `/tap`, `/fill`). On Android the host resolves
against the tree and acts by coordinate (`/tap-at-norm-coord`,
`/input-text`). The verbs behave the same; the route lists do not match, and
that is why.
## Tap family
| verb | iOS | Android | notes |
|---|---|---|---|
| `tapOn` / `tap` | ✅ | ✅ | Selectors resolved via a11y tree; native tap dispatch; `fallback:` chains containing `ocrText` poll for `SMIX_TAP_OCR_POLL_MS` (default 3000 ms) |
| `doubleTapOn` / `doubleTap` | ✅ | ✅ | Resolved on the host and judged like `tapOn`: a double tap delivered to something else fails `TAP_MISSED`. iOS sends both touches in one synthesised event, 80 ms apart; Android two clicks 150 ms apart |
| `repeatTap` / `tapOn: { repeat }` | ✅ | ✅ | Found once and held still, then every touch goes in one request, so the interval is the number you state (on iOS no shorter than one touch takes to deliver, about 280 ms). The first touch is judged like `tapOn` (a miss fails `TAP_MISSED`); the later ones are not, since the screen may change after the first |
| `longPressOn` / `longPress` | ✅ | ✅ | 500 ms by default (maestro's documented 0.5s); `{ duration: N }` sets it. Resolved on the host and judged like `tapOn` on both platforms |
| `tapOn: { point: "X%,Y%" }` | ✅ | ✅ | Normalized [0, 1] coordinates; the escape hatch for screens with no a11y semantics. Not a verb of its own — there is no `tapByCoord` |
## Input family
| verb | iOS | Android | notes |
|---|---|---|---|
| `inputText` / `fill` | ✅ | ✅ | `--force-key-events` opt-in bypasses a11y-focus resolution for RN hidden-input patterns |
| `eraseText` / `clear` | ✅ | ✅ | iOS deletes proportionally to the field's own length; Android empties the focused node exactly (`ACTION_SET_TEXT`), falling back to bounded deletes for a field the tree cannot address |
| `pasteText` | ✅ | ❌ | Since Android 10 the clipboard serves only the focused app, and the runner cannot be focused while driving yours. Use `inputText` |
| `setClipboard` | ✅ | ❌ | Same clipboard restriction as `pasteText`. On a registered physical iPhone it needs Xcode 27 (`devicectl device pasteboard`) |
| `copyTextFrom` | ✅ | ❌ | Same clipboard restriction as `pasteText`. Assert on what the app renders instead |
## Assert family
| verb | iOS | Android | notes |
|---|---|---|---|
| `assertVisible` / `expect` | ✅ | ✅ | Visibility check via a11y tree bounds + visible flag |
| `assertNotVisible` / `expectNotVisible` | ✅ | ✅ | |
| `extendedWaitUntil` | ✅ | ✅ | `timeout` field; polls at 250 ms; `ocrText` in `fallback:` fires OCR per iteration; auto-captures screenshot + tree JSON to `.smix/timeouts/` on timeout |
| `expect: { signal }` | ✅ | ✅ | Metro log signal; consumer configures `.smix/config.json` metroLog |
| `expect: { signals }` | ✅ | ✅ | Ordered / any-order variants |
| `expectLogClean` | ✅ | ✅ | Allowlist multi-source merge |
| `assertTrue` | ✅ | ✅ | Expression engine — `${output.name}`, `${env.NAME}`, arithmetic |
| `assertScreenshot` | ✅ | ✅ | Two comparisons, the same on both: `threshold` (default 5) is a 64-bit dhash distance; `thresholdPercentage` is maestro's share of matching pixels (RGB within 10%), and a size mismatch fails. Writing both is an error. **Differs from maestro**: a flow that writes neither compares by hash, where maestro compares pixels at 95. `cropOn` compares one element's region and the baseline is the cropped image. `mask:` regions count neither way; masks covering everything are refused. `label` / `optional` refused by name |
| `rememberBounds` | ✅ | ✅ | smix's own — maestro has no verb for it. Keeps where an element is, under a name, in device-independent pixels (points on iOS; pixels ÷ density on Android). Takes any selector except `ocrText` and `anchorRelative`, which name no box to measure |
| `assertBoundsUnchanged` | ✅ | ✅ | smix's own — maestro has no verb for it. The element's box now matches the one `rememberBounds` kept under `was`, every edge within `within` device-independent pixels (default 0). A failure prints both boxes and how far each edge moved |
| `neverVisible` | ✅ | ✅ | smix's own — maestro has no verb for it. Runs the steps under `during` and, beside them, keeps asking the question `assertNotVisible` asks, as fast as the device answers; one sighting fails it, with the time since the span began and the inner step that was running. A pass says how many times it looked and the longest stretch nobody was looking. Refuses `ocrText` and `anchorRelative` for the same reason `assertNotVisible` does |
## Control flow
| verb | iOS | Android | notes |
|---|---|---|---|
| `runFlow` | ✅ | ✅ | Path resolution: cwd → `std/` catalogue |
| `runFlow: { when, commands }` | ✅ | ✅ | Inline conditional; `when` takes `platform` / `true` / `visible` / `notVisible` / `label`, combined with AND in maestro's order; OCR fires when a gate selector contains `ocrText`; `env` / `label` / `optional` on the block; unknown keys are parse errors; skips emit `SKIPPED: <reason>` to stderr |
| `retry` | ✅ | ✅ | `maxRetries` field; default 3 |
| `repeat` | ✅ | ✅ | `while:` takes the same conditions as `runFlow.when`; `label` / `optional` on the block |
| `pressKey` | ✅ | ✅ | One key table for the flow, `smix press-key` and MCP, read when the flow is read: maestro's spellings, the wire names, shorthands. enter/return, delete, tab, space, escape, and the four arrows on both. `back` is the `back` verb below. lock / volumeUp / volumeDown are pressed on Android (`KEYCODE_POWER` and the volume keys); on iOS the step fails with `no_such_button` and the reason — no lock button in XCUIDevice, no volume buttons on the simulator. maestro's TV remote keys are refused by name; `Power` points to `lock` |
| `back` | ✅ | ✅ | Navigation back — iOS nav-bar back / edge swipe, Android KEYCODE_BACK. `pressKey: back` and `smix press-key back` are this, not a keystroke. Closes a system share sheet under gesture navigation, where there is no back button to tap. Both platforms answer whether the screen changed, not whether the key was delivered, and both report which reading decided (`settledBy`); a back an app swallows is a failure |
## Lifecycle
| verb | iOS | Android | notes |
|---|---|---|---|
| `launchApp` | ✅ | ✅ | `clearState`, `clearKeychain`, `arguments`, `permissions` |
| `stopApp` / `terminate` | ✅ | ✅ | |
| `killApp` | ✅ | ✅ | |
| `clearState` / `reset` | ✅ | ⚠️ | Android clears via `pm clear`, which also reverts the app's runtime permissions — app data is app-private, so the host has no way to wipe one without the other. iOS clears the sandbox and privacy separately |
| `clearKeychain` / `resetKeychain` | ✅ | ❌ | Credentials live in each app's own KeyStore, out of the host's reach. Use `clearState` (a full `pm clear`), or have the app expose a sign-out path — `clearAppData` also errors on Android |
| `clearUserDefaults` | ✅ | ❌ | v1.0.27 — per-key NSUserDefaults deletion via `simctl spawn defaults delete`; Android SharedPreferences has no host-side per-key path (explicit error; use `clearState` for a full wipe — `clearAppData` is iOS-only) |
## Media
| verb | iOS | Android | notes |
|---|---|---|---|
| `takeScreenshot` | ✅ | ✅ | Long form with `annotate: [...]` (5 primitives) + auto-mkdir + PNG ext inference; `cropOn:` writes only that element (a baseline for `assertScreenshot` `cropOn`). Other keys, `label` / `optional` included, are refused by name |
| `startRecording` | ✅ | ⚠️ | Android records on the device with `screenrecord`, whose `--time-limit` help calls 180 s the maximum, not a default to raise. iOS has no such cap |
| `stopRecording` | ✅ | ✅ | Android interrupts `screenrecord` rather than killing it — the mp4's moov atom is written on interrupt, and without it the file will not play — then pulls the file |
| `addMedia` | ✅ | ✅ | Android pushes to `/sdcard/Pictures/` and fires a media-scan broadcast. Landing the bytes is not enough: a file MediaStore has not indexed is invisible to the app |
## Gesture
| verb | iOS | Android | notes |
|---|---|---|---|
| `scroll` | ✅ | ✅ | |
| `scrollUntilVisible` | ✅ | ✅ | One host-side loop on both: swipe, look (tree, then each `ocrText`), stop when the element is wholly on screen and has stopped moving. `visibilityPercentage` / `centerElement` / `timeout` / `label` / `optional` read; `speed` / `waitToSettleTimeoutMs` refused by name |
| `swipe` (`direction:` or `start:`/`end:` or `from:`/`to:`) | ✅ | ✅ | Absolute + relative coord shapes |
| `hideKeyboard` | ✅ | ✅ | |
## Device
| verb | iOS | Android | notes |
|---|---|---|---|
| `openLink` / `openUrl` | ✅ | ✅ | System URL handler |
| `setLocation` | ✅ | ✅ | Android sends `geo fix` on the emulator console. The fix persists and replays when an app starts listening, so setting it early is not a race. On a registered physical iPhone it needs Xcode 27, and the location stays until `xcrun devicectl device simulate location clear --device <UDID>` |
| `travel` | ✅ | ⚠️ | iOS hands the route to CoreSimulator. Android has no route primitive — the emulator console takes one position at a time — so smix walks it from the host, one `geo fix` a second. Both return immediately and travel in the background. A registered physical iPhone takes the route through `devicectl` (Xcode 27), with the same caveat as `setLocation` |
| `setPermissions` | ✅ | ✅ | `pm grant` / `pm revoke` per permission on Android; `simctl privacy` on iOS |
| `setOrientation` | ✅ | ⚠️ | An app that has locked its orientation stays where it is; neither platform reports that as a failure. Android reads the display's rotation back before answering, so a rotation that does not arrive is a failure rather than a silent no-op |
## smix-native extensions
These are verbs — write them in a flow.
| verb | iOS | Android | notes |
|---|---|---|---|
| `fixture` | ✅ | ✅ | JSON registry OR TS registry |
| `webview_eval` / `webviewEval` / `webViewEval` | ✅ | ✅ | RN WebView / native WebView bridge |
| `clearLocation` | ✅ | ⚠️ | The way back from `setLocation` / `travel`, which outlive the flow that ran them — maestro has no verb for it. iOS clears through `simctl location clear`, a registered iPhone through `devicectl device simulate location clear`. On Android it stops the route smix is walking and leaves the device where it stands: the emulator console has no inverse of `geo fix`, and an emulator has no real position to be given back |
### Coordinates and OCR are not verbs
Coordinate taps, coordinate swipes and OCR are capabilities, reached
through the verbs and selectors that already exist. They were listed
here as verbs once; a flow that wrote `tapById:` or `tapAtCoord:` got
`unsupported command`.
| capability | iOS | Android | how you write it |
|---|---|---|---|
| tap by id | ✅ | ✅ | `tapOn: { id: "btn" }` — the id path skips OCR and the a11y walk |
| tap at a coordinate | ✅ | ✅ | `tapOn: { point: "50%,80%" }` — normalized 0..1 |
| swipe between coordinates | ✅ | ✅ | `swipe: { from: …, to: … }`; on the CLI `smix swipe --from 50%,80% --to 50%,20%`, and `swipe_from` / `swipe_to` through MCP |
| find text by OCR | ✅ | ✅ | the `ocrText` selector, below — the `find_text_by_ocr` wire route has no verb of its own |
### Selector forms
Written inside a selector, not as a step.
| form | iOS | Android | notes |
|---|---|---|---|
| `ocrText` | ✅ | ✅ | Vision framework (iOS) / ML Kit (Android) |
| `anchored` (alias `anchorRelative`) | ✅ | ✅ | Selector-relative anchoring |
## Utility
| verb | iOS | Android | notes |
|---|---|---|---|
| `waitForAnimationToEnd` | ✅ | ✅ | Bare form compares frames until the screen holds still. A screen that never settles — a spinner, a caret — is not a failure; the wait just ends at its ceiling. `: N` / `{ timeout: N }` sets that ceiling |
| `evalScript` | ❌ | ❌ | Errors unconditionally on both platforms ("a complete JS runtime is not supported") with an `assertTrue` pointer. No debug-bridge path exists |
| `runScript` | ❌ | ❌ | Sibling of `evalScript`; same unconditional error |
| `clearAppData` | ✅ | ❌ | iOS session-scoped in-place wipe (cooperative terminate → sandbox rm → relaunch). Android errors — use `clearState` |
| `resetAppData` | ✅ | ✅ | App-owned URL-scheme wipe via `openurl` / `am start VIEW`; `waitFor.logLinePattern` needs `--metro-log` on both |
| `assertCondition` | ✅ | ✅ | Host-side AI judge over a screenshot (local `claude` CLI); platform-independent |
| `extractWithAI` | ✅ | ✅ | Same host-side AI lane, writes into `output.*` |
## Names in the table that are not verbs
There are none. Eleven rows in `VERB_TABLE` once named things the parser
never dispatched, and every one is settled: ten were deleted — `ocrText` and
`anchorRelative` are selector fields, `tapAtCoord` is `tapOn: {point}`,
`tapById` is `tapOn: {id}`, `toggleAirplaneMode` was implemented nowhere —
and `back` now parses directly.
Deleting those rows is what made `doubleTap` and `longPress` start working:
a row whose maestro and smix names are identical shadows the alias when the
parser normalizes a verb, so the name never reached the lookup that would
have mapped it onto `doubleTapOn`. The row promising the verb was what
stopped it.
A test in the adapter reads the parser's dispatch out of the source and
compares it with the table in both directions, so neither can drift from the
other in silence.
## Not supported
- `fillAtCoord` — no coordinate escape hatch for typing; `tapAtCoord` is the
only one, by design
- Real devices — the simulator and the emulator only
- One log-signal syntax across platforms — each platform's log tail is read
on its own terms
<!-- ===== docs/migrating-to-4.md ===== -->
# Migrating to smix 4.0
Device records moved. Where a simulator's UDID lives, who is recorded as
holding it, and which port a runner has on it are now facts about **the
machine**, not about the checkout you happen to be standing in.
If you drive smix by hand or from YAML flows, there is **nothing to
undo** — run `smix sim migrate` and `smix lease migrate` once and carry
on. Everything below is for code that calls the Rust crates.
---
## Why it moved
A simulator is an operating-system object. Its UDID, its runtime version,
whether it is booted and who booted it do not change when you `cd`.
They were stored in whichever `.smix/` sat above the working directory,
so a machine with four checkouts held four answers about the same
simulators. Measured on one machine: a runner was found holding port
22087 with no record of it. The rule is to find a runner's owner before
touching it; the check came back empty, so it could neither be confirmed
an orphan nor stopped. It was on the books the whole time — in another
workspace's books.
---
## 1. Your existing records keep working, once
Nothing is lost and nothing is moved out from under you. The old
locations are still read; the migrations copy and never remove.
```bash
smix sim migrate --dry-run # say what would move
smix sim migrate # move it
smix lease migrate # the same, for who-holds-what
```
Run them from each checkout that has a `.smix/`, or name the trees with
`--from <DIR>`. Running either twice does nothing the second time, so
"run it again if you are not sure" is safe advice.
Until you do, a device only one checkout knows about is named as such
every time you list devices — no other tree can see it.
To check where you stand:
```bash
smix sim list --registered # every recorded device, and whose book it is in
smix runner list # every runner on this machine, and who knows about it
```
---
## 2. `smix_lease::store` takes a `LeaseDir`, not a path
**Before**: every ledger function took a workspace root and appended
`.smix/leases` to it itself.
```rust
smix_lease::store::write(&workspace_root, &lease)?;
let lease = smix_lease::store::read(&workspace_root, udid)?;
```
**Now**: they take the ledger directory as a type.
```rust
let leases = smix_lease::store::LeaseDir::machine()
.ok_or("no HOME or XDG_DATA_HOME")?;
smix_lease::store::write(&leases, &lease)?;
let lease = smix_lease::store::read(&leases, udid)?;
```
`LeaseDir::machine()` is the answer you want in a program. `LeaseDir::at`
exists for tests, where a temporary directory stands in for a whole
machine.
This is a type rather than a path on purpose. When the first argument
changed meaning, twenty-five call sites inside smix went on compiling and
went on writing device facts into checkouts — nothing but a type could
have caught that, and nothing but a type can catch it in your code
either.
`store::lease_dir(root)` is gone. It existed to build `.smix/leases` from
a workspace root, which is the thing that no longer happens.
Reading a checkout's old book — to report a divergence, never to act on
one — is `CheckoutLedgers`:
```rust
use smix_lease::store::CheckoutLedgers;
if let Some(book) = CheckoutLedgers::discover(&cwd) {
for id in book.device_ids() { /* book.read(&id)? */ }
}
```
It has no write path, deliberately: what a checkout holds was written by
whichever smix that tree last ran.
---
## 3. Two SDK calls take the ledger directory
`Leased::acquire` and `App::hold_device_lease` each gained one argument,
in the same position and for the same reason: the tree they were given
used to mean two things — where the ledger is, and where a dead holder's
build products would be settled. Only the second is still a tree.
```rust
// before
Leased::acquire(&control, &workspace_root, udid, &executor)?;
app.hold_device_lease(&workspace_root, udid, &reconciler)?;
// now
let leases = smix_lease::store::LeaseDir::machine().unwrap();
Leased::acquire(&control, &workspace_root, &leases, udid, &executor)?;
app.hold_device_lease(&workspace_root, &leases, udid, &reconciler)?;
```
---
## 4. Two things smix now refuses that it used to do
Neither needs a code change. Both change what happens on a machine
somebody else is also using, so they are worth knowing.
**A live holder is no longer reclaimed for going quiet.** The heartbeat
is written when the ledger is touched, so a holder that takes a device
and then serves requests for hours is silent by design. Ninety seconds of
silence used to make its lease reclaimable; a long-lived MCP session was
exactly that shape. `smix lease reconcile` now reports it as held.
`StaleReason::HeartbeatExpired` remains as a name and is never produced.
If you relied on that to clear a genuinely wedged holder, end the process
— which you can establish and smix cannot — and the ordinary
holder-is-gone path settles it.
**`smix down` leaves a live holder alone.** It used to close every ledger
it found, which was right while the ledgers were per checkout. The
directory is the machine's now, and it holds other people's sessions. It
closes what is no longer held, or what this process holds, and names the
rest.
---
## 5. Nothing falls back to the working directory any more
A device record used to fall back to `.smix/sims.json` beside you when
the machine location could not be resolved. It now says so and stops.
There is no good place to put such a record silently — one written where
nobody else reads it is the failure this release is about.
Set `SMIX_MACHINE_DIR` if you need to point smix's machine data
somewhere else; it moves all of it together.
<!-- ===== docs/migrating-to-3.md ===== -->
# Migrating to smix 3.0
Three things in 3.0 change what code you already have does. Everything
else is additive, and this page is only about those three.
Read it if you have flows, scripts, or an SDK integration written
against 2.x. If you drive smix by hand, all three make smix do what it
already said it did, and there is nothing to undo.
---
## 1. `fill` replaces the field it names
**Before**: filling a field typed on the end of whatever was already
there. Returning to a form and filling the same field again left both
values concatenated.
**Now**: **you can only replace a field you named.**
| what you write | 2.x | 3.0 |
|---|---|---|
| `fill(id:email)` / `inputText: {id, text}` | appends | **replaces** |
| `inputText: "text"` (scalar, no field named) | appends | appends |
| `pasteText` | appends | appends |
| `App::fill(&focused(), …)` | appends | appends |
Typing into whatever holds focus still appends, because there is no
named field to empty — and that is also what maestro's verbs of that
shape do, so a flow ported from maestro still means what it meant.
### What to check in your flows
Search for a field filled twice without a clear between:
```bash
grep -n -A3 'inputText:' your-flows/*.yaml | grep -B1 'id:'
```
Three shapes, and what to do with each:
- **Filled once.** Nothing to do — the field was empty, and emptying an
empty field changes nothing.
- **`eraseText` then `inputText: {id, …}`.** Still correct. The
`eraseText` is now redundant, not wrong; leaving it costs a round
trip.
- **Two `inputText: {id, …}` on the same field, expecting the values to
join.** This is the one that changes. Rewrite it as one step with the
whole value, or tap the field and use the scalar form for the second
part:
```yaml
# 2.x: relied on appending
- inputText: { id: "search", text: "hello " }
- inputText: { id: "search", text: "world" }
# 3.0: say what the field should hold
- inputText: { id: "search", text: "hello world" }
# 3.0: or keep the two steps, appending deliberately
- tapOn: { id: "search" }
- inputText: "hello "
- inputText: "world"
```
### Why the default flipped rather than gaining a flag
The guides have described this verb as replacing since it existed
("Fill — replaces focused field content"). The implementation appended.
In a password field the difference is invisible — the dots look right —
so it surfaces as a login rejecting a correct password, which is what it
cost the person who reported it.
A flag would have left that bug in place for everyone who did not know
to set it. The wire carries `clearFirst` on `POST /fill` and it defaults
to true; a runner too old to know the field appends, which is what it
did before, so the field is additive on the wire.
---
## 2. A selector naming two things means both
**Before**: `{ id: X, text: Y }` parsed to `Text { Y }` and the id was
dropped. It matched **any** element reading Y.
**Now**: it matches the one element that is both — the id decides which
element, the text narrows it.
```yaml
# examples/hello.yaml, unchanged, and now asserting what it says
- assertVisible:
id: "home-counter-label"
text: "1"
```
The form was already written this way in these guides and in flows;
what changed is that it means it. `id` > `label` > `text` decides which
key becomes the form, and any verb taking a selector reads it alike.
### What to check
- **An assertion that passed because *something else* on screen carried
the text.** It fails now, and it was never checking what it looked
like it was checking. This is the change most likely to turn a green
flow red, and the red is the honest answer.
- **`{ id, text }` where the text is a stale copy.** A label that was
edited without the flow being updated used to be ignored; now it is
part of the match.
- **`role` + `name` is unaffected** — one form spelled with two keys,
not a conjunction.
To go back to the looser check, drop the key you did not mean:
`{ text: "1" }` asserts something reads 1, `{ id: "counter" }` asserts
the counter is there.
## 3. `describe` and `tree` leave out the keyboard's keys
**Before**: every key of the software keyboard appeared as its own
element — a summary per letter, plus `Next keyboard`, `Dictate`, shift
and delete. Around sixty of them, the same sixty on every screen of
every app.
**Now**: `describe` never enumerates them. `tree` collapses them and
prints how many it left out; `smix tree --keyboard` includes them.
**The keyboard element itself still appears.** A keyboard covering the
thing you wanted to tap is the explanation for a failure, and hiding it
would turn a legible failure into a mystery. Only the keys go.
### What to check
- **Counting elements from `describe --json`.** The count drops on any
screen with the keyboard up. If you were using it as a fingerprint,
it is a different fingerprint now.
- **Selecting a key by label** (`text:a`, `label:return`). Use
`pressKey` / `smix press-key`, which names keys directly and does not
depend on the tree at all.
- **A tree snapshot committed as a fixture.** Regenerate it, or pass
`--keyboard` to keep the old shape.
---
## Rust API
Two crates changed shape. This affects you only if you depend on them
directly — `smix-sdk`'s `App` is unchanged.
```rust
// smix-driver: the Driver trait
async fn fill(&self, selector: &Selector, text: &str,
include: Option<IncludeScope>,
clear_first: bool) -> Result<(), ExpectationFailure>;
// smix-runner-client
pub async fn fill(&self, selector: &Selector, text: &str,
include: Option<IncludeScope>,
clear_first: bool) -> Result<RunnerKeyboardResult, RunnerTransportError>;
```
Pass `true` for the 3.0 behaviour. `App::fill` derives it from whether
the selector names a field, which is the rule above expressed once.
`smix-runner-client` also gained `clear_text()` for the Android runner's
`POST /clear-text`, and `KNOWN_UNAVAILABLE_CATEGORIES`, which now
includes `not-running`.
Discussion
Did this work in your project? Say what you used it for and what you changed. People and their agents can both post here.
Posts are public.Sign in to post
No one has posted yet. Be the first.