drive-a-mirrored-iphone
Skill22 starsChanged 3 months agoApache-2.0View on GitHub
Drive a physical iPhone through the mirror backend (iPhone Mirroring + mirrord) - which action kinds work, long-press/wheel-scroll, OCR observe semantics (capture_scale, interactable as compatibility constant, interactive_only no-op), coordinate spaces, and testing without a Mac via fakeMirrord and the mock daemon. Use when a task targets a mirror-backed device or extends the mirror action layer.
In this repository8 other files it gives its agents.
- broker-federationSkill
- build-and-testSkill
- journal-evidence-exportSkill
- lease-and-drive-a-simSkill
- park-thaw-semanticsSkill
- run-evalsSkill
- run-the-daemonSkill
- ui-audit-and-predicatesSkill
The file
We split this skill into the rules it gives an agent
4 rules. About 0 are common, 0 are distinctive, and 4 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈0
Distinctive · estimate
This author's own choices. Worth a look.
4
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 15
ALL mirror coordinates (tap/swipe/scroll input AND observed frames) are
- Very specific: line 62
Use" panel. Poll for state:"ready" instead of guessing.
- Very specific: line 63
After a timeout panel, one tap on the on-screen Connect button (≈ centre-x, y≈476 in a 318x701 window) resumes the session.
- Very specific: line 68
Every input result carries verified + verify_delta (pixel-delta).
≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Drive a mirrored iPhone (mirror backend)
Mirror-backed physical devices are driven by injecting CoreGraphics input
into the iPhone Mirroring window via the helpers/mirrord Swift helper
(Unix-socket HTTP/JSON). The Go side lives in
internal/actions/device_mirror.go + internal/actions/mirror/client.go.
Coordinate space and scale
- ALL mirror coordinates (tap/swipe/scroll input AND observed frames) are
capture-image pixels, not device points.
observe/screenshotresults carryimg_w/img_h(the tap coordinate space) andcapture_scale— the capture's pixels-per-window-point factor (≈2.0 on a HiDPI/Retina host, 1.0 headless; 0/absent means the helper predates the field). Divide bycapture_scaleto get window points. - Size-capped screenshots still report the pre-transform
img_w/img_h, so map detected pixels back through those before tapping.
Action kinds specific to mirror
- Long-press:
tap {x,y,duration_ms}withduration_msin 1..10000 (the helper is single-threaded and sleeps the whole press — an unbounded press would wedge every later action). Sim (touch-down → delay → touch-up) and WDA (W3C pointer chain) implement the same field with the same bound. - Wheel scroll:
scroll {x,y,dy}(dynon-zero; positive scrolls content down). Mirror-only — it's the trackpad-scroll analog for fine list movement without swipe momentum. Sim/WDA returnnot_implemented(useswipe). Note velocity swipes can't advance paged feeds like TikTok (#190) — wheel scroll doesn't either; that needs the phone-harness movement work (#196). scrollandtypeneed focus, everything else doesn't (real-phone findings, #241/#244):tap/swipe/buttonall land ininput_mode: backgroundwith iPhone Mirroring behind other windows, butscrollfailsunavailable("could not be brought frontmost") whenever another app holds focus on a shared Mac — fall back to a shortswipe.typedegrades automatically: background key events reach the mirroring process (Cmd shortcuts work) but iOS routes keys into text input only while the window is frontmost, so the helper briefly activates the window, types, and hands focus back best-effort — the ack saysinput_mode:"focused-degrade"with aninput_note. Expect the mirroring window to steal focus for the duration of atypeeven in background mode.- The iOS on-screen keyboard never appears during
type— iPhone Mirroring attaches a virtual hardware keyboard, so iOS suppresses the soft keyboard by design. A missing keyboard is NOT a focus failure; verify typing by the ack'sverify:"ocr-delta"/pixel delta or by re-observing the field. - CLI:
manzanas tap X Y --duration-ms N,manzanas scroll X Y DY.
Getting a session started (real phone)
GET /v1/statuson the mirrord socket is the health check that matters:state:"blocked"+marker:"iphone in use"means the phone is unlocked in someone's hand — mirroring will NOT connect until it is locked, and every action returns a screenshot of the "iPhone in Use" panel. Poll forstate:"ready"instead of guessing.- After a timeout panel, one
tapon the on-screen Connect button (≈ centre-x, y≈476 in a 318x701 window) resumes the session. - Launch apps by Spotlight, not
launch_app(no devicectl on a mirror target):button home→tapthe Search pill →type "<app>"→tapthe Top Hit. - Every input result carries
verified+verify_delta(pixel-delta). Averify_deltanear 0 withverified:truemeans the tap landed but the UI didn't change — usually a mis-aimed coordinate; re-screenshot before firing more taps.
Observe = OCR, and its honesty rules
observeruns Vision OCR over a window capture: flat "Text" leaves with label+frame only. No roles/ids/values/hierarchy — predicates can only match visible text. Icon-only buttons are invisible; fall back to coordinatetapfrom ascreenshot.- OCR nodes carry
interactable: trueas a uniform compatibility constant (v0 additivity), NOT a measurement — OCR can't tell a button from a caption, so never treat the flag as signal on mirror trees.interactive_only: trueconsequently returns the full tree, with an explanatory suffix onnote. Don't rely on that filter on mirror devices. wait_for_stablepixel-diffs captures directly (no OCR) — the cheap way to wait out animations; eachwait_tree_stablepoll costs a full OCR pass.
Developing/testing mirror changes without a Mac
- fakeMirrord (
internal/actions/device_mirror_test.go): in-process httptest server impersonating mirrord; recordscalls/bodyper request, serves cannedboxes, andtestMirrorBackend(t)wires it into a realDeviceBackend. Assert request bodies (e.g.duration_mspresent) and response plumbing (e.g.capture_scale) here — this is the required e2e smoke path for Go-only slices. - Mock daemon:
manzanasd --mock+ the CLI exercises routing and the sim-sidenot_implementedanswers (boot the leased mock target first or you gettarget_not_booted). Lease acquire returns the ID in theidfield. - Any
helpers/mirrordSwift change must at least compile on a real Mac: on the Mac mini claim thebuild-manzanaslock, build in a throwawaygit worktree(never touch the checked-out branch or the running LaunchAgent),cd helpers/mirrord && ./build.sh, then remove the worktree and release the lock.
Protocol discipline
- v0 changes must be ADDITIVE (CONTRIBUTING.md): never reject a
previously-accepted payload field or drop a response field outright —
prefer no-op +
note, or a new optional field. CI review enforces this on integration PRs. - New action kinds/fields go in
proto/PROTOCOL.md§5 anddocs/devices.mdin the same PR.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

