agentleFS
BariBariGoodBariBariGood/manzanas/.agents/skills/drive-a-mirrored-iphone/SKILL.md

drive-a-mirrored-iphone

Skill22 starsChanged 3 months agoApache-2.0View on GitHub

Drive a physical iPhone through the mirror backend (iPhone Mirroring + mirrord) - which action kinds work, long-press/wheel-scroll, OCR observe semantics (capture_scale, interactable as compatibility constant, interactive_only no-op), coordinate spaces, and testing without a Mac via fakeMirrord and the mock daemon. Use when a task targets a mirror-backed device or extends the mirror action layer.

The file

We split this skill into the rules it gives an agent

4 rules. About 0 are common, 0 are distinctive, and 4 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈0

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 4

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    ALL mirror coordinates (tap/swipe/scroll input AND observed frames) are

    line 15
  2. Very specific:

    Use" panel. Poll for state:"ready" instead of guessing.

    line 62
  3. Very specific:

    After a timeout panel, one tap on the on-screen Connect button (≈ centre-x, y≈476 in a 318x701 window) resumes the session.

    line 63
  4. Very specific:

    Every input result carries verified + verify_delta (pixel-delta).

    line 68

≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Drive a mirrored iPhone (mirror backend)

Mirror-backed physical devices are driven by injecting CoreGraphics input into the iPhone Mirroring window via the helpers/mirrord Swift helper (Unix-socket HTTP/JSON). The Go side lives in internal/actions/device_mirror.go + internal/actions/mirror/client.go.

Coordinate space and scale

  • ALL mirror coordinates (tap/swipe/scroll input AND observed frames) are capture-image pixels, not device points. observe/screenshot results carry img_w/img_h (the tap coordinate space) and capture_scale — the capture's pixels-per-window-point factor (≈2.0 on a HiDPI/Retina host, 1.0 headless; 0/absent means the helper predates the field). Divide by capture_scale to get window points.
  • Size-capped screenshots still report the pre-transform img_w/img_h, so map detected pixels back through those before tapping.

Action kinds specific to mirror

  • Long-press: tap {x,y,duration_ms} with duration_ms in 1..10000 (the helper is single-threaded and sleeps the whole press — an unbounded press would wedge every later action). Sim (touch-down → delay → touch-up) and WDA (W3C pointer chain) implement the same field with the same bound.
  • Wheel scroll: scroll {x,y,dy} (dy non-zero; positive scrolls content down). Mirror-only — it's the trackpad-scroll analog for fine list movement without swipe momentum. Sim/WDA return not_implemented (use swipe). Note velocity swipes can't advance paged feeds like TikTok (#190) — wheel scroll doesn't either; that needs the phone-harness movement work (#196).
  • scroll and type need focus, everything else doesn't (real-phone findings, #241/#244): tap/swipe/button all land in input_mode: background with iPhone Mirroring behind other windows, but scroll fails unavailable ("could not be brought frontmost") whenever another app holds focus on a shared Mac — fall back to a short swipe. type degrades automatically: background key events reach the mirroring process (Cmd shortcuts work) but iOS routes keys into text input only while the window is frontmost, so the helper briefly activates the window, types, and hands focus back best-effort — the ack says input_mode:"focused-degrade" with an input_note. Expect the mirroring window to steal focus for the duration of a type even in background mode.
  • The iOS on-screen keyboard never appears during type — iPhone Mirroring attaches a virtual hardware keyboard, so iOS suppresses the soft keyboard by design. A missing keyboard is NOT a focus failure; verify typing by the ack's verify:"ocr-delta"/pixel delta or by re-observing the field.
  • CLI: manzanas tap X Y --duration-ms N, manzanas scroll X Y DY.

Getting a session started (real phone)

  • GET /v1/status on the mirrord socket is the health check that matters: state:"blocked" + marker:"iphone in use" means the phone is unlocked in someone's hand — mirroring will NOT connect until it is locked, and every action returns a screenshot of the "iPhone in Use" panel. Poll for state:"ready" instead of guessing.
  • After a timeout panel, one tap on the on-screen Connect button (≈ centre-x, y≈476 in a 318x701 window) resumes the session.
  • Launch apps by Spotlight, not launch_app (no devicectl on a mirror target): button home → tap the Search pill → type "<app>" → tap the Top Hit.
  • Every input result carries verified + verify_delta (pixel-delta). A verify_delta near 0 with verified:true means the tap landed but the UI didn't change — usually a mis-aimed coordinate; re-screenshot before firing more taps.

Observe = OCR, and its honesty rules

  • observe runs Vision OCR over a window capture: flat "Text" leaves with label+frame only. No roles/ids/values/hierarchy — predicates can only match visible text. Icon-only buttons are invisible; fall back to coordinate tap from a screenshot.
  • OCR nodes carry interactable: true as a uniform compatibility constant (v0 additivity), NOT a measurement — OCR can't tell a button from a caption, so never treat the flag as signal on mirror trees. interactive_only: true consequently returns the full tree, with an explanatory suffix on note. Don't rely on that filter on mirror devices.
  • wait_for_stable pixel-diffs captures directly (no OCR) — the cheap way to wait out animations; each wait_tree_stable poll costs a full OCR pass.

Developing/testing mirror changes without a Mac

  • fakeMirrord (internal/actions/device_mirror_test.go): in-process httptest server impersonating mirrord; records calls/body per request, serves canned boxes, and testMirrorBackend(t) wires it into a real DeviceBackend. Assert request bodies (e.g. duration_ms present) and response plumbing (e.g. capture_scale) here — this is the required e2e smoke path for Go-only slices.
  • Mock daemon: manzanasd --mock + the CLI exercises routing and the sim-side not_implemented answers (boot the leased mock target first or you get target_not_booted). Lease acquire returns the ID in the id field.
  • Any helpers/mirrord Swift change must at least compile on a real Mac: on the Mac mini claim the build-manzanas lock, build in a throwaway git worktree (never touch the checked-out branch or the running LaunchAgent), cd helpers/mirrord && ./build.sh, then remove the worktree and release the lock.

Protocol discipline

  • v0 changes must be ADDITIVE (CONTRIBUTING.md): never reject a previously-accepted payload field or drop a response field outright — prefer no-op + note, or a new optional field. CI review enforces this on integration PRs.
  • New action kinds/fields go in proto/PROTOCOL.md §5 and docs/devices.md in the same PR.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.