device-and-ui
Skill9 starsChanged 6 months agoMITView on GitHub
Work with physical devices and UI automation. Use when implementing device commands, touch/swipe simulation, IDB integration, accessibility inspection, or editing files under src/core/devices.ts or src/core/ui-interaction.ts.
- Installs packages
In this repository7 other files it gives its agents.
- cli-developmentSkill
- example-projectSkill
- mcp-setupSkill
- simulator-workflowSkill
- swift-agent-debug-logSkill
- tool-handler-developmentSkill
- swift-agent-debug-logSkill
The file
We split this skill into the rules it gives an agent
2 rules. About 0 are common, 0 are distinctive, and 2 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈0
Distinctive · estimate
This author's own choices. Worth a look.
2
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 13
All physical-device interactions use Apple's CoreDevice framework through xcrun devicectl.
- Very specific: line 45
Log workarounds: Use Xcode's Console.app, or sudo pymobiledevice3 remote tunneld + pymobiledevice3 syslog live --udid <UDID>.
≈ “Common” is an estimate: at least 1 in 5 of the 350,495 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Device & UI Automation
Physical Device Commands (via xcrun devicectl)
All physical-device interactions use Apple's CoreDevice framework through xcrun devicectl. This is the only reliable CLI for iOS 17+ / macOS 15+ (Sonoma/Tahoe) devices.
Tool Summary
| Tool | Core function | Status |
bazel_ios_list_devices | listDevices() → devicectl list devices | ✅ Stable |
bazel_ios_device_info | deviceInfo() → devicectl device info details | ✅ Stable |
bazel_ios_device_build_and_run | Build + install + launch chain | ✅ Stable |
bazel_ios_device_install_app | installAppOnDevice() → devicectl device install app | ✅ Stable |
bazel_ios_device_launch_app | launchAppOnDevice() → devicectl device process launch | ✅ Stable |
bazel_ios_device_stop_app | terminateAppOnDevice() → lookup PID via apps+procs, then devicectl device process terminate --pid | ✅ Stable |
bazel_ios_device_screenshot | screenshotDevice() → pymobiledevice3 > idevicescreenshot -n > idevicescreenshot | ⚠️ See connectivity notes |
bazel_ios_device_log_start/stop | startDeviceLogCapture() → pymobiledevice3 > idevicesyslog | ⚠️ See connectivity notes |
bazel_ios_device_test | bazel test with --test_arg=--destination id=<UDID> | ✅ Stable |
bazel_ios_device_pair/unpair | devicectl manage pair/unpair | ✅ Stable |
bazel_ios_device_list_pairs | devicectl list devices | ✅ Stable |
Device Connectivity & iOS 17+ Limitations
Starting with iOS 17 and macOS 15 (Sonoma), Apple replaced the legacy lockdownd/usbmuxd protocol with CoreDevice (remoted). This affects third-party tools:
| Transport | devicectl | idevicescreenshot | idevicesyslog | pymobiledevice3 |
| USB cable | ✅ | ❌ Invalid service (iOS 17+) | ❌ Cannot connect | ❌ needs tunneld |
| Wi-Fi/Network | ✅ | ❌ Device not found | ❌ Device not found | ❌ needs tunneld |
Key implications:
devicectlis the only reliable CLI for list/info/install/launch/terminate on iOS 17+.- Screenshot & log tools depend on
libimobiledeviceorpymobiledevice3, which cannot access the CoreDevice tunnel withoutsudo pymobiledevice3 remote tunneldrunning in a separate terminal. - Screenshot workarounds: Connect via USB and use Xcode's device screen mirror, or run
sudo pymobiledevice3 remote tunneldin background and thenpymobiledevice3 developer dvt screenshot <path> --udid <UDID>. - Log workarounds: Use Xcode's Console.app, or
sudo pymobiledevice3 remote tunneld+pymobiledevice3 syslog live --udid <UDID>.
Process Termination (terminateAppOnDevice)
devicectl device info processes returns process entries with only executable (a file:// URL string) and processIdentifier — no bundleIdentifier field. The terminate flow is:
- Query installed apps via
devicectl device info appsto find the app's executable name from itsurlfield (e.g.,IdentityIntelligenceDemo.app→IdentityIntelligenceDemo). - Query running processes via
devicectl device info processes. - Match the executable URL's last path component against the known executable name.
- Terminate by PID:
devicectl device process terminate --device <UDID> --pid <PID>.
Device Resolution
resolveDevice() finds a device by UDID or name from connected devices:
- Unicode normalization handles smart quotes in device names (e.g.,
Matheus\u2019s iPhone→Matheus's iPhone). - Connection state derived from
connectionProperties.tunnelState === 'connected'orpairingState === 'paired'withvisibilityClass === 'default'.
Build for Device
buildCommandArgs() conditionally excludes simulator flags when platform === 'device':
- Device builds use
--ios_multi_cpus=arm64(nosim_prefix). - Simulator-specific args like
--ios_simulator_deviceare omitted.
App Bundle Discovery for Device
findAppBundle() supports both explicit (//pkg:target) and implicit (//pkg) Bazel labels:
- Implicit
//Apps/Fooresolves to target nameFoo(last path component). - Searches
bazel-bin/<pkg>/<name>.appand<name>_archive-root/Payload/<name>.app.
UI Automation — IDB (preferred, simulators only)
Facebook IDB (idb) sends real HID touch events to the simulator runtime. All UI interaction functions in src/core/ui-interaction.ts check findIdb() first, fall back to CGEvent/osascript.
IDB Commands
idb ui tap --udid <UDID> -- X Y
idb ui swipe --udid <UDID> -- X1 Y1 X2 Y2
idb ui text --udid <UDID> -- "hello"
idb ui key --udid <UDID> -- <hid_code>
idb ui button --udid <UDID> -- HOME
idb ui describe-all --udid <UDID> # accessibility tree- Coordinates are in iOS points (app coordinate space), not macOS screen pixels.
idb ui describe-allreturns real accessibility elements withAXFrame,AXLabel,AXUniqueId,role,type.- Install:
brew install idb-companion. TheidbPython CLI wrapsidb_companion.
UI Automation — Fallback (CGEvent/osascript, simulators only)
- Touch/drag/swipe/pinch: dynamically compiled Swift scripts with
CoreGraphicsCGEvent— posts mouse events at screen coordinates. - Text typing/key presses:
osascript(AppleScript) withSystem Eventskeystroke/key code. osascriptbrings Simulator.app to the front before touch events (CGEvent uses absolute screen coordinates).
Important: macOS CGEvent does NOT translate to iOS touch events in the simulator. IDB is required for reliable touch simulation.
Accessibility
simulatorAccessibilitySnapshot has no reliable programmatic API from outside the app process. Falls back to simctl listapps with a descriptive message. Proper implementation would require an in-app accessibility bridge or XCTest.
Logging
Simulator Logs
Two distinct log channels:
| Channel | Captured via | Visibility |
print() / stdout | simctl launch --console-pty | Only with console-pty flag |
os.log / Logger | simctl spawn <UDID> log stream | Unified log system |
Logger.info()is transient — use.noticeor higher for guaranteed capture.--predicatesupportssubsystem == "...",process == "...",composedMessage CONTAINS "..."withAND/ORlogic.
Device Logs
startDeviceLogCapture() mirrors the screenshot strategy — tries pymobiledevice3 first, falls back to idevicesyslog:
- pymobiledevice3 (
syslog live --udid <UDID>) — works on iOS 17+ with CoreDevice tunnel. Detects early failures (device not found, no tunneld) within 1.5s and falls through. - idevicesyslog (
-u <UDID> -n) — fallback for pre-iOS 17 or when pymobiledevice3 is unavailable.
- Process filtering via
--process-name(pymobiledevice3) or-p(idevicesyslog). - Logs are captured in memory and returned on
log_stop. - The response includes which backend was selected (
toolfield).
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

