handsy
Skill5k starsChanged 2 days agoMITView on GitHub
See and drive a real iPhone through an ESP32 Agent (Handsy) board with the esp32agent CLI or MCP server. Use when asked to look at, tap, swipe, type on, or operate an iPhone, or to test an iOS app on a real device.
- Installs packages
Adopted by
Steady: 0 new projects took it in the last 4 weeks, none the 4 weeks before.thin data
Who's talking about onehumanbeing/esp32-iphone-agent
- A Hacker News story“ESP32-iPhone-agent: Let an AI model use an iPhone via ESP32, AirPlay, USB HID”Hacker News · Oct 15, 2026 · 3 points
- A Hacker News story“ESP32 iPhone Agent”Hacker News · Oct 17, 2026 · 2 points
- davila7/claude-code-templates“Add esp32agent MCP (real iPhone control via ESP32-S3) to devtools”GitHub · Nov 3, 2026 · 1 reactions
- heilcheng/awesome-agent-skills“Add onehumanbeing/handsy to Development and Testing”GitHub · Nov 3, 2026 · 1 reactions
- ultrametricai/ultrametric“Add the AI Video and Device Agents arenas (issue #378 placements)”GitHub · Oct 23, 2026 · 1 reactions
The biggest 5 of 35 mentions linking to the repository that our crawl found. Each line is their own headline.
See all 35
Talked about on GitHub, Hacker News and Bluesky (35 times).
On GitHub · 32
- Nov 3, 2026Add esp32agent MCP (real iPhone control via ESP32-S3) to devtools — davila7/claude-code-templates1 reaction · 3 comments
- Nov 3, 2026Add onehumanbeing/handsy to Development and Testing — heilcheng/awesome-agent-skillssays it tested it1 reaction · 3 comments
- Oct 23, 2026Add the AI Video and Device Agents arenas (issue #378 placements) — ultrametricai/ultrametric1 reaction · 6 comments
- Nov 9, 2026Weekly discovery: new DeepSeek Harness ecosystem candidates — fendouai/awesome-deepseek-harness0 reactions · 1 comment
- Nov 7, 2026Curate CUA resources batch w4-036 (OmniInfer, pyautogui_mcp_server, ...) — reacher-z/Awesome-CUA0 reactions
27 more on GitHub
- Nov 7, 2026[Candidate] onehumanbeing/esp32-iphone-agent/skills/handsy - handsy — ProSkillsMD/proskills0 reactions · 1 comment
- Nov 7, 2026[Candidate] onehumanbeing/esp32-iphone-agent/ios/iphoneagent/skills.skillsbundle/skills/handsy-run-tasks - handsy-run-tasks — ProSkillsMD/proskills0 reactions · 1 comment
- Nov 7, 2026[Candidate] onehumanbeing/esp32-iphone-agent/integrations/dsh/skills/handsy - handsy — ProSkillsMD/proskills0 reactions · 1 comment
- Nov 3, 2026Submit MCP Server: Handsy AI (io.github.onehumanbeing/esp32agent) — chatmcp/mcpso0 reactions
- Nov 3, 2026Add ESP32 iPhone Agent to Automation — kyrolabs/awesome-agents0 reactions
- Nov 3, 2026Add handsy skill to Development & Code Tools — BehiSecc/awesome-claude-skills0 reactions
- Nov 3, 2026Add ESP32 iPhone Agent (esp32agent) to Hardware & IoT — TensorBlock/awesome-mcp-servers0 reactions · 2 comments
- Nov 3, 2026Add onehumanbeing/esp32-iphone-agent to Mobile Device Control — MobinX/awesome-mcp-list0 reactions · 1 comment
- Nov 3, 2026Add Handsy AI (iOS): let an AI agent operate your real iPhone with a $7 ESP32 board — pluwen/awesome-testflight-link0 reactions · 1 comment
- Nov 3, 2026Add Handsy AI (ESP32 iPhone Agent) — dkhamsing/open-source-ios-apps0 reactions · 2 comments
- Nov 3, 2026Add Handsy (XIAO ESP32-S3 iPhone AI agent) to the project catalog — Seeed-Studio/OSHW-XIAO-Series0 reactions · 1 comment
- Nov 3, 2026Add esp32agent (Handsy) metadata — Xwordsman/awesome-mcp0 reactions
- Nov 3, 2026Add esp32agent — toolleeo/awesome-cli-apps-in-a-csv0 reactions
- Nov 3, 2026feat: add Handsy — AI agent that operates a real iPhone with a XIAO ESP32-S3 — Seeed-Studio/OSHW-XIAO-Series0 reactions
- Nov 3, 2026Add onehumanbeing/esp32-iphone-agent (Embedded System) 🤖🤖🤖 — punkpeye/awesome-mcp-servers0 reactions · 1 comment
- Nov 3, 2026[Server Submission]: Handsy (esp32agent): let AI agents operate a real iPhone through an ESP32-S3 board — cline/mcp-marketplace0 reactions
- Oct 19, 2026vendor-radar: candidates needing a founder call — ultrametricai/ultrametric0 reactions · 2 comments
- Oct 19, 2026vendor-radar: candidates 2026-10-19 — ultrametricai/ultrametric0 reactions · 1 comment
- Oct 18, 2026Add ESP32 iPhone Agent (onehumanbeing/esp32-iphone-agent) — reacher-z/Awesome-CUA0 reactions
- Oct 15, 2026onehumanbeing 'esp32-iphone-agent' — 7달러 ESP32 보드로 AI 모델이 실제 iPhone 조작, Mac·탈옥 없이 (10/11, GitHub) — roomedia/ax-trend0 reactions
- Oct 14, 2026업스트림에 PR로 보내주실 수 있을까요? 🙌 — jinwon-int/esp32-iphone-agent0 reactions · 1 comment
- Oct 14, 2026Discovery watchlist — rishibhaskar777/whichai0 reactions
- Oct 13, 2026Project submission: ESP32 iPhone Agent — slavakurilyak/awesome-ai-agents0 reactions
- Oct 13, 2026【开源自荐】ESP32 iPhone Agent:用 7 美元的 ESP32 让 AI 操作真实 iPhone — OpenGithubs/weekly0 reactions
- Oct 13, 2026[开源推荐] ESP32 iPhone Agent:用 ESP32 让 AI 操作真实 iPhone — 521xueweihan/HelloGitHub0 reactions
- Oct 13, 2026【开源自荐】ESP32 iPhone Agent:用 7 美元的 ESP32 让 AI 操作真实 iPhone — ruanyf/weekly0 reactions
- Oct 13, 2026iphone: tool y puente esp32-iphone-agent (listo para enchufar la placa) — nicogarrr/asistenta0 reactions
On Hacker News · 2
- Oct 15, 2026ESP32-iPhone-agent: Let an AI model use an iPhone via ESP32, AirPlay, USB HID3 points
- Oct 17, 2026ESP32 iPhone Agent2 points
On Bluesky · 1
- Oct 15, 2026Control stock iphone with screen scraping and AI: https://github.com/onehumanbeing/esp32-iphone-agent #iphone #ai — @wraptile.fosstodon.org.ap.brid.gy0 likes and reposts · 1 reply
In this repository3 other files it gives its agents.
- AGENTS.mdAGENTS.md
- CLAUDE.mdCLAUDE.md
- esp32agentSkill
The file
We split this skill into the rules it gives an agent
10 rules. About 0 are common, 6 are distinctive, and 4 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈6
Distinctive · estimate
This author's own choices. Worth a look.
4
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 59
Start Screen Mirroring on the iPhone to ESP32 Agent.
- Very specific: line 79
Every tool takes an optional device (board id) when more than one board is nearby.
- Very specific: line 83
Every command prints JSON: {"status":"ok","data":...} or {"status":"error","error":"..."} (exit code 1).
- Very specific: line 111
Do that one action, with coordinates in pixels of that full-size screenshot.
- Very specific: line 118
If a tap missed, retry with io tap X,Y --aim.
- Distinctive: line 126
After a task succeeds, save it so the next run is faster:
- Distinctive: line 139
Ask the user to confirm before anything that spends money, sends a message, email or post, calls someone, deletes data, changes passwords …
- Distinctive: line 141
Do not read or copy private content (messages, photos, codes, passwords) beyond what the task needs.
- Distinctive: line 142
If a screen asks for a password, passcode, Face ID or a one-time code, stop and hand it to the user.
- Distinctive: line 143
Stop and ask when the screen is not what you expected.
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Handsy: drive a real iPhone with esp32agent
An ESP32 Agent board (Handsy) plugs into the iPhone (USB-C models directly, Lightning models through a Lightning to
USB Camera Adapter) and acts as the phone's USB mouse and keyboard. It also receives the iPhone's AirPlay screen mirroring. Your computer reaches
the board over Bluetooth with the esp32agent command line tool, which is also an MCP server. No jailbreak is
needed.
iPhone only for now. iPad support is coming soon.
Prerequisites
- A Seeed Studio XIAO ESP32S3 board (8 MB flash, 8 MB PSRAM), or a similar ESP32-S3 board with native USB.
- Board firmware v0.2.2 or later. Install it in Chrome or Edge at https://handsy.tech/install (one USB data cable, nothing else to install).
- On the iPhone:
- AssistiveTouch on (Settings > Accessibility > Touch > AssistiveTouch), so the mouse pointer shows and taps.
- Screen Mirroring to the board: join the board's Wi-Fi, then Control Center > Screen Mirroring > ESP32 Agent.
- The phone unlocked while the agent works.
- On the computer: Bluetooth, Node.js 18 or later (for
npx), andffmpeg(needed for screenshots).
Install
Nothing to install up front; npx fetches the CLI:
npx -y esp32agent versionOr install it globally with npm install -g esp32agent and call esp32agent directly. The examples below use
npx -y esp32agent; with a global install, drop the npx -y.
MCP server (stdio), if your agent does not already have it from this plugin:
claude mcp add esp32agent -- npx -y esp32agent mcpOther MCP clients:
{ "mcpServers": { "esp32agent": { "command": "npx", "args": ["-y", "esp32agent", "mcp"] } } }Pair (once per computer)
- Plug the board into the iPhone. Within 3 minutes run
npx -y esp32agent pair. - Start Screen Mirroring on the iPhone to ESP32 Agent.
npx -y esp32agent screenshotshould now save a PNG of the phone's screen.npx -y esp32agent calibratemeasures the pointer. It also runs by itself before the earliest tap. It needs the pointer visible on screen.
MCP tools
| Tool | What it does |
screenshot | The screen now, after the last action settled. Other tools take pixel coordinates of this image |
tap | Tap at x, y; count: 2 double-taps |
longpress | Press and hold at x, y (duration_ms, default 1000) |
swipe | Swipe from x1, y1 to x2, y2 |
type_text | Type into the focused field (ASCII; Chinese is typed as pinyin) |
press_key | enter, esc, tab, backspace, space, arrows, cmd+a, shift+tab ... |
press_button | HOME only (go to the Home Screen) |
device_info | Board status: mirroring (ap), video frozen (fz), app busy (busy), usb, ble |
history | action: list (recent runs) or show (one run's steps with their screenshots) |
intent_list, intent_show, intent_save | Saved tasks: list, show one, save what worked |
Every tool takes an optional device (board id) when more than one board is nearby.
CLI commands
Every command prints JSON: {"status":"ok","data":...} or {"status":"error","error":"..."} (exit code 1).
| Command | What it does |
devices | Boards nearby (id, paired, connected) |
pair, unpair | Pair this computer with the board; forget it here |
screenshot [--format png|jpeg] [--quality N] [--scale F] [--max-size N] [--output PATH|DIR/|-] | Save the screen; prints filePath |
io tap X,Y [--aim] [--count N] | Tap; --aim finds the pointer and corrects before tapping |
io longpress X,Y [--duration MS] | Press and hold |
io swipe X1,Y1,X2,Y2 | Swipe |
io text 'TEXT' | Type ASCII text |
io keys COMBO | Press a key or combo |
io button HOME | Home Screen |
device info | Board status |
calibrate | Measure the pointer |
history list, history show RUN|latest, history new | What was done on the phone; start a new run |
intent list, intent show NAME, intent save ..., intent delete NAME | Saved tasks |
intent export NAME [--screenshots] [--output FILE], intent import FILE | Move saved tasks to and from the iOS app |
upgrade [--check], version | Update the CLI; print its version |
mcp | Run the MCP server on stdio |
With several boards nearby pass --device <id> or set ESP32AGENT_DEVICE.
The see-act loop
- Take a screenshot (
screenshottool, oresp32agent screenshotand read the PNG atfilePath). It waits until the screen has settled after your last action. - Decide one action from what you see.
- Do that one action, with coordinates in pixels of that full-size screenshot. If you used
--scaleor--max-size, convert the coordinates back to full size. - Take another screenshot and check the result before the next step. Never chain several taps blind.
Tips:
- To scroll a list, swipe up from the lower middle to the upper middle of the screen.
- To open an app: press HOME, swipe down in the middle of the Home Screen, type the app name, press
enter. - If a tap missed, retry with
io tap X,Y --aim. - Tap a text field so it has focus before typing.
cmd+athenbackspaceclears it. - The board is a hardware keyboard, so iOS may hide the on-screen keyboard; the field still takes the keys.
- Chinese text is typed as pinyin (
io text '拿铁'typesnatie). With the Chinese pinyin keyboard active, pressspaceto accept the top candidate, or screenshot and tap the right one.
Save what worked
After a task succeeds, save it so the next run is faster:
history show latest(MCPhistory, actionshow) lists the run's numbered steps in 0-1000 screen units.- Pick the steps that did the job, skipping misses and retries.
intent save --name 'Alarm 7:00' --prompt 'Set an alarm for 7:00 AM every day' --hint 'Type the time' --from-run RUN --steps 2,3,5(MCPintent_save). Keep the prompt short and use at most 6 hints.
To reuse one, intent show NAME, convert its 0-1000 coordinates to pixels (x * width / 1000,
y * height / 1000), and check each screen against the step's reference screen. If the screen looks different,
drop the recorded steps and continue with the normal loop.
Safety
You are using a real phone with the owner's accounts, money and contacts.
- Ask the user to confirm before anything that spends money, sends a message, email or post, calls someone, deletes data, changes passwords or security settings, or accepts terms.
- Do not read or copy private content (messages, photos, codes, passwords) beyond what the task needs.
- If a screen asks for a password, passcode, Face ID or a one-time code, stop and hand it to the user.
- Stop and ask when the screen is not what you expected.
Troubleshooting
| Error or symptom | What to do |
not paired | Replug the board into the iPhone and run pair within 3 minutes |
screen mirroring is off | Ask the user to start Screen Mirroring to ESP32 Agent |
video frozen | Ask the user to stop and restart Screen Mirroring |
phone usb asleep | The phone is locked; ask the user to unlock it |
busy: the app is using the phone | The ESP32 Agent iOS app is driving the phone. Wait and retry later |
no cli service (firmware too old?) | Update the board firmware to v0.2.2 or later at https://handsy.tech/install |
not supported on ESP32 Agent | There is no app list, UI tree or file access; work from screenshots |
| Taps land off target | Make sure AssistiveTouch is on and the pointer is visible, run calibrate, or use io tap --aim |
| Screenshot fails | Check that ffmpeg is installed and Screen Mirroring is on |
Limits: screenshots can be up to about a second behind and animations look choppy. Typing is ASCII only, one key per character. HOME is the only button: no volume or power buttons, and no installing or launching apps directly.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

