esp32agent
Skill5k starsChanged 2 days agoMITView on GitHub
See and drive a real iPhone through an ESP32 Agent board (USB mouse/keyboard + AirPlay screen mirroring, reached over Bluetooth). Use when asked to look at, tap, type on, or operate the iPhone, or to test an iOS app on the device.
1 other project uses a version of this, 1 of them edited; most edited versions we compared are an older text (1 of 1); 1 new project in the last 4 weeks, against 0 the 4 before.
How people change it
Most projects run an older version: 1 of 1 add nothing and lack rules this one has, most likely added since.
Of the 1 edited version we compared: 1 is an older text.
Not in older versions
- When the task depends on what the phone plays (a video, a call, a voice message, a spoken code),
esp32agent listengives its words from the last 30 s, or says why the phone can't be heard (listening off, the app not sharing, no Wi-Fi). …missing from 1 of 1 - Before a task,
esp32agent memory search 'TASK'(MCP:memory_recall): the user's preferences and how their apps work, the same entries the iOS app uses.missing from 1 of 1 - When the user says to remember something,
esp32agent memory add 'TEXT' --kind preference(MCP:memory_add). Only the user's own words; never passwords, codes or numbers off the screen.missing from 1 of 1 - Only the user's own words; never passwords, codes or numbers off the screen.missing from 1 of 1
We split this file and each edited version into the rules they give an agent and match them by wording. A version that adds no rule of its own and has not changed since this file last did reads as an older text, so what it lacks is not counted as dropped. We keep no past texts of a file, so that is inferred, and an edited version that started from an older text still counts its gaps as drops.
Adopted by
Surging: 1 new project took it in the last 4 weeks, none the 4 weeks before.thin data
Most recently taken up by
- FreitasCSpace/iphone-agent-linuxedited, first seen Nov 15, 2026
Who's talking about onehumanbeing/esp32-iphone-agent
- A Hacker News story“ESP32-iPhone-agent: Let an AI model use an iPhone via ESP32, AirPlay, USB HID”Hacker News · Oct 15, 2026 · 3 points
- A Hacker News story“ESP32 iPhone Agent”Hacker News · Oct 17, 2026 · 2 points
- davila7/claude-code-templates“Add esp32agent MCP (real iPhone control via ESP32-S3) to devtools”GitHub · Nov 3, 2026 · 1 reactions
- heilcheng/awesome-agent-skills“Add onehumanbeing/handsy to Development and Testing”GitHub · Nov 3, 2026 · 1 reactions
- ultrametricai/ultrametric“Add the AI Video and Device Agents arenas (issue #378 placements)”GitHub · Oct 23, 2026 · 1 reactions
The biggest 5 of 35 mentions linking to the repository that our crawl found. Each line is their own headline.
See all 35
Talked about on GitHub, Hacker News and Bluesky (35 times).
On GitHub · 32
- Nov 3, 2026Add esp32agent MCP (real iPhone control via ESP32-S3) to devtools — davila7/claude-code-templates1 reaction · 3 comments
- Nov 3, 2026Add onehumanbeing/handsy to Development and Testing — heilcheng/awesome-agent-skillssays it tested it1 reaction · 3 comments
- Oct 23, 2026Add the AI Video and Device Agents arenas (issue #378 placements) — ultrametricai/ultrametric1 reaction · 6 comments
- Nov 9, 2026Weekly discovery: new DeepSeek Harness ecosystem candidates — fendouai/awesome-deepseek-harness0 reactions · 1 comment
- Nov 7, 2026Curate CUA resources batch w4-036 (OmniInfer, pyautogui_mcp_server, ...) — reacher-z/Awesome-CUA0 reactions
27 more on GitHub
- Nov 7, 2026[Candidate] onehumanbeing/esp32-iphone-agent/skills/handsy - handsy — ProSkillsMD/proskills0 reactions · 1 comment
- Nov 7, 2026[Candidate] onehumanbeing/esp32-iphone-agent/ios/iphoneagent/skills.skillsbundle/skills/handsy-run-tasks - handsy-run-tasks — ProSkillsMD/proskills0 reactions · 1 comment
- Nov 7, 2026[Candidate] onehumanbeing/esp32-iphone-agent/integrations/dsh/skills/handsy - handsy — ProSkillsMD/proskills0 reactions · 1 comment
- Nov 3, 2026Submit MCP Server: Handsy AI (io.github.onehumanbeing/esp32agent) — chatmcp/mcpso0 reactions
- Nov 3, 2026Add ESP32 iPhone Agent to Automation — kyrolabs/awesome-agents0 reactions
- Nov 3, 2026Add handsy skill to Development & Code Tools — BehiSecc/awesome-claude-skills0 reactions
- Nov 3, 2026Add ESP32 iPhone Agent (esp32agent) to Hardware & IoT — TensorBlock/awesome-mcp-servers0 reactions · 2 comments
- Nov 3, 2026Add onehumanbeing/esp32-iphone-agent to Mobile Device Control — MobinX/awesome-mcp-list0 reactions · 1 comment
- Nov 3, 2026Add Handsy AI (iOS): let an AI agent operate your real iPhone with a $7 ESP32 board — pluwen/awesome-testflight-link0 reactions · 1 comment
- Nov 3, 2026Add Handsy AI (ESP32 iPhone Agent) — dkhamsing/open-source-ios-apps0 reactions · 2 comments
- Nov 3, 2026Add Handsy (XIAO ESP32-S3 iPhone AI agent) to the project catalog — Seeed-Studio/OSHW-XIAO-Series0 reactions · 1 comment
- Nov 3, 2026Add esp32agent (Handsy) metadata — Xwordsman/awesome-mcp0 reactions
- Nov 3, 2026Add esp32agent — toolleeo/awesome-cli-apps-in-a-csv0 reactions
- Nov 3, 2026feat: add Handsy — AI agent that operates a real iPhone with a XIAO ESP32-S3 — Seeed-Studio/OSHW-XIAO-Series0 reactions
- Nov 3, 2026Add onehumanbeing/esp32-iphone-agent (Embedded System) 🤖🤖🤖 — punkpeye/awesome-mcp-servers0 reactions · 1 comment
- Nov 3, 2026[Server Submission]: Handsy (esp32agent): let AI agents operate a real iPhone through an ESP32-S3 board — cline/mcp-marketplace0 reactions
- Oct 19, 2026vendor-radar: candidates needing a founder call — ultrametricai/ultrametric0 reactions · 2 comments
- Oct 19, 2026vendor-radar: candidates 2026-10-19 — ultrametricai/ultrametric0 reactions · 1 comment
- Oct 18, 2026Add ESP32 iPhone Agent (onehumanbeing/esp32-iphone-agent) — reacher-z/Awesome-CUA0 reactions
- Oct 15, 2026onehumanbeing 'esp32-iphone-agent' — 7달러 ESP32 보드로 AI 모델이 실제 iPhone 조작, Mac·탈옥 없이 (10/11, GitHub) — roomedia/ax-trend0 reactions
- Oct 14, 2026업스트림에 PR로 보내주실 수 있을까요? 🙌 — jinwon-int/esp32-iphone-agent0 reactions · 1 comment
- Oct 14, 2026Discovery watchlist — rishibhaskar777/whichai0 reactions
- Oct 13, 2026Project submission: ESP32 iPhone Agent — slavakurilyak/awesome-ai-agents0 reactions
- Oct 13, 2026【开源自荐】ESP32 iPhone Agent:用 7 美元的 ESP32 让 AI 操作真实 iPhone — OpenGithubs/weekly0 reactions
- Oct 13, 2026[开源推荐] ESP32 iPhone Agent:用 ESP32 让 AI 操作真实 iPhone — 521xueweihan/HelloGitHub0 reactions
- Oct 13, 2026【开源自荐】ESP32 iPhone Agent:用 7 美元的 ESP32 让 AI 操作真实 iPhone — ruanyf/weekly0 reactions
- Oct 13, 2026iphone: tool y puente esp32-iphone-agent (listo para enchufar la placa) — nicogarrr/asistenta0 reactions
On Hacker News · 2
- Oct 15, 2026ESP32-iPhone-agent: Let an AI model use an iPhone via ESP32, AirPlay, USB HID3 points
- Oct 17, 2026ESP32 iPhone Agent2 points
On Bluesky · 1
- Oct 15, 2026Control stock iphone with screen scraping and AI: https://github.com/onehumanbeing/esp32-iphone-agent #iphone #ai — @wraptile.fosstodon.org.ap.brid.gy0 likes and reposts · 1 reply
Other versions1 project uses a version of this.
Word for word · 0
None found.
Edited · 1
- FreitasCSpace/iphone-agent-linux★ 098% same
Wrote their own version · 0
None found.
How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.
The file
We split this skill into the rules it gives an agent
7 rules. About 0 are common, 3 are distinctive, and 4 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈3
Distinctive · estimate
This author's own choices. Worth a look.
4
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 32
When the task depends on what the phone plays (a video, a call, a voice message, a spoken code), esp32agent listen gives its words from the …
- Distinctive: line 45
After a task succeeded, save it so the next run is faster:
- Distinctive: line 58
Follow the steps as references, not blindly: coordinates are 0-1000 units of the screen, so convert them to pixels of your screenshot (x * …
- Very specific: line 65
Before a task, esp32agent memory search 'TASK' (MCP: memory_recall): the user's preferences and how their apps work, the same entries the …
- Very specific: line 67
When the user says to remember something, esp32agent memory add 'TEXT' --kind preference (MCP: memory_add).
- Distinctive: line 68
Only the user's own words.
- Very specific: line 80
Only HOME as a button; no volume, power, or app install/launch (open apps from the Home Screen or Spotlight: io button HOME, swipe down in …
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
esp32agent
esp32agent talks to an ESP32 Agent board plugged into the iPhone. The board is the phone's USB mouse and
keyboard and receives its AirPlay screen mirroring; this computer reaches it over Bluetooth. Every command prints
JSON: {"status":"ok","data":...} or {"status":"error","error":"..."}. The MCP server (esp32agent mcp)
offers the same as tools: screenshot, tap, longpress, swipe, type_text, press_key, press_button, device_info,
plus history, intent_list, intent_show, intent_save, memory_recall and memory_add.
Setup (once)
esp32agent pairwithin 3 minutes of plugging the board into the iPhone.- Screen Mirroring on the iPhone: Control Center > Screen Mirroring > ESP32 Agent.
esp32agent calibrate(also runs by itself on the first tap). It needs the mouse pointer visible on screen.
The loop
esp32agent screenshot(writes a PNG in the current directory and prints itsfilePath; read the image). It waits until the screen has settled after your last action, so it never shows the screen from before it.- Decide one action from what you see.
- Act, with coordinates in pixels of that screenshot (full size: don't pass
--scale/--max-sizeif you take coordinates from it):esp32agent io tap X,Y(add--aimafter a missed tap: finds the pointer and corrects first;--count 2double-taps, e.g. to zoom a map or select a word)esp32agent io longpress X,Y --duration 1500esp32agent io swipe X1,Y1,X2,Y2(scroll a list: swipe up from the lower to the upper middle)esp32agent io text 'hello',esp32agent io keys enter,esp32agent io keys cmd+aesp32agent io button HOME(Home Screen)
- Screenshot again and check the result before the next step. Never chain several taps blind.
When the task depends on what the phone plays (a video, a call, a voice message, a spoken code),
esp32agent listen gives its words from the last 30 s, or says why the phone can't be heard (listening off,
the app not sharing, no Wi-Fi). The words can be partial or misheard; check the screen when it matters.
Typing
- The board types printable ASCII only, one key per character.
- Chinese is typed as pinyin:
io text '拿铁'typesnatie(the reply saystyped_as). With the iPhone's Chinese (pinyin) keyboard active, commit the first candidate withio keys space, or screenshot and tap the right candidate. Many search boxes accept the plain pinyin as well. - The board is a hardware keyboard, so iOS may hide the on-screen keyboard; the text field still takes the keys.
- Tap the text field first so it has focus;
io keys cmd+athenio keys backspaceclears it.
Save what worked
After a task succeeded, save it so the next run is faster:
esp32agent history show latest(MCP:historywith action show): the run's steps in 0-1000 screen units, numbered, each with the screenshot it was done on. Every action and screenshot is recorded by itself; a new run starts after 10 minutes idle, or withesp32agent history newbefore a new task.- Pick the steps that did the job (skip detours, misses and retries).
esp32agent intent save --name 'Alarm 7:00' --prompt 'Set an alarm for 7:00 AM every day' --hint 'Type the time instead of scrolling the wheel' --from-run RUN --steps 2,3,5(MCP:intent_savewith name, prompt, hints, from_run, steps). One short, clear instruction as the prompt and at most 6 concrete hints. Only replayable moves can be steps: tap, swipe, type, key, home. The same name replaces the old intent.
Run a saved intent
esp32agent intent list, thenesp32agent intent show NAME(MCP:intent_list,intent_show): the prompt, hints and recorded steps, each with its reference screen (a JPEG path).- Follow the steps as references, not blindly: coordinates are 0-1000 units of the screen, so convert them to pixels of your screenshot (x * width / 1000, y * height / 1000).
- Take a screenshot after each step and check it looks like the next step's reference screen.
- On a mismatch (another screen, a popup, a changed layout) drop the recorded steps and go on with the normal loop, using the prompt and hints.
Memory
- Before a task,
esp32agent memory search 'TASK'(MCP:memory_recall): the user's preferences and how their apps work, the same entries the iOS app uses. - When the user says to remember something,
esp32agent memory add 'TEXT' --kind preference(MCP:memory_add). Only the user's own words; never passwords, codes or numbers off the screen.
Errors to act on
busy: the app is using the phone: the iOS app is driving right now. Wait and retry later; don't fight it.screen mirroring is off: ask the user to start Screen Mirroring to ESP32 Agent.video frozen: ask the user to stop and restart Screen Mirroring.not paired: ask the user to replug the board and runesp32agent pairwithin 3 minutes.phone usb asleep: the phone is locked; ask the user to unlock it.not supported on ESP32 Agent: there is no app list, UI tree, file access or webview; work from screenshots.
Limits
- Video comes over Bluetooth: a screenshot can be up to a second behind, animations look choppy.
- Only HOME as a button; no volume, power, or app install/launch (open apps from the Home Screen or Spotlight:
io button HOME, swipe down in the middle,io text 'Settings',io keys enter).
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

