agent-desktop
Skill2 starsChanged 17 days agoApache-2.0View on GitHub
Operate desktop applications on macOS, Windows and Linux through native accessibility trees, using the agent-desktop CLI. Use when an AI agent must see or act on a desktop GUI: click buttons, fill forms, read UI state, navigate menus, toggle checkboxes, scroll, drag, type text, press shortcuts, take screenshots, manage windows, use the clipboard, launch or close apps, read notifications. Triggers on: "click button", "fill form", "open app", "read UI", "computer use", "operate desktop", "accessibility tree", "snapshot app", "type into field", "navigate menu", "toggle checkbox", "take screenshot", "desktop automation", "agent-desktop", or any desktop GUI task on macOS, Windows or Linux.
10 other projects use a version of this, 2 of them edited.
Adopted by
Most recently taken up by
- kinhluan/skillstheir own version, first seen Nov 8, 2026
- SwiftyJunnos/skillsword for word, first seen Nov 1, 2026
- travisjneuman/.claudetheir own version, first seen Nov 1, 2026
- vercel-labs/agent-desktopedited, first seen Nov 1, 2026
- lahfir/agent-desktopedited, first seen Oct 30, 2026
- superagent-ai/grok-clitheir own version, first seen Oct 8, 2026
Used with
Projects with it also use shelra-workspace-quality, agent-memory-systems and 6 more.
Other files people use alongside this one.
Files found in the same public projects as a version of this one, more often than they turn up anywhere else. Each project counts once, large collections count for less, and this file's own repository is left out.
ShelraSkillshelra-workspace-quality
Audit or refine ShelraCode's main OpenTUI coding workspace when chat activity, live status, plan/context sidebar, agent visibility, or verification UX must match a supplied visual specification. Do not use for onboarding, settings, or unrelated screens.
Also in 3 projects that use this
xfstudioSkillagent-memory-systems
Memory is the cornerstone of intelligent agents. Without it, every interaction starts from zero. This skill covers the architecture of agent memory: short-term (context window), long-term (vector stores), and the cognitive architectures that organize them. Key insight: Memory isn't just storage - it's retrieval. A million stored facts mean nothing if you can't find the right one. Chunking, embedding, and retrieval strategies determine whether your agent remembers or forgets. The field is fragm
Also in 3 projects that use this
pproencaSkillterminal-ui
Terminal User Interface (TUI) performance and UX guidelines for TypeScript applications using Ink and Clack. This skill should be used when building CLI tools, interactive terminal prompts, or developer tooling with TUI components. Triggers on tasks involving TUI components, CLI prompts, terminal rendering, keyboard input handling, or developer tooling.
Also in 3 projects that use this
guaderramaSkillspec-driven-development
Generate comprehensive specifications for software projects using spec-driven development methodology. Leverages Bright Data MCP tools to research documentation, analyze competitor implementations, gather real-world examples, and enrich specs with architectural insights. Use when user requests specifications, technical plans, project specs, or architecture documentation.
Also in 3 projects that use this
VercelSkillagent-browser
Automates browser interactions for web testing, form filling, screenshots, and data extraction. Use when the user needs to navigate websites, interact with web pages, fill forms, take screenshots, test web applications, or extract information from web pages.
Also in 6 projects that use this
MicrosoftSkillimpeccable
Use when the user wants to design, redesign, shape, critique, audit, polish, simplify, harden, animate, or otherwise improve a frontend interface: websites, landing pages, dashboards, product UI, components, forms, settings, onboarding, empty states, and native iOS or Android screens. Handles UX review, hierarchy, layout, typography, color, motion, UX copy, accessibility, responsive behavior, edge cases, and capturing or documenting design systems (DESIGN.md, tokens). Also for making bland UI bolder, loud UI quieter, or ambitious visual effects. Not for backend-only or non-UI tasks.
Also in 3 projects that use this
Other versions10 projects use a version of this.
Word for word · 1
- SwiftyJunnos/skills★ 1100% same
Edited · 2
- lahfir/agent-desktop★ 24k69% same
- vercel-labs/agent-desktop★ 24k69% same
Wrote their own version · 7
- superagent-ai/grok-cli★ 3.5ksame name
- travisjneuman/.claude★ 106same name
- kinhluan/skills★ 6same name
- Shelra/ShelraCode★ 3same name
- yosoyjavieruiz/Distroia★ 3same name
- yosoyjavieruiz/ShelraCode★ 3same name
- ferris-joy/axiomdesk★ 2same name
How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.
In this repository8 other files it gives its agents.
- AGENTS.mdAGENTS.md
- architecture-diagramSkill
- technical-writerSkill
- agent-desktop-ffiSkill
- agent-desktop-linuxSkill
- agent-desktop-macosSkill
- agent-desktop-windowsSkill
- jev-desktopSkill
The file
We split this skill into the rules it gives an agent
6 rules. 0 are common, 1 is distinctive, and 5 are very specific to this project.
0
Common
Most of the 9 edited versions we read keep these.
1
Distinctive
This author's own choices. Worth a look.
5
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 28
Run agent-desktop status. It reports the platform, permissions and supported_surfaces.
- Very specific: line 29
Run agent-desktop skills get platform.
- Very specific: line 34
Use find when you know the role or exact name.
- Very specific: line 46
Never skeleton a surface (--surface menu, sheet, alert).
- Very specific: line 48
After an async step, use wait (see commands-system.md).
- Distinctive: line 54
Only addressable elements get refs.
“Common” is measured: kept by at least 6 in 10 of the 9 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
agent-desktop
agent-desktop is a CLI that reads and operates desktop apps through the OS accessibility tree. Every command prints one JSON envelope (ok, data or error). It is a tool, not an agent: the observe, act and verify loop runs in you.
Shell syntax
Examples use a POSIX shell. In PowerShell, quote every ref ('@s8f3k2p9:e1'), because a bare @token is the splatting operator. Use $env:NAME = 'value' instead of export, and | ConvertFrom-Json instead of $(...) and jq. The Windows skill has the PowerShell forms.
First run
- Run
agent-desktop status. It reports the platform, permissions andsupported_surfaces. - Run
agent-desktop skills get platform. It loads the skill for the OS the binary runs on (macOS, Windows or Linux). - For flags and syntax, run
agent-desktop <command> --help. This skill does not repeat flag lists.
The loop
Use find when you know the role or exact name. Otherwise map the app first and drill into one region.
agent-desktop snapshot --skeleton --app "Notes" -i --compact # shallow map, children_count on cut branches
agent-desktop snapshot --root @s8f3k2p9:e3 -i --compact # drill into one region
agent-desktop click @s8f3k2p9:e12 # act on a qualified ref
agent-desktop snapshot --root @s8f3k2p9:e3 -i --compact # verify: re-read the same region- A small app needs no skeleton. Take a full
snapshot -i. -ikeeps interactive nodes only, so a row's static-text label is dropped. To tell which row a repeated button belongs to, drill that row's container with--rootand leave out-i. On Windows and Linux that reads the row but does not make its button actionable: a button repeated in other rows is a twin in the window and its ref is refused (see Refs).- A
--rootdrill adds its refs to the same snapshot id. Refs outside the drilled subtree, and the drilled root itself, stay valid; refs that were already inside it are replaced, so an old ref there returnsSTALE_REF. - Never skeleton a surface (
--surface menu,sheet,alert). A surface is already focused. - An action that opens a menu, sheet or alert lists it in
data.surfaces. Target that overlay next. - After an async step, use
wait(seecommands-system.md). A fixed sleep is a race.
Refs
- Output refs are qualified:
@<snapshot_id>:e<N>, for example@s8f3k2p9:e1. Pass them to later commands as they are. - Refs are valid for one snapshot. After any UI change, re-drill the affected region or take a new snapshot.
- Only addressable elements get refs. Static text and inert containers do not.
- A command re-identifies the element before it acts. On macOS the ref carries the element's own address, so identical twins each keep their ref. On Windows and Linux a ref resolves only when it was the only element in the window its snapshot captured with its role, native id, name and description, counted even below a skeleton or depth cut, and exactly one live element still matches. Identical twins there have no ref path: only
--headed mouse-click --xy X,Ycan choose one, it takes the foreground, and a Linux Wayland session refuses it. STALE_REF: the element is gone or no longer matches.details.kindissaved_ref_absent, orretained_object_staleon macOS. Take a fresh snapshot and use the new ref.AMBIGUOUS_TARGET: an identical element appeared (saved_ref_ambiguous), or the ref was not proven unique at capture (identity_not_unique). Repeating the ref or re-snapshotting identical twins gives the same refusal; act on a control that is unique in the window.identity_not_uniquealso follows a snapshot that could not read the whole window in time: retake it with a larger--timeout-ms.TIMEOUTwithdetails.kind: resolution_identity_unknown: a live read did not answer, so identity is unproven. Nothing was sent; retry.SNAPSHOT_NOT_FOUND: the snapshot id expired. Snapshot again.
Delivery and retry
Action results carry data.disposition and errors carry error.disposition, each with delivery and retry. Observation commands such as snapshot have none on success; their errors, and every INVALID_ARGS refusal, carry not_delivered and retry: safe.
- Retry or substitute an action only when
disposition.retryissafe. Onlynot_deliveredis safe. - Any other delivery value means the action may have landed. A repeat can click, type or submit twice, and
togglecan undo itself. - On a result that is not safe, read the live state (
get,is, or a fresh snapshot) and decide from that.details.post_statehelps when present. - Branch on
error.codeanderror.disposition. Message and suggestion text can change. details.retryable: falsemeans waiting cannot change the outcome. Try a different approach.
Headless and headed
Ref actions are headless by default. They use semantic accessibility calls and do not take focus, move the cursor, synthesize keys or use the clipboard. If no semantic path exists, the action fails closed. Use the global --headed flag only when the user explicitly permits physical input, such as double-click, hover, drag, or focus. Never switch to headed mode to recover from a headless failure. Raw coordinates (--xy) never imply focus. Which actions need --headed differs by OS: read the platform skill.
References
Load one only when you need it: agent-desktop skills get desktop <reference>.
| Reference | Open it when |
commands-observation | You need snapshot, find, get, is, screenshot, list-surfaces semantics, partial trees, or the --root rules |
commands-interaction | You choose between click, type, set-value, select, toggle, scroll, press and mouse commands, or you debug an action error |
commands-system | You launch or close apps, manage windows, use the clipboard, wait, batch, sessions, multi-agent cursors, tracing, or load skills |
workflows | You want worked patterns (forms, menus, dialogs, waiting for a new window, scroll-find, drag) and the anti-patterns list |
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

