agentleFS
betoxfbetoxf/universal-computer-use/skills/agent-desktop/SKILL.md

agent-desktop

Skill2 starsChanged 17 days agoApache-2.0View on GitHub

Operate desktop applications on macOS, Windows and Linux through native accessibility trees, using the agent-desktop CLI. Use when an AI agent must see or act on a desktop GUI: click buttons, fill forms, read UI state, navigate menus, toggle checkboxes, scroll, drag, type text, press shortcuts, take screenshots, manage windows, use the clipboard, launch or close apps, read notifications. Triggers on: "click button", "fill form", "open app", "read UI", "computer use", "operate desktop", "accessibility tree", "snapshot app", "type into field", "navigate menu", "toggle checkbox", "take screenshot", "desktop automation", "agent-desktop", or any desktop GUI task on macOS, Windows or Linux.

10 other projects use a version of this, 2 of them edited.

Adopted by

Most recently taken up by

Used with

Projects with it also use shelra-workspace-quality, agent-memory-systems and 6 more.

Other files people use alongside this one.

Files found in the same public projects as a version of this one, more often than they turn up anywhere else. Each project counts once, large collections count for less, and this file's own repository is left out.
  • ShelraSkill

    shelra-workspace-quality

    Audit or refine ShelraCode's main OpenTUI coding workspace when chat activity, live status, plan/context sidebar, agent visibility, or verification UX must match a supplied visual specification. Do not use for onboarding, settings, or unrelated screens.

    Also in 3 projects that use this

  • xfstudioSkill

    agent-memory-systems

    Memory is the cornerstone of intelligent agents. Without it, every interaction starts from zero. This skill covers the architecture of agent memory: short-term (context window), long-term (vector stores), and the cognitive architectures that organize them. Key insight: Memory isn't just storage - it's retrieval. A million stored facts mean nothing if you can't find the right one. Chunking, embedding, and retrieval strategies determine whether your agent remembers or forgets. The field is fragm

    Also in 3 projects that use this

  • pproencaSkill

    terminal-ui

    Terminal User Interface (TUI) performance and UX guidelines for TypeScript applications using Ink and Clack. This skill should be used when building CLI tools, interactive terminal prompts, or developer tooling with TUI components. Triggers on tasks involving TUI components, CLI prompts, terminal rendering, keyboard input handling, or developer tooling.

    Also in 3 projects that use this

  • guaderramaSkill

    spec-driven-development

    Generate comprehensive specifications for software projects using spec-driven development methodology. Leverages Bright Data MCP tools to research documentation, analyze competitor implementations, gather real-world examples, and enrich specs with architectural insights. Use when user requests specifications, technical plans, project specs, or architecture documentation.

    Also in 3 projects that use this

  • VercelSkill

    agent-browser

    Automates browser interactions for web testing, form filling, screenshots, and data extraction. Use when the user needs to navigate websites, interact with web pages, fill forms, take screenshots, test web applications, or extract information from web pages.

    Also in 6 projects that use this

  • MicrosoftSkill

    impeccable

    Use when the user wants to design, redesign, shape, critique, audit, polish, simplify, harden, animate, or otherwise improve a frontend interface: websites, landing pages, dashboards, product UI, components, forms, settings, onboarding, empty states, and native iOS or Android screens. Handles UX review, hierarchy, layout, typography, color, motion, UX copy, accessibility, responsive behavior, edge cases, and capturing or documenting design systems (DESIGN.md, tokens). Also for making bland UI bolder, loud UI quieter, or ambitious visual effects. Not for backend-only or non-UI tasks.

    Also in 3 projects that use this

The file

We split this skill into the rules it gives an agent

6 rules. 0 are common, 1 is distinctive, and 5 are very specific to this project.

  • 0

    Common

    Most of the 9 edited versions we read keep these.

  • 1

    Distinctive

    This author's own choices. Worth a look.

  • 5

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    Run agent-desktop status. It reports the platform, permissions and supported_surfaces.

    line 28
  2. Very specific:

    Run agent-desktop skills get platform.

    line 29
  3. Very specific:

    Use find when you know the role or exact name.

    line 34
  4. Very specific:

    Never skeleton a surface (--surface menu, sheet, alert).

    line 46
  5. Very specific:

    After an async step, use wait (see commands-system.md).

    line 48
  6. Distinctive:

    Only addressable elements get refs.

    line 54

“Common” is measured: kept by at least 6 in 10 of the 9 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

agent-desktop

agent-desktop is a CLI that reads and operates desktop apps through the OS accessibility tree. Every command prints one JSON envelope (ok, data or error). It is a tool, not an agent: the observe, act and verify loop runs in you.

Shell syntax

Examples use a POSIX shell. In PowerShell, quote every ref ('@s8f3k2p9:e1'), because a bare @token is the splatting operator. Use $env:NAME = 'value' instead of export, and | ConvertFrom-Json instead of $(...) and jq. The Windows skill has the PowerShell forms.

First run

  1. Run agent-desktop status. It reports the platform, permissions and supported_surfaces.
  2. Run agent-desktop skills get platform. It loads the skill for the OS the binary runs on (macOS, Windows or Linux).
  3. For flags and syntax, run agent-desktop <command> --help. This skill does not repeat flag lists.

The loop

Use find when you know the role or exact name. Otherwise map the app first and drill into one region.

agent-desktop snapshot --skeleton --app "Notes" -i --compact   # shallow map, children_count on cut branches
agent-desktop snapshot --root @s8f3k2p9:e3 -i --compact        # drill into one region
agent-desktop click @s8f3k2p9:e12                              # act on a qualified ref
agent-desktop snapshot --root @s8f3k2p9:e3 -i --compact        # verify: re-read the same region
  • A small app needs no skeleton. Take a full snapshot -i.
  • -i keeps interactive nodes only, so a row's static-text label is dropped. To tell which row a repeated button belongs to, drill that row's container with --root and leave out -i. On Windows and Linux that reads the row but does not make its button actionable: a button repeated in other rows is a twin in the window and its ref is refused (see Refs).
  • A --root drill adds its refs to the same snapshot id. Refs outside the drilled subtree, and the drilled root itself, stay valid; refs that were already inside it are replaced, so an old ref there returns STALE_REF.
  • Never skeleton a surface (--surface menu, sheet, alert). A surface is already focused.
  • An action that opens a menu, sheet or alert lists it in data.surfaces. Target that overlay next.
  • After an async step, use wait (see commands-system.md). A fixed sleep is a race.

Refs

  • Output refs are qualified: @<snapshot_id>:e<N>, for example @s8f3k2p9:e1. Pass them to later commands as they are.
  • Refs are valid for one snapshot. After any UI change, re-drill the affected region or take a new snapshot.
  • Only addressable elements get refs. Static text and inert containers do not.
  • A command re-identifies the element before it acts. On macOS the ref carries the element's own address, so identical twins each keep their ref. On Windows and Linux a ref resolves only when it was the only element in the window its snapshot captured with its role, native id, name and description, counted even below a skeleton or depth cut, and exactly one live element still matches. Identical twins there have no ref path: only --headed mouse-click --xy X,Y can choose one, it takes the foreground, and a Linux Wayland session refuses it.
  • STALE_REF: the element is gone or no longer matches. details.kind is saved_ref_absent, or retained_object_stale on macOS. Take a fresh snapshot and use the new ref.
  • AMBIGUOUS_TARGET: an identical element appeared (saved_ref_ambiguous), or the ref was not proven unique at capture (identity_not_unique). Repeating the ref or re-snapshotting identical twins gives the same refusal; act on a control that is unique in the window. identity_not_unique also follows a snapshot that could not read the whole window in time: retake it with a larger --timeout-ms.
  • TIMEOUT with details.kind: resolution_identity_unknown: a live read did not answer, so identity is unproven. Nothing was sent; retry.
  • SNAPSHOT_NOT_FOUND: the snapshot id expired. Snapshot again.

Delivery and retry

Action results carry data.disposition and errors carry error.disposition, each with delivery and retry. Observation commands such as snapshot have none on success; their errors, and every INVALID_ARGS refusal, carry not_delivered and retry: safe.

  • Retry or substitute an action only when disposition.retry is safe. Only not_delivered is safe.
  • Any other delivery value means the action may have landed. A repeat can click, type or submit twice, and toggle can undo itself.
  • On a result that is not safe, read the live state (get, is, or a fresh snapshot) and decide from that. details.post_state helps when present.
  • Branch on error.code and error.disposition. Message and suggestion text can change.
  • details.retryable: false means waiting cannot change the outcome. Try a different approach.

Headless and headed

Ref actions are headless by default. They use semantic accessibility calls and do not take focus, move the cursor, synthesize keys or use the clipboard. If no semantic path exists, the action fails closed. Use the global --headed flag only when the user explicitly permits physical input, such as double-click, hover, drag, or focus. Never switch to headed mode to recover from a headless failure. Raw coordinates (--xy) never imply focus. Which actions need --headed differs by OS: read the platform skill.

References

Load one only when you need it: agent-desktop skills get desktop <reference>.

ReferenceOpen it when
commands-observationYou need snapshot, find, get, is, screenshot, list-surfaces semantics, partial trees, or the --root rules
commands-interactionYou choose between click, type, set-value, select, toggle, scroll, press and mouse commands, or you debug an action error
commands-systemYou launch or close apps, manage windows, use the clipboard, wait, batch, sessions, multi-agent cursors, tracing, or load skills
workflowsYou want worked patterns (forms, menus, dialogs, waiting for a new window, scroll-find, drag) and the anti-patterns list

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.