agentleFS
ferris-joyferris-joy/axiomdesk/skills/agent-desktop/SKILL.md

agent-desktop

Skill2 starsChanged 6 months agoApache-2.0View on GitHub

Desktop automation via native OS accessibility trees using the agent-desktop CLI. Use when an AI agent needs to observe, interact with, or automate desktop applications (click buttons, fill forms, navigate menus, read UI state, toggle checkboxes, scroll, drag, type text, take screenshots, manage windows, use clipboard, manage notifications). Covers 54 commands across observation, interaction, keyboard/mouse, app lifecycle, notifications (macOS), clipboard, wait, and a `skills` command that prints these bundled docs straight from the binary. Triggers on: "click button", "fill form", "open app", "read UI", "automate desktop", "accessibility tree", "snapshot app", "type into field", "navigate menu", "toggle checkbox", "take screenshot", "desktop automation", "agent-desktop", or any desktop GUI interaction task. Supports the macOS Phase 1 adapter, with Windows and Linux planned against the same core contracts.

  • Installs packages

10 other projects use a version of this.

Adopted by

Most recently taken up by

Used with

Projects with it also use shelra-workspace-quality, agent-memory-systems and 6 more.

Other files people use alongside this one.

Files found in the same public projects as a version of this one, more often than they turn up anywhere else. Each project counts once, large collections count for less, and this file's own repository is left out.
  • ShelraSkill

    shelra-workspace-quality

    Audit or refine ShelraCode's main OpenTUI coding workspace when chat activity, live status, plan/context sidebar, agent visibility, or verification UX must match a supplied visual specification. Do not use for onboarding, settings, or unrelated screens.

    Also in 3 projects that use this

  • xfstudioSkill

    agent-memory-systems

    Memory is the cornerstone of intelligent agents. Without it, every interaction starts from zero. This skill covers the architecture of agent memory: short-term (context window), long-term (vector stores), and the cognitive architectures that organize them. Key insight: Memory isn't just storage - it's retrieval. A million stored facts mean nothing if you can't find the right one. Chunking, embedding, and retrieval strategies determine whether your agent remembers or forgets. The field is fragm

    Also in 3 projects that use this

  • pproencaSkill

    terminal-ui

    Terminal User Interface (TUI) performance and UX guidelines for TypeScript applications using Ink and Clack. This skill should be used when building CLI tools, interactive terminal prompts, or developer tooling with TUI components. Triggers on tasks involving TUI components, CLI prompts, terminal rendering, keyboard input handling, or developer tooling.

    Also in 3 projects that use this

  • guaderramaSkill

    spec-driven-development

    Generate comprehensive specifications for software projects using spec-driven development methodology. Leverages Bright Data MCP tools to research documentation, analyze competitor implementations, gather real-world examples, and enrich specs with architectural insights. Use when user requests specifications, technical plans, project specs, or architecture documentation.

    Also in 3 projects that use this

  • VercelSkill

    agent-browser

    Automates browser interactions for web testing, form filling, screenshots, and data extraction. Use when the user needs to navigate websites, interact with web pages, fill forms, take screenshots, test web applications, or extract information from web pages.

    Also in 6 projects that use this

  • MicrosoftSkill

    impeccable

    Use when the user wants to design, redesign, shape, critique, audit, polish, simplify, harden, animate, or otherwise improve a frontend interface: websites, landing pages, dashboards, product UI, components, forms, settings, onboarding, empty states, and native iOS or Android screens. Handles UX review, hierarchy, layout, typography, color, motion, UX copy, accessibility, responsive behavior, edge cases, and capturing or documenting design systems (DESIGN.md, tokens). Also for making bland UI bolder, loud UI quieter, or ambitious visual effects. Not for backend-only or non-UI tasks.

    Also in 3 projects that use this

The file

We split this skill into the rules it gives an agent

9 rules. 0 are common, 4 are distinctive, and 5 are very specific to this project.

  • 0

    Common

    Most of the 10 edited versions we read keep these.

  • 4

    Distinctive

    This author's own choices. Worth a look.

  • 5

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Use progressive skeleton traversal as the default approach.

    line 52
  2. Distinctive:

    Only interactive elements get refs: button, textfield, checkbox, link, menuitem, tab, slider, combobox, treeitem, cell

    line 86
  3. Very specific:

    Every snapshot returns snapshot_id.

    line 90
  4. Distinctive:

    After any action that changes UI, re-drill the affected region or re-snapshot

    line 92
  5. Distinctive:

    Every command returns a JSON envelope on stdout:

    line 97
  6. Very specific:

    Skeleton first, drill second: Start with --skeleton -i --compact for dense apps.

    line 229
  7. Very specific:

    Refs are snapshot-scoped: Keep snapshot_id for deterministic multi-step use.

    line 231
  8. Very specific:

    Use wait for async UI: After launch/dialog triggers, wait for expected state.

    line 233
  9. Very specific:

    Check permissions first: Run permissions on first use.

    line 234

“Common” is measured: kept by at least 6 in 10 of the 10 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

agent-desktop

CLI tool enabling AI agents to observe and control desktop applications via native OS accessibility trees.

Core principle: agent-desktop is NOT an AI agent. It is a tool that AI agents invoke. It outputs structured JSON with ref-based element identifiers. The observation-action loop lives in the calling agent.

Installation

npm install -g agent-desktop
# or
bun install -g --trust agent-desktop

Requires macOS 12+ with Accessibility permission granted to your terminal. Screen Recording permission is also required for screenshots.

Reference Files

Detailed documentation is split into focused reference files. Read them as needed:

ReferenceContents
references/commands-observation.mdsnapshot, find, get, is, screenshot, list-surfaces — all flags, output examples
references/commands-interaction.mdclick, type, set-value, select, toggle, scroll, drag, keyboard, mouse — choosing the right command
references/commands-system.mdlaunch, close, windows, clipboard, wait, batch, status, permissions, version
references/workflows.md12 common patterns: forms, menus, dialogs, scroll-find, drag-drop, async wait, anti-patterns
references/macos.mdmacOS permissions/TCC, AX API internals, smart activation chain, surfaces, Notification Center, troubleshooting

The Observe-Act Loop (Progressive Skeleton Traversal)

Use progressive skeleton traversal as the default approach. It reduces token consumption 78-96% for dense apps by exploring the UI in two phases: a shallow skeleton overview, then targeted drill-downs into regions of interest.

1. SKELETON → agent-desktop snapshot --skeleton --app "App" -i --compact
   Parse the overview. Identify the region containing your target.
   Regions show children_count (e.g., "Sidebar" with children_count: 42).
   Named containers at truncation boundary have refs for drill-down.
   Keep the returned snapshot_id.

2. DRILL    → agent-desktop snapshot --root @e3 --snapshot <snapshot_id> -i --compact
   Expand the target region. Now you see its interactive elements.

3. ACT      → agent-desktop click @e12 --snapshot <snapshot_id>  (or type, select, toggle...)

4. VERIFY   → agent-desktop snapshot --root @e3 --snapshot <snapshot_id> -i --compact
   Re-drill the same region to confirm the state change.
   Scoped invalidation: only @e3's subtree refs are replaced.

5. REPEAT   → Continue drilling other regions or acting as needed.

When to skip skeleton and use full snapshot instead:

  • Simple apps with few elements (Finder, Calculator, TextEdit)
  • You already know the exact element name — use find instead
  • Surface snapshots (menus, sheets, alerts) — these are already focused

When skeleton shines:

  • Dense Electron apps (Slack, VS Code, Discord, Notion)
  • Any app where full snapshot exceeds ~50 refs
  • Multi-region workflows (sidebar + main content + toolbar)

Ref System

  • Refs assigned depth-first: @e1, @e2, @e3...
  • Only interactive elements get refs: button, textfield, checkbox, link, menuitem, tab, slider, combobox, treeitem, cell
  • In skeleton mode, named/described containers at truncation boundary also get refs (drill-down targets with empty available_actions)
  • Static text, groups, containers remain in tree for context but have no ref
  • Refs are deterministic within a snapshot but NOT stable across snapshots if UI changed
  • Every snapshot returns snapshot_id; ref-consuming commands accept --snapshot <id>
  • last_refmap.json is only a latest-snapshot inspection artifact. The command path uses snapshot-scoped storage.
  • After any action that changes UI, re-drill the affected region or re-snapshot
  • Scoped invalidation: re-drilling --root @e3 only replaces refs from @e3's previous drill — refs from other regions and the skeleton itself are preserved

JSON Output Contract

Every command returns a JSON envelope on stdout:

Success: { "version": "2.0", "ok": true, "command": "snapshot", "data": { ... } } Error: { "version": "2.0", "ok": false, "command": "click", "error": { "code": "STALE_REF", "message": "...", "suggestion": "..." } }

Exit codes: 0 success, 1 structured error, 2 argument error.

Error Codes

CodeMeaningRecovery
PERM_DENIEDAccessibility or Screen Recording permission not grantedGrant the named permission in System Settings
ELEMENT_NOT_FOUNDRef cannot be resolved against the live UIRe-run snapshot, use fresh ref
APP_NOT_FOUNDApp not runningLaunch it first
ACTION_FAILEDAX action rejectedTry an explicit alternative command
ACTION_NOT_SUPPORTEDElement can't do thisUse different command
STALE_REFRef from old snapshotRe-run snapshot
SNAPSHOT_NOT_FOUNDSnapshot ID is missing or expiredRun snapshot again and use the returned ID
POLICY_DENIEDA physical/headed path was blockedUse an explicit mouse/focus/keyboard command if physical interaction is intended
WINDOW_NOT_FOUNDNo matching windowCheck app name, use list-windows
PLATFORM_NOT_SUPPORTEDAdapter method not implemented on this platformUse a supported platform adapter
TIMEOUTWait condition not metIncrease --timeout
INVALID_ARGSBad argumentsCheck command syntax
NOTIFICATION_NOT_FOUNDNotification index no longer existsRe-run list-notifications

Command Quick Reference (54 commands)

Observation

agent-desktop snapshot --skeleton --app "App" -i --compact  # Skeleton overview (preferred)
agent-desktop snapshot --root @e3 -i --compact              # Drill into region
agent-desktop snapshot --app "App" -i                       # Full tree (simple apps)
agent-desktop snapshot --app "App" --surface menu -i        # Surface snapshot
agent-desktop screenshot --app "App" out.png                # PNG screenshot
agent-desktop find --app "App" --role button                # Search elements
agent-desktop get @e1 --snapshot s... --property text       # Read element property
agent-desktop is @e1 --snapshot s... --property enabled     # Check element state
agent-desktop list-surfaces --app "App"                     # Available surfaces

Interaction

agent-desktop click @e5 --snapshot s...         # AX-first click, no cursor move by default
agent-desktop double-click @e3                  # AXOpen; physical double-click uses mouse-click --count 2
agent-desktop triple-click @e2                  # Physical triple-click uses mouse-click --count 3
agent-desktop right-click @e5                   # Right-click; menu returned when verified
agent-desktop type @e2 --snapshot s... "hello"  # Headless AX text insertion when supported
agent-desktop set-value @e2 "new value"         # Set value directly
agent-desktop clear @e2                         # Clear element value
agent-desktop focus @e2                         # Set keyboard focus
agent-desktop select @e4 "Option B"             # Select dropdown/list option
agent-desktop toggle @e6                        # Toggle checkbox/switch
agent-desktop check @e6                         # Idempotent check
agent-desktop uncheck @e6                       # Idempotent uncheck
agent-desktop expand @e7                        # Expand disclosure
agent-desktop collapse @e7                      # Collapse disclosure
agent-desktop scroll @e1 --direction down       # Scroll element
agent-desktop scroll-to @e8                     # Scroll into view

Keyboard & Mouse

agent-desktop press cmd+c                       # Key combo
agent-desktop press return --app "App"          # Targeted key press
agent-desktop key-down shift                    # Hold key
agent-desktop key-up shift                      # Release key
agent-desktop hover @e5                         # Explicit cursor movement
agent-desktop hover --xy 500,300                # Cursor to coordinates
agent-desktop drag --from @e1 --to @e5          # Drag between elements
agent-desktop mouse-click --xy 500,300          # Click at coordinates
agent-desktop mouse-move --xy 100,200           # Move cursor
agent-desktop mouse-down --xy 100,200           # Press mouse button
agent-desktop mouse-up --xy 300,400             # Release mouse button

App & Window

agent-desktop launch "System Settings"          # Launch and wait
agent-desktop close-app "TextEdit"              # Quit gracefully
agent-desktop close-app "TextEdit" --force      # Force kill
agent-desktop list-windows --app "Finder"       # List windows
agent-desktop list-apps                         # List running GUI apps
agent-desktop focus-window --app "Finder"       # Bring to front
agent-desktop resize-window --app "App" --width 800 --height 600
agent-desktop move-window --app "App" --x 0 --y 0
agent-desktop minimize --app "App"
agent-desktop maximize --app "App"
agent-desktop restore --app "App"

Notifications

agent-desktop list-notifications                # List all notifications
agent-desktop list-notifications --app "Slack"  # Filter by app
agent-desktop list-notifications --text "deploy" --limit 5  # Filter by text
agent-desktop dismiss-notification 1            # Dismiss by index
agent-desktop dismiss-all-notifications         # Dismiss all
agent-desktop dismiss-all-notifications --app "Slack"  # Dismiss all from app
agent-desktop notification-action 1 "Reply" --expected-app Slack   # Click action (with NC reorder guard)

Clipboard

agent-desktop clipboard-get                     # Read clipboard
agent-desktop clipboard-set "text"              # Write to clipboard
agent-desktop clipboard-clear                   # Clear clipboard

Wait

agent-desktop wait 1000                         # Pause 1 second
agent-desktop wait --element @e5 --snapshot s... --timeout 5000 # Wait for element
agent-desktop wait --window "Title"             # Wait for window
agent-desktop wait --text "Done" --app "App"    # Wait for text
agent-desktop wait --menu --app "App"           # Wait for menu surface
agent-desktop wait --menu-closed --app "App"    # Wait for menu dismissal
agent-desktop wait --notification --app "App"   # Wait for new notification

System

agent-desktop status                            # Health check
agent-desktop permissions                       # Check permission
agent-desktop permissions --request             # Trigger permission dialog
agent-desktop version --json                    # Version info
agent-desktop batch '[...]' --stop-on-error     # Batch uses the same typed command path as CLI
agent-desktop skills                            # List bundled skill docs
agent-desktop skills get desktop --full         # Load this skill + all references

Key Principles for Agents

  1. Skeleton first, drill second. Start with --skeleton -i --compact for dense apps. Drill into regions with --root @ref. Full snapshot only for simple apps.
  2. Use -i --compact flags. Filters to interactive elements and collapses empty wrappers, minimizing tokens.
  3. Refs are snapshot-scoped. Keep snapshot_id for deterministic multi-step use; re-drill the affected region after any UI-changing action. Scoped invalidation keeps other refs intact.
  4. Prefer refs over coordinates. click @e5 > mouse-click --xy 500,300.
  5. Use wait for async UI. After launch/dialog triggers, wait for expected state.
  6. Check permissions first. Run permissions on first use; screenshots also need Screen Recording.
  7. Handle errors. Parse error.code and follow error.suggestion.
  8. Use find for targeted searches. Faster than any snapshot when you know role/name.
  9. Use surfaces for overlays. snapshot --surface menu for menus, --surface sheet for dialogs. Never --skeleton for surfaces — they're already focused.
  10. Batch for performance. Multiple commands in one invocation.
  11. Headless by default. Ref actions use semantic AX paths and block silent focus stealing, cursor movement, keyboard synthesis, and pasteboard insertion. Use explicit focus, press, hover, drag, or mouse-* commands only when physical/headed interaction is intended.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.