desk
Skill0 starsChanged 40 days agoMITView on GitHub
Use when a task needs your Mac itself — clicking, typing into a form or app, or seeing what is on screen — and there is no CLI, API or MCP route. Drives the mouse and keyboard with the `desk` command (screenshot, click, type, key, scroll, open).
3 other projects use a version of this.
Adopted by
Most recently taken up by
- conikeec/desktheir own version, first seen Oct 15, 2026
- angletech2026-arch/desktop-sensetheir own version, first seen Oct 13, 2026
- aylee/agent-ostheir own version, first seen Jul 13, 2026
Other versions3 projects use a version of this.
Word for word · 0
None found.
Edited · 0
None found.
Wrote their own version · 3
- angletech2026-arch/desktop-sense★ 16same name
- aylee/agent-os★ 0same name
- conikeec/desk★ 0same name
How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.
In this repository3 other files it gives its agents.
- brain-polishSkill
- cusSkill
- wildSkill
The file
We split this skill into the rules it gives an agent
8 rules. 0 are common, 4 are distinctive, and 4 are very specific to this project.
0
Common
Most of the 3 edited versions we read keep these.
4
Distinctive
This author's own choices. Worth a look.
4
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 11
Every desk call needs dangerouslyDisableSandbox: true (the sandbox blocks screen capture and input).
- Distinctive: line 15
Look with the cheapest tool that answers the question:
- Very specific: line 42
Look with ui --find or ui first.
- Distinctive: line 46
Check the result before going on.
- Very specific: line 49
Prefer the keyboard when it's reliable (tab between fields, cmd+l for the address bar).
- Distinctive: line 61
Stop and ask in chat before any click that:
- Distinctive: line 70
Before starting, tell the user you're taking the mouse, since they may be using the machine.
- Very specific: line 73
Prefer a real route over the GUI when one exists: a CLI, an API, an MCP tool, or open <url>.
“Common” is measured: kept by at least 6 in 10 of the 3 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
desk: use your Mac
desk (from screen-intelligence, screen_intelligence/desk.py, installed on PATH) gives you
eyes and hands. You are the vision model and the planner; it only captures and acts.
Every desk call needs dangerouslyDisableSandbox: true (the sandbox blocks screen capture and input).
Commands
Look with the cheapest tool that answers the question:
| Command | Does | Cost |
desk ui --find TEXT [App] | Only the elements whose label contains TEXT. Use this to locate a known button or field. | ~40 tokens |
desk ui [--full] [App] | The window's buttons, fields, links and labels as numbered lines (first 80; --all for everything). | ~700 tokens |
desk shot | Screenshot to runtime/desk.png (1280 wide, faint grid, red circle at the mouse); then Read it. For apps that draw their own pixels (the ui says "nothing readable"), or to see layout, colours and images. | ~1,200 tokens |
[App] is the name as macOS shows it (Safari, Google Chrome, Finder); without it, the frontmost app, which is usually the terminal.
Act:
| Command | Does |
desk click #N | Click element N from the last ui. Brings its app to the front first, since the terminal often covers it. |
desk click X Y [right|double] | Click at (X, Y) in the last screenshot's pixels. |
desk move X Y / desk move #N | Put the cursor there without clicking. Do this before scroll: macOS scrolls whatever is under the cursor, which is often the terminal. |
desk drag X1 Y1 X2 Y2 | Press, move slowly, release (screenshot pixels, or #N for either end). |
desk type "text" | Pastes the text (your clipboard is restored after). Exact, including : and capitals. |
desk type --keys "text" | Real keystrokes, for native menus and pickers that ignore paste (e.g. choosing "September" in a select). |
desk key cmd+l · desk scroll down 5 · desk open "App" · desk wait 0.5 | |
desk expect "Add project" Safari | Stops unless a label containing the text shows up within 3s (--timeout S). ~40 tokens. |
desk until "log out|logout" Safari --timeout 900 | Wait (checking every 2s) until a label or the window title contains any of the texts; --gone waits for it to disappear. For handing off to the user. |
desk run "click #4; expect 'First name'; type Ada; key tab; type Lovelace; ui --find Submit Safari" | Several steps in one call. Every click waits up to 1.5s for the clicked window to change; if it doesn't, the batch stops and names the step. If the user moves the mouse, the batch finishes the current step and stops; hand control back, don't retry. |
Loop
- Look with
ui --findoruifirst;shotonly when text isn't enough. - Act in batches. Clicks are change-checked, so batch whole sequences with
run: put anexpectafter each step that should open or reveal something (a form, a menu, a new page), and end with one cheap look. Screenshot only when a batch stops or the final look is ambiguous. Still take risky clicks (submit, send, delete) one at a time. - Check the result before going on. If it didn't do what you expected, look again (a screenshot if the text is ambiguous); don't click harder.
Element numbers only stay valid until the page changes; after any click that navigates or opens something, run ui again.
Prefer the keyboard when it's reliable (tab between fields, cmd+l for the address bar).
Hand off to the user (logins, 2FA, captchas)
Get the user to the right screen first (open the URL, click "Log in"). Then start
desk until "<something only visible once they're done>" <App> --timeout 900 with run_in_background: true, and in the
same turn tell them exactly what to do ("log in with your school account; I'll carry on when you're in"). When the
background task finishes, carry on without waiting for them to say so. Pick a marker that can't appear on the
login page itself: "Log out" or their name, not "Dashboard".
Ask the user first
Stop and ask in chat before any click that:
- submits a form or application, sends a message or email, posts, or buys anything;
- deletes, overwrites, or changes a setting or permission;
- answers a legal or eligibility question (work authorization, attestations, privacy consent) — those are their answers;
- needs a password, 2FA code or payment detail — never type these; hand back to them.
Fill only facts from ~/Notes or that the user gave in this conversation. Leave anything else blank and list it for them.
Captchas are theirs.
Before starting, tell the user you're taking the mouse, since they may be using the machine. They can abort any time by slamming the mouse into a screen corner (pyautogui's fail-safe).
Prefer a real route over the GUI when one exists: a CLI, an API, an MCP tool, or open <url>.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.