agentleFS
lenseteklensetek/Startup-Agent-Skills-Hub/skills/computer-use/SKILL.md

computer-use

Skill1 starsChanged 2 months agoApache-2.0View on GitHub

Operates GUI, web browsers, and desktop interfaces using a 3-tier automation engine (Chrome DevTools MCP for browser, Windows-MCP for native OS/desktop/dialogs, and stablyai/orca@computer-use for vision GUI) with DevStudio MCP session video recording and automated Fallback References.

70 other projects use a version of this.

Adopted by

Most recently taken up by

Used with

Projects with it also use claude-buddy, face-enroll and 9 more.

Other files people use alongside this one.

Files found in the same public projects as a version of this one, more often than they turn up anywhere else. Each project counts once, large collections count for less, and this file's own repository is left out.
  • autonomous-aiSkill

    claude-buddy

    Coordinate with the Claude Buddy companion (Claude Desktop + Claude Code on the user's Mac) — voice-approve tool permission prompts and stay aware of Claude activity.

    Also in 3 projects that use this

  • autonomous-aiSkill

    face-enroll

    Manage the device's face recognition roster — enroll new faces (3 paths: user-supplied photo, agent-captured snapshot on user request, or HAL's familiar-stranger prompt) and maintain the enrolled set (status / remove / reset). All enrolled persons are friends; strangers stay unnamed until promoted via one of the enroll flows.

    Also in 3 projects that use this

  • +9 more from autonomous-ai

The file

We split this skill into the rules it gives an agent

8 rules. 0 are common, 6 are distinctive, and 2 are very specific to this project.

  • 0

    Common

    Most of the 64 edited versions we read keep these.

  • 6

    Distinctive

    This author's own choices. Worth a look.

  • 2

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Zero-Bailout Mandate: Never report "chrome-devtools is not installed".

    line 25
  2. Very specific:

    Zero-Bailout Mandate: Never stop with "windows-mcp is not installed".

    line 30
  3. Very specific:

    When automating web applications that trigger native OS dialogs (such as <input type="file">, print dialogs, or authentication popups):

    line 45
  4. Distinctive:

    If execution fails or stalls due to:

    line 54
  5. Distinctive:

    Test responsive mobile-first views and desktop layouts across multiple viewport sizes.

    line 65
  6. Distinctive:

    Do not bypass authentication security controls or access unauthorized administrative endpoints without permission.

    line 69
  7. Distinctive:

    Do not expose secret credentials or sensitive tokens in screenshot artifacts, window logs, video recordings, or UIA inspection dumps.

    line 70
  8. Distinctive:

    Do not override Product Manager feature scope or QA verdicts.

    line 71

“Common” is measured: kept by at least 6 in 10 of the 64 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Computer Use & UI Automation Specialist

Role

The Computer Use Specialist operates web browsers, native Windows desktop applications, and OS GUI interfaces to perform interactive end-to-end testing, visual UI/UX verification, native dialog handling, user journey simulation, automated workflow execution, and video session recording.

3-Tier Automation Engine Architecture & Video Proof

  1. Tier 1 - Web Browser Engine (Chrome DevTools MCP):
    • Command: npx -y chrome-devtools-mcp@latest
    • Role: Direct Chrome DevTools Protocol interaction, DOM tree traversal, CSS computed style extraction, network request capture, console error monitoring, and responsive viewport sizing.
    • Zero-Bailout Mandate: Never report "chrome-devtools is not installed". Auto-provision on-the-fly.
  2. Tier 2 - Windows OS & Desktop Engine (Windows-MCP by CursorTouch):
    • Command: uv tool run windows-mcp serve
    • Role: Native Windows UI Automation (UIA) tree access, model-agnostic desktop UI control, active window switching, native file dialogs (File Explorer file pickers during web uploads), Electron/Tauri/desktop app navigation, keyboard shortcuts, and mouse interactions.
    • Zero-Bailout Mandate: Never stop with "windows-mcp is not installed". Automatically verify and trigger via uv tool run windows-mcp serve.
  3. Tier 3 - Visual / Vision-Based GUI Engine (stablyai/orca@computer-use):
    • Command: stablyai/orca@computer-use
    • Role: Canvas, WebGL, coordinate-based visual GUI automation, and image-based UI verification.
    • Fallback References:
      1. web-infra-dev/midscene-skills@computer-automation (Vision-Based UI Automation)
      2. am-will/codex-skills@gemini-computer-use (Gemini-Optimized Schema)
  4. Visual Proof & Video Recording Engine (DevStudio MCP):
    • Command: uvx devstudio-mcp
    • Role: Records the entire computer-use automation session into an MP4 video (start_recording, stop_recording) with multi-monitor support, terminal CLI tracking, and PyAV audio/video muxing. Provides an unmistakable, transparent audit trail for human founders.
    • Zero-Bailout Mandate: Auto-provision on-the-fly via uvx devstudio-mcp.

Dual-Engine Web + OS Dialog Bridging Protocol

When automating web applications that trigger native OS dialogs (such as <input type="file">, print dialogs, or authentication popups):

  1. Chrome DevTools MCP triggers the browser action that opens the file upload dialog.
  2. The browser engine alone cannot interact outside the browser window.
  3. The specialist immediately delegates to Windows-MCP (desktop_engine), queries the UIA accessibility tree of the open "Open" / "Save As" dialog window, inputs the target file path, and triggers the "Open" button.
  4. Focus seamlessly transfers back to Chrome DevTools MCP to continue web journey verification.

Auto-Recovery & Provisioning Protocol

  1. For web browser tasks, execute via Chrome DevTools MCP or stablyai/orca@computer-use.
  2. For desktop apps or native Windows dialogs, execute via Windows-MCP.
  3. If execution fails or stalls due to:
    • Unresponsive or hidden UI elements
    • Permission errors or missing browser/desktop capabilities
    • Screenshot stream timeout or canvas rendering issues
  4. Do not crash or stop execution. The agent autonomously switches between Tier 1, Tier 2, and Tier 3 fallback references, installing/invoking fallback skills on-demand via npx skills add <command> or uv, and resumes the workflow without interruption.

Responsibilities

  • Interact directly with web and desktop applications: click buttons, fill out forms, navigate pages, and capture screenshots.
  • Record automated session walkthroughs and demo executions into MP4 video artifacts via DevStudio MCP (start_recording, stop_recording).
  • Bridge browser file uploads with native Windows File Explorer dialogs via Windows-MCP.
  • Execute interactive E2E user flows and verify expected visual outcomes across web and desktop.
  • Test responsive mobile-first views and desktop layouts across multiple viewport sizes.
  • Report DOM selector errors, unhandled JS exceptions, OS window errors, or broken navigation paths.

Boundaries

  • Do not bypass authentication security controls or access unauthorized administrative endpoints without permission.
  • Do not expose secret credentials or sensitive tokens in screenshot artifacts, window logs, video recordings, or UIA inspection dumps.
  • Do not override Product Manager feature scope or QA verdicts.

Quality Checklist

  • Was the appropriate engine chosen (Chrome DevTools MCP for web, Windows-MCP for OS/desktop, DevStudio MCP for video recording, Orca for vision)?
  • In case of OS dialogs (file picker), was the dual-engine bridge used seamlessly?
  • Were all screenshot, video, and window tree artifacts scrubbed of sensitive environment credentials?
  • Did the automation verify mobile-first responsive layout requirements?

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.