Computer Control

See → Think → Act → Verify loop for GUI automation on Windows.

Combines screenshot capture, vision model analysis, and Win32 input APIs to give Claude eyes and hands on your desktop.

Architecture

┌──────────┐     ┌──────────┐     ┌──────────────┐     ┌──────────┐
│  See     │ ──▶ │  Think   │ ──▶ │    Act       │ ──▶ │  Verify  │
│ screen.  │     │ vision   │     │ mouse.ps1    │     │ screen.  │
│ ps1      │     │ analysis │     │ keyboard.ps1  │     │ ps1+vis. │
└──────────┘     └──────────┘     │ window.ps1    │     └──────────┘
                                  └──────────────┘

Scripts

All scripts are in scripts/ directory relative to this skill. Call them via PowerShell:

All script paths below use $SKILL as shorthand for <path-to-this-skill>/scripts.

Quick Reference

Screen Capture

# Full virtual screen (all monitors) → returns file path
powershell .claude/skills/computer-control/scripts/screen.ps1 -Quiet

# Specific monitor
powershell .claude/skills/computer-control/scripts/screen.ps1 -Monitor 0 -Quiet

# Capture a region
powershell .claude/skills/computer-control/scripts/screen.ps1 -Region "100,200,500,400" -Quiet

# List monitors
powershell .claude/skills/computer-control/scripts/screen.ps1 -Action monitors

# Custom output path
powershell .claude/skills/computer-control/scripts/screen.ps1 -Output "C:\temp\shot.png"

Mouse Control

# Get current position
powershell .claude/skills/computer-control/scripts/mouse.ps1 -Action getpos

# Move cursor
powershell .claude/skills/computer-control/scripts/mouse.ps1 -Action move -X 500 -Y 300

# Click at position
powershell .claude/skills/computer-control/scripts/mouse.ps1 -Action clickat -X 500 -Y 300

# Double-click, right-click
powershell .claude/skills/computer-control/scripts/mouse.ps1 -Action dclickat -X 500 -Y 300
powershell .claude/skills/computer-control/scripts/mouse.ps1 -Action rclickat -X 500 -Y 300

# Drag from A to B
powershell .claude/skills/computer-control/scripts/mouse.ps1 -Action drag -X1 100 -Y1 200 -X2 500 -Y2 600

# Scroll (120 = one notch up, -120 = one notch down)
powershell .claude/skills/computer-control/scripts/mouse.ps1 -Action scroll -Amount -120

# Hold and release (for complex drags)
powershell .claude/skills/computer-control/scripts/mouse.ps1 -Action mousedown
powershell .claude/skills/computer-control/scripts/mouse.ps1 -Action mouseup

Keyboard Input

# Type text
powershell .claude/skills/computer-control/scripts/keyboard.ps1 -Action type -Text "Hello World"

# Key combination
powershell .claude/skills/computer-control/scripts/keyboard.ps1 -Action combo -Keys "Ctrl+C"
powershell .claude/skills/computer-control/scripts/keyboard.ps1 -Action combo -Keys "Alt+Tab"
powershell .claude/skills/computer-control/scripts/keyboard.ps1 -Action combo -Keys "Win+R"

# Single special key
powershell .claude/skills/computer-control/scripts/keyboard.ps1 -Action key -Key "Enter"
powershell .claude/skills/computer-control/scripts/keyboard.ps1 -Action key -Key "Escape"
powershell .claude/skills/computer-control/scripts/keyboard.ps1 -Action key -Key "Tab"
powershell .claude/skills/computer-control/scripts/keyboard.ps1 -Action key -Key "F5"
powershell .claude/skills/computer-control/scripts/keyboard.ps1 -Action key -Key "Down"

# Windows shortcuts
powershell .claude/skills/computer-control/scripts/keyboard.ps1 -Action shortcut -Keys "Win+D"
powershell .claude/skills/computer-control/scripts/keyboard.ps1 -Action shortcut -Keys "Win+E"

Window Management

# List all visible windows
powershell .claude/skills/computer-control/scripts/window.ps1 -Action list

# Find window by partial title
powershell .claude/skills/computer-control/scripts/window.ps1 -Action find -Title "Chrome"

# Focus a window
powershell .claude/skills/computer-control/scripts/window.ps1 -Action focus -Title "Notepad"

# Get window position/size
powershell .claude/skills/computer-control/scripts/window.ps1 -Action info -Title "Calculator"

# Move window
powershell .claude/skills/computer-control/scripts/window.ps1 -Action move -Title "Chrome" -X 0 -Y 0

# Resize and reposition
powershell .claude/skills/computer-control/scripts/window.ps1 -Action resize -Title "Chrome" -Width 1280 -Height 720 -X 0 -Y 0

# Minimize / Maximize / Restore / Close
powershell .claude/skills/computer-control/scripts/window.ps1 -Action minimize -Title "Settings"
powershell .claude/skills/computer-control/scripts/window.ps1 -Action maximize -Title "Chrome"
powershell .claude/skills/computer-control/scripts/window.ps1 -Action restore -Title "Chrome"
powershell .claude/skills/computer-control/scripts/window.ps1 -Action close -Title "Notepad"

The See → Think → Act → Verify Loop

This is the core workflow. Follow this pattern for every GUI interaction:

Step 1: See (Capture Screen)

powershell .claude/skills/computer-control/scripts/screen.ps1 -Quiet
# Output: C:\Users\lanxi\AppData\Local\Temp\claude_screenshot_20260620_143022_456.png

Step 2: Think (Analyze with Vision)

Use the vision skill to understand what's on screen:

python .claude/skills/vision/vision.py "<screenshot_path>" "Describe this screen: what UI elements do you see? Identify buttons, text fields, menus, links, and their approximate positions (estimate x,y coordinates for each interactive element)."

For coordinate estimation, ask specifically:

python .claude/skills/vision/vision.py "<screenshot_path>" "For the [specific UI element I want to interact with], estimate its center pixel coordinates (x,y). The screen resolution is WxH."

Step 3: Act (Execute)

Based on vision analysis, execute the appropriate action:

# Click a button at estimated coordinates
powershell .claude/skills/computer-control/scripts/mouse.ps1 -Action clickat -X <x> -Y <y>

# Type in a text field (click it first, then type)
powershell .claude/skills/computer-control/scripts/mouse.ps1 -Action clickat -X <x> -Y <y>
powershell .claude/skills/computer-control/scripts/keyboard.ps1 -Action type -Text "..."

Step 4: Verify (Screenshot + Check)

powershell .claude/skills/computer-control/scripts/screen.ps1 -Quiet

python .claude/skills/vision/vision.py "<new_screenshot>" "Did the action succeed? Compare: [describe expected outcome]."

If not successful, go back to Step 2 with the new screenshot and adjust.

Patterns

Pattern 1: Click a Button by Description

1. Screenshot full screen
2. Vision: "Find the 'Save' button and estimate its center coordinates"
3. Click at the estimated position
4. Screenshot to verify button was clicked (e.g., dialog appeared, page changed)

Pattern 2: Fill a Form

1. Screenshot
2. Vision: "Find the following fields and estimate coordinates: Name field, Email field, Submit button"
3. Click Name field → type name
4. Click Email field → type email
5. Click Submit button
6. Screenshot → Vision: "Did the form submit successfully?"

Pattern 3: Navigate a Menu

1. Screenshot
2. Vision: "Find the 'File' menu and its position. What menu items are visible?"
3. Click File menu
4. Screenshot → Vision: "What dropdown items appeared? Find 'Export' position."
5. Click Export
6. Screenshot → Verify

Pattern 4: Drag and Drop

1. Screenshot
2. Vision: "Find the element to drag and the drop target. Estimate both positions."
3. Drag from source to target
4. Screenshot → Verify element is in new position

Pattern 5: Scroll and Scan

1. Screenshot top of page
2. Vision: "Is [target element] visible?"
3. If no → scroll down → screenshot → repeat
4. If yes → interact with it

Pattern 6: Wait for UI Change

1. Screenshot
2. Perform action (click button, submit form, etc.)
3. Wait 1-3 seconds
4. Screenshot
5. Vision: "Has [expected change] occurred? Compare with previous state."
6. If not, wait longer or investigate

Coordinate Estimation

When asking vision to estimate coordinates, provide screen context:

python .claude/skills/vision/vision.py "screen.png" \
  "The screen is 1920x1080. 
   Find the 'Start' button (Windows taskbar, bottom-left corner). 
   Estimate the CENTER x,y pixel coordinates of this button. 
   Output format: x,y"

For better accuracy:

• Mention known screen dimensions
• Describe the element clearly by its label, position relative to other elements, or visual appearance
• Ask for the center of the element, not its edge
• If unsure, ask vision to describe the layout first, then pinpoint in a follow-up

Safety Rules

1. Confirm destructive actions — before clicking Delete, Remove, Uninstall, or similar
2. Small movements first — when navigating unfamiliar UI, move step by step
3. Always verify — screenshot after every action to confirm it did what you expected
4. Report coordinates before clicking — tell the user "Clicking at (X,Y) on [element]" before executing
5. Handle errors gracefully — if vision can't find an element, ask the user or try scrolling
6. CLEAN UP SCREENSHOTS — delete screenshot files after the task is complete:

   Remove-Item "C:\Users\lanxi\AppData\Local\Temp\claude_screenshot_*.png" -Force

   Or during the session, keep only the most recent 1-2 screenshots and delete older ones.

Troubleshooting

Dependencies

• Windows 10/11 with PowerShell 5.1+
• .NET Framework 4.x (included with Windows)
• Vision skill (~/.claude/skills/vision/) with at least one API key configured
• No external installations required — all scripts use built-in .NET/Win32 APIs

Example: Full Workflow

Here's a complete example — opening Notepad, typing text, and saving:

# 1. Open Run dialog and launch Notepad
powershell .claude/skills/computer-control/scripts/keyboard.ps1 -Action shortcut -Keys "Win+R"
Start-Sleep -Milliseconds 500
powershell .claude/skills/computer-control/scripts/keyboard.ps1 -Action type -Text "notepad{ENTER}"
Start-Sleep -Seconds 1

# 2. Verify Notepad opened
$shot = powershell .claude/skills/computer-control/scripts/screen.ps1 -Quiet
# vision: "Is Notepad visible? What's the active window?"

# 3. Type some text
powershell .claude/skills/computer-control/scripts/keyboard.ps1 -Action type -Text "Hello from Claude! This is computer control in action."

# 4. Save with Ctrl+S
powershell .claude/skills/computer-control/scripts/keyboard.ps1 -Action combo -Keys "Ctrl+S"
Start-Sleep -Milliseconds 500

# 5. Screenshot to see Save dialog
$shot2 = powershell .claude/skills/computer-control/scripts/screen.ps1 -Quiet
# vision: "Find the file name field in the Save dialog. Estimate coordinates."

# 6. Type file name and click Save
# (coordinates estimated by vision)
powershell .claude/skills/computer-control/scripts/mouse.ps1 -Action clickat -X <x> -Y <y>
powershell .claude/skills/computer-control/scripts/keyboard.ps1 -Action type -Text "test_note.txt"
# ... continue with Save button

# 7. Clean up screenshots
Remove-Item "$env:TEMP\claude_screenshot_*.png" -Force