Remote GUI Operation

Enables Claude Code to see and interact with GUI applications on remote headless Linux servers.

When to use

• You need to operate a desktop application on a server that has no monitor
• Examples: VPN clients, IDEs, database GUIs, file managers, browser-based tools
• The application has no CLI alternative

Auto-Setup Flow

When this skill is triggered, Claude should automatically:

1. Check and install dependencies (if missing):

# Check what's installed
which Xvfb xdotool scrot fluxbox 2>/dev/null

# Install missing packages (auto-detect package manager)
if command -v apt &>/dev/null; then
    echo 'PASSWORD' | sudo -S apt install -y xvfb fluxbox scrot xdotool imagemagick
elif command -v yum &>/dev/null; then
    echo 'PASSWORD' | sudo -S yum install -y xorg-x11-server-Xvfb fluxbox scrot xdotool ImageMagick
fi

2. Check if virtual display is already running (don't start duplicates):

# Check existing Xvfb
ps aux | grep Xvfb | grep -v grep
# If not running, start it:
Xvfb :99 -screen 0 1280x720x24 &>/dev/null &
export DISPLAY=:99

3. Check if window manager is running (fixes black screenshot):

# If fluxbox not running → screenshots will be black!
ps aux | grep fluxbox | grep -v grep || fluxbox &>/dev/null &
sleep 2

4. Launch the target GUI app
5. Enter screenshot-action loop

Auto-Fix: Black Screenshot

If screenshot is all black, Claude should automatically:

# 1. Check if fluxbox is running
if ! pgrep -x fluxbox > /dev/null; then
    export DISPLAY=:99
    fluxbox &>/dev/null &
    sleep 2
fi

# 2. Check if the app window exists
xdotool search --name "" | head -5
# If no windows → app didn't start, re-launch it

# 3. Retry screenshot
scrot /tmp/screen.png

Auto-Fix: App Won't Launch

# Check DISPLAY is set
echo $DISPLAY  # Should be :99

# Check Xvfb is running
pgrep Xvfb || (Xvfb :99 -screen 0 1280x720x24 &>/dev/null &)

# Check app error output
/path/to/app 2>&1 | head -20
# Fix missing libraries, permissions, etc.

Auto-Fix: Click Misses Target

# Window may have moved. Take fresh screenshot and recalculate coordinates.
scrot /tmp/screen_fresh.png
# Re-read screenshot and find new button position

Commands Reference

Setup

Xvfb :99 -screen 0 1280x720x24 &>/dev/null &
export DISPLAY=:99
fluxbox &>/dev/null &

Screenshot

CRITICAL: scrot PNG files contain ICC profile metadata that causes Claude API "Could not process image" errors. ALWAYS convert to JPEG with Pillow to strip ICC profile.

# Take screenshot and convert to JPEG (strips ICC profile)
scrot /tmp/screenshot.png && python3 -c "
from PIL import Image
Image.open('/tmp/screenshot.png').convert('RGB').save('/tmp/screenshot.jpg', 'JPEG', quality=95)
" && rm /tmp/screenshot.png

# Transfer JPEG (not PNG!) to local
scp server:/tmp/screenshot.jpg /local/path/

If you accidentally sent a bad PNG to Claude:

1. /restore — best option, rolls back to before the bad image entered context
2. /compact — compresses history, bad image disappears but loses detail
3. /clear or new conversation — last resort

Prevention (PostToolUse hook): Add to settings.json to auto-convert every screenshot after Bash commands:

{
  "hooks": {
    "PostToolUse": [{
      "matcher": "Bash",
      "command": "python3 -c \"import glob; from PIL import Image; [Image.open(f).convert('RGB').save(f.replace('.png','.jpg'),'JPEG',quality=95) for f in glob.glob('/tmp/*.png')]\" 2>/dev/null || true"
    }]
  }
}

Mouse

xdotool mousemove X Y          # Move to coordinates
xdotool click 1                # Left click
xdotool click 3                # Right click
xdotool click --repeat 2 1     # Double click
xdotool mousemove X Y click 1  # Move and click in one command
xdotool click 4                # Scroll up
xdotool click 5                # Scroll down

Keyboard

xdotool type 'text'                  # Type text
xdotool type --delay 50 'slow text'  # Type with delay (for slow apps)
xdotool key Return                    # Enter
xdotool key Tab                       # Tab
xdotool key ctrl+a                    # Select all
xdotool key ctrl+c                    # Copy
xdotool key ctrl+v                    # Paste
xdotool key Escape                    # Escape
xdotool key BackSpace                 # Backspace

Window Management

xdotool search --name "Window Title"  # Find window by title
xdotool windowactivate WINDOW_ID      # Focus window
xdotool getactivewindow                # Get current window ID

Limitations

• ~3-5 seconds per action cycle
• Requires sudo for initial package installation
• No audio support
• Very small text may be hard to read (use higher resolution: 1920x1080x24)