uitars-mcp

# uitars-mcp — Full LLM Documentation

## Project Overview

uitars-mcp is a desktop + browser GUI agent MCP server. It gives AI agents eyes and hands
via visual grounding: capture screenshots → VLM → execute actions. 8 MCP tools total.

Single uvicorn serves FastAPI REST API (port 10976) + FastMCP HTTP (/mcp).
Vite + React webapp (port 10977) with Desktop, Browser, and Help tabs.

## Quick Start

```powershell
git clone https://github.com/sandraschi/uitars-mcp.git
cd uitars-mcp
uv sync
# Browser operator (optional):
uv sync --extra browser
uv run playwright install chromium
# Configure VLM:
$env:UITARS_VLM_BASE_URL = "http://127.0.0.1:11434/v1"
$env:UITARS_VLM_MODEL = "qwen2.5-vl:7b"
.\web_sota\start.ps1
```

## MCP Tools (8)

### Desktop (5)
1. **uitars_execute(task, max_steps=15)** — GUI task end-to-end
2. **uitars_screenshot()** — Desktop capture as base64 PNG
3. **uitars_click(x, y, button="left")** — Click at coordinates
4. **uitars_type(text)** — Type text at focus
5. **uitars_help()** — Inline help, examples, config summary

### Browser (3)
6. **uitars_browser_navigate(url)** — Open URL, return page screenshot
7. **uitars_browser_execute(task, start_url?, max_steps=15)** — Browser task via VLM
8. **uitars_browser_close()** — Close browser, free Playwright resources

## REST API

| Method | Path | Description |
|--------|------|-------------|
| GET | /api/health | Backend readiness |
| GET | /api/status | VLM model health |
| GET | /api/screenshot | Desktop capture |
| POST | /api/execute | Execute desktop task |
| POST | /api/browser/navigate | Navigate browser to URL |
| POST | /api/browser/execute | Execute browser task |
| POST | /api/browser/close | Close browser |
| * | /mcp | FastMCP HTTP endpoint |

## Configuration

| Variable | Default | Description |
|----------|---------|-------------|
| UITARS_PORT | 10976 | Backend port |
| UITARS_FRONTEND_PORT | 10977 | Frontend port |
| UITARS_VLM_BASE_URL | http://127.0.0.1:11434/v1 | VLM endpoint |
| UITARS_VLM_MODEL | qwen2.5-vl:7b | Model name |
| UITARS_VLM_API_KEY | ollama | API key |
| UITARS_MAX_STEPS | 15 | Max actions per task |

## VLM Providers

- **Ollama (local):** qwen2.5-vl:7b — ~5.5 GB VRAM
- **vLLM (local):** UI-TARS-1.5-7B — ~18 GB VRAM (FP16)
- **Anthropic API:** claude-sonnet-4 — no GPU
- **OpenAI API:** gpt-4o — no GPU

## Architecture

```
Agent → MCP tool call → FastMCP (/mcp)
  → operators/computer.py: capture_screenshot (mss) → VLM → execute (pyautogui)
  → operators/browser.py: page.screenshot → VLM → execute (Playwright)
  → loop until finished() or max_steps
```

## Safety

- pyautogui.FAILSAFE = True — move mouse to corner (0,0) to abort
- max_steps limit prevents infinite loops
- Browser: headless Chromium, isolated session
- Local VLM: screenshots never leave your machine

## Dependencies

Python: fastmcp>=3.2, fastapi, uvicorn, httpx, ui-tars, mss, pyautogui, pillow
Browser (optional): playwright>=1.48
Node (webapp): react 18, vite 5, typescript 5.6