unix_computer_use

---
name: unix_computer_use
description: Local and remote desktop observation/input tools with coordinate normalization (local macOS/Linux by default; optional OSWorld HTTP and SSH Mac backends).
version: 0.4.2
type: extension
entry: plugin.py
plugin_api: "2.0"
runtime: python3
permissions: [tool, subprocess, net]
env_from_settings: []
when_to_use: The user asks Ouroboros to inspect a desktop, take a screenshot, click/type/press keys, drag, or operate a local/explicitly configured remote GUI under human or benchmark supervision.
model_experience:
  what_model_sees: Roughly two dozen desktop tools (screenshot, click/type/key/scroll input, connection registry) join the tool list, and each screenshot auto-attaches a WXGA-bounded image to the conversation in the same round.
  token_effect: The tool schemas add a moderate fixed cost to every round while the skill is enabled; each screenshot adds one image attachment (the dominant per-use cost), so prefer batching actions per capture.
---

# Unix Computer Use

This bundled extension exposes a reviewable computer-use substrate for local
macOS/Linux desktops and explicitly configured remote targets. **All actions are
for supervised, low-risk workflows under human or benchmark observation.** Windows
support is deferred to a separate backend/skill.

Tools:

- `capabilities` reports the platform, display-session type, and detected
  screenshot/input backends plus the active connection/backend.
- `list_connections`, `add_connection`, `test_connection`, `activate_connection`,
  and `use_local` manage the local/remote connection registry in skill state.
- `screenshot` captures the desktop, downscales the image to fit WXGA
  (1280x800, configurable) and persists the exact image-to-input coordinate
  transform. Pass coordinates read off the returned image directly to the
  input tools — they are remapped automatically. The result carries the typed
  `auto_attach_image` field, so the host attaches the image to the
  conversation in the same round (v6.81.1) — no `view_image` call is needed
  for a fresh capture; the returned `path` stays `view_image`-readable for
  re-viewing later.
- `click` (left/right/middle, double/triple), the thin aliases
  `double_click`/`triple_click` (models call these names unprompted), `move`,
  `left_click_drag`, `mouse_down`/`mouse_up`, `cursor_position`, `type_text`,
  `key`, `hold_key`, `scroll`, and `wait` execute input through platform tools
  when available. Pointer coordinates pass through one normalizer that accepts
  the malformations models actually emit (the pair packed into `x` with `y`
  absent or duplicated) and fails loudly on anything ambiguous.
- `window_list` lists visible windows/processes where the platform exposes a
  lightweight backend.
- `ax_tree` returns a set-of-marks accessibility snapshot of the frontmost
  window on macOS: numbered interactive elements (role, title, center
  coordinates in input space — click them with `raw=true`). It degrades
  honestly to a process/window list when the AX walk fails or on Linux.
- `remote_exec` runs a shell command on the active remote backend only; it
  refuses on local backend. Use ordinary Ouroboros shell/file tools for local
  work.

Coordinate contract:

- Input tools accept coordinates in the LAST screenshot's image space and
  remap them through the stored transform; `raw=true` bypasses remapping.
- macOS input consumes LOGICAL points while screenshots are physical pixels;
  the stored transform already folds the Retina scale in. On multi-display
  Macs the scale is approximate (`approx` flag) — prefer `ax_tree` marks.

Backend matrix (honest limitations):

- macOS: `screencapture` + `cliclick` + `osascript`; `sips` for downscaling.
  `scroll` is **unsupported** (cliclick has no scroll-wheel command — page via
  `key` with `page-down`/`page-up`); `middle` click and non-left
  `mouse_down`/`mouse_up` are unsupported via cliclick. `hold_key` holds
  PURE-MODIFIER combos only (`cmd`, `cmd+shift`, ...): cliclick `kd:`/`ku:`
  accept only modifiers and `kp:` cannot hold — non-modifier holds report an
  honest error. In `key` combos the base key is tapped while modifiers are
  held (`cmd+s` = hold cmd, tap s). TCC permission state (Screen Recording /
  Accessibility) is NOT verified: tools exit 0 even when denied
  (wallpaper-only capture / dropped input) — ensure grants.
- Linux X11: `xdotool` (input), `gnome-screenshot`/`scrot` (capture),
  `wmctrl` (windows), ImageMagick (downscaling). Function keys and
  case-sensitive keysyms are supported (`f5`→`F5`, `XF86AudioPlay` as-is).
- Linux Wayland: `ydotool` (pointer + typing; requires a running `ydotoold`)
  or `wtype` (typing only), `grim` or `gnome-screenshot` (capture). `xdotool`
  does NOT work on Wayland. `key` and `hold_key` are **unsupported** on
  Wayland (`ydotool key` takes raw keycodes only — combos would silently
  no-op, which this skill refuses to fake); use `type_text` for text.
  Pointer press/release uses ydotool button masks (`0x40`/`0x80`).
- Remote OSWorld HTTP: `screenshot` uses `GET <target>/screenshot`; input and
  `remote_exec` use `POST <target>/execute`, matching OSWorld's in-VM
  `pyautogui` channel. The runner writes an `osworld_http` connection before
  each task. Honesty guarantees on this backend: `scroll` clicks are 1:1 wheel
  detents (not multiplied); non-ASCII `type_text` is pasted via the in-VM
  clipboard (`pyperclip` + `ctrl+v`, ASCII `typewrite` fallback) so Unicode is
  not silently dropped; every action's `ok` reflects the guest process
  `returncode`/`status` (HTTP 200 alone is not success); `key` `backspace`
  presses BackSpace and `fwd-delete` presses forward Delete; screenshots are
  capped at 20 MB and PNG-validated before use.
- Remote SSH macOS: `screenshot` uses remote `screencapture` + `scp`; input uses
  a remote input helper path. The remote Mac must have Screen Recording and
  Accessibility grants configured by the owner. This skill does not install or
  store private SSH keys.

Connection registry:

- Connections live in `data/state/skills/unix_computer_use/connections.json`.
- Default active connection is `local`.
- `add_connection` can add `osworld_http` or `ssh_macos` connection metadata, but
  it never accepts private key material.
- For SSH, configure `~/.ssh/config`, `IdentityFile`, and/or `ssh-agent`
  yourself. If auth fails, `test_connection` tells you what to fix.
- Agents should add/test/activate connections only when the user or a benchmark
  runner provides an explicit target. They should not scan for arbitrary remote
  machines.
- A disabled active connection FAILS CLOSED: action tools return an error rather
  than silently falling back to the local desktop (an action aimed at a remote
  target must never land on the host).

Permissions note: the remote backends require the `net` permission. `net` needs
no owner grant (skill grants only gate `inject_chat`/`subscribe_event`), but it
does remove this skill from the launcher's native auto-enable class — so on a
fresh install the owner must ENABLE it explicitly (a benchmark runner does this
via `save_enabled`). Local-only computer use is unaffected once enabled.

The skill intentionally does not hide OS permission requirements, and missing
backends produce explicit errors with the capability report instead of
guessing. The agent should prefer semantic application APIs when available and
should ask the human before destructive or sensitive UI actions.