agent-desktop-linux

The observe-act loop, refs, JSON envelope, sessions and tracing match every platform. This file covers only what differs on Linux.

Needs Ubuntu 24.04+ (x86_64 or aarch64), a graphical session of the signed-in user and a running AT-SPI bus. Other distributions need the at-spi2-core bridge fix dc0dc33; Ubuntu 22.04's bridge aborts on Action.GetActions. The archive loader floor is glibc 2.35. Run the CLI inside the target apps' desktop session: the session bus, display and runtime directory come from its environment. A TTY login, an SSH shell without session variables, a container without a bus or another user's session sees no apps. GNOME is the measured desktop; X11 and Wayland sessions both work and differ in input, capture and window control. Observation and semantic actions need no helper tool.

Capability Table

First Contact

• Run agent-desktop status first. An empty supported_surfaces means the accessibility bus did not answer, not an empty desktop. Read references/bus-and-permissions.md; do not restart a bus you did not start.
• --app takes the name list-apps prints (gnome-calculator, nautilus) or a desktop-entry name or id backed by verified executable and argument identity, never a window title or an arbitrary registry label. A desktop-entry name such as Files is accepted once the entry exists. list-windows --app is exact (aliases included); list-apps --app combines a substring of the listed name with exact adapter aliases, ignoring case.
• Window ids are opaque addresses of 34 to 70 characters, valid for the life of the window's process; run list-windows again after a restart. list-apps rows carry bundle_id (desktop id when known) and process_instance.
• GTK4 apps nest deeply. --max-depth and --skeleton count semantic levels, so empty layout wrappers add output levels. -i --compact still shows nested nameless group nodes: read past them.
• An empty file view does not prove an empty folder. If an icon view exposes no file items, try an available list or detailed view, then take a fresh snapshot and verify which items the application exposes.
• A window appears before its content. Run wait --window "Text Editor" --app gnome-text-editor first, then wait --text, which matches a node's own name, value or description. For a web-engine app (detected from its AT-SPI toolkit metadata), a window observation waits for three equal full reads and a web-document role, or a requested depth cut. A shell with no document yet stays incomplete when the budget ends. A --timeout shorter than the first read returns TIMEOUT, never a partial tree.
• Geometry-only refs require unchanged geometry. Position and size must both match, in the ref's recorded coordinate frame; a one-pixel change breaks identity. After a layout change or a window move, take a fresh snapshot instead of retrying the old ref. Indistinguishable siblings stay AMBIGUOUS_TARGET: in a Text Editor window with three tabs, the text views share one rectangle and one state, so a ref to any of them answers AMBIGUOUS_TARGET with candidate_count 3 until the extra tabs close. A ref belongs to the window it was taken in, so two windows with the same title each resolve their own refs.
• Chromium and Qt apps need a launch flag or variable to appear on the bus; Electron apps need none, only --no-sandbox where chrome-sandbox is not setuid. See references/troubleshooting.md.
• Headed ref actions need a window that can take focus. A window the window manager never activated ends activation_not_granted. See references/windows-and-lifecycle.md.
• A GTK4 secret field can show its text. Only a password role is withheld. Treat a GTK4 entry with hidden characters as readable by every observation command.
• Blocked key combos refuse without --force. See references/display-servers-and-input.md.
• After opening a menu-role popup, inspect data.surfaces. A menu entry leads to snapshot --surface menu --app <name> -i. If absent, use wait --menu and list-surfaces before assuming the opener failed. A list-backed dropdown is part of the window, not a menu surface; see the shell reference.
• Shell pointer input requires receiver proof. On X11 a headed shell click is proven from the shell's own tree: the receiver must be the target and stand still, or the click refuses before any pointer event. Wayland shell clicks refuse. Notifications need the GNOME notification service; tray snapshots also need a StatusNotifierWatcher.
• The state root must be private. A group-writable or world-accessible AGENT_DESKTOP_HOME or ~/.agent-desktop makes snapshot fail with INTERNAL; run chmod 700 on it or let the tool create it.

Wait for a window

Wait for an app window to appear (--window requires a title or title substring):

agent-desktop wait --window "Text Editor" --app gnome-text-editor --timeout 10000

To wait for a close that you cause, batch it so the baseline is taken before the close (Calculator has no unsaved-document prompt to interrupt it):

agent-desktop batch '[{"command":"close-app","args":{"app":"gnome-calculator"}},{"command":"wait","args":{"event":"window-closed","app":"gnome-calculator","timeout":10000}}]' --stop-on-error

To watch someone else close a window, start agent-desktop wait --event window-closed --app gnome-text-editor --timeout 10000 before they close the window. There is no --window-closed flag.

Cursor overlay

Start a session and pass its returned ID to every overlay command:

session_id=$(agent-desktop session start --name docs-example | jq -r '.data.session_id')
agent-desktop cursor-overlay enable --session "$session_id"
agent-desktop cursor-overlay disable --session "$session_id"
agent-desktop session end "$session_id"

Without a session, cursor-overlay enable and disable return INVALID_ARGS. On X11, enable returns data.rendered: true; on Wayland it saves the setting and returns data.rendered: false without drawing an overlay.

session start --cursor --multi-agent returns data.rendered: false, because nothing is drawn until an agent acts. On X11 each agent id then gets its own cursor; on Wayland none is drawn.

What differs on Linux

• Firefox/Gecko select can change the selected value without firing change or input events, so dependent page handlers do not run. The result stays delivered_unverified even when value readback matches. Check the dependent application effect before continuing: value readback alone is no commit proof.
• Two click calls on one GTK4 button less than about 250 ms apart act once. Both answer ok: true with delivered_unverified. To click one button several times, wait for the effect between the clicks: use wait --element <ref> --predicate value --value <expected> on the element that shows the result, or wait 300.
• Ref actions are semantic AT-SPI calls that never move the pointer or change the active window. Under --headed, ref click, right-click and type try physical input first and fall back to the semantic action only after a not_delivered refusal (Windows works the other way round). On Wayland, headed pointer commands and a headed select refuse at once with ACTION_NOT_SUPPORTED, cause wayland_session; a headed type needs the Mutter RemoteDesktop capability. references/display-servers-and-input.md has the full rules.
• select tries the semantic route first and the pointer second; the pointer route needs an explicit --headed request on X11. A Qt combo's semantic selection only highlights its popup row, so headless selection refuses with POLICY_DENIED, kind pointer_required, before changing that highlight. Headed, on a list whose toolkit index cannot be mapped to the row (a GTK4 list) it clicks the row and checks its selected state; on a combo with no semantic opener (a libadwaita combo row) it clicks the combo open, selects the choice through the popup's own list and checks the value the combo shows. Headless, a list refuses with selection_index_unproven, and a combo with no semantic opener with POLICY_DENIED, kind pointer_required, unless the combo already shows the choice. A GTK4 drop-down that opens through its own toggle takes select without --headed and checks the value it shows.
• A ref can expose no safe semantic mutation route. Headless requests then return an unsupported-action error without inventing hidden or disabled state and without switching to physical input. GTK4 entry and password Activate actions are one such case: use type to edit the field, or request --headed explicitly when a pointer click is intended.
• Opening a native Wayland Qt combo returns ACTION_NOT_SUPPORTED, kind native_popup_requires_input, before invoking its action. Qt needs a user input serial to show that popup. An already open combo may be collapsed; Qt windows proven to use Xwayland retain their semantic opener.
• Headless type writes through EditableText. A terminal's text node is a textfield named Terminal whose only action is RightClick; headless type on it is POLICY_DENIED with check supported_action, and its value is null. It receives keys only under --headed.
• In key combos cmd and meta are the Super key. Key names are the macOS and Windows names plus X keysym names. press takes mapped keys only; type handles characters the layout cannot produce.
• Window ops act on X11 clients; focus-window and headed scoped press require an X11 session. Headless scoped press requires live target focus on either display server. Focusing a window on another workspace can switch the user's current desktop.
• Linux exact FFI refs carrying bounds or a bounds hash require window-relative geometry, even when the ref also carries a name or identifier. Geometry-free identity refs are supported.

References

• references/bus-and-permissions.md: bus bootstrap, the three bus states, status and permissions output, state-root rules.
• references/display-servers-and-input.md: X11 versus Wayland table, press, type, pointer, wheel, drag, blocked combos, key names.
• references/windows-and-lifecycle.md: launch, close-app, window ops, focus, workspaces, stopped apps, background press.
• references/capture-and-clipboard.md: screenshot on X11 and Wayland, the Wayland consent dialog, clipboard owner and helper.
• references/shell-tray-and-notifications.md: menus and GTK4 popovers, GNOME Overview/top bar, shell reads and input, notification identity guards, SNI and AppIndicator requirements and refusals, sandbox guidance.
• references/troubleshooting.md: symptom-to-cause map, Chromium, Electron and Qt flags, toolkit limits, error codes with platform_detail.

Read any file from the binary: agent-desktop skills get agent-desktop-linux references/troubleshooting.md.