tomekdottomekdot/hermes-skills/skills/software-development/computer-use-cua-firefox/SKILL.md

computer-use-cua-firefox

Skill1 starsChanged 5 months agoMITView on GitHub

Reliable patterns for driving Firefox in AI Studio / Gemini through cua-driver on Windows. Use when browser automation must enter prompts into AI Studio, read back replies, or interact with the bottom input bar.

In this repository10 other files it gives its agents.
- github-actions-node24-scannerSkill
- github-pages-deploySkill
- maniaplanet-items-organizerSkill
- self-improvementSkill
- angular-frontendSkill
- my-frontend-starterSkill
- openplanet-plugin-devSkill
- pursuit-mapsSkill
- pyplanetSkill
- rename-git-commitsSkill

The file

We split this skill into the rules it gives an agent

6 rules. About 0 are common, 3 are distinctive, and 3 are very specific to this project.

≈0
Common · estimate
Many files like this have a rule of this kind.
≈3
Distinctive · estimate
This author's own choices. Worth a look.
3
Very specific
Tied to their project's names, files or tools. Probably skip.

Common
Distinctive
Very specific

Very specific:
Critical: Do NOT call SOM UIA walk for Firefox MozillaWindowClass — it times out after 4s.
line 39
Very specific:
If UIA SOM walk is attempted, expect error: get_window_state timed out after 4s (UIA provider unresponsive on hwnd ..., class …
line 41
Distinctive:
If text is missing, re-focus the field and retry typing.
line 66
Very specific:
Call list_windows(on_screen_only=true) → confirm Firefox AI Studio window
line 111
Distinctive:
Verify via vision that old text is gone / replaced
line 121
Distinctive:
Wait for AI response after manual send
line 124

≈ “Common” is an estimate: at least 1 in 5 of the 350,495 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.

Files

SKILL.md

🖥️ CUA Firefox + AI Studio Automation

📋 Overview

Firefox on Windows is not safely queryable by UIA get_window_state for complex pages like Google AI Studio. This skill encodes the proven fallback sequence based on real testing on MozillaWindowClass.

Always use this workflow when:

Typing a prompt into Google AI Studio in Firefox
Reading back the current AI Studio conversation via vision
Interacting with the bottom input bar / Run button

🔁 Canonical Workflow (tested, working)

1. Locate the Firefox window

mcp_cua_driver_list_windows(on_screen_only=true)

Target window title contains: Mozilla Firefox or Google AI Studio
Known identifiers for AI Studio on this machine:
- pid=15356
- window_id=11470172
- Window class: MozillaWindowClass

2. Read current page state via vision

mcp_cua_driver_get_window_state(capture_mode="vision", pid=..., window_id=...)

Critical: Do NOT call SOM UIA walk for Firefox MozillaWindowClass — it times out after 4s.
Vision capture returns a screenshot without UIA tree. Use this to inspect UI state.
If UIA SOM walk is attempted, expect error: get_window_state timed out after 4s (UIA provider unresponsive on hwnd ..., class 'MozillaWindowClass').

3. Focus the input field

mcp_cua_driver_click(x=493, y=905, pid=15356, window_id=11470172, dispatch="background")

These coordinates target the cdk-textarea-autosize textarea element in AI Studio's bottom input bar.
This element exposes: ValuePattern, InvokePattern, TextPattern, LegacyIAccessible.
Why click instead of hotkey?
- hotkey(["ctrl","a"], ...) on MozillaWindowClass arrives as bare a because PostMessage does not set system modifier state.
- Triple-click or direct pixel click is more reliable for selection/positioning.

4. Type the prompt

mcp_cua_driver_type_text(text=..., pid=15356, window_id=11470172, dispatch="background")

Uses WM_CHAR path; works reliably for text entry in MozillaWindowClass.
Delay between characters: default 30ms. Adjust if autocomplete/IME is sensitive.
This will overwrite or append depending on field state. If unsure, click field first (step 3).

5. Verify input landed

mcp_cua_driver_get_window_state(capture_mode="vision", pid=..., window_id=...)

Inspect screenshot to confirm text is present in the input area.
If text is missing, re-focus the field and retry typing.

6. Send the message

Default: manual send. User presses Ctrl+Enter in the focused input field.
Optional automation:
- Pixel-click the Run button using coordinates from vision zoom.
- hotkey(["ctrl","enter"], ...) only after explicit user confirmation.

🚨 Known Issues and Workarounds

Issue	Workaround
`get_window_state` times out on Firefox	Use `capture_mode="vision"`; skip SOM UIA walk
`execute_javascript` unavailable in Firefox	Use CUA pixel/vision-based interactions; switch to Chrome with CDP if DOM needed
`hotkey(["ctrl","a"])` inserts literal "a"	Use click(x,y) to focus field instead
Firefox window is minimized/behind others	Use `list_windows` to find correct `window_id`; vision capture still works on minimized windows
Text input doesn't appear after `type_text`	Re-focus field with click(x,y); verify via vision screenshot

✅ Verified Working Setup

OS: Windows 10 x86_64
Firefox: Mozilla Firefox (MozillaWindowClass)
cua-driver: 0.6.8
Target app: Google AI Studio (aistudio.google.com)
Input method: mcp_cua_driver_type_text via PostMessage WM_CHAR
Focus method: mcp_cua_driver_click at (493, 905) in window-local coordinates

📚 References

cua-driver: mcp_cua_driver_* tools (list_windows, get_window_state, click, type_text, hotkey, zoom)
Skill location (local): C:\Users\tomekdot\AppData\Local\hermes\skills\software-development\computer-use-cua-firefox\SKILL.md
Skill location (repo): https://github.com/tomekdot/hermes-skills/tree/main/skills/software-development/computer-use-cua-firefox
Related skill: windows-desktop/computer-use-cua-firefox (categorized under windows-desktop for generic CUA patterns)

🧪 Test Cases

TC-1: Basic prompt entry

Call list_windows(on_screen_only=true) → confirm Firefox AI Studio window
Call get_window_state(capture_mode="vision", ...) → screenshot
click(x=493, y=905) on input field
type_text("Hello from CUA test")
get_window_state(capture_mode="vision", ...) → verify text visible
Manual: User presses Ctrl+Enter

TC-2: Replace existing prompt

Focus input field with click(x=493, y=905)
type_text("New prompt replacing old one")
Verify via vision that old text is gone / replaced

TC-3: Read AI response

Wait for AI response after manual send
get_window_state(capture_mode="vision", ...)
Inspect screenshot for response text in chat history

⚡ Quick Reference Card

Step	Command	Notes
List windows	`list_windows(on_screen_only=true)`	Filter by `MozillaWindowClass`
Screenshot	`get_window_state(capture_mode="vision", pid, window_id)`	Fast, no UIA walk
Focus input	`click(x=493, y=905, pid, window_id)`	Targets textarea
Type prompt	`type_text(text, pid, window_id)`	WM_CHAR path
Verify	vision screenshot	Always verify after typing
Send (manual)	User presses Ctrl+Enter	Do not auto-send

Last updated: 2026-07-04 — verified working on cua-driver 0.6.8 + Firefox + Google AI Studio

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

It worked
Partly
It didn't

What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

Check your setup, free