computer-use
Skill0 starsChanged 3 days agoMITView on GitHub
Windows computer-use 通用技能:在桌面应用、浏览器、IDE、办公软件和其他 Windows 场景中,按环境选择 UI Automation 语义操作、坐标鼠标键盘注入或 Trae/TraeWork computer-use MCP;覆盖启动、查找、聚焦、点击、拖拽、输入、窗口/标签管理、权限弹窗处理与结果验证。
70 other projects use a version of this.
Adopted by
Most recently taken up by
- Eymistaken/Pcbridgetheir own version, first seen Nov 16, 2026
- rodos4312/Meu-Agenttheir own version, first seen Nov 15, 2026
- codewhale-hq/codewhale-cu-plugintheir own version, first seen Nov 15, 2026
- mdiri-ai/SoduWorktheir own version, first seen Nov 15, 2026
- NousResearch/hermes-agenttheir own version, first seen Nov 13, 2026
- okou-ai/okou-skillstheir own version, first seen Nov 11, 2026
Used with
Projects with it also use claude-buddy, face-enroll and 9 more.
Other files people use alongside this one.
Files found in the same public projects as a version of this one, more often than they turn up anywhere else. Each project counts once, large collections count for less, and this file's own repository is left out.
autonomous-aiSkillclaude-buddy
Coordinate with the Claude Buddy companion (Claude Desktop + Claude Code on the user's Mac) — voice-approve tool permission prompts and stay aware of Claude activity.
Also in 3 projects that use this
autonomous-aiSkillface-enroll
Manage the device's face recognition roster — enroll new faces (3 paths: user-supplied photo, agent-captured snapshot on user request, or HAL's familiar-stranger prompt) and maintain the enrolled set (status / remove / reset). All enrolled persons are friends; strangers stay unnamed until promoted via one of the enroll flows.
Also in 3 projects that use this
- +9 more from autonomous-ai
Other versions70 projects use a version of this.
Word for word · 0
None found.
Edited · 0
None found.
Wrote their own version · 70
- NousResearch/hermes-agent★ 264ksame name
- bojieli/ai-agent-book-projects★ 53ksame name
- bojieli/ai-agent-book★ 53ksame name
- xuzhougeng/wisp-science★ 1.1ksame name
- mdiri-ai/SoduWork★ 930same name
- avibe-bot/avibe★ 640same name
- cyhhao/vibe-remote★ 638same name
- autonomous-ai/autonomous-os★ 569same name
- autonomous-ai/lampOS★ 569same name
- autonomous-ai/Physical-AI-Operating-System★ 569same name
60 more not listed
How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.
In this repository46 other files it gives its agents.
- api-engineeringSkill
- baidu-api-devSkill
- business-managementSkill
- copywritingSkill
- doubao-web-imageSkill
- doubao-web-videoSkill
- douyin-pip-shortsSkill
- douyin-script-studioSkill
- geoSkill
- huarui-devSkill
- huarui-videoSkill
- huarui-zhisuanSkill
- juhe-nexgate-devSkill
- juhe-nexgateSkill
- kuaishou-script-studioSkill
- ops-btpanelSkill
- ops-deploySkill
- ops-linuxSkill
- ops-tencentcloudSkill
- seedance-devSkill
- seoSkill
- short-dramaSkill
- site-auditSkill
- site-deliverySkill
- site-incidentSkill
- site-launchSkill
- site-maintenanceSkill
- site-promotionSkill
- site-rescueSkill
- site-visual-designSkill
- svg-lecture-deckSkill
- task-deliverySkill
- toutiao-article-studioSkill
- trae-screen-tutorialSkill
- tutorial-series-planSkill
- tutorial-video-assembleSkill
- versus-script-studioSkill
- video-post-productionSkill
- wechat-mp-studioSkill
- workbuddy-foodSkill
- workbuddy-imageSkill
- wp-adminSkill
- wp-devSkill
- wxchannels-script-studioSkill
- xhs-note-studioSkill
- xingqiao-mediaSkill
The file
Files
Computer Use(Windows 通用电脑控制)
将本技能用于需要实际操作 Windows 图形界面的任务:打开或切换应用、点击控件、输入文本、拖拽、滚动、关闭标签页/窗口、操作浏览器、IDE、办公软件和其他桌面应用。先识别当前可用的电脑控制后端,再选择语义操作或坐标注入;不要把某个平台的调用方式误套到其他环境。
后端选择
1. WorkBuddy / 已连接 computer-use 工具
优先使用当前会话已经提供的电脑控制能力。遵循该后端自己的工具参数和返回结果;每次动作后重新读取界面状态,不能凭“动作已发送”宣称完成。
2. Trae / TraeWork
仅在 Trae Exec 沙箱中使用专属 MCP:
- server:
ide_mcp.config.ext.computer-use - 调用形式:
tools.run_mcp({ server_name, tool_name, args }) - 工具:
list_apps、launch_app、get_app_state、click、type_text、press_key、set_value、select_text、scroll、drag、perform_action - 示例:
scripts/trae-exec-examples.js
tools.run_mcp、text()、image() 只在 Trae Exec 中存在。其他 Agent、CLI 或桌面工具不能照抄这段代码,应改用其自身 computer-use 接口或本地脚本。
3. 本地 Windows 脚本
需要自研或命令行执行时使用:
scripts/uia_text.py:pywinauto UIA Value 模式,精确写入文本,支持多行,不移动物理鼠标。scripts/mouse_click.py:pyautogui 坐标注入,移动真实光标并发送点击/组合键。scripts/cursor_position.ps1:读取系统光标位置,用于验证坐标注入。
这些脚本需要相应依赖,使用前先检查环境,不要默认安装或下载依赖。
标准执行流程
- 确认目标和动作边界:明确目标应用、窗口/标签、要执行的操作、允许的写入范围,以及是否涉及删除、发送、付款、发布、权限授予等高影响动作。
- 发现应用:优先列出运行中的窗口/应用;找不到目标时再启动。记录本次观测到的 pid、app_id、窗口标题或控件属性。
- 读取界面状态:先取 UI 语义树;只有需要坐标、截图或视觉定位时才取截图。Trae 中可用
get_app_state({disableScreenshot: true})先取语义树。 - 优先语义操作:使用控件
element_id、标题、control type、Value/Invoke/Focus 模式。语义操作通常更稳定,不依赖分辨率,也不移动物理鼠标。 - 必要时使用坐标注入:先重新取得带截图的状态,再使用相对当前窗口或屏幕的坐标。坐标依赖分辨率、缩放和窗口位置,窗口变化后必须重新定位。
- 文本输入选择:精确文本、多行文本优先 UIA
set_value/set_text;只有需要模拟真人键盘输入时才使用type_text或键盘注入。注意输入法可能污染模拟键盘内容。 - 处理权限弹窗:遇到“允许访问设备”“授予权限”“发送/付款/发布”等敏感确认,不自动点击允许;暂停并请求用户确认,或在不影响安全的情况下关闭/阻止弹窗。
- 执行小步动作:每次动作后重新观察。旧的 element_id、UI revision、窗口 pid 或坐标在界面变化后可能失效。
- 验证结果:以窗口标题、控件值、文本内容、字符数、标签数量、任务状态或其他可读 UI 证据确认完成。失败时报告实际状态,不把调用返回成功当成业务完成。
常见操作规则
窗口和应用
list_apps找到运行中的 pid;未运行时使用观测到的app_id启动。- 启动返回已带 pid 时直接复用,不重复查找或重复启动。
- 窗口不存在时先重新列出应用;不要对已失效 pid 盲目重试。
标签页与窗口关闭
- 优先点击目标标签页的关闭控件。
- 找不到标签栏时,先聚焦目标窗口,重新确认活动标签就是用户授权关闭的目标,并检查是否有未保存内容,再发送
Ctrl+W;目标或保存状态不明确时暂停询问。 - 出现保存或放弃更改提示时,不自动丢弃未保存内容;按用户明确选择处理。
- 关闭最后一个标签可能直接关闭整个浏览器窗口;执行前确认目标范围。
Alt+F4属于窗口级关闭,只有用户明确要求关闭窗口时使用。
拖拽、滚动和坐标
- Trae 的坐标点击要求先用带截图的
get_app_state建立坐标上下文,并把目标窗口element_id一起传入。 - 坐标相对哪个窗口或屏幕必须以当前后端文档为准,不自行假设。
- 拖拽使用后端提供的 drag 参数;拖动后重新读取 UI 状态确认对象位置或值已改变。
后端差异与故障排查
- Trae 后端中,
element_id操作通常走 UIA,不移动物理鼠标;坐标点击会注入真实鼠标事件。 type_text可能受输入法影响;精确输入改用set_value。No window found for pid:窗口已关闭或 pid 已失效,重新发现应用,不要原样重试。No cached UI revision:丢弃旧控件 ID,重新获取完整 UI 状态。diff/no_change:只代表增量状态,需完整树时关闭 diff。- 坐标点击无效:确认截图上下文、窗口焦点、缩放比例和坐标参考系,重新截图后再操作。
安全边界
- 不绕过操作系统权限、登录验证、验证码、支付确认或用户授权。
- 不自动确认不可逆删除、付款、发送敏感信息、发布内容、授予设备权限和远程控制。
- 处理个人文件时遵守只读扫描、逐项确认、备份和回收站规则;不使用广泛删除命令。
- 不把截图、窗口树、剪贴板、Token、Cookie、密码或私人内容写入日志、共享包或代码仓库。
- 完成后报告实际执行的动作、验证证据、未完成项和需要用户确认的下一步。
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

