omni-pilot
Skill1 starsChanged 49 days agoApache-2.0View on GitHub
MANDATORY: Execute whenever the user requests full OS control, GUI automation outside of browsers, or bypassing HTML DOM scripting. Uses PyAutoGUI and ImageGrab to physically move the mouse and type based on raw visual coordinates.
In this repository21 other files it gives its agents.
- min-tokSkill
- academic-scholarSkill
- agent-mcp-craftSkill
- anti-overengineeringSkill
- api-security-reviewerSkill
- data-verifySkill
- deep-researcherSkill
- design-no-slopSkill
- enterprise-teamSkill
- git-craftSkill
- human-codingSkill
- laravel-architectSkill
- markdown-docSkill
- modern-stack-horizonSkill
- modern-web-guidanceSkill
- perf-benchmarkSkill
- python-bug-fixerSkill
- react-ui-architectSkill
- reverse-engineer-anythingSkill
- typescript-nodeSkill
- tyw-auditSkill
The file
Files
Omni-Pilot (Pixel-Level OS Embodiment)
Tujuan
Menghentikan ketergantungan AI pada kode HTML/DOM yang rapuh. Omni-Pilot memungkinkan AI "melihat" layar penuh (seluruh OS, bukan cuma Chrome) menggunakan Multimodal Vision, lalu menggerakkan mouse dan keyboard secara fisik layaknya manusia.
Workflow Eksekusi (Computer Use)
- Visual Capture: Jalankan perintah
python C:\Users\MSI\.gemini\config\utils\omni_pilot.py --action capture. Ini akan menyimpan gambar layar diC:\Users\MSI\.gemini\antigravity\scratch\omni_screen.png. - Visual Processing: Baca file
omni_screen.pngtersebut menggunakan alatview_file(Mata Multimodal Anda). Cari letak elemen/tombol yang ingin Anda klik (perkirakan koordinat X dan Y secara piksel dari kiri atas). - Physical Click: Jalankan perintah
python C:\Users\MSI\.gemini\config\utils\omni_pilot.py --action click --x [X_COORD] --y [Y_COORD]untuk menggerakkan kursor secara nyata dan mengekliknya. - Physical Typing: Jika ingin mengetik, klik dulu area teksnya, lalu jalankan
python C:\Users\MSI\.gemini\config\utils\omni_pilot.py --action type --text "teks anda". - Physical Press: Jika ingin menekan Enter/Escape, jalankan
python C:\Users\MSI\.gemini\config\utils\omni_pilot.py --action press --text "enter".
Aturan Absolut
- BANNED (Puppeteer/DevTools): Anda DILARANG KERAS menggunakan
puppeteer_evaluate,chrome-devtools, atau mencari DOM dengan skrip jika User mengaktifkan mode Omni-Pilot ini. - Fail-Safe: Jika kursor bergerak liar, arahkan segera koordinat X=0, Y=0 untuk memicu Abort.
- Selalu capture ulang layar setiap kali tata letak GUI berubah (misal: setelah klik). Jangan menebak koordinat buta.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.