sumo-qa
AGENTS.md7 starsChanged 40 days agoApache-2.0View on GitHub
If you're an AI agent setting up sumo-qa for the user, follow this exactly. One command for the common case; per-host flags if you only need to wire one host. When creating GitHub issues, use the closest template in .github/ISSUE_TEMPLATE/ and include every required field in the issue body. For implementation tasks raised by an AI agent, use .github/ISSUE_TEMPLATE/ai-task.yml unless a more specific template fits better. pip install sumo-qa creates both sumo-qa (the MCP server binary) and sumo-qa-install (this configurator).…
- Installs packages
Where files are headingNew AGENTS.md files have more rules about keeping secrets out of the code, and fewer about code style.
Of the AGENTS.md files we read, the 31,774 first seen in the last six months, against the older ones: the kind of rule that gained most share, and the kind that lost most. About files like this in general, not this one, and an estimate, since a kind is a pattern in the words.
In this repository21 other files it gives its agents.
- .github/copilot-instructions.mdCopilot instructions
- regen-eval-baselineSkill
- scaffold-sumo-qa-skillSkill
- sumo-qa-answering-testing-questionSkill
- sumo-qa-closing-qa-gapsSkill
- sumo-qa-creating-test-planSkill
- sumo-qa-deciding-approachSkill
- sumo-qa-executing-qa-rolloutSkill
- sumo-qa-finding-test-dataSkill
- sumo-qa-finishing-qa-workSkill
- sumo-qa-implementing-with-tddSkill
- sumo-qa-measuring-coverageSkill
- sumo-qa-planning-qa-rolloutSkill
- sumo-qa-preparing-for-workSkill
- sumo-qa-reviewing-before-mergeSkill
- sumo-qa-security-testingSkill
- sumo-qa-strategisingSkill
- sumo-qa-strengthening-testsSkill
- sumo-qa-suggesting-external-skillSkill
- sumo-qa-triaging-test-failuresSkill
- using-sumo-qaSkill
The file
We split this AGENTS.md into the rules it gives an agent
6 rules. About 0 are common, 3 are distinctive, and 3 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈3
Distinctive · estimate
This author's own choices. Worth a look.
3
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 3
If you're an AI agent setting up sumo-qa for the user, follow this exactly.
- Very specific: line 7
When creating GitHub issues, use the closest template in .github/ISSUE_TEMPLATE/ and include every required field in the issue body.
- Distinctive: line 28
Use these when you only want to configure one host:
- Very specific: line 60
After running sumo-qa-install, in VS Code: Cmd+Shift+P → Developer: Reload Window.
- Distinctive: line 61
Use Agent mode (not Ask, not Edit) in Copilot Chat.
- Distinctive: line 62
Use a capable model: Claude Sonnet 4.5 or GPT-5 (full).
≈ “Common” is an estimate: at least 1 in 5 of the 40,246 AGENTS.md files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
AGENTS.md, sumo-qa setup
If you're an AI agent setting up sumo-qa for the user, follow this exactly. One command for the common case; per-host flags if you only need to wire one host.
GitHub issue creation
When creating GitHub issues, use the closest template in .github/ISSUE_TEMPLATE/ and include every required field in the issue body. For implementation tasks raised by an AI agent, use .github/ISSUE_TEMPLATE/ai-task.yml unless a more specific template fits better.
Default: configure every detected host
python -m pip install sumo-qa
python -m sumo_qa.installerpip install sumo-qa creates both sumo-qa (the MCP server binary) and sumo-qa-install (this configurator). Use python -m sumo_qa.installer for setup because it works even when the pip Scripts directory is not on PATH yet.
What sumo-qa-install does:
- Locates the
sumo-qaMCP binary. Ifpip install sumo-qaalready put it on PATH (the common case), uses that path directly: nouvinvocation. Ifsumo-qais not on PATH yet, falls back touv tool installand printsuvinstall instructions ifuvis also missing. - Claude Code (if
~/.claude/exists): symlinks eachskills/<name>/into~/.claude/skills/<name>/. Writesclaude_desktop_config.jsonwith the MCP entry pointing at the absolute binary path. - VS Code + Copilot (if cwd is a workspace with
.git/.vscode/package.json/ etc.): writes.vscode/mcp.jsonwith the VS Code-native schema (serverskey,type: stdio). - JetBrains IDEs (if JetBrains config dir exists): prints exact Settings UI steps. JetBrains' MCP plugin requires Settings UI registration: external XML writes don't reliably register the runtime coroutine.
- Verifies the binary responds to JSON-RPC
initialize.
Per-host flags
Use these when you only want to configure one host:
python -m sumo_qa.installer --claude-code # Claude Code only
python -m sumo_qa.installer --vscode # VS Code workspace (cwd-based)
python -m sumo_qa.installer --vscode --workspace /path/to/repo
python -m sumo_qa.installer --jetbrains # Prints JetBrains UI steps only
python -m sumo_qa.installer --vscode --skip-mcp-install # Skip uv reinstall for speed
python -m sumo_qa.installer --profile core # Save the MCP tool profile, then configure every detected host
python -m sumo_qa.installer --profile core --vscode # Save the profile, then configure VS Code onlyRe-runs are idempotent.
Updating
python -m pip install --upgrade sumo-qa # refresh server + bundled skills/knowledge
python -m sumo_qa.installer # refresh host symlinks + MCP configs
# Restart the host or open a fresh chat — SessionStart hook re-injects new contentYou only strictly need the second step when new skills are added or a host's MCP config schema changes; routine content updates flow through the existing symlinks automatically.
VS Code specifics
- VS Code reads
.vscode/mcp.jsonfrom the workspace root, not from$HOME.sumo-qa-installrefuses to write to~/.vscode/mcp.jsonand prints a clear error if you're cd'd into$HOME. - VS Code uses a different MCP schema than Claude Desktop:
{ "servers": { "sumo-qa": { "type": "stdio", "command": "...", "args": [] } } }sumo-qa-installwrites this format. Earlier versions wrotemcpServers(the Claude Desktop schema) which VS Code silently ignored. On re-run, the installer strips the stalemcpServerskey. - After running
sumo-qa-install, in VS Code: Cmd+Shift+P → Developer: Reload Window. VS Code caches the MCP server list at startup; it doesn't re-read the file mid-session. - Use Agent mode (not Ask, not Edit) in Copilot Chat. Tools require Agent mode.
- Use a capable model: Claude Sonnet 4.5 or GPT-5 (full). Mini/fast variants don't reliably call MCP tools.
JetBrains specifics
JetBrains AI Assistant's MCP config (llm.mcpServers.xml) is not reliably writable externally on IDEA 2026.1, the plugin requires its in-process Settings UI to register the runtime coroutine. sumo-qa-install prints the exact UI fields (with the absolute binary path):
Settings → Tools → AI Assistant → Model Context Protocol → + Add server
Name: sumo-qa
Command: <absolute path printed by sumo-qa-install>
Args: (empty)
Working directory: (empty)
Apply.For Junie (JetBrains' agentic coding agent), drop this into ~/.junie/mcp/sumo-qa.json:
{
"mcpServers": {
"sumo-qa": {
"command": "<absolute path printed by sumo-qa-install>"
}
}
}Or per-project at <repo>/.junie/mcp/sumo-qa.json. Junie reads MCP configs from those directories on startup.
Verify
In any host, ask in chat:
load the QA classificationsYou should get the canonical change-classification names back. If yes, the MCP is wired correctly.
After install
The host needs to re-read its MCP config:
- Claude Code: restart the app
- JetBrains IDEs: restart the IDE (or restart the AI Assistant tool window)
- VS Code: Developer: Reload Window
What an AI agent can vs. must ask the user for
| You (the agent) CAN | You MUST ASK the user |
Run python -m pip install sumo-qa then python -m sumo_qa.installer (with whatever host flags) | Restart their host application(s) after install |
| Run the verification call | Anything that requires admin / sudo elevation |
Re-run python -m sumo_qa.installer to refresh | The JetBrains AI Assistant Settings UI add (it's a clicks-only flow; can't be scripted) |
Verify Junie's ~/.junie/mcp/sumo-qa.json file content |
Review guidelines
- On release-please PRs (
chore(main): release X.Y.Z), do not flag a file in thegit addlist of the "Regenerate version-bearing assets" step in.github/workflows/release-please.ymlas stale against the new version (for example.claude-plugin/marketplace.json,assets/preview-doctor.*,plugin_packaging/generated/manifest.json). That step regenerates those files after the version bump and commits any that changed to the release PR branch.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

