scaffold-sumo-qa-skill
Skill7 starsChanged 43 days agoApache-2.0View on GitHub
Scaffolds a new sumo-qa sub-skill end-to-end — SKILL.md skeleton, matching promptfoo eval YAML, and an approaches.md catalogue entry — through a single bundled script. Use this whenever the user wants to add a new sumo-qa-* skill, even if they describe it as "creating a skill", "adding an approach", "starting a new routing branch", or just "writing skills/sumo-qa-X". Enforces the sumo-qa- name prefix and checks for collisions against both the source tree and the installed MCP server, so a clean handoff to the next contributor is always possible.
- Installs packages
In this repository21 other files it gives its agents.
- AGENTS.mdAGENTS.md
- .github/copilot-instructions.mdCopilot instructions
- regen-eval-baselineSkill
- sumo-qa-answering-testing-questionSkill
- sumo-qa-closing-qa-gapsSkill
- sumo-qa-creating-test-planSkill
- sumo-qa-deciding-approachSkill
- sumo-qa-executing-qa-rolloutSkill
- sumo-qa-finding-test-dataSkill
- sumo-qa-finishing-qa-workSkill
- sumo-qa-implementing-with-tddSkill
- sumo-qa-measuring-coverageSkill
- sumo-qa-planning-qa-rolloutSkill
- sumo-qa-preparing-for-workSkill
- sumo-qa-reviewing-before-mergeSkill
- sumo-qa-security-testingSkill
- sumo-qa-strategisingSkill
- sumo-qa-strengthening-testsSkill
- sumo-qa-suggesting-external-skillSkill
- sumo-qa-triaging-test-failuresSkill
- using-sumo-qaSkill
The file
We split this skill into the rules it gives an agent
2 rules. About 0 are common, 0 are distinctive, and 2 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈0
Distinctive · estimate
This author's own choices. Worth a look.
2
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 56
Add the routing line in skills/sumo-qa-deciding-approach/SKILL.md manually.
- Very specific: line 57
Run sumo-qa-validate so the catalogue still parses.
≈ “Common” is an estimate: at least 1 in 5 of the 350,495 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
scaffold-sumo-qa-skill
Drives the four-step recipe for adding a new sumo-qa-<name> sub-skill so all the pieces land together. The deterministic file generation lives in scripts/scaffold.py; this document is the thin guide for gathering inputs and checking the surfaces the script cannot reach (the installed MCP server).
When to use
Trigger this skill whenever the user asks to add a new sumo-qa sub-skill — phrases like "scaffold a new skill", "add a sumo-qa-X skill", "I want a skill that handles Y for QA", or even just "let's start the routing for Z". The user invokes it explicitly with /scaffold-sumo-qa-skill; it doesn't auto-trigger.
Inputs
Gather these from the user up front. If they only give a partial set, ask for the rest in one round of questions — don't drip-feed.
- Skill name (without prefix) — kebab-case, descriptive of the user-intent the skill handles. The script prepends
sumo-qa-automatically. - One-line description — for the SKILL.md frontmatter
description:. Convention across the estate: start with "Use when ..." so the trigger context is obvious. - Approach tag — the kebab-case routing key that lands in
knowledge/approaches.md. The user'ssumo-qa-deciding-approachskill consults this catalogue when picking which sub-skill to route to.
Verify the surfaces the script can't reach
The script checks for collisions against the source tree (skills/, tests/evals/promptfoo/, knowledge/approaches.md). It can't check the installed MCP server, which serves a frozen-at-install-time view of the catalogues. Before invoking the script, call:
sumo_qa_load_approachesthrough the MCP server and confirm the proposed approach tag isn't already shipped. The source tree may have unreleased entries the MCP doesn't yet see, and vice versa — checking both surfaces is what catches both classes of collision.
Run the script
python3 .claude/skills/scaffold-sumo-qa-skill/scripts/scaffold.py \
--name <kebab-name> \
--description "Use when <triggering context>." \
--approach-tag <kebab-tag>The script exits non-zero with a clear message on any source-tree collision. If that happens, surface the collision to the user verbatim rather than trying to work around it — the namespace policy exists to keep routing unambiguous.
On success, the script prints the three files it created plus the next-step checklist. Pass that summary back to the user so they know what they still need to do by hand.
What the script does and does not do
It writes three files (the SKILL.md skeleton, the eval YAML stub, the approaches.md block) and prints the follow-up checklist. It deliberately does not edit skills/sumo-qa-deciding-approach/SKILL.md — that file has context-sensitive routing tables that resist mechanical patching, and a half-correct edit there breaks routing for every other sub-skill. The contributor adds the routing line by hand using the approach tag they just registered.
After the script runs
Remind the user, in this order:
- Flesh out the new SKILL.md via
/skill-creator:skill-creator(or/superpowers:writing-skills) — hand it the scaffolded file as the starting point. This is the repo convention perfeedback_use_skill_creator_for_skills: skill-creator handles description-trigger optimisation, format consistency with the rest of the estate, and the eval-driven quality loop. Hand-authoring the body by pattern-matching sibling skills drifts on all three and is exactly the failure mode the scaffolder exists to make easier to avoid. Don't skip this step even when the skill feels "obvious" — every skill in the estate ran through skill-creator at least once. - Replace the seed scenario in the new eval YAML with a concrete case before running
npm run eval. The placeholder will fail every assertion. The eval YAML stub is intentionally<REPLACE>-marked throughout so this can't be missed. - Add the routing line in
skills/sumo-qa-deciding-approach/SKILL.mdmanually. The script prints the exact markdown row to paste. - Run
sumo-qa-validateso the catalogue still parses. - Reinstall the package (
pip install -e .) and restart the MCP host before the new approach tag becomes visible tosumo_qa_load_approaches. Until then, source and installed views disagree — that's expected, not a bug.
The scaffold is a starting point, not a finished skill. Don't open a PR with just the scaffold contents; the eval will fail against the placeholder content, and the repo policy (feedback_eval_fixes_target_skill_not_rubric) is to fix the SKILL.md until the eval passes rather than ship with known gaps.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

