Skill quality suite

Validates a SKILL.md against the Agent Skills specification, lints the instructions an agent will actually follow, scans a skill for secrets and prompt injection before you install it, checks whether it will work on another agent, and measures whether it improves the agent's work at all.

A quality gate in two halves. The static half reads the skill: eight checks, one command each, separate because they fail at different moments - structure breaks today and in silence, the specification breaks on publication, compatibility breaks on somebody else's machine, routing breaks when a neighbour's description moves. Running them as one undifferentiated pass reports all four with the same urgency, which is how a report stops being read.

The evaluation half runs an agent: does the skill fire, does it make the work come out better, did the last edit make it worse. It costs money and minutes and needs an agent installed, so nothing in it runs unless you ask for it by name.

Everything lives in scripts/sqs.py. The static half is standard library only and works offline; it finds the skills folder on its own, or takes --skills-dir.

Paid layers: offer, warn, never start

sqs.py eval in every form, and any check a model has to read or judge, spends the user's money or the usage window of their subscription. The rule for all of it:

1. Never run a paid layer on your own initiative, however well it fits the task. Run it only after the user said yes to that run in this conversation.
2. When a paid layer would help, offer it - and say, every time, that it is paid and what one run costs: a capped trigger run measured at about $0.21 (23.09.2026); a trigger set of twenty queries at three runs each is sixty of those.
3. Say that it pays off only over time. One run of a model that answers differently each time is an anecdote. The value is a measurement repeated across edits - the regression gate comparing this version with the last one - not a single number today.
4. Offer the free step first: check, route --prompt, evals, cases --from-history. Most "does it fire" questions are answered there.

Plain wording that works: "This would need eval --trigger, which is paid - about $0.21 per run, sixty runs for a proper set - and only pays off if you keep measuring across edits. The free checks say X. Run it?"

Start here

check is the everyday one: structure, spec, quality, compat and security together. all adds the publish module. Neither runs an agent - that is eval, and it is always opt-in.

From your history to a proposal

discover, improve and --transcripts print evidence and decide nothing. Each edit passes the bar in from-history.md first; most fail it, and saying why is a result.

Reading the report

⛔ SP004  `name: video-tools` does not match the folder `video` (SKILL.md)
⚠️  ST006  SKILL.md is 21370 B > the 15000 B budget (SKILL.md)
·   QL006  при необходимости - lines 191, 295 (SKILL.md:191)

• ⛔ error - fix it. Nothing here is cosmetic: a broken pointer into references/ does not crash anything, the agent just skips the step in silence, and from outside it looks like the work came out weaker than usual. That is why this class of breakage survives for months.
• ⚠️ warning - a decision, not a defect. An orphan file is either unwired or no longer needed, and only you know which.
• · info - a nudge. Real, small, and safe to leave.

--format board prints one line per layer instead of a finding list - the view for deciding rather than fixing, where a layer nobody ran says NOT RUN instead of looking like a pass. --score adds an aggregate number and the arithmetic that produced it: additional, never the headline, because a number hides which question failed. sarif feeds GitHub code scanning, github annotates a diff, json everything else.

Every finding carries a rule code. sqs.py explain ST008 gives the code's reasoning and its fix; sqs.py rules lists all of them. Reach for explain rather than guessing at what a message means - the reasoning is the part that decides whether the finding applies to your case.

The modules

sqs.py structure   links, orphans, budgets, section pointers, outbound paths
sqs.py spec        layout, fences, description shape, asset weight, nesting
sqs.py quality     the description as a pointer, placeholders, vague bounds
sqs.py compat      will the skill work on somebody else's harness
sqs.py security    secrets, destructive commands, injection, hidden characters
sqs.py evals       the eval files and the routing invariants - free
sqs.py publish     what has to be true before the skill leaves the machine
sqs.py fix         the repairs with exactly one correct answer
sqs.py eval        runs the agent: triggering, task success, regressions - costs money
sqs.py baseline    the findings a tree has agreed to live with, for now

evals reads the eval files and costs nothing. eval runs the agent and costs money. The two are one letter apart because they are about the same thing at two prices.

Four of them need a word of their own.

security is the one to run on a skill somebody else wrote, before the agent reads it for work. A skill is executable text; an installed skill is a supply chain. It reports credentials, destructive commands, text addressed at the agent rather than the task, and hidden or bidirectional characters that make the rendered file differ from the file the model reads.

compat is the check your own machine can never make for you. It reads the skill as a list of features - a frontmatter field, a directory, a tool name, a hard-coded path into someone's skill folder - and asks ten harness adapters what each one means to them: Claude Code, Codex, Cursor, Gemini CLI, Antigravity, OpenCode, Cline, Roo Code, Windsurf, GitHub Copilot. Every row rests on that project's own documentation, and what the documentation does not state comes back UNKNOWN rather than as an invented incompatibility. sqs.py harnesses lists them with the page each rests on.

evals is the free half of testing. Per skill: does this one carry a usable eval set (evals/trigger/*.yaml or evals/eval_queries.json, and evals/evals.json; --init scaffolds them). Tree-wide: do the routing invariants still hold, delegated to evals/run_evals.py when one sits beside the skills.

eval is the half that runs an agent, and the only part of the suite that observes behaviour instead of reasoning about text:

• --trigger sends each query to a headless session and counts what loaded. The output is the confusion matrix - true and false positives and negatives, precision, recall, F1 - printed beside the cases it came from. F1 is printed, not scored: it weighs a miss and a false fire equally, and in a tree of skills they are not equal.
• --runtime runs the task set twice, once with the skill loaded and once with no skills at all, and reports task success, tool calls, turns, time, tokens, cost and whether following the skill made the agent do something destructive. A task with no assertions comes back ungraded, never as a pass.
• --save <label> stores a run; --compare v1 v2 diffs two. Quality falling fails the gate, cost rising is reported and does not, because cheaper-and-worse and dearer-and-better are both trades somebody has to look at.

sqs.py eval <skill> with no layer named runs nothing: it prints what each layer would cost and stops. See evaluating.md.

fix is a dry run unless you pass --apply. It only repairs what the broken state forces: the name the folder already dictates, characters that should never have been in the file. It deliberately will not shorten an over-long description - there is no single right shortening, and picking one for you deletes a trigger you needed.

Fixing what it found

1. sqs.py explain <CODE> - the reasoning, then the fix.
2. sqs.py fix --apply for the mechanical ones.
3. The rest by hand. When the finding is about what the skill says rather than how it is wired, the report has taken you as far as counting can: open writing-rubric.md and do the reading pass.
4. Re-run until clean. A run that ends 0 error and a run you assumed would are different things.

Three scopes of escape hatch, for findings that are correct-and-intended:

• A line - write sqs-allow: SE002 on it or just above it. This is how one quotation of a dangerous pattern avoids being reported as a use of one.
• A file - sqs-allow-file: SE001, SE002 in its first 25 lines, for a file whose whole job is to hold the patterns. Annotating each occurrence there would be noise pretending to be care.
• The tree - sqs.config.json beside the skills: rules maps a code to off/info/warning/error, ignore skips skills, allow_dirs accepts a directory the spec does not name, harnesses sets the target environments and lang the publication language.
• A subtree - .sqsignore at the skill root, one path glob per line: not read at all, so nothing is claimed about it. It hides a path from this suite only, not from the installer (PB015).

Two more ways to make a report survive contact with an old tree, neither of which hides anything:

• --baseline reports only what the baseline file does not already carry, so switching the gate on over three hundred existing findings does not turn the build red on day one. The recorded findings stay in the file, dated, and sqs.py baseline show lists them: it is a queue, not a bin.
• --min-confidence high drops the findings the suite is less sure of - the prose heuristics - and keeps the filesystem and parse facts. Every rule carries a detection confidence and a false-positive risk, and sqs.py explain <CODE> prints both.

Record why in the config next to the entry. A silenced rule with no reason gets un-silenced by the next person who reads the file, including you.

The references

• creating-a-skill.md - the order for building a new skill, and for reworking an old one. Open it before writing any skill from scratch.
• writing-rubric.md - the reading pass: the pointer, the two loads, the information hierarchy, completion criteria, leading words, pruning. Open it when the problem is what the skill says rather than how it is wired.
• agent-compatibility.md - the ten harnesses, what each row rests on, the five portability verdicts, and how to add a harness without inventing one. Open it before answering "will this work on X".
• evaluating.md - the three eval questions, the trigger loop with its train/validation split, the baseline/treatment comparison, the regression gate, and the part that stays a human's job. Open it before touching a description that already fires, and before the first sqs.py eval.
• publishing.md - the gate before a skill leaves the machine, and what the publish module cannot see.
• from-history.md - from the user's own sessions to a proposal, and the bar every edit to a skill passes. Open it before discover or any edit.
• judging-sessions.md - a skill's loads judged: two verdicts each, then drafts.
• mistakes-journal.md - recording a mistake, reviewing the journal into rules and gates, and what improve reads from it.
• porting.md - moving a skill to another harness: read the target's own page now, plan against it, write a copy, never the original. Open it before changing a skill for a harness it was not written for.
• editing-this-suite.md - the audit, the golden corpus and the generated pages, which all have to agree before a change is done. Open it before changing a rule, a harness or the evaluation layer of this suite itself.