Writing Skills

Overview

Writing skills IS Test-Driven Development applied to process documentation.

A skill is behavior-shaping code, not prose. You write a failing test (a pressure scenario an agent gets wrong without the skill), watch it fail, write the minimal skill that fixes it, watch it pass, then close loopholes.

Core principle: if you didn't watch an agent fail without the skill, you don't know whether the skill teaches the right thing — or anything at all.

The Iron Law:

NO SKILL WITHOUT A FAILING BASELINE FIRST

This applies to new skills AND to edits of existing skills. "Simple addition", "just a section", "docs update" — all the same violation. Write the skill before testing it? Delete it and start over.

When to Use

• Creating a new skill for this repo (also read CONTRIBUTING.md — new skills must be agreed in an issue first and ship with eval coverage)
• Editing any skills/**/SKILL.md content
• Reviewing a PR that changes skill content

When NOT to use: for project-specific conventions (those go in the team-ai-directives repo as CDRs), or for mechanical constraints a script or test can enforce — automate those, save documentation for judgment calls.

Core Process

RED — Baseline the failure

Run the scenario WITHOUT the skill loaded. Document verbatim:

• What the agent actually did
• The exact rationalizations it used to skip the correct behavior
• Which pressures triggered the failure (time, sunk cost, authority, "it works either way")

If the agent already behaves correctly without the skill, stop — there is nothing to fix. A skill that enforces behavior that comes for free is context noise (this repo exists to fight context noise, not add to it).

GREEN — Write the minimal skill

Start from the template (templates/SKILL-template.md in this directory). Address only the failures you observed in RED — not hypothetical ones.

Run the same scenario WITH the skill. The agent should now comply.

REFACTOR — Close the loopholes

New rationalization found? Add an explicit counter, re-test. Loop until bulletproof under pressure.

Micro-test wording before full scenarios

Full pressure runs are slow. Verify wording cheaply first: one fresh API/subagent call per sample, 5+ reps per variant, always including a no-guidance control. If the control doesn't exhibit the failure, there is nothing to fix. Read every flagged match manually — automated counts overstate both failure and success.

Structure Rules (this repo)

Skill Discovery Optimization (SDO)

The description is the trigger surface: factory-mission hands every skill's name+description to its subagents, and the model decides what to load from that list.

Description = when to use. NEVER what the skill does.

Measured failure mode (see superpowers' writing-skills, which this skill adapts): when a description summarizes the workflow, agents follow the description instead of reading the skill body — a description saying "code review between tasks" produced one review where the skill's flowchart clearly required two.

# BAD: workflow summary — agents will follow this shortcut
description: Bootstraps the session by injecting the team context index and pulls rules on demand

# GOOD: trigger only
description: Use when a session starts and the team AI directives context is not yet loaded

• Start with "Use when…" and name concrete symptoms and situations
• Third person (it is injected into prompts)
• Keywords an agent would search for: error messages, symptoms, tool names
• Skills with disable-model-invocation: true are exempt (not routed by the model) but must still not mislead

Match the Form to the Failure

Before writing guidance, classify the baseline failure. The form that bulletproofs one failure type backfires on another.

No nuance clauses either way: "don't X unless it matters" reopens negotiation; a single nuance clause on a winning recipe degrades it.

Bulletproofing Discipline Skills

For rule-enforcing skills, close every loophole explicitly: forbid the specific workarounds ("don't keep it as reference — delete means delete"), add the foundational principle early ("violating the letter of the rules is violating the spirit"), build the rationalization table from your RED transcript, and list Red Flags the agent can self-check against.

Red Flags

• Skill written before the baseline run — delete and start over
• Description summarizes what the skill does instead of when to use it
• "The agent will obviously do the right thing" — that's what RED is for
• Skill grew past ~500 lines and everything is inline instead of references/
• New skill shipped without its eval criterion + grader
• Hypothetical failure modes addressed but observed ones missing

All of these mean: stop, re-run the baseline, fix the skill.

Verification

A skill change is done when ALL of these are true:

• Baseline scenario run WITHOUT the skill; verbatim failures recorded
• Same scenario passes WITH the skill
• pytest tests/unit/test_playbook_integrity.py -q green (depth, frontmatter, description rules)
• Eval criterion + binary grader added, and the grader has a unit test (evals/promptfoo/tests/)
• Regenerated install artifacts are in sync: pytest tests/unit/test_generated_artifacts_sync.py -q
• Manual acceptance check in a real agent (CONTRIBUTING "Manual testing") — skill triggers when it should, stays silent when it shouldn't

References

• templates/SKILL-template.md — starting point for a new skill
• references/skill-testing.md — pressure scenarios, pressure types, micro-testing methodology
• CONTRIBUTING.md — PR process, eval coverage requirement, manual test protocol