agentleFS
Sign inStart free
EskyeeEskyee/agentbot-opensource/skills/aeon-skill-evals/SKILL.md

aeon-skill-evals

Skill17 starsChanged 5 months agoMITView on GitHub

Validate the output of any installed skill against an assertion manifest — word counts, required patterns, forbidden phrases, required sections, source citation. Detects regressions by diffing vs prior runs (NEW_FAIL / NEW_PASS / CHRONIC / STABLE_FAIL). Bootstrap mode generates a starter manifest from a skill's recent successful runs so manifests aren't written speculatively. Triggers: "evaluate this skill's output", "check skill X for regressions", "bootstrap evals for Y", "did this skill output pass quality gates".

2 other projects use a version of this.

Adopted by

Most recently taken up by

The file

Create a free account to mount
Files
SKILL.md

aeon-skill-evals

Quality net for installed skills. Each skill can declare an assertion manifest; outputs are checked against it; failing assertions surface regressions and route concrete fixes.

Manifest format

token-movers:
  min_words: 200
  required_patterns: ["Top movers", "24h"]
  forbidden_patterns: ["I cannot", "as an AI"]
  must_cite_source: true
  min_distinct_items: 5

narrative-tracker:
  min_words: 400
  required_sections: ["TRANSITIONS", "POSITIONS", "MAP"]
  forbidden_patterns: ["exciting", "consider"]
  must_have_position_call: true

Supported assertions: min_words / max_words, required_patterns / forbidden_patterns, required_sections, must_cite_source, min_distinct_items, output_pattern (regex), and per-skill-family custom binary checks.

Operations

  • eval — run every manifest-defined skill against its latest output.
  • eval --skill=NAME — one skill.
  • bootstrap --skill=NAME — generate a starter manifest from recent successful runs.

Regression states

StateAction
NEW_FAILPassing last run, failing now. Severity scales with pass streak.
NEW_PASSFailing last run, passing now. Log the win.
CHRONICFailing > 3 consecutive runs. Recommend operator review.
STABLE_FAILAlways failing. Manifest assertion mismatch — flag for review.

State in local evals-state.json.

Bootstrap mode

Samples last 5 successful runs of a skill. Computes:

  • min_words at p25 of historical runs.
  • Required patterns from common section headers.
  • Forbidden patterns from default list (refusals, hedging filler).

Emits the proposed manifest for review. Never auto-commits — assertions need a human signoff.

Rules

  • Assertions are observations, not specifications. Bootstrap before writing speculatively.
  • Forbidden patterns catch hallucination markers and refusals. Keep the list tight; don't lint stylistic choices.
  • Chronic failures get a recommendation, not a re-file.
  • Manifest changes are reviewed; never auto-edited by this skill.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

Reviews can't be read right now.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

Check your setup, free