Skill: continuous-evals

Load trigger: "load continuous-evals skill" > DORA: AI Capability 7: Quality internal platforms Token cost: Low to load; a run costs one cheap-model call per task

Purpose

Agent configuration is code: a reworded rule or a retired skill can silently change what agents do. This skill makes configuration changes regression-safe the same way tests make code changes safe.

How it works

Each task runs in a throwaway copy of the working tree (CI: OpenCode; locally claude -p by default) with a per-task timeout, and is scored on three rubric dimensions from its transcript:

• task success — expect: output_regex, file_absent, file_present;
• tool use — rubric.tool_use.forbidden / required: anchored, case-insensitive regexes over tool names, MCP tools included;
• trajectory — rubric.trajectory.max_steps: model steps taken.

The run fails if any dimension's rate drops below baseline.json, if there are no tasks, or if a task has no expectation or no rubric. CI runs it on config changes, on pushes to main, and weekly. A non-zero agent exit also fails a task, unless it sets "ignore_agent_exit": true in expect: use that only for "must not happen" tasks (e.g. protected-env) where a blocked tool call may legitimately exit non-zero. A crashed agent also leaves a file absent, so never use it for tasks that expect a positive result.

Procedure

1. Changing configuration? Run scripts/run-evals.sh before pushing. CI runs it anyway on any change to AGENTS.md, .agents/, .claude/, .opencode/, hooks, or .pre-commit-config.yaml.

2. An agent got something wrong in real work? Add a task that reproduces it:

   {
     "id": "short-kebab-id",
     "why": "which rule/skill/agent this protects",
     "prompt": "the real request, answerable in a few turns",
     "permission_mode": "acceptEdits",
     "expect": { "output_regex": "…" },
     "rubric": {
       "tool_use": { "forbidden": ["write|edit|serena_(create|replace|insert)_.*"] },
       "trajectory": { "max_steps": 6 }
     }
   }

   Prefer a grader that cannot pass by accident: a file check, or a regex that only the correct answer matches.

3. Raising the baseline: allowed any time the suite passes at the higher rate. Lowering it needs a reason in the PR description — that is the point.

Limits

• A regex grader checks the answer, not the reasoning. Keep tasks narrow.
• Model output varies; tasks must be robust to phrasing, or they flake and someone lowers the baseline to get green.