evaluate-agent-behavior
Skill1 starsChanged 2 months agoMITView on GitHub
Measure how an agent skill, prompt, tool, grader, or harness behaves with predeclared cases, controlled trials, and calibrated grading. Not for application tests or implementing the behavior under test.
In this repository21 other files it gives its agents.
- change-production-data-safelySkill
- craft-contextual-uiSkill
- diagnose-root-causesSkill
- engineer-code-changeSkill
- engineer-reliable-data-pipelinesSkill
- engineer-secure-systemsSkill
- handle-sensitive-data-safelySkill
- infer-intent-before-codingSkill
- migrate-contracts-safelySkill
- operate-production-safelySkill
- orchestrate-long-running-workSkill
- plan-decision-complete-changeSkill
- ponytailSkill
- publish-verified-github-changeSkill
- release-production-changesSkill
- respond-to-production-incidentsSkill
- review-code-with-evidenceSkill
- route-ultracode-subagentsSkill
- verify-end-to-endSkill
- write-durable-documentationSkill
- write-tests-that-prove-behaviorSkill
The file
We split this skill into the rules it gives an agent
8 rules. About 0 are common, 8 are distinctive, and 0 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈8
Distinctive · estimate
This author's own choices. Worth a look.
0
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 12
Before any trial, record: the immutable candidate version.
- Distinctive: line 14
Test invocation and capability separately: explicit invocation, automatic routing, and over-triggering are three different failures.
- Distinctive: line 18
Use the smallest falsifying matrix: representative work, target edge, tempting shortcut, non-trigger near match, and regression.
- Distinctive: line 32
Never rank candidates on UNKNOWN or blocked evidence.
- Distinctive: line 37
Run mutating cases only in disposable copies or isolated worktrees.
- Distinctive: line 38
Use multiple trials when variance could change the decision.
- Distinctive: line 39
Do not rescue trials with hints, accept unsupported verification claims, or change a grader after an inconvenient result without versioning …
- Distinctive: line 41
Keep author and subjective evaluator separate for high-impact or taste-dependent behavior.
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Evaluate Agent Behavior
Own the cases, graders, controlled trials, and evidence that show how an agent behaves. Do not implement the instruction or product under test; if remediation is requested, hand the bounded change to its implementation owner, then evaluate the revised candidate. Evaluation work is read-only except for requested cases, fixtures, graders, or harness artifacts.
Define a measurable contract
Before any trial, record: the immutable candidate version; the observable outcome, pass conditions, anti-behaviors, safety boundaries, and preserved behavior; trigger, non-trigger, and near-match expectations; allowed tools, starting state, environment, and cost or latency budget; case and trial budget, aggregation rule, and release threshold.
Test invocation and capability separately: explicit invocation, automatic routing, and over-triggering are three different failures.
Build cases and graders
Use the smallest falsifying matrix: representative work, target edge, tempting shortcut, non-trigger near match, and regression. Add adversarial cases only for a real recovery or safety contract. Specify each case before running it using the case schema.
Prefer deterministic artifact, state, command, or runtime checks; then stable reference comparison; then a narrow calibrated rubric. Grade outcomes before process; require process only to protect an observable invariant; never require hidden chain-of-thought. Prove the grader with a known-good result it accepts and a known-wrong result it rejects. Use UNKNOWN when evidence is unavailable.
Gate comparative claims
For a single candidate, ordinary records suffice. For any comparison or causal claim:
- Freeze an immutable manifest: candidate or treatment, context, model and tools, fixture, budget, grader, allocation.
- Record exposure and isolation evidence. Instruction-scoped access is not process, filesystem, or credential isolation; missing evidence is UNKNOWN.
- Give trials unique IDs; share no mutable state across arms unless declared.
- Calibrate with good and wrong controls; keep intention-to-treat assignments; report load and tool failures.
- For authenticated flows, verify the account or environment and credential rotation without recording secrets; block if unproven.
Never rank candidates on UNKNOWN or blocked evidence. Changed inputs invalidate the comparison.
Run controlled trials
- Fresh agents and contexts with only real-invocation inputs. Hold task, tools, starting state, candidate, and grader constant.
- Run mutating cases only in disposable copies or isolated worktrees. Never mutate production, shared systems, or user work without authority.
- Use multiple trials when variance could change the decision. Declare whether success means one pass, a minimum rate, or consistency. Never average away a safety miss.
- Do not rescue trials with hints, accept unsupported verification claims, or change a grader after an inconvenient result without versioning and rerunning.
- Preserve prompts, outputs, tool evidence, grader versions, and scores; redact secrets.
- Keep author and subjective evaluator separate for high-impact or taste-dependent behavior. Treat timeouts, malformed output, and inaccessible evidence as UNKNOWN or infrastructure failure, not as pass or fail.
Diagnose without implementing
For each failure, find the earliest wrong decision and classify its cause: instructions, trigger, context, tool, capability, harness, fixture, or grader. Look for reward hacking: changed tests, skipped steps, inert artifacts, brittle matching, verbose claims that satisfy the rubric without the outcome. Recommend the causal change; after it lands, rerun failed, regression, and non-trigger cases. Keep exploratory cases separate from release-blocking regressions until reliability justifies promotion.
Stop and ask when
- the release threshold or aggregation rule was never declared and the result depends on it;
- a trial needs credentials, production systems, or user data;
- the candidate changed mid-evaluation.
Done when
The candidate meets the predeclared threshold with no disqualifying safety or scope failure and no material regression or unjustified context cost, or the failing task class is reported plainly rather than hidden in an aggregate. Report versions, cases, graders, trial counts, raw evidence locations, failures and causes, threshold disposition, recommendations, and limits. Never claim untested improvement.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

