agentleFS
S3n0001S3n0001/agent-skills/evaluate-agent-behavior/SKILL.md

evaluate-agent-behavior

Skill1 starsChanged 2 months agoMITView on GitHub

Measure how an agent skill, prompt, tool, grader, or harness behaves with predeclared cases, controlled trials, and calibrated grading. Not for application tests or implementing the behavior under test.

The file

We split this skill into the rules it gives an agent

8 rules. About 0 are common, 8 are distinctive, and 0 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈8

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 0

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Before any trial, record: the immutable candidate version.

    line 12
  2. Distinctive:

    Test invocation and capability separately: explicit invocation, automatic routing, and over-triggering are three different failures.

    line 14
  3. Distinctive:

    Use the smallest falsifying matrix: representative work, target edge, tempting shortcut, non-trigger near match, and regression.

    line 18
  4. Distinctive:

    Never rank candidates on UNKNOWN or blocked evidence.

    line 32
  5. Distinctive:

    Run mutating cases only in disposable copies or isolated worktrees.

    line 37
  6. Distinctive:

    Use multiple trials when variance could change the decision.

    line 38
  7. Distinctive:

    Do not rescue trials with hints, accept unsupported verification claims, or change a grader after an inconvenient result without versioning …

    line 39
  8. Distinctive:

    Keep author and subjective evaluator separate for high-impact or taste-dependent behavior.

    line 41

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Evaluate Agent Behavior

Own the cases, graders, controlled trials, and evidence that show how an agent behaves. Do not implement the instruction or product under test; if remediation is requested, hand the bounded change to its implementation owner, then evaluate the revised candidate. Evaluation work is read-only except for requested cases, fixtures, graders, or harness artifacts.

Define a measurable contract

Before any trial, record: the immutable candidate version; the observable outcome, pass conditions, anti-behaviors, safety boundaries, and preserved behavior; trigger, non-trigger, and near-match expectations; allowed tools, starting state, environment, and cost or latency budget; case and trial budget, aggregation rule, and release threshold.

Test invocation and capability separately: explicit invocation, automatic routing, and over-triggering are three different failures.

Build cases and graders

Use the smallest falsifying matrix: representative work, target edge, tempting shortcut, non-trigger near match, and regression. Add adversarial cases only for a real recovery or safety contract. Specify each case before running it using the case schema.

Prefer deterministic artifact, state, command, or runtime checks; then stable reference comparison; then a narrow calibrated rubric. Grade outcomes before process; require process only to protect an observable invariant; never require hidden chain-of-thought. Prove the grader with a known-good result it accepts and a known-wrong result it rejects. Use UNKNOWN when evidence is unavailable.

Gate comparative claims

For a single candidate, ordinary records suffice. For any comparison or causal claim:

  1. Freeze an immutable manifest: candidate or treatment, context, model and tools, fixture, budget, grader, allocation.
  2. Record exposure and isolation evidence. Instruction-scoped access is not process, filesystem, or credential isolation; missing evidence is UNKNOWN.
  3. Give trials unique IDs; share no mutable state across arms unless declared.
  4. Calibrate with good and wrong controls; keep intention-to-treat assignments; report load and tool failures.
  5. For authenticated flows, verify the account or environment and credential rotation without recording secrets; block if unproven.

Never rank candidates on UNKNOWN or blocked evidence. Changed inputs invalidate the comparison.

Run controlled trials

  • Fresh agents and contexts with only real-invocation inputs. Hold task, tools, starting state, candidate, and grader constant.
  • Run mutating cases only in disposable copies or isolated worktrees. Never mutate production, shared systems, or user work without authority.
  • Use multiple trials when variance could change the decision. Declare whether success means one pass, a minimum rate, or consistency. Never average away a safety miss.
  • Do not rescue trials with hints, accept unsupported verification claims, or change a grader after an inconvenient result without versioning and rerunning.
  • Preserve prompts, outputs, tool evidence, grader versions, and scores; redact secrets.
  • Keep author and subjective evaluator separate for high-impact or taste-dependent behavior. Treat timeouts, malformed output, and inaccessible evidence as UNKNOWN or infrastructure failure, not as pass or fail.

Diagnose without implementing

For each failure, find the earliest wrong decision and classify its cause: instructions, trigger, context, tool, capability, harness, fixture, or grader. Look for reward hacking: changed tests, skipped steps, inert artifacts, brittle matching, verbose claims that satisfy the rubric without the outcome. Recommend the causal change; after it lands, rerun failed, regression, and non-trigger cases. Keep exploratory cases separate from release-blocking regressions until reliability justifies promotion.

Stop and ask when

  • the release threshold or aggregation rule was never declared and the result depends on it;
  • a trial needs credentials, production systems, or user data;
  • the candidate changed mid-evaluation.

Done when

The candidate meets the predeclared threshold with no disqualifying safety or scope failure and no material regression or unjustified context cost, or the failing task class is reported plainly rather than hidden in an aggregate. Report versions, cases, graders, trial counts, raw evidence locations, failures and causes, threshold disposition, recommendations, and limits. Never claim untested improvement.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.