agent-evaluation-operations
Skill136 starsChanged 5 months agoMITView on GitHub
Evaluate agent and skill behavior or routing.
In this repository80 other files it gives its agents.
- AGENTS.mdAGENTS.md
- accessibility-audit-websitesSkill
- accessibility-performance-polishSkill
- agent-memory-provenanceSkill
- agent-orchestration-architectureSkill
- animation-easing-languageSkill
- animation-jank-qaSkill
- animation-tool-routerSkill
- anti-generic-website-reviewSkill
- brand-voice-to-ui-copySkill
- browser-inspection-workflowSkill
- code-change-safety-checkpointSkill
- code-change-undo-revertSkill
- color-material-lighting-directionSkill
- content-pruning-reviewSkill
- content-structure-information-architectureSkill
- conversion-forms-checkout-polishSkill
- creativitySkill
- critical-thinkingSkill
- cross-browser-qaSkill
- css-grid-editorial-systemsSkill
- cta-language-systemsSkill
- deep-researchSkill
- design-motion-principlesSkill
- editorial-typography-systemsSkill
- evidence-reconciliationSkill
- final-client-handoffSkill
- frontend-quality-auditorSkill
- frontend-tooling-trust-gateSkill
- gsap-performanceSkill
- hero-image-art-directionSkill
- image-crop-responsive-qaSkill
- image-video-loading-qaSkill
- logical-thinkingSkill
- mobile-responsive-qaSkill
- modern-site-engineeringSkill
- motion-language-directorSkill
- motion-performance-qaSkill
- motion-react-layout-transitionsSkill
- motion-react-microinteractionsSkill
- motion-react-page-transitionsSkill
- motion-react-patternsSkill
- nextjs-app-router-routingSkill
- nextjs-server-client-boundariesSkill
- nextjs-site-architectureSkill
- non-generic-web-copySkill
- performance-audit-websitesSkill
- performance-budget-labSkill
- premium-visual-reference-librarySkill
- premium-web-build-gateSkill
- radix-accessible-primitivesSkill
- react-accessible-componentsSkill
- react-component-compositionSkill
- react-component-craftSkill
- react-error-loading-empty-statesSkill
- react-form-patternsSkill
- react-responsive-componentsSkill
- react-state-and-effects-disciplineSkill
- reduced-motion-designSkill
- responsive-visual-polish-qaSkill
- The first 60 shown.
The file
We split this skill into the rules it gives an agent
8 rules. About 0 are common, 6 are distinctive, and 2 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈6
Distinctive · estimate
This author's own choices. Worth a look.
2
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 12
Agent workflow: test the real prompt, model, tools, approvals, state, and failure paths.
- Distinctive: line 13
Skill behavior: test whether the skill triggers on the right requests, stays out of unrelated requests, cooperates with adjacent skills, …
- Distinctive: line 30
Prefer deterministic assertions for file state, structured fields, tool calls, authorization boundaries, and exact completion status.
- Very specific: line 44
Run positive, negative, adjacent, overlap, precedence, and co-invocation cases.
- Distinctive: line 50
Run baseline and treatment under the same harness.
- Distinctive: line 53
Test external calls in a sandbox, fixture, or dry-run path before production.
- Distinctive: line 57
Do not infer safety, production readiness, or broad behavioral improvement from one successful demonstration.
- Very specific: line 65
Use Promptfoo or another project-local harness only when its telemetry, credentials, remote execution, and configuration have been reviewed.
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Agent Evaluation Operations
Evaluate the claim the change is supposed to support, not the amount of new prompt text or the number of passing examples. Routine copyediting of a test comment, ordinary execution of one existing test, or a task that merely mentions agents does not require an evaluation program.
Choose the evaluation mode
- Agent workflow: test the real prompt, model, tools, approvals, state, and failure paths.
- Skill behavior: test whether the skill triggers on the right requests, stays out of unrelated requests, cooperates with adjacent skills, and improves outcomes without excessive context or rigidity.
- Release comparison: hold the harness constant and compare the current baseline with the proposed change.
For evaluating a skill creation or behavior-bearing modification, read skill behavior evaluation. Use the regression corpus when testing synthetic public agent and Codex operating behavior. Validate the corpus with python3 "${CODEX_HOME:-$HOME/.codex}/skills/agent-evaluation-operations/scripts/validate_regression_corpus.py".
Specify before measuring
Record the evaluation claim, tested system, model/reasoning setting, prompt and skill versions, tool access, side-effect policy, attempt budget, acceptance threshold, and what would falsify the claim. Do not compare two runs that silently differ on these dimensions.
Build cases from real work: ordinary success, ambiguous input, missing data, conflicting evidence, missing access, unavailable tools, duplicate events, unsafe external actions, escalation, recovery, and every confirmed historical failure. Keep a small smoke set plus a growing regression set.
Trace run ID, tested-system version, model/reasoning setting, prompt and skill versions, tools, input class, structured result, error, latency, cost, and approval path. Redact or avoid sensitive payload capture by default.
Test retrieval and application separately. A skill may fail to trigger even when its rules are sound, or it may trigger and still fail to change behavior. Keep a small matched baseline and treatment set with identical prompts, inputs, model settings, tools, and viewports; score first attempts blind when subjective judgment matters.
Score observable behavior
Prefer deterministic assertions for file state, structured fields, tool calls, authorization boundaries, and exact completion status. Use written rubrics for judgment. Calibrate model graders against examples and human review; do not let the candidate skill be the sole judge of its own success.
Measure:
- factual grounding and evidence quality;
- scope coverage and completion-state accuracy;
- tool and skill routing, including false-positive triggers;
- action correctness, policy/approval adherence, and reversibility;
- correction quality after contradictory evidence;
- latency, tokens/cost, retries, and human review;
- privacy, recoverability, and sensitive-data handling.
Test collisions, not only positive triggers
Run positive, negative, adjacent, overlap, precedence, and co-invocation cases. Use python3 "${CODEX_HOME:-$HOME/.codex}/skills/agent-evaluation-operations/scripts/analyze_skill_collisions.py" --root "${CODEX_HOME:-$HOME/.codex}/skills" only to generate candidate pairs; lexical similarity is discovery evidence, not proof of a semantic collision.
Reject a skill change when it attracts unrelated work, duplicates an existing owner without a routing reason, weakens a capability floor, improves only the curated examples, or adds more context cost than demonstrated value.
Gate releases
- Run baseline and treatment under the same harness.
- Require the proposed change to fix its target regressions without material degradation elsewhere.
- Fail the release when a required scenario, authorization boundary, cost budget, or quality threshold is missed.
- Test external calls in a sandbox, fixture, or dry-run path before production.
- Separate local structural validation, simulated behavior, and live production evidence.
- Preserve run identifiers, versions, aggregate scores, failures, and reviewer overrides while redacting sensitive payloads.
Do not infer safety, production readiness, or broad behavioral improvement from one successful demonstration.
Correction loop
For recurring review feedback, preserve the original artifact, the correction, its source, the proposed destination, exceptions, and a holdout case. Treat collector, reviewer and maintainer as logical roles, using separate staffing only when authorized and justified. Gather raw evidence, verify and group it, then decide whether it becomes guidance, a component or token, a deterministic check, an exemplar, an evaluation fixture, a coverage gap, or no change. Rerun affected cases after accepted changes and watch whether the same complaint actually becomes less common.
The corpus validator checks structure only. Use the concrete synthetic dossiers and local observer example in fixture execution to connect cases to real inputs and observed task state; an unexecuted case is not a model result.
Use Promptfoo or another project-local harness only when its telemetry, credentials, remote execution, and configuration have been reviewed. Route agent-system architecture to $agent-orchestration-architecture; route repository security to $repository-release-security.
Optional specialists
For repository release checks, use repository-release-security only if installed; otherwise run the repository-required checks, inspect the patch for secrets and dependency changes, and preserve rollback. Its absence does not block an otherwise verified release.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

