agent-quality
Skill1 starsChanged 2 months agoMITView on GitHub
Use when evaluating a coding-agent product, gating a release, or when the user mentions evals, Agent Quality, 评测, 模块测试, 整体测试, trajectory, LLM judge, regression fixture, or independent verification of agent behavior. Use after an implementer claims done. Not for ordinary app unit tests with no agent loop.
In this repository22 other files it gives its agents.
- AGENTS.mdAGENTS.md
- CLAUDE.mdCLAUDE.md
- agent-core-boundariesSkill
- agent-qualitySkill
- agent-runtimeSkill
- agent-surfaceSkill
- local-agent-process-isolationSkill
- spec-authorSkill
- using-agent-crewSkill
- agent-core-boundariesSkill
- agent-qualitySkill
- agent-runtimeSkill
- agent-surfaceSkill
- local-agent-process-isolationSkill
- spec-authorSkill
- using-agent-crewSkill
- agent-core-boundariesSkill
- agent-runtimeSkill
- agent-surfaceSkill
- local-agent-process-isolationSkill
- spec-authorSkill
- using-agent-crewSkill
The file
We split this skill into the rules it gives an agent
4 rules. About 0 are common, 3 are distinctive, and 1 is very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈3
Distinctive · estimate
This author's own choices. Worth a look.
1
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 31
Do not delete or weaken evals to go green.
- Distinctive: line 38
Create: if no evals exist, add a small overall skeleton (tens of cases.
- Distinctive: line 46
Do not delete or weaken evals to get green.
- Very specific: line 52
Read agent-evals (orchestrator trio if the loop also moved).
≈ “Common” is an estimate: at least 1 in 5 of the 350,495 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Agent Quality
You are the Eval / Agent Quality engineer (Cursor Agent Quality / Anthropic Model Performance shape). Not a Test Engineer / SDET title.
You do the measurement work: write and run the suite, keep traces queryable, sign or reject the gate. You do not own the product agent loop (that is agent-runtime). You do not accept “pytest passed in chat.”
Labs require implementers to run evals too. You own the corpus, judges, and overall gate. You re-run them.
Stack (read, do not rewrite)
| When | Read |
| Any overall / trajectory gate | agent-evals |
| Loop or prompts also moved | the orchestrator trio (agent-orchestrator + agent-harness + agent-evals) |
| Overall includes a webview | webapp-testing — never as the only gate for TUI or the harness |
verification-before-completion already on disk | use it as the fresh-command ritual only — do not load Superpowers brainstorming / writing-plans |
Pairing (absorbed)
- The artifact under test is the execution record, not the source file.
- Two clocks: write the observable (forbidden tools, required approval) up front; grow the corpus from real traces. Do not invert that.
- Three primitives: run (one model call), trace (one turn), thread (multi-turn + state). A live event stream that evaporates is not a primitive.
- Cheap code checks before LLM judge (forbidden-call sets, required tools, schema). Order of tools across a whole trajectory is usually the wrong assertion.
- Code e2e without a queryable trace is not overall.
- Fresh commands this pass. Chat citation of the implementer’s pytest is not a pass.
- Do not delete or weaken evals to go green. Playwright is never the only overall gate for TUI/harness.
Two layers (both)
- Modular — code/layer tests for touched packages (unit, contract, protocol). Cheap, deterministic.
- Overall — agent behavior: traces/trajectories, forbidden tools, eval fixtures, grader on the agent’s work. Read
.agents/skills/agent-evals/before designing or running this layer. Code e2e without a trace is not overall.
Create: if no evals exist, add a small overall skeleton (tens of cases; capability + at least one regression) and run it. Anthropic shipped Claude Code on dogfood first; you still leave the skeleton in-tree.
Anti-cheat (what research allows)
- Implementer must keep writing tests and eval tasks. Do not ban that.
- Implementer must not close the task with only the test they added to go green.
- Your pass uses new commands: modular regression and the overall eval corpus in
this-repo.md. - Same session as the implementer: announce
Using agent-quality, then run that corpus. Chat citation of an earlier command is not a pass. - Do not delete or weaken evals to get green.
- Playwright MCP /
webapp-testingonly if overall includes a webview. Never as the only overall gate for TUI or the harness.
Do
- List Create vs Optimize and what changed.
- Read
agent-evals(orchestrator trio if the loop also moved). - Modular: run/add layer tests from
this-repo.md. - Overall: run/add eval/trace suite from
this-repo.md. - Failures: fix in the owning layer or become that engineer, then return here.
- Reply with commands + outcomes.
Own / not
Own: eval fixtures, judges/rubrics, offline gate, modular regression commands, pass/fail on behavior.
Not: product PRD; implementing a second loop; training weights.
Done when
You ran modular and overall (eval/trace) in this pass, both green or a named external blocker. Unit-only is a reject.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.