agent-quality
Skill0 starsChanged yesterdayMITView on GitHub
Use when measuring or improving the answer quality of an AI agent (support bot, RAG, call scoring, classifier) rather than its code - "collect metrics on the agent", "improve the answer quality", "the agent escalates too often", "run the golden set", "which model is better", "compare the prompts", "check the RAG", "call scoring has drifted" - and before claiming that a prompt, model or retrieval change made the agent better.
1 other project uses a version of this.
Adopted by
Most recently taken up by
- xin-yi33/coding-agent-crewtheir own version, first seen Sep 1, 2026
Other versions1 project uses a version of this.
Word for word · 0
None found.
Edited · 0
None found.
Wrote their own version · 1
- xin-yi33/coding-agent-crew★ 1same name
How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.
In this repository23 other files it gives its agents.
- AGENTS.mdAGENTS.md
- docs/agents.mdAGENTS.md
- global/AGENTS.mdAGENTS.md
- CLAUDE.mdCLAUDE.md
- blockersSkill
- changesSkill
- codex-orchestrationSkill
- context-resumeSkill
- copy-checkSkill
- debug-profileSkill
- evidence-researchSkill
- healthSkill
- project-initSkill
- relaySkill
- reportSkill
- shipSkill
- spec-analyzeSkill
- spec-auditSkill
- spec-dispatchSkill
- spec-nextSkill
- spec-planSkill
- ui-checkSkill
- work-statusSkill
The file
We split this skill into the rules it gives an agent
2 rules. About 0 are common, 2 are distinctive, and 0 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈2
Distinctive · estimate
This author's own choices. Worth a look.
0
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 21
Cost before the run: Before a paid run name: the number of calls, the model, the amount, the stop condition.
- Distinctive: line 28
Before a conclusion check the schema, the period, the units and what exactly the number means.
≈ “Common” is an estimate: at least 1 in 5 of the 350,495 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
agent-quality: agent quality is proven by comparison
An agent is the prompt, the model, retrieval, escalation rules and data together. "It got better" counts only when the new version beat the previous one on the same set of cases and did not make the critical ones worse.
Order
- Version. Record what is compared: the model, the prompt (a file or a commit), retrieval settings, collections and the data date, escalation rules.
- Case set. Take an existing one: a golden set is the user's labelled table, use it. No set: collect 20-50 real cases from logs, including the critical categories, and show them to the user before the run.
- What counts as an answer. An internal AI note (a draft) is not an answer to the customer: it may not have been used. Only a sent message counts as an answer. Do not mix them.
- Metrics on one set:
- answer correctness by the labels;
- escalations: justified, unneeded (not from the allowed categories), missed;
- grounding: the answer is backed by the retrieved documents;
- latency and cost per case.
- Baseline. First a run of the current version. Without it the change is not assessed.
- Cost before the run. Before a paid run name: the number of calls, the model, the amount, the stop condition. Preliminary checks run on saved answers and the cheapest models.
- Change and a repeated run on the same set. Change one thing at a time, otherwise it is unclear what worked.
- Live check. Separately look at 5-10 real dialogues after the change: the case set does not cover everything.
Data
- New scoring rules apply to new records. Recalculating historical data only on a direct request and after describing what changes in the accumulated base.
- Before a conclusion check the schema, the period, the units and what exactly the number means.
Report (in the user's language)
Compared: <version A> → <version B>, set <name, N cases>
Correctness: <A> → <B>
Escalations unneeded / missed: <A> → <B>
Grounding: <A> → <B>
Latency, cost per case: <A> → <B>
Got worse: <cases and categories> | nothing
Calls and amount: plan <…>, actual <…>
Conclusion: accept B | keep A | need more <what>Mistakes
| Thought | What to do |
| "The answers got better, I looked" | A table of A versus B on one set is needed |
| "I will run 500 cases on a strong model" | First the amount estimate and consent |
| "The AI note matched the labels, so the agent answered right" | Count only sent answers |
| "I will change the prompt, the model and retrieval at once" | One change per run |
| "I will rescore old calls by the new rules" | Only new ones, unless asked otherwise |
| "There are many logs, so there are metrics" | A metric is a number on a labelled set |
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.