tool-use-eval
Skill1 starsChanged 28 days agoMITView on GitHub
Evaluation pattern for agents that call tools / functions - measures tool selection accuracy, argument correctness, recovery from tool errors, and avoidance of unnecessary tool calls. Pairs with eval-suite-planner. Use when: tool-use eval, function-calling eval, agent tool selection, tool invocation accuracy, tool argument validation, tool error recovery, agent tools, function-calling agent, MCP eval.
In this repository6 other files it gives its agents.
- cost-quality-frontierSkill
- eval-generatorSkill
- eval-result-interpreterSkill
- eval-suite-plannerSkill
- eval-triage-and-improvementSkill
- find-skillsSkill
The file
We split this skill into the rules it gives an agent
2 rules. About 0 are common, 2 are distinctive, and 0 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈2
Distinctive · estimate
This author's own choices. Worth a look.
0
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 76
Every test case captures:
- Distinctive: line 103
Adopt / extend the seed suite: - start with the 18 canonical cases
≈ “Common” is an estimate: at least 1 in 5 of the 350,495 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Tool-Use Eval
Evaluation pattern for agents whose job is to invoke tools / functions / MCP capabilities. Conversational quality and task-completion evals miss the failure modes specific to tool-use; this skill adds the four dimensions that actually predict whether a tool-using agent is production-ready.
Core Principle
A tool-using agent has four ways to fail that pure-conversation agents don't. Generic quality scores can be high while every one of these is broken. This eval pattern separates them so improvements target the right failure mode.
The Four Tool-Use Dimensions
| Dimension | What It Measures | Failure Example |
| Tool Selection Accuracy | Did the agent pick the right tool for the request? | Used search_web when lookup_internal_kb was the right call |
| Argument Correctness | Were the arguments well-formed and schema-valid? | Required field missing, type mismatch, hallucinated parameter |
| Error Recovery | When a tool returned an error, did the agent recover sensibly? | Repeats the same failing call, or gives up silently |
| Restraint | Did the agent avoid unnecessary tool calls? | Calls 5 tools when 1 would do; calls a tool when no tool was needed |
The four dimensions are independently scorable - improvements often help one and hurt another (e.g., tightening argument validation can increase Restraint failures).
Scoring Anchors (0-3 per dimension)
Tool Selection Accuracy
| Score | Anchor |
| 0 | Wrong tool; result has no chance of satisfying the request |
| 1 | Plausible-looking tool but mismatched to the request's actual intent |
| 2 | Right tool family, but a more specific / cheaper tool existed and was missed |
| 3 | Most appropriate tool selected on the first call |
Argument Correctness
| Score | Anchor |
| 0 | Schema-invalid (missing required field, type mismatch, hallucinated param) |
| 1 | Schema-valid but semantically wrong |
| 2 | Schema-valid and semantically reasonable, with one suboptimal field |
| 3 | Schema-valid, semantically optimal, no extraneous fields |
Error Recovery
| Score | Anchor |
| 0 | Agent ignored the error or repeated the same failing call > 1× |
| 1 | Agent stopped without explaining the failure to the user |
| 2 | Agent tried a different approach but did not surface the failure clearly |
| 3 | Sensible alternative tried or failure clearly surfaced with next-step guidance |
Restraint
| Score | Anchor |
| 0 | Tool called when no tool was needed (request answerable from prior context) |
| 1 | Multiple tools called for what should have been a single call |
| 2 | Tool count is right but ordering wasted a call |
| 3 | Minimum-necessary tool calls, ordered well |
Canonical Test Suite
The skill ships an 18-case seed suite covering common failure shapes:
| Case Type | Count | Tests |
| No tool needed | 3 | Restraint baseline - request answerable from context |
| Single obvious tool | 3 | Selection + Argument basics |
| Two plausible tools | 3 | Selection discrimination |
| Multi-tool chain | 3 | Sequencing + Restraint |
| First call errors | 3 | Error Recovery |
| Ambiguous request | 3 | Pre-call clarification vs. guessing |
Extend with 5-10 cases from your actual production traffic.
Trace Capture Schema
Every test case captures:
request- the user prompttool_calls[]- each call's name, arguments, timestamp, result, error (if any)final_response- the agent's reply to the userexpected_tools[]- gold-standard tool sequence (for Selection scoring)expected_no_tools- boolean (for Restraint scoring)
Without the trace, post-hoc scoring is impossible.
Failure Taxonomy
Standardized labels so triage rolls up cleanly:
| Code | Meaning |
SEL-WRONG | Wrong tool selected |
SEL-SUBOPTIMAL | Right family, wrong specific tool |
ARG-INVALID | Schema validation failed |
ARG-SEMANTIC | Schema-valid but semantically wrong |
REC-SILENT | Failure not surfaced to user |
REC-LOOP | Same failing call repeated |
RES-OVERCALL | More tool calls than needed |
RES-UNNECESSARY | Tool call when none was needed |
Process
- Inventory the tools - list with schemas and one-line use cases
- Adopt / extend the seed suite - start with the 18 canonical cases
- Run with full trace capture - request + every call + final response
- Score each case on all four dimensions independently
- Aggregate by failure type - which dimension is dragging?
- Track over time - re-run on every model / prompt / tool-schema change
Pairs With
- eval-suite-planner - designs the broader plan; tool-use-eval is the tool-use chapter
- eval-result-interpreter - produces SHIP / ITERATE / BLOCK verdicts
- eval-triage-and-improvement - walks through failures, recommends fixes
- cost-quality-frontier - Restraint failures show up as cost regressions
Tips
- Score the trace, not the response. Wasteful tool sequences hide if you only judge output.
- Restraint regressions are the most expensive at scale - a 2× tool-call regression doubles inference + tool-execution costs.
- Don't conflate Selection and Argument failures - different fixes (prompt vs. schema clarity).
- Test the no-tool baseline - over-calling is the hardest failure mode to detect without it.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.