kit-evals
Skill0 starsChanged 14 days agoMITView on GitHub
Run, add to or report Kit's agent evals — the in-repo Kit suite (kit eval), Terminal-Bench via Harbor, a SWE-bench Verified subset and Aider polyglot. Use before claiming the agent got better or worse, when changing the loop, a tool, a prompt or a provider, or before publishing any benchmark number.
In this repository45 other files it gives its agents.
- AGENTS.mdAGENTS.md
- CLAUDE.mdCLAUDE.md
- api-and-interface-designSkill
- browser-testing-with-devtoolsSkill
- ci-cd-and-automationSkill
- code-review-and-qualitySkill
- code-simplificationSkill
- context-engineeringSkill
- debugging-and-error-recoverySkill
- deprecation-and-migrationSkill
- documentation-and-adrsSkill
- doubt-driven-developmentSkill
- frontend-ui-engineeringSkill
- git-workflow-and-versioningSkill
- idea-refineSkill
- incremental-implementationSkill
- interview-meSkill
- observability-and-instrumentationSkill
- performance-optimizationSkill
- planning-and-task-breakdownSkill
- security-and-hardeningSkill
- shipping-and-launchSkill
- source-driven-developmentSkill
- spec-driven-developmentSkill
- test-driven-developmentSkill
- using-agent-skillsSkill
- kit-ci-triageSkill
- kit-provider-adaptersSkill
- kit-releaseSkill
- kit-tuiSkill
- a11y-passSkill
- add-readmeSkill
- api-docsSkill
- changelogSkill
- cli-helpSkill
- code-reviewSkill
- completeness-qaSkill
- data-checkSkill
- deps-hygieneSkill
- fix-bugSkill
- project-setupSkill
- pr-readySkill
- ship-checklistSkill
- workspace-setupSkill
- write-testsSkill
The file
We split this skill into the rules it gives an agent
2 rules. About 0 are common, 0 are distinctive, and 2 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈0
Distinctive · estimate
This author's own choices. Worth a look.
2
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 11
No number goes in the README, a PR description or a post without a committed result file in docs/evals/ (date, Kit commit, model id, task …
- Very specific: line 38
Never run a paid benchmark without the owner's go and a --budget.
≈ “Common” is an estimate: at least 1 in 5 of the 350,495 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Kit evals
The plan is docs/PRD.md §13.3; the suite is evals/README.md.
The rule for numbers
No number goes in the README, a PR description or a post without a committed result file in
docs/evals/ (date, Kit commit, model id, task set and its pin, pass rate, cost, turns) and a link to the
run. Comparisons name the same model, date and task subset for both sides.
Kit suite (kit eval)
- 33 tasks in
evals/tasks/<id>/:files/(the fixture repo, with akit.tomlgate that is the grader and a[gate.scope]that denies the check files),task.toml(category,expect,prompt) andsolution.json(the reference solution as scripted tool calls). kit eval --baseline(no agent: every gate must fail) andkit eval --scripted(every reference solution must pass throughkit run --agent kit) run in CI on three OSes. Run both after adding or changing a task.- A model run:
kit eval --model <id> --budget 0.50 --out docs/evals/<date>-<model>.json. It costs money: the owner's call. SetKIT_HOMEto a scratch folder when testing locally so runs stay out of~/.kit. - Adding a task: write the fixture, prompt and solution; the gate must fail on the fixture, pass with the
solution, and finish in under 2 minutes. A reference solution that only works with a blanket
replace (
replace_allover a common word) is a bug: real agents make that mistake, and the gate should catch it.
External benchmarks (manual or weekly, always budget-capped)
- Terminal-Bench: Harbor (Apache-2.0).
uv tool install 'harbor[modal]', run the oracle agent first, thenharbor run -d terminal-bench/terminal-bench@<pinned> --agent <kit adapter> --model <id>. Version 4.0 is not comparable with 2.x results. - SWE-bench Verified: a fixed 50-task subset, MIT harness, Docker on x86_64 (about 120 GB disk).
swebench eval verified -p preds --run-id <id>. A regression check, not a headline (saturated). - Aider polyglot:
Aider-AI/polyglot-benchmarkin Docker viabenchmark/docker_build.sh. Use it to compare edit formats.
Never run a paid benchmark without the owner's go and a --budget. Report what ran, what it cost, and
what failed, including infrastructure failures.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.