getting-started
Skill15 starsChanged 16 days agoApache-2.0View on GitHub
Use when a developer wants to set up evalkit for their agent for the first time, try evalkit, or follow the evalkit quickstart — takes them from installing evalkit-py to a connected agent, a dataset mined from the agent's real traces, a baseline experiment and one measured improvement. Not for a team that already runs experiments (use evalkit-usage) or for deploying a server (use evalkit-deployment).
36 other projects use a version of this.
Adopted by
Most recently taken up by
- vmoranv/jshookmcptheir own version, first seen Nov 15, 2026
- seayniclabs/privacyopstheir own version, first seen Nov 15, 2026
- Jackywine/jackywine-digital-garden-skillstheir own version, first seen Nov 15, 2026
- orgiggly/claude-plugintheir own version, first seen Nov 15, 2026
- aivrar/portable-hermes-agenttheir own version, first seen Nov 14, 2026
- seayniclabs/homelab-infra-mcptheir own version, first seen Nov 8, 2026
Other versions36 projects use a version of this.
Word for word · 0
None found.
Edited · 0
None found.
Wrote their own version · 36
- vmoranv/jshookmcp★ 2.1ksame name
- NVIDIA-NeMo/Curator★ 1.9ksame name
- davekilleen/Dex★ 509same name
- aivrar/portable-hermes-agent★ 247same name
- GoogleCloudPlatform/race-condition★ 232same name
- taskade/mcp★ 170same name
- claude-world/director-mode-lite★ 80same name
- dylantarre/design-system-skills★ 50same name
- crewAIInc/skills★ 45same name
- aks129/HealthClawGuardrails★ 34same name
26 more not listed
How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.
In this repository7 other files it gives its agents.
- AGENTS.mdAGENTS.md
- CLAUDE.mdCLAUDE.md
- authoring-eval-casesSkill
- connect-tracesSkill
- eval-driven-developmentSkill
- evalkit-deploymentSkill
- evalkit-usageSkill
The file
We split this skill into the rules it gives an agent
11 rules. 0 are common, 7 are distinctive, and 4 are very specific to this project.
0
Common
Most of the 35 edited versions we read keep these.
7
Distinctive
This author's own choices. Worth a look.
4
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 21
Before anything else, ask the developer:
- Distinctive: line 31
After that, stop for the developer only at the points under "When to stop".
- Very specific: line 35
When a step fails and the linked docs do not cover it, show the developer the exact command and error, and offer to open an issue for it on …
- Distinctive: line 88
Stop and ask only for what you cannot do yourself:
- Very specific: line 99
Do not ask about the mapping preset: read the agent's code, pick the preset for its framework from evalkit traces mapping presets, and …
- Distinctive: line 105
Do not move on until each holds:
- Very specific: line 107
After step 3: evalkit runs list <agent> --source app returns runs, and one session checked with evalkit traces import (Phoenix) or evalkit …
- Very specific: line 110
After step 5: every run has a trajectory.
- Distinctive: line 114
Before step 6: the baseline has at least one case that fails or runs slow.
- Distinctive: line 115
If every case passes, mine a wider window or add cases for the slowest runs.
- Distinctive: line 117
After step 6: a second experiment of the whole dataset exists.
“Common” is measured: kept by at least 6 in 10 of the 35 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Getting started
Walk the developer through the evalkit quickstart in their agent's project folder. At the end they have evalkit connected to their agent, a dataset mined from the agent's own traces, a baseline experiment, and a second experiment that shows what one round of fixes gained.
The quickstart page is the source of truth for every step and command:
Quickstart. (evalkit-docs/ sits beside the
installed skills; in the evalkit repository the page is
docs/user/quickstart.md.) Read it first and follow it in order. This skill
only adds what you need to run it for someone.
Ask two things first
Before anything else, ask the developer:
- Which pace. "Just set it up": run straight through and explain at the
end. "Walk me through it": say what each step does as you do it, from the
quickstart and the pages it links to (
../evalkit-docs/concepts.mdfirst). - Which path (the tabs of the quickstart's step 3). "My agent sends traces to Phoenix": evalkit imports its history. "No traces yet": the agent sends its traces straight to evalkit. Read the agent's code first and suggest the path it fits.
After that, stop for the developer only at the points under "When to stop".
Rules
- When a step fails and the linked docs do not cover it, show the developer
the exact command and error, and offer to open an issue for it on
https://github.com/inferedge-ai/evalkit/issues (with
gh issue create --repo inferedge-ai/evalkitwhenghis available). Show them the title and body first and open it only when they agree. The repo is public: leave out keys, tokens, internal URLs, customer data and the agent's private code. - Work from the root of the developer's agent project. Cases go in
evalkit/cases/<dataset>/and the run script isrun_evals.pyat the project root (inevalkit/,from myagent import ...fails). Never put an__init__.pyin theevalkit/folder. Addevalkit-runs/to.gitignorebefore the first pull. - Log in with a bare
evalkit login: it uses hosted evalkit. Pass--urlonly when the developer names another server. - Find exact command shapes with
evalkit --helpand--helpon any subcommand; do not guess flags. - Install
evalkit-pyinto the agent's own project with the tool it already uses (pip in its active virtualenv, uv or poetry), pinned to the server's version:evalkit-py==<version>, wherecurl <server url>/api/cliprints<version>(hosted evalkit:https://evalkit.inferedge.com). Runs refuse to start when the versions differ; after a server upgrade,evalkit updatemoves the pin. evalkit needs Python 3.12 or newer; if the project cannot use it, say so and stop. Run everyevalkitcommand in that environment (active virtualenv,uv runorpoetry run); there is no separate global install. - A non-Python agent, or one in another process: follow the note at the top of the quickstart (a Python task that calls the agent, the agent's own OpenTelemetry SDK, and a session id passed in or returned).
- On the "No traces yet" path, tracing goes in the agent's own start-up code,
never only in
run_evals.py: app runs do not go through it. Add the framework's instrumentor (the Instrumentation column of../evalkit-docs/framework-mappings.md) and one session per run. Keep the agent token out of files the repo tracks. - One run per case in the quickstart. For real comparisons, pass
repetitions(for examplerepetitions=3). - An async agent gets an
async def task. Never wrap it inasyncio.run. - Hand over to the other skills where the quickstart does:
- Step 3, a
!or ✗ on a required line ofevalkit traces importorevalkit traces check(session,root,mapping,llm,trajectory, andtoolswhen the agent called a tool): the connect-traces skill. A!onunmappedis fine. - Step 4, mining cases: the eval-driven-development skill, Loop 2.
- Step 6, fixing the agent: the eval-driven-development skill, Loop 1,
with
--apply.
- Step 3, a
- Mine a small first batch: pick the window so it holds 30 app runs or
fewer, and narrow it if
evalkit runs listshows more. On the Phoenix path, import only that window. - The cases stay in the developer's own workspace. Skip the review steps of the mining loop.
- An agent that runs as a service: restart it after each fix, before the next run.
When to stop
Stop and ask only for what you cannot do yourself:
- The sign-in in the browser that
evalkit loginopens. - Phoenix path: the trace store's URL, project name and API key. Ask for the name of the environment variable that holds the key; never ask for the key itself. Creating the trace source needs an admin: if it is refused, say so and stop.
- "No traces yet" path: 5 to 10 real questions to ask the agent, if the developer wants to pick them.
- The case plan from the mining loop, before any case is written.
Do not ask about the mapping preset: read the agent's code, pick the preset
for its framework from evalkit traces mapping presets, and check it on one
session.
Checks between steps
Do not move on until each holds:
- After step 3:
evalkit runs list <agent> --source appreturns runs, and one session checked withevalkit traces import(Phoenix) orevalkit traces check(no traces yet) has a ✓ on every required line. - After step 5: every run has a trajectory. A run with none says why in its
trajectory note; follow the session id rule in
../evalkit-docs/sdk-runs.md("An agent with its own trace store" or "An agent in another process"), change the task and run again. - Before step 6: the baseline has at least one case that fails or runs slow. If every case passes, mine a wider window or add cases for the slowest runs.
- After step 6: a second experiment of the whole dataset exists.
Finish
Print both experiment links (each run of run_evals.py prints one) and
the output of evalkit summary <new id> --baseline <old id>. Then say in a
few lines what was set up, which cases were fixed, and how response time,
cost and tokens moved. Point at the quickstart's "Where next" links.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

