evalkit usage

For any developer on a team whose evalkit server is already running. You do not need to know how the server was deployed; you need its URL and a username from whoever deployed it.

No server yet, or want a sandbox of your own? The try-it stack runs evalkit on your machine with your LLM as the judge, and walks a sample agent through an SDK run with traces: the kit is try/ in the evalkit-deployment skill, and the walkthrough is Try locally.

evalkit skills install puts the evalkit user docs in evalkit-docs/ beside this skill; links to ../evalkit-docs/ point there. In the evalkit repository the same pages are in docs/user/, and every evalkit server shows them under /docs. In the docs, {{origin}} means your evalkit server's URL.

The CLI is the reference. evalkit --help lists every command and evalkit <command> --help documents arguments, options and an example. Read those instead of this file for syntax; this file only says which command does what and explains the two things the help text cannot: how to get connected and how case review works.

Install and log in

1. Install the CLI from PyPI at the server's version. evalkit is on PyPI as evalkit-py and needs Python 3.12 or newer, in the tool and in any project that adds the SDK. curl <server url>/api/cli prints the server's version:

   uv tool install --reinstall "evalkit-py==<version>"

   With pipx: pipx install --force "evalkit-py==<version>". <server url>/docs/install in the web UI shows every command with the version filled in. evalkit --version prints what you have.

   For SDK runs, add evalkit to the project at the same version: uv add "evalkit-py[otel]==<version>"; uv.lock pins it. To keep evalkit out of the app's dependencies, run the eval script with uv run --with "evalkit-py[otel]==<version>" python eval/run.py instead. After a server deploy, local and SDK runs refuse to start until the versions match: run evalkit update from the project root, which reinstalls the CLI, moves the evalkit-py pin in pyproject.toml (and runs uv sync) and refreshes the skills evalkit skills install put in your repos, in one step.

2. evalkit login --url <url> with your username, e.g. evalkit login --url https://evalkit.internal. The login is stored in ~/.evalkit/config.toml. The CLI talks to one server at a time; logging in again replaces the stored login. Dev and prod are shared workspaces on that server (@dev, @prod), not separate logins. With single sign-on configured on the server, the command opens your browser at your identity provider and stores its tokens, which refresh themselves; on a machine without a browser, evalkit login --url <url> --no-browser prints a URL and a code to enter on any device. Without single sign-on (or with --username) it asks for a password and stores an evalkit token that expires after the TTL the deployer configured (default 30 days). When a command says your login has expired, log in again. evalkit whoami shows the login kind and when the token expires.

3. Scripts and CI authenticate as a machine user. With single sign-on, set EVALKIT_SERVER_URL, EVALKIT_OIDC_CLIENT_ID and EVALKIT_OIDC_CLIENT_SECRET (a machine client at the identity provider, bound once by an admin with evalkit users add <name> --subject <sub>); no login step is needed. Without single sign-on, use a dedicated non-admin password user and run a non-interactive evalkit login at the start of every job (see evalkit login --help). Never the admin account. See CI sync and Single sign-on. EVALKIT_TIMEOUT (seconds, default 120) sets how long the CLI and SDK wait for each server call; it does not change the server's limits on its own calls to a harness. Every server call needs a login, lint included; only evalkit lint --offline runs without one, and it skips the checks that need server state. The server lint covers everything --offline does, fixture keys included, so CI needs only the server lint. The lint job and the sync job both use the machine user. The sync job targets a shared workspace (--workspace dev); a machine user is recorded as the changelog actor (--actor is admin only). Install the CLI at the start of the job so CI always runs the server's build:

   uv tool install --reinstall "evalkit-py==$(curl -fsSL "$EVALKIT_SERVER_URL/api/cli" | jq -r .version)"

The web UI serves at the server URL with the same credentials.

Workspaces, references and local runs

Every dataset and agent lives in a workspace:

• Every user owns a personal workspace named after their username.
• An admin creates shared workspaces, e.g. dev and prod, and adds members as writer (may sync and create code-driven agents there) or reviewer (may approve cases there). Only an admin may point a shared agent or remote evaluator at a URL, since the server sends that workspace's API keys there. evalkit workspaces list and evalkit workspaces members list <ws> show them.
• CI syncs into the shared workspace the deployer picked (usually dev). Setting that CI up is on the CI sync page.

References:

• The CLI and SDK accept a bare name (finance, myagent). It means your personal workspace.
• name@workspace names any other one: finance@dev, ouragent@prod.
• API paths and UI URLs always carry @workspace.

The agent you pick decides who drives the run:

• An agent without a harness_url is code-driven: your own code runs it with run_experiment (an SDK run), and its trajectories come from its OpenTelemetry traces. This is the default way to start, and the try-it stack walks through it.
• An agent with a harness_url serves the Harness API. Use it for a deployed agent: in a shared workspace the server drives it (a server run); in your personal workspace your CLI drives it from this machine (a local run).

Workspaces has the full table, with examples for dev and prod, and what happens when you run from several branches.

Agents and remote evaluators

Agents and remote (HTTP) custom evaluators are rows on the server. You create them once with the commands below.

evalkit agents create myagent --trajectory-source otel --mapping mapping.json   # code-driven, your workspace
evalkit agents create ouragent --workspace dev --harness-url https://ouragent.internal/evalkit/v1   # a deployed agent
evalkit evaluators create citation_check --endpoint http://localhost:9000/evaluate
evalkit agents list --workspace dev

• agents create|edit|show|list|archive|restore and evaluators create|edit|list|archive. Creating one in a shared workspace needs the writer role there.
• An OTel agent (--trajectory-source otel) needs --mapping <file.json> when you create it (see the connect-traces skill).
• agents edit <agent> --trajectory-source otel|atif switches an existing agent. Switching to otel takes --mapping <file.json>, or reuses the agent's last active mapping. It's refused while one of the agent's experiments is running; finished experiments keep the source they ran with.
• A run scores with the remote evaluators in the agent's workspace.
• Agents cannot be renamed. Archive one and create another instead.

evalkit.yaml and credentials

Put an evalkit.yaml in your repo. evalkit run finance --agent myagent then syncs <cases>/finance to finance@<you>, drives myagent from this machine, scores, and writes everything to the server. Run finance@dev to use the shared cases instead (Approved ones only, no sync). evalkit run drives agents with a harness_url; for a code-driven agent, run the eval script with case_ids=[...] (see sdk-runs.md). Fixture bytes are read from the fixtures directory and uploaded by sha256 on every run or sync.

cases: eval/cases            # optional: directory whose children are datasets
fixtures: eval/fixtures      # optional: fixture bytes, flat, keyed by filename
credentials:                 # env:NAME references only, never the key itself
  agents:
    myagent: env:MYAGENT_API_KEY             # bare name = your workspace
  evaluators:
    citation_check: env:CITATION_API_KEY

• credentials.agents here is only for your personal Harness API agents. The server drives shared agents (ouragent@dev) with keys stored in its database (evalkit agents create|edit … --api-key-env NAME, by an admin or a writer), so an entry for one here is never read. The same holds for shared remote evaluators.
• Keys come from the environment or from a .env file next to evalkit.yaml. A variable set in the environment wins over .env.
• An agent or evaluator with no credentials entry gets no auth header, and the run prints one line saying so.
• credentials holds agent and evaluator keys only; a file with credentials.llm_providers (or credentials.judges) fails to load. Your own judge's key goes with the run (below).
• cases and fixtures are only needed to sync your own datasets by name.

The judge

The judge is chosen per experiment. By default it is the server's default judge, which an admin sets with evalkit judge-default set (evalkit judge-default show prints it). A judge names either an LLM provider configured on the server or a base URL, never both:

• Served by an LLM provider on the server (the default): the evaluators that use a judge (answer_rubrics, presentation_judge, trajectory_rubrics) run on the evalkit server with its stored key, also in local and SDK runs. No judge key is needed on your machine. evalkit run --judge-provider NAME (evalkit llm-providers list) picks another one.
• Your own judge: evalkit run --judge-base-url URL --judge-model M --judge-api-key-env NAME (the env var or .env entry holding the key), or Judge(base_url=..., model=..., api_key=...) in the SDK. Those evaluators run on your machine. The key is never stored: evalkit evaluate <id> and evalkit run … --resume <id> need --judge-api-key-env NAME again; in the SDK, pass the same Judge(..., api_key=...) to evaluate_experiment(..., judge=...). A different judge is refused. Server runs refuse --judge-base-url.

--judge-model and --judge-effort change the rest for one experiment. evalkit evaluate reuses the experiment's judge. tool_trajectory_match, required_tools_called and max_response_time always run on your machine. Each score records where it ran (scored_on: local or server).

Inside a dataset directory, subfolders become tags (finance/statement-analysis/x.yaml gets the tag statement-analysis). evalkit run finance --tag statement-analysis runs the cases carrying that tag; repeat --tag to run the cases carrying any of several tags. Local runs require the CLI and server versions to match.

SDK runs

An agent that is a library, a notebook, or a service with its own job runner does not need a Harness API. Install the SDK into the project (the web UI's Docs → Install page has the exact command for your server; it is evalkit-py with the otel extra), create a code-driven agent (no --harness-url), and run the experiment from your own code. A local run uses your own login for the API calls (the one from evalkit login, or EVALKIT_SERVER_URL and the machine client variables). The exported traces need an agent token from evalkit tokens create --agent <name> in EVALKIT_OTEL_TOKEN, required by tracing.register: it never expires, while a login token runs out under a long-running process and the export would fail silently.

# eval/run.py — uv run python eval/run.py
from evalkit.sdk import run_experiment, evaluator, tracing
from myagent import MyAgent, setup_tracing

setup_tracing()  # your own OpenTelemetry provider
tracing.register(agent="myagent")  # add evalkit as an export destination
agent = MyAgent()


def task(input: dict, files: list) -> str:
    return agent.ask(input["question"], attachments=[f.path for f in files])


@evaluator(name="citation_check")
def citation_check(case, run) -> float:
    return 1.0 if run.output and run.output.get("citations") else 0.0


if __name__ == "__main__":
    print(
        run_experiment(
            dataset="finance", agent="myagent", task=task, evaluators=[citation_check]
        ).url
    )

# no --harness-url: a code-driven agent. Trajectories come from its spans.
cp <skills>/connect-traces/references/openinference.json eval/mapping.json
evalkit agents create myagent --trajectory-source otel --mapping eval/mapping.json

Rules of the road:

• run_experiment is a push model: your process runs the task once per case and repetition, exports the agent's OpenTelemetry spans to the server, waits for each session to settle, and records runs and scores through the API. dataset, case_ids, tags, repetitions, model and resume mean what they mean on evalkit run (see evalkit --help).
• The task's parameters bind by name from input, files, model and metadata, in any order; any other required parameter is an error, and a task with one parameter of another name (def task(q)) gets the input.
  • input: the case input, minus files.
  • files: a list of Fixture with name (the fixture key), path (the downloaded file) and context (the file's description from the fixture registry, the text a Harness API agent gets with the file; show it to the agent).
  • model: the model argument of run_experiment, else None. It is the only way the override reaches the agent; run_experiment warns when model is passed and the task has no model parameter.
  • metadata: session_id (the SDK's session id), experiment_id (None on a dry run), case_id, repetition (from 1) and tags.
• In a notebook use await arun_experiment(...); the sync form refuses to run inside an event loop.
• A task may return an ATIF document instead of tracing (RunResult(answer=..., trajectory=...)).
• An agent in another process joins the run in one of two ways: the task injects the W3C trace context (opentelemetry.propagate.inject(headers)) and the agent continues the trace, or the agent stamps its own session id (the attribute its mapping's session_id rule names) on its root span and the task returns RunResult(answer=..., session_id=...). A queue-backed agent that opens its session in a worker returns RunResult(session_id=...) with the id its own API returns, or the task passes metadata["session_id"] along with the job and the agent uses it as its session id. The connect-traces skill has the details, including how a shared worker exports only its eval runs if the developer wants that: tracing.eval_run(metadata) marks a job, and tracing.processor(agent=..., only=tracing.is_eval_span) keeps every span of a marked trace, bridged spans included. Spans from another process go through that process's own exporter, which makes its own choice.
• A pipeline that should not touch the global tracer provider, or code that hands spans straight to a processor, uses tracing.processor(agent=...) or tracing.exporter(agent=...) instead of tracing.register.
• evalkit records fixed capabilities for a code-driven agent (thinking, artifacts and workspace false) and never reads them from spans. Pass --provides thinking=true to evalkit agents create (or edit) when its spans carry the model's reasoning.
• A code-driven agent's case inputs must pass its input schema, by default an object with a required string question. For other inputs (multi-turn turns, structured fields), write a JSON Schema file and run evalkit agents edit <agent> --input-schema schema.json. run_experiment(dry_run=...) checks it too.
• Python evaluators (@evaluator) see the same payload as the HTTP Custom Evaluator API and are selected by name under a case's evaluators.custom. Re-score with evaluate_experiment(...) from the same script; evalkit evaluate refuses those experiments.
• dry_run=True runs one case without creating an experiment and prints what the server made of the session. Use it while wiring up tracing, then follow the connect-traces skill if the span kinds or messages look wrong. The mapping you created the agent with decides how spans are read.
• evalkit traces check <agent> <session id> is the diagnostic; evalkit tokens create --agent <name> mints a collector-only token for a deployed agent that cannot carry your login.
• An agent token is bound to one agent on the server, and the agent must exist first (evalkit agents create). Neither tokens create nor the first span export creates an agent. You mint, list and revoke only your own tokens.

Already have a Harness API?

Keep it. Create the agent with --trajectory-source otel --mapping <file> next to its --harness-url, and only the trajectory moves to OpenTelemetry: evalkit still drives the agent through the Harness API (evalkit run, server runs, the Workspace tab), and builds the trajectory from the spans instead of GET /runs/{id}/trajectory. The harness then has to:

• tag the spans of each run with the run id it returned from POST /runs, in the attribute its mapping's session_id rule names (session.id in the OpenInference reference, gen_ai.conversation.id in gen_ai). The root span is enough; children inherit. A queue-backed harness whose worker opens its own session can return that session id as the run_id instead;
• export them to the collector (see the connect-traces skill). The harness learns that a run is an eval run from the metadata of POST /runs (experiment_id, case_id, repetition); a harness that exports only eval runs passes that to tracing.eval_run, with the run_id as session_id;
• declare provides.trajectory: false, or drop the trajectory endpoint.

Move to the SDK (run_experiment, no harness_url) only to stop hosting the harness. You lose server-started runs and the Workspace tab, and run_experiment refuses an agent that has a harness_url, so it is one or the other per agent. Do not create a second agent for the same code just to use run_experiment: the two names get separate run histories, tokens and mappings.

What evalkit can see of your agent

Each Harness API endpoint lets evalkit do one thing. This table is about agents that have a Harness API (deployment paths 3b and 3c). On the SDK + traces path (3a) there is no harness and none of these endpoints exist; those runs show only what the task returns and the spans it exports.

The Workspace tab needs the files endpoints, so a code-driven agent never has one. If a developer wants to see the files an agent wrote, the agent needs a Harness API with at least the files endpoints; its trajectory can still come from traces (a harness agent with --trajectory-source otel). The web UI's Docs → Harness API page has the full table.

Which command

Read evalkit <command> --help before you run a command. It is the source of truth for how the command behaves: for example, run executes only Approved cases on shared datasets (personal datasets run every case) and checks inputs against the agent's schema, sync on a directory prunes, and collect bundles only failures unless --all.

Cases that need input files name a fixture key; see references/suite-fixtures.md for how to register the file and upload its bytes before you sync.

Case review

Review applies to datasets in shared workspaces. Datasets in a personal workspace have no review state. Review never filters a run: evalkit run executes every case the selection picks, whatever its state. Review decides what gets promoted (below).

Every case in a shared suite dataset is in exactly one review state, derived from its current content hash and the latest decisive event recorded at that hash. Editing a case changes the hash, so an edit resets it to Unreviewed on the next sync unless carry-over applies (below).

stateDiagram-v2
    [*] --> Unreviewed: sync creates case

    Unreviewed --> ReviewRequested: request review
    Unreviewed --> Approved: approve
    Unreviewed --> ChangesRequested: request changes

    ReviewRequested --> Approved: approve
    ReviewRequested --> ChangesRequested: request changes
    ReviewRequested --> ReviewRequested: request review (new reviewer)

    ChangesRequested --> ReviewRequested: request review
    ChangesRequested --> Approved: approve

    Approved --> ChangesRequested: request changes (retrospective)
    Approved --> ReviewRequested: request review (retrospective)

    Approved --> Unreviewed: edit and sync
    Unreviewed --> Unreviewed: edit and sync
    ReviewRequested --> ReviewRequested: edit and sync (carry-over)
    ChangesRequested --> ReviewRequested: edit and sync (carry-over)

    Approved --> [*]: promoted to the target workspace (if linked)

Rules the diagram encodes:

• Any decisive event may be recorded from any state; the latest one wins. Requesting review or changes on an Approved case is a retrospective re-review.
• An edit is not an event. Re-syncing a case that was in Review requested or Changes requested carries the request over to the new content, so a fix goes straight back to the reviewer instead of silently becoming Unreviewed.
• Only reviewers of that workspace (evalkit workspaces members list <ws> --role reviewer) may approve or request changes. Everyone can author, sync to their own workspace, request review and comment; a comment is resolved by its author or a reviewer. Syncing to a shared workspace needs the writer role there, which is why CI usually does it as admin.

Promotion between workspaces

An admin can link two shared workspaces, a promotion source and a promotion target: evalkit workspaces promote-to dev prod. From then on, approving a case in finance@dev also writes that exact version into finance@prod, already Approved, together with the dev overlays that extend it. Nothing else reaches the target:

• evalkit sync --workspace prod and review decisions on prod datasets are refused with a 409 that names the source. Comments are still allowed; the case page on the target links to the same case in the source.
• Removing a case from the source, or requesting changes on a newer version, leaves the target untouched. The target keeps the last approved version until a newer one is approved.
• Approving the same content again changes nothing in the target except the overlays, which are refreshed.
• The target dataset must exist first: evalkit datasets create finance@prod (a writer of prod). Datasets synced into the target before the link keep their cases; those stay Unreviewed there, and still run, until a version is approved in the source.

Both case pages show the twin: on the source, "prod: same version · approved …" / "ahead of prod" / "not in prod"; on the target, "newer in dev" / "removed in dev". The twin field on GET /api/datasets/<ref>/cases carries the same information. evalkit workspaces list shows the link (promotes to prod on the source row, promoted from dev on the target row); promote-to <source> --none clears it. One target per source, one source per target, no chains.

Commands, in the order a case usually moves: evalkit review request → reviewer runs evalkit review queue → evalkit review approve or evalkit review request-changes → author edits, syncs (carry-over) → reviewer approves. evalkit review show prints state, changelog and comments for a case; evalkit review comment and evalkit review resolve handle discussion. The same actions are available on the case page in the web UI.

From an MCP client

The server is also a read-only MCP server at {{origin}}/mcp, for answering questions from a coding agent or chat client without the CLI: an agent's runs in a window, why a run failed, usage by day, user or tool, a run's trajectory and files, and experiments with per-case scores. It cannot change anything.

Connect with your evalkit login token (the one evalkit login stored in ~/.evalkit/config.toml):

claude mcp add --transport http evalkit {{origin}}/mcp --header "Authorization: Bearer $EVALKIT_TOKEN"

If your deployer configured mcp.oauth, an OAuth client (Claude, Cursor, Codex) connects with the URL alone and signs in through your identity provider. Agent tokens are refused. The tools, their limits and the shape of what they return are in MCP.

Deeper work

• Connecting an agent's OpenTelemetry traces, and writing a span mapping when its instrumentation is custom: the connect-traces skill.
• Writing or editing cases: the authoring-eval-cases skill.
• Turning a failing experiment into agent fixes: the eval-driven-development skill.
• Anything about the server itself — config, credentials, shared workspaces, upgrades, user accounts — is the deployer's job and the evalkit-deployment skill.

For bugs, feature requests and feedback on evalkit, open an issue at https://github.com/inferedge-ai/evalkit/issues; questions go to https://github.com/inferedge-ai/evalkit/discussions.