evalkit — agent notes

Terms

Use this vocabulary everywhere — code, specs, PRs, UI copy.

Layout

• src/evalkit/contracts/ — pydantic models for the wire contracts (ATIF, Harness API, Custom Evaluator API, case YAML, ESF and the span mapping document)
• src/evalkit/core/ — business logic: runner (prepare_experiment server-side; drive_experiment / evaluate_experiment anywhere; the experiment state machine and lease), evaluators, gauges, importer, lint, scorecard, pull (core/pull.py, the per-run layout evalkit runs pull writes), run digests (core/run_digest.py, per-run and windowed metrics from ATIF), the Blob protocol, judge, workspace references, access rules (core/access.py), review (core/review.py, the review state model); core/traces/ holds the mapping engine, the ESF → ATIF converter, the settle rule, the SessionReader protocol and the Phoenix import adapter; core/python_eval.py runs Python evaluators
• src/evalkit/store/ — store/base.py holds the store models and the runner's Store protocol; store/blob.py is the obstore Blob
• src/evalkit/client/ — the runner's Store, Blob and SessionReader over the evalkit HTTP API (HttpStore, HttpBlobStore, HttpSessionReader), the sync SyncApi, the stored server login (client/connection.py) and the dataset sync helpers; used by the CLI and the SDK
• src/evalkit/sdk/ — run_experiment / evaluate_experiment, task binding, the @evaluator decorator and tracing.register
• src/evalkit/db/ — Postgres and the server's Store (PgStore): users, workspaces and members, tokens, agents, remote evaluators, trace sources, cases and case versions, datasets and syncs, overlays, fixture blobs, review events and comments, experiments, agent runs, experiment runs, evaluations, SQL scorecard aggregation, spans and mappings + alembic
• src/evalkit/api/ — FastAPI server (api/otel.py is the collector, api/traces.py sessions and mappings, api/tokens.py agent tokens, api/runs.py and api/pull.py an agent's runs, api/mcp/ the read-only MCP server at /mcp); src/evalkit/cli/ — typer CLI (local runs, thin HTTP calls for everything else)
• docs/user/ — the user pages (Try it locally, Quickstart, Concepts and the web UI's docs pages). The UI imports these files directly, the wheel bundles them, and evalkit skills install copies them to evalkit-docs/ beside the skills, so skills link to ../evalkit-docs/<page>.md; there is no second copy. Pages link to each other as page.md
• ui/ — React web UI, bundled into the wheel as evalkit/bundled/ui (paths.ui_root()); skills/ — distributable skills
• mkdocs.yml, docs/site/ — the GitHub Pages site: MkDocs Material over docs/user/, plus docs/site/sdk-reference.md (mkdocstrings), hooks, theme and overrides; mike versions it
• deploy/ — Dockerfile (server image build; clients deploy via the evalkit-deployment skill's bundled compose); contracts/ — generated OpenAPI/JSON Schemas

Layering rules

• api → core → store; db may import core, store/base.py and contracts; cli → client → core; sdk → client → core. Arrows never reverse; core imports neither sdk, client nor api; sdk imports nothing from cli. core/traces has no database or HTTP-server imports; the runner reaches sessions only through the SessionReader on RunnerDeps.
• Only store/blob.py imports obstore. store/ never imports core or db. client and cli may import store/base.py (pure models and the Store protocol) but never db, store/blob, or any server-only package (server-only commands live behind the evalkit[server] extra boundary in cli/server_commands.py).
• The CLI never holds database or blob credentials: local runs reach the store and blob through the evalkit API. The server's API keys are stored keys, encrypted in the database with encryption_key; the CLI's keys live in evalkit.yaml credentials: and env.
• The base package depends on opentelemetry-api only; the OTel SDK and the OTLP exporter are the evalkit[otel] extra, opentelemetry-proto is in server.
• evalkit code has zero knowledge of any client's agent internals: no client tool names, paths, buckets, or env vars.

Commands

• uv run pytest — tests (e2e tests excluded by default)
• uv run ruff check / uv run ruff format — lint/format
• uv run mypy src/evalkit — types
• cd ui && npm test — UI tests
• uv run python scripts/gen_schemas.py — regenerate the OpenAPI contract docs under contracts/ (a drift test fails CI when they're stale)
• uv run --group docs mkdocs serve — the docs site on :8000 (mkdocs build --strict is the CI check)
• uv run python scripts/fetch_cost_map.py — fetch the LiteLLM cost map (gitignored; the Docker build and the wheel build fetch it, and the server fetches it at runtime when missing)

Local server

• uv sync --extra server, a Postgres database, and skills/evalkit-deployment/config.example.yaml copied to config.yaml and filled in.
• uv run evalkit db upgrade --config config.yaml, then EVALKIT_ADMIN_PASSWORD=<password> uv run evalkit serve --config config.yaml (API and UI on :8020). The password creates admin on the first start. It listens on 127.0.0.1; pass --host 0.0.0.0 when an agent or harness in Docker must reach it through host.docker.internal.
• cd ui && npm run dev serves a hot-reloading UI on :5180 that proxies /api to :8020.
• Without a checkout: pip install "evalkit-py[server]", then evalkit init-server (creates the database, writes config.yaml + .env, migrates) and evalkit serve in the folder it made.
• serve and db upgrade load the .env beside config.yaml; the environment wins.
• A source checkout reports the plain semver; GET /api/cli returns {"version", "package"}.
• The try-it stack (skills/evalkit-deployment/try/, walkthrough in docs/user/try-locally.md) builds the image from this checkout: docker compose up -d --wait --build.
• uv run python -m tests.e2e.services serves a fake harness, custom evaluator and judge on ports 18080, 19000 and 19090, and a fake identity provider on port 19100 (issuer http://127.0.0.1:19100, client id evalkit-ui, machine client ci / e2e-secret; see tests/e2e/README.md), for e2e runs against a built image.

UI changes in a PR

• Run the server and UI locally (see "Local server"; a throwaway Postgres, dummy keys and the fake services in python -m tests.e2e.services are enough).
• Click through every page and flow the PR changes and take screenshots.
• Add them to the PR description under "Screenshots" with gh pr edit <n> --body-file body.md --attach ./shot.png ...: write ![caption](./shot.png) in body.md and gh uploads each file to GitHub and rewrites the link. Never commit screenshots to the repo.
• tsc, vitest and vite build passing is not a UI test.
• No browser alert(), confirm() or prompt() in new UI code: use inline fields or dialogs.

Extending evalkit

Another package can serve evalkit with API routes and UI pages of its own.

API routes. Build the app with create_app(cfg, ui_dist=..., resolve_tenant=...), then add plain FastAPI routes with app.include_router(...).

• The UI is served by FastAPI's app.frontend(), which is tried only after every other route, so routes added after create_app answer GET.
• evalkit's routes come first and win a path clash, including ones with parameters such as /api/agents/{ref}. Keep yours under a prefix of your own (/api/billing/...).
• A route that needs a signed-in user is added with dependencies=SIGNED_IN from evalkit.api.auth; without it the route is public. SIGNED_IN covers path operations (@router.get and the like), not router.add_route() or mounts. A handler that needs the user takes CurrentUser; an admin-only one takes AdminUser.

UI pages. The UI takes extra pages and nav items from ui/src/extensions/index.tsx, which exports extensions of type UiExtensions (ui/src/lib/extensions.ts). In evalkit it is empty.

• Clone evalkit at the tag vX.Y.Z of the evalkit-py the server runs, replace the ui/src/extensions/ folder with yours, build the UI there (npm ci && npx tsc -b && npx vite build) and pass ui/dist as ui_dist. tsc -b checks your pages against evalkit's types. A Vite alias to a module outside ui/ does not work: imports in that module, such as react, do not resolve.
• routes render for signed-in users inside the app shell. nav items join the top bar, moreNav the More menu, and adminNav the admin part of that menu. adminNav only hides the item from non-admins; the page's endpoints must check admin themselves.
• Keep the folder path, the extensions export, UiExtensions and SIGNED_IN stable: other packages build against them.

CI

• Open pull requests as drafts (gh pr create --draft). CI skips drafts, so run the tests locally and mark the PR ready (gh pr ready <number>) when it should be reviewed and run CI.

.github/workflows/ci.yml runs on PRs to develop/main (non-draft; jobs are path-filtered against the PR base) and on pushes to those branches:

• python — ruff check, ruff format --check, mypy src/evalkit, pytest -n auto
• ui — npm ci, tsc -b, vitest run, vite build in ui/

Release

• .github/workflows/publish.yml runs on pushes to develop and main. When code changed (src/, ui/, docs/user/, skills/, contracts/, packaging, Dockerfile), it builds the server image and pushes it to ghcr.io/inferedge-ai/evalkit, tagged with the full git sha and the branch name. It always builds the wheel (web UI included) as a workflow artifact, and on main, when code changed, it publishes it to PyPI as evalkit-py (job pypi, trusted publishing through the pypi environment).
• main only: the image is also tagged with the version, and the workflow creates GitHub release vX.Y.Z with the wheel attached and the version's CHANGELOG.md section as notes.
• Deploying an image is the operator's job; nothing here deploys.
• Version string: main builds are MAJOR.MINOR.PATCH; every other build is the PEP 440 pre-release MAJOR.MINOR.PATCH.dev<run number>. Git holds the plain semver in pyproject.toml; the workflow stamps .devN with uv version before building the image and the wheel, so both share one version. Feature PRs never bump it: develop builds carry the last released semver plus .devN.
• Releasing: a PR into develop bumps pyproject.toml and renames the ## Unreleased changelog section to the new version; then a develop → main PR (merge commit, not squash) ships it. CI's version job blocks a PR into main unless the version was bumped above main's and the changelog has that section. Entries that land under ## Unreleased while the release PR is open ship in that release without notes; move them into the version's section if they matter. Bump MAJOR when the CLI and server can no longer talk, MINOR for new commands, endpoints or fields, PATCH otherwise. While evalkit is on 0.x, a MINOR difference also counts as incompatible: the CLI refuses every command except evalkit update --server <url>.
• PyPI is the only package index. GET /api/cli returns the server's version; every response carries X-Evalkit-Version. The CLI compares it to its own version on every call, and evalkit update reinstalls evalkit-py==<server version> from PyPI.
• The docs site: ci.yml's python job runs mkdocs build --strict on PRs; publish.yml's docs job deploys it with mike to the gh-pages branch, dev from develop and X.Y.Z (alias latest) from main.

Changelog

CHANGELOG.md has one section per released version plus ## Unreleased at the top. It is a general record of what changed in each version, for anyone reading it: users, operators, contributors. Every PR that changes behaviour, an interface, a dependency, the schema, packaging or the build adds its entry under ## Unreleased, in the same PR; the release PR renames that section. An entry says what changed and, when it is not obvious, why; name the command, flag, endpoint, field or config key so a reader can find it. Note anything refused or different from before in the same entry. Test-only changes and refactors with no visible effect do not need an entry.

Specs and plans

Design specs, implementation plans and other planning documents are not checked in to this repo. Share them in the pull request instead: paste them into the description or link to them.

Code style

• No new docstrings or inline comments unless they state a constraint the code cannot show. Code is the single source of truth. The one exception: the public evalkit.sdk API (evalkit.sdk.__all__) has Google-style docstrings, because they are its published reference.
• f-strings everywhere except stdlib logging (%s placeholders); structlog uses plain kwargs.
• X | None, never Optional[X]. No local imports. No try/except unless the spec calls for it (fail fast).
• Manage deps with uv add / uv remove; run tooling via uv run.