Terminal-Bench harness adapter

Driving a bro as an agent under Harbor, the harness Terminal-Bench runs on. The distribution is bro-benchmark and its package is bro.benchmark, a portion of the framework's bro namespace; nothing in the framework imports it. What the project is and how to drive it is README.md. The launcher-side bro-bench member is mapped by ../bench/AGENTS.md.

Development

This is not a uv workspace member: harbor reaches openai < 3 through litellm while the wire distributions' client extras pin openai == 3, so no single lock satisfies both. The directory therefore locks and syncs on its own: uv sync --directory benchmark --all-groups builds .venv from the uv.lock committed beside this file. It depends through editable path sources on core, ride, both trail-format distributions, the bare provider plug-ins, and bro-bench, whose launcher-side credential and command registrations, and the preset composition the job runner reads, are shared with the root environment. Bare provider plug-ins supply declarations and pricing without pulling wire clients into Harbor's interpreter. No harness is a dependency: Harbor's process runs none, and a trial gets both from the relocatable bundle, which builds and installs bro plus both harnesses and their trail-format distributions from the root workspace. Its requirements export requests the client extras, and the build validates every registered wire client in the bundled interpreter.

Formatting and linting stay the repository root's, which walks this directory through its [tool.ruff] src. The suite's environment rebuild is the root's too, but reaches a run only through the conftest at its pytest root — so the conftest.py beside this file applies it, and a suite that did not would inherit the session running it. pytest and pyright run from here instead, inside .venv: run-tests syncs it and drives both, and --skip benchmark skips that whole stage. Build the wheel with uv build from this directory rather than uv build --package.

The *_e2e_test.py modules sit in a roster no gate stage runs (BENCHMARK_E2E_PYTEST_FILES): they build a bundle and drive the host docker daemon, the way ride/ride/e2e_test.py does. The two that grade a trial spend real tokens as well, so conftest.py holds them out of directory collection: pytest reaches them only as files named on its command line. Run them explicitly, from .venv:

uv run --directory benchmark pytest bro/benchmark/bundle_e2e_test.py
uv run --directory benchmark pytest bro/benchmark/harbor_e2e_test.py
uv run --directory benchmark pytest bro/benchmark/benchmark_job_e2e_test.py

Components

• bro/benchmark/cli.py (benchmark) — owns every post-run verb: bundle, retain, publish, query, and import-trails
• bro/benchmark/bundle.py (benchmark bundle) — builds the relocatable runtime a foreign container runs ride from, in the materialized venv/ + bin/ layout --runtime-bundle takes: a pinned standalone CPython with the dependencies WHEEL_PACKAGES resolves to against the workspace lock and those distributions themselves as built wheels installed into its own site-packages, every console script relocated to find the interpreter beside itself, ride's shim farm over the session commands, and runtime assets installed by the bundle’s own registered harnesses. Its manifest records the benchmark installation home and those inputs plus each harness’s files and checksums and gives the bundle a content-derived identity. The bundle writes its installation-home marker beside the interpreter's environment. Bundle is the layout a consumer addresses — interpreter, scripts, shim farm, manifest, and identity; built(root) reports a stale, incomplete, or malformed bundle rather than building one behind the caller's back; cached(workspace, root) builds a stale one first and refuses the rest
• bro/benchmark/harbor_agent.py — BroAgent, the BaseInstalledAgent harbor imports. Its Harbor version is the uploaded bundle's identity, and its recorded name carries the bro and, off the default, the harness. install() has the uploaded bundle's own framework hydrate a trial store from the host store's defaults, rooted at the preset's credentials list and including any preserved-reference dependencies, then uploads both and runs bro show <bro> through the uploaded bundle — the one validation the host cannot make, and a smoke test of the bundle in the task's own image — then ride check-harness <name>, so the selected harness owns its runtime probe. The launcher validates the trial's harness against the uploaded bundle's manifest rather than against its own installation. run() is a single ride solo --unboxed --tree "$PWD" --runtime-bundle <bundle> … under setsid, a join-only ride in the task's directory whose records land under the collected agent directory, reaped through a fresh root exec when harbor cancels the phase. The root revokes bro.placement=boxed and grants bro.placement=join, so its summons may join the task tree and start no new party (bro/reference/permissions.md, "Start surfaces"). Harbor's model_name carries <provider>/<rest of recipe text>, required by every preset. The trial writes a host configuration whose defaults select its harness and recipe for roots and summons; ride solo passes neither --harness nor --llm; the retry classification is the roster providers' declared failure signatures; the token counters harbor keeps are summed from the trial's trail store
• bro/benchmark/trial_store.py — the trail store a trial's run leaves under its collected agent directory: its path, whether a run recorded into it, and the token totals over its trails
• bro/benchmark/harbor_environment.py — UnmountedDockerEnvironment, the environment the job config names: it keeps a trial's logs, artifacts and reward inside the task container for harbor to copy out, so a job runs wherever the docker daemon is reachable and leaves nothing on the docker host
• bro/benchmark/trajectory.py — converts the projected local trail in each finished trial into Harbor's ATIF v1.7 models at agent/trajectory.json, including normalized counts for reporting and step/final metrics priced through caller-supplied provider tables
• bro/benchmark/job.py (bro.benchmark.job) — runs Harbor on the cached bundle against a known concrete job directory and adds the run's provenance to that raw result: the built bundle's manifest, and the presets record its config carries. A config Harbor would read lossily, or a docker-environment job on a host without the compose plugin, fails before Harbor runs
• bro/benchmark/retention.py (benchmark retain) — resolves one raw job from an artifact ref or local path. It derives the manifest's trial rows from Harbor records and local trail stores. It copies the snapshotted files under the flat date/job key with conditional checksummed puts and the manifest last. It also owns the reads of a retained run the later verbs share: the prefix shape, the manifest, its file list, and the verified download
• bro/benchmark/publication.py (benchmark publish) — verifies and downloads a retained run into scratch. It derives Harbor trajectories and result costs from the manifest's captured rates. It uploads with the scoped Harbor credential and appends an immutable publication record
• bro/benchmark/query.py (benchmark query) — loads DuckDB's S3 extensions and credential-chain secret, then defines typed runs and trials views over the current-format retention markers for inline, file, or interactive SQL
• bro/benchmark/import_trails.py (benchmark import-trails) — downloads each trial's retained trail store from the manifest's file list and imports it whole, parents first, into the configured trails store
• agents/, settings/, datasets/, tasks/ — the presets a run is composed from, a JSON file each, read by bro.bench.presets (../bench/AGENTS.md); with the bundle, the whole of what a score depends on