benchsdk-cli
Skill131 starsChanged 18 days agoMITView on GitHub
Install, authenticate, and use the unified bench CLI to query and run ComputeSDK benchmarks. Covers device-code OAuth, API keys, config files, and non-interactive/CI use.
- Reads credentials
- Installs packages
In this repository4 other files it gives its agents.
- llms.txtllms.txt
- local-platform-e2eSkill
- writing-benchmarksSkill
- add-sandbox-providerSkill
The file
We split this skill into the rules it gives an agent
2 rules. About 0 are common, 1 is distinctive, and 1 is very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈1
Distinctive · estimate
This author's own choices. Worth a look.
1
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 20
Run via the built binary:
- Very specific: line 192
If no credentials are present, the CLI fails fast with an AuthError.
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
bench CLI
The bench binary is the unified CLI for the ComputeSDK benchmarks platform.
bench run executes benchmark files; all other commands query platform data.
Installation
The CLI ships with @benchsdk/runner. From the benchmarks repo root:
pnpm install
pnpm -r --filter "./packages/**" buildRun via the built binary:
node packages/benchsdk-runner/dist/bin.cjs --versionOr add packages/benchsdk-runner to your PATH so bench --version works.
Authentication
The CLI tries credentials in this order:
--api-key <key>BENCHMARKS_PLATFORM_TOKEN(a better-auth session token — does not refresh)BENCHMARKS_PLATFORM_API_KEY(an org-scopedbp_...key)- OAuth tokens from
~/.benchsdk/credentials.json(frombench auth login)
API keys and tokens are useful for non-interactive environments. OAuth is
interactive and stores a refresh token under ~/.benchsdk. At least one
credential is required for any command that touches the platform.
Device-code login
bench auth login --base-url https://platform.computesdk.comThis prints a URL and a user code. Approve the device in the browser; the CLI polls and stores the access/refresh token pair. Access tokens last 1 hour; refresh tokens are 30-day sliding windows and rotate on every refresh.
Check status
bench auth status
bench auth status --jsonLog out
bench auth logoutThis removes ~/.benchsdk/credentials.json.
Configuration file
~/.benchsdk/config.json stores non-secret defaults:
{
"baseUrl": "https://platform.computesdk.com",
"org": "computesdk",
"format": "table"
}--base-url, --org, and --format override this file. Credentials live in
~/.benchsdk/credentials.json and are created with 0600 file permissions.
Common commands
Organizations
bench org list
bench org use computesdkbench org use switches the active organization and rewrites the stored access
token with the new organization claim.
Benchmarks
bench benchmarks list
bench benchmarks list --limit 20 --offset 40
bench benchmarks list --jsonRuns
bench runs list my-benchmark
bench runs list my-benchmark --participant e2b --since 7d
bench runs list my-benchmark --day 2026-10-22 --tz America/Los_Angeles
bench runs show my-benchmark <runId>Results
bench results my-benchmark
bench results my-benchmark --run <runId>
bench results my-benchmark --format json
bench results my-benchmark --participant e2b --steps create --since 7d
bench results my-benchmark --steps all --day 2026-10-22Investigating one provider
"How did <participant>'s <step> do over the last N days?" — the answer is three commands:
# Runs the provider took part in, newest first:
bench runs list my-benchmark --participant e2b --since 7d
# Per-run results for just that provider's step (table: one row per run × participant × step):
bench results my-benchmark --participant e2b --steps create --since 7d
# The actual worker logs for a suspicious run:
bench logs my-benchmark <runId> --participant e2bFilter flags: --participant <list> (comma-separated), --steps <list> or --steps all (results only), --since / --until (YYYY-MM-DD, an ISO timestamp, or a relative 24h / 7d / 2w), --day YYYY-MM-DD (sugar for since+until on that day — cannot be combined with --since/--until), --tz <IANA zone> for date-only bounds (defaults to the machine's zone), --source-org <slug> when a slug is ambiguous across orgs. The response echoes the filter the server applied; a platform without the feature fails loudly instead of returning unfiltered results. --json prints the raw response including the filter echo.
Reliability
Uptime views for a type=reliability benchmark (e.g. the actions-reliability canary) — the same surface as the dashboard's reliability page and GET /api/v1/benchmarks/:slug/reliability:
# Aggregated range (default: last 7 days, day buckets, one row per participant + incidents):
bench reliability my-benchmark
bench reliability my-benchmark --from 2026-10-01 --to 2026-10-28
bench reliability my-benchmark --bucket hour --from 2026-10-27T00:00:00Z
# Per-slot view of one participant on one local day:
bench reliability my-benchmark --day 2026-10-27 --participant e2bThe range view prints each participant's uptime/passed/degraded/failed counts, current status, and scheduler stats (expected vs fired vs missed slots — missed slots attribute downtime to the scheduler rather than the provider), plus incidents and top errors. The day view prints a one-mark-per-slot strip (. passed, ~ degraded, x failed, ! missed, pending) with the failing slots listed. Flags: --day requires --participant and cannot combine with --from/--to/--bucket/--metrics/--breakdown; --metrics/--breakdown take comma-separated task-data keys; --tz defaults to the machine's zone; --source-org disambiguates shared slugs. --json prints the raw response.
Artifacts
bench artifacts list my-benchmark <runId>
bench artifacts list my-benchmark <runId> --worker <workerId>Export
bench export my-benchmark --out ./exports
bench export my-benchmark --run <runId> --out ./exportsRunning benchmarks
bench run still executes benchmark files via @benchsdk/runner:
bench run path/to/file.bench.ts --iterations 4 --concurrency 2You can also use an org-scoped bp_ key for non-interactive runs:
BENCHMARKS_PLATFORM_API_KEY=bp_... bench run path/to/file.bench.tsNon-interactive / CI use
Set environment variables so the CLI never tries to open a browser:
export BENCHMARKS_PLATFORM_URL=https://platform.computesdk.com
export BENCHMARKS_PLATFORM_API_KEY=bp_...
bench benchmarks list --json --limit 100If no credentials are present, the CLI fails fast with an AuthError.
Environment variables
BENCHMARKS_PLATFORM_URL— platform root URL (default:https://platform.computesdk.com)BENCHMARKS_PLATFORM_API_KEY— API keyBENCHMARKS_PLATFORM_TOKEN— bearer/session tokenCI— when set, the CLI avoids interactive prompts
Global flags
--base-url <url>— override the platform URL--api-key <key>— use an API key for this command--org <slug>— override the active organization--format json|table— output format--json— shortcut for--format json--verbose— print stack traces and extra diagnostics on errors--version--helpandbench <command> --help
Exit codes
0— success1— general error or API failure2— authentication error
Troubleshooting
bench --versionshows0.0.0— thedist/output is stale or thepackage.jsonwas not found. Rebuild withpnpm --filter @benchsdk/cli build.Cannot find module .../dist/bin.cjs— build@benchsdk/runnerfirst:pnpm --filter @benchsdk/runner build.Your session has expired— the refresh token is expired or revoked. Runbench auth loginagain, or useBENCHMARKS_PLATFORM_API_KEY.- 401/403 on list commands — verify the active org with
bench auth statusor pass--org <slug>. - The CLI auto-refreshes access tokens within 5 minutes of expiry. If a command fails with an auth error, the refresh token may also be expired.
Secrets needed
None for local development with BENCHMARKS_PLATFORM_API_KEY if one is
provided. For OAuth login in CI, prefer BENCHMARKS_PLATFORM_API_KEY or
BENCHMARKS_PLATFORM_TOKEN instead of bench auth login.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

