agentleFS
computesdkcomputesdk/benchmarks/.agents/skills/benchsdk-cli/SKILL.md

benchsdk-cli

Skill131 starsChanged 18 days agoMITView on GitHub

Install, authenticate, and use the unified bench CLI to query and run ComputeSDK benchmarks. Covers device-code OAuth, API keys, config files, and non-interactive/CI use.

  • Reads credentials
  • Installs packages

The file

We split this skill into the rules it gives an agent

2 rules. About 0 are common, 1 is distinctive, and 1 is very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈1

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 1

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Run via the built binary:

    line 20
  2. Very specific:

    If no credentials are present, the CLI fails fast with an AuthError.

    line 192

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

bench CLI

The bench binary is the unified CLI for the ComputeSDK benchmarks platform. bench run executes benchmark files; all other commands query platform data.

Installation

The CLI ships with @benchsdk/runner. From the benchmarks repo root:

pnpm install
pnpm -r --filter "./packages/**" build

Run via the built binary:

node packages/benchsdk-runner/dist/bin.cjs --version

Or add packages/benchsdk-runner to your PATH so bench --version works.

Authentication

The CLI tries credentials in this order:

  1. --api-key <key>
  2. BENCHMARKS_PLATFORM_TOKEN (a better-auth session token — does not refresh)
  3. BENCHMARKS_PLATFORM_API_KEY (an org-scoped bp_... key)
  4. OAuth tokens from ~/.benchsdk/credentials.json (from bench auth login)

API keys and tokens are useful for non-interactive environments. OAuth is interactive and stores a refresh token under ~/.benchsdk. At least one credential is required for any command that touches the platform.

Device-code login

bench auth login --base-url https://platform.computesdk.com

This prints a URL and a user code. Approve the device in the browser; the CLI polls and stores the access/refresh token pair. Access tokens last 1 hour; refresh tokens are 30-day sliding windows and rotate on every refresh.

Check status

bench auth status
bench auth status --json

Log out

bench auth logout

This removes ~/.benchsdk/credentials.json.

Configuration file

~/.benchsdk/config.json stores non-secret defaults:

{
  "baseUrl": "https://platform.computesdk.com",
  "org": "computesdk",
  "format": "table"
}

--base-url, --org, and --format override this file. Credentials live in ~/.benchsdk/credentials.json and are created with 0600 file permissions.

Common commands

Organizations

bench org list
bench org use computesdk

bench org use switches the active organization and rewrites the stored access token with the new organization claim.

Benchmarks

bench benchmarks list
bench benchmarks list --limit 20 --offset 40
bench benchmarks list --json

Runs

bench runs list my-benchmark
bench runs list my-benchmark --participant e2b --since 7d
bench runs list my-benchmark --day 2026-10-22 --tz America/Los_Angeles
bench runs show my-benchmark <runId>

Results

bench results my-benchmark
bench results my-benchmark --run <runId>
bench results my-benchmark --format json
bench results my-benchmark --participant e2b --steps create --since 7d
bench results my-benchmark --steps all --day 2026-10-22

Investigating one provider

"How did <participant>'s <step> do over the last N days?" — the answer is three commands:

# Runs the provider took part in, newest first:
bench runs list my-benchmark --participant e2b --since 7d

# Per-run results for just that provider's step (table: one row per run × participant × step):
bench results my-benchmark --participant e2b --steps create --since 7d

# The actual worker logs for a suspicious run:
bench logs my-benchmark <runId> --participant e2b

Filter flags: --participant <list> (comma-separated), --steps <list> or --steps all (results only), --since / --until (YYYY-MM-DD, an ISO timestamp, or a relative 24h / 7d / 2w), --day YYYY-MM-DD (sugar for since+until on that day — cannot be combined with --since/--until), --tz <IANA zone> for date-only bounds (defaults to the machine's zone), --source-org <slug> when a slug is ambiguous across orgs. The response echoes the filter the server applied; a platform without the feature fails loudly instead of returning unfiltered results. --json prints the raw response including the filter echo.

Reliability

Uptime views for a type=reliability benchmark (e.g. the actions-reliability canary) — the same surface as the dashboard's reliability page and GET /api/v1/benchmarks/:slug/reliability:

# Aggregated range (default: last 7 days, day buckets, one row per participant + incidents):
bench reliability my-benchmark
bench reliability my-benchmark --from 2026-10-01 --to 2026-10-28
bench reliability my-benchmark --bucket hour --from 2026-10-27T00:00:00Z

# Per-slot view of one participant on one local day:
bench reliability my-benchmark --day 2026-10-27 --participant e2b

The range view prints each participant's uptime/passed/degraded/failed counts, current status, and scheduler stats (expected vs fired vs missed slots — missed slots attribute downtime to the scheduler rather than the provider), plus incidents and top errors. The day view prints a one-mark-per-slot strip (. passed, ~ degraded, x failed, ! missed, pending) with the failing slots listed. Flags: --day requires --participant and cannot combine with --from/--to/--bucket/--metrics/--breakdown; --metrics/--breakdown take comma-separated task-data keys; --tz defaults to the machine's zone; --source-org disambiguates shared slugs. --json prints the raw response.

Artifacts

bench artifacts list my-benchmark <runId>
bench artifacts list my-benchmark <runId> --worker <workerId>

Export

bench export my-benchmark --out ./exports
bench export my-benchmark --run <runId> --out ./exports

Running benchmarks

bench run still executes benchmark files via @benchsdk/runner:

bench run path/to/file.bench.ts --iterations 4 --concurrency 2

You can also use an org-scoped bp_ key for non-interactive runs:

BENCHMARKS_PLATFORM_API_KEY=bp_... bench run path/to/file.bench.ts

Non-interactive / CI use

Set environment variables so the CLI never tries to open a browser:

export BENCHMARKS_PLATFORM_URL=https://platform.computesdk.com
export BENCHMARKS_PLATFORM_API_KEY=bp_...

bench benchmarks list --json --limit 100

If no credentials are present, the CLI fails fast with an AuthError.

Environment variables

  • BENCHMARKS_PLATFORM_URL — platform root URL (default: https://platform.computesdk.com)
  • BENCHMARKS_PLATFORM_API_KEY — API key
  • BENCHMARKS_PLATFORM_TOKEN — bearer/session token
  • CI — when set, the CLI avoids interactive prompts

Global flags

  • --base-url <url> — override the platform URL
  • --api-key <key> — use an API key for this command
  • --org <slug> — override the active organization
  • --format json|table — output format
  • --json — shortcut for --format json
  • --verbose — print stack traces and extra diagnostics on errors
  • --version
  • --help and bench <command> --help

Exit codes

  • 0 — success
  • 1 — general error or API failure
  • 2 — authentication error

Troubleshooting

  • bench --version shows 0.0.0 — the dist/ output is stale or the package.json was not found. Rebuild with pnpm --filter @benchsdk/cli build.
  • Cannot find module .../dist/bin.cjs — build @benchsdk/runner first: pnpm --filter @benchsdk/runner build.
  • Your session has expired — the refresh token is expired or revoked. Run bench auth login again, or use BENCHMARKS_PLATFORM_API_KEY.
  • 401/403 on list commands — verify the active org with bench auth status or pass --org <slug>.
  • The CLI auto-refreshes access tokens within 5 minutes of expiry. If a command fails with an auth error, the refresh token may also be expired.

Secrets needed

None for local development with BENCHMARKS_PLATFORM_API_KEY if one is provided. For OAuth login in CI, prefer BENCHMARKS_PLATFORM_API_KEY or BENCHMARKS_PLATFORM_TOKEN instead of bench auth login.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.