writing-benchmarks
Skill131 starsChanged 18 days agoMITView on GitHub
Author or improve a ComputeSDK benchmark. Turn a unit of work into a declarative *.bench.ts file (config + task with benchsdk steps), set up participants, scoring, and display, verify with a dry run, and wire it into package.json scripts and CI. Use when asked to create, extend, or improve a benchmark.
- Reads credentials
In this repository4 other files it gives its agents.
- llms.txtllms.txt
- benchsdk-cliSkill
- local-platform-e2eSkill
- add-sandbox-providerSkill
The file
We split this skill into the rules it gives an agent
7 rules. About 0 are common, 2 are distinctive, and 5 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈2
Distinctive · estimate
This author's own choices. Worth a look.
5
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 23
Before writing code, pin down:
- Very specific: line 82
Always destroy in finally: step('destroy', () => withTimeout(sandbox.destroy(), timeout), { reportConcurrency: false }).catch(err => …
- Very specific: line 99
Place under benchmarks/<area>/<name>.bench.ts.
- Very specific: line 126
Prefer scoring (validated at startup).
- Very specific: line 146
No provider creds? Exercise the file with a stub participant (requiredEnvVars: []), or use the local-platform-e2e skill to run against a …
- Distinctive: line 149
Check the run page URL printed by the runner when not dry-running.
- Very specific: line 165
Read WRITING_BENCHMARKS.md § "Common gotchas" before debugging: stale dist/, --iterations scaling per-phase, env-filtered participants, the …
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Writing a benchmark
A benchmark is a unit of work (what one iteration measures) expressed as a
declarative *.bench.ts file that exports exactly config + task. The
bench run CLI owns orchestration — the file never calls the runner itself.
Canonical API reference: WRITING_BENCHMARKS.md (repo root) — read it for the
full field tables. Runnable templates: examples/01-hello.bench.ts through
11-logging.bench.ts. Reference implementations:
benchmarks/sandbox/tti.bench.ts— minimal sandbox benchmark (create → exec → destroy, one metric).benchmarks/sandbox/dax.bench.ts— rich benchmark (work measured inside the sandbox, pre-measured phase steps,TaskError, legacy results writer).
1. Define the unit of work
Before writing code, pin down:
- Timed boundary — what starts the clock and what ends it. For TTI that's
sandbox.create()→ first command succeeding; destroy is excluded. Write the boundary in the file's header comment. - Uniformity — the task body must run identically for every participant.
Push provider differences into participant config (
sandboxOptions,timeout,destroyTimeoutMs), never intoif (provider === ...)branches inside the task. - Headline metric(s) — the numbers the task reports via
measure()and what counts as a success (exit-code checks,scoring.success.requireData). - Load profile — which of
iterations,concurrency,staggerDelayMsdefine the benchmark vs. stay CLI-overridable. Ashapemay set slug/name/staggerDelayMs, neveriterations/concurrency. See § Horizontal load.
2. Horizontal load — iterations × concurrency
Horizontal load (how many units of work run at once, and how fast they ramp) is controlled entirely by three config/CLI knobs — the task function itself never manages parallelism:
iterations— total tasks per participant. Each iteration is one full unit of work (one sandbox create → work → destroy). This is the horizontal volume knob:iterations: 100launches 100 sandboxes per participant.concurrency— max tasks in flight at once.1= sequential (one-at-a-time cold starts),N= up to N units of work running simultaneously.iterations: N, concurrency: Nis a burst: all N sandboxes launch at once.staggerDelayMs— task i launches ati * staggerDelayMs. Combined withconcurrency: Nthis is a staggered ramp (e.g. 100 sandboxes, one every 200ms) — measures how TTI degrades as load ramps.
The named run shapes are just presets over these knobs:
sequential = {concurrency: 1}, burst = {concurrency: N},
staggered = {concurrency: N, staggerDelayMs: D}.
Two related knobs that change ordering/fan-out, not load:
groupBy—'participant'(default: each participant runs to completion before the next) vs'round'(participants take turns per iteration, so every participant's Nth task sees the same external conditions).phases— split iterations into named segments (cold/warm). Total iterations = sum of phase iterations;--iterationsscales each phase.
In CI, horizontal scale comes from the matrix: one job per provider, each
passing --provider <name> plus the same --run-key so they share one
platform run, and --iterations/--concurrency from workflow inputs.
3. Decompose the work into steps
Wrap each phase in step('name', fn) — each step becomes a timed, labeled row
on the platform run page.
Sandbox-benchmark conventions (follow tti.bench.ts):
- Steps:
create→ the work →destroy. Keep names short;display.stepsmaps them to labels. - Always destroy in
finally:step('destroy', () => withTimeout(sandbox.destroy(), timeout), { reportConcurrency: false }).catch(err => log('destroy failed', { level: 'warn', ... })). - Guard every external call with
withTimeout(benchmarks/src/util/timeout.ts) and a provider overridable timeout. log()with{ level, meta }for diagnosis — the worker log uploads as a run artifact. Log the sandbox id after destroy (sandboxId()util).- Step return values shaped like
{ stdout, stderr, error, exitCode, ... }are captured as step output, not returned — passcaptureOutput: falseor wrap. - Work measured inside the sandbox (a script printing
BENCH_*lines): return{ data, steps, latencyMs }from the task with pre-measuredTaskStepRecord[], and setgroupBy: 'round'so the runner honors them (participant mode ignores pre-measured steps). Seedax.bench.ts. - Errors: throw
TaskError(code + data) for domain failures; let plain Errors bubble for unexpected ones.
4. Write the file
- Place under
benchmarks/<area>/<name>.bench.ts. Start from the closest existing benchmark, not from scratch. import '../src/env.js'first (loadsbenchmarks/.env).configessentials:benchmarkSlug,benchmarkName,iterations,concurrency,participants,display,scoring. Optional:shapes,phases,groupBy,defaultProviders,dimensions,onComplete,customCliFlags— see WRITING_BENCHMARKS.md before reaching for them.
5. Participants
- Sandbox benchmarks share
benchmarks/sandbox/providers.ts(ProviderConfig:name,requiredEnvVars,createCompute,sandboxOptions,timeout,destroyTimeoutMs). Other areas have their ownproviders.ts; new domains should copy the pattern. - Missing
requiredEnvVars→ participant is skipped with a log; all skipped → cleanNoAvailableParticipantsErrorexit.--providerfilters further. - Vercel always gets
sandboxOptions: { persistent: false }— the default auto-snapshots on everystop()and accrues Snapshot Storage. defaultProviderslimits what runs without--provider(e.g. dax runs a subset by default).
6. Scoring and display
scoring.metrics:weights.median + p95 + p99across all metrics must sum to 1.0;ceiling= worst acceptable value;higherIsBetter: true+floorfor throughput-style metrics;trim(default 0.05) drops outliers before median/p95/p99.- Prefer
scoring(validated at startup); useonScoreonly when a metric needs a function extractor. display.steps[].keymust exactly matchstep()names;display.metrics[].keymatchesmeasure()keys. Undeclared keys still render — declarations only improve labels/units/direction.scoring.groupByrenders a breakout per distinctdatavalue (only with 2–12 values) — different from top-levelgroupBy(execution ordering).
7. Verify
# packages/*/dist is not committed — build once first
pnpm -r --filter "./packages/**" build
pnpm typecheck
# Dry run still requires platform auth, but skips ingestion
BENCHMARKS_PLATFORM_API_KEY=bp-... pnpm exec bench run \
benchmarks/<area>/<name>.bench.ts --provider <name> --iterations 2 --dry-run- No provider creds? Exercise the file with a stub participant
(
requiredEnvVars: []), or use thelocal-platform-e2eskill to run against a local benchmarks-platform. - Check the run page URL printed by the runner when not dry-running.
8. Wire it in
package.json: add"bench:<name>"(andbench:<name>:<provider>variants when useful) following the existing script shape.- CI: copy the closest workflow in
.github/workflows/— provider matrix,load-vault-secrets.shregex (add any new env var names), shared--run-key "$GITHUB_RUN_ID-$GITHUB_RUN_ATTEMPT",--no-ingeston push,SHOULD_RUNguard for single-provider dispatch. results/legacy JSON +generate-*.svgonly if the README/results pipeline consumes the new benchmark — otherwise skip the legacy writers.- Draft PR by default.
Gotchas
Read WRITING_BENCHMARKS.md § "Common gotchas" before debugging: stale
dist/, --iterations scaling per-phase, env-filtered participants, the
weight-sum rule, outcome-shaped step returns, round-vs-participant step
ownership, Vercel persistent: false.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

