local-platform-e2e
Skill131 starsChanged 18 days agoMITView on GitHub
Stand up benchmarks-platform locally (Postgres + MinIO + ClickHouse in docker) and run a real @benchsdk/runner benchmark against it, with no cloud or provider credentials. Use when testing @benchsdk/client / @benchsdk/runner against the platform end to end, or when debugging benchmark reporting, worker planning, artifacts, or dashboard results locally.
- Reads credentials
- Installs packages
In this repository4 other files it gives its agents.
- llms.txtllms.txt
- benchsdk-cliSkill
- writing-benchmarksSkill
- add-sandbox-providerSkill
The file
We split this skill into the rules it gives an agent
5 rules. About 0 are common, 0 are distinctive, and 5 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈0
Distinctive · estimate
This author's own choices. Worth a look.
5
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 42
Use ClickHouse >= 25.x: 24.8 fails ch:migrate with TTL expression result column should have DateTime or Date type, but has …
- Very specific: line 145
Check imported/failed/failureSamples in the response.
- Very specific: line 151
Write a throwaway bench inside the repo (untracked, e.g.
- Very specific: line 166
Run it via the bench run CLI (under tsx so the .bench.ts module loads without a build):
- Very specific: line 186
When probing the heartbeat in-process cache, the body must carry currentStep, all four progressDone/InFlight/Errors/Total fields and …
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Local end-to-end: @benchsdk/runner ↔ benchmarks-platform
Goal: exercise upsert benchmark → create run → planWorkers → claim → heartbeat → task_results → artifact upload → complete → dashboard, with zero external credentials.
1. Infra (docker)
docker run -d --name pg -p 5432:5432 -e POSTGRES_PASSWORD=postgres -e POSTGRES_DB=bench postgres:16
docker run -d --name minio -p 9000:9000 -e MINIO_ROOT_USER=minioadmin -e MINIO_ROOT_PASSWORD=minioadmin \
quay.io/minio/minio server /data
docker run -d --name ch -p 8123:8123 -e CLICKHOUSE_PASSWORD=chpass clickhouse/clickhouse-server:25.6
docker run --rm --network host --entrypoint sh quay.io/minio/mc -c \
"mc alias set local http://127.0.0.1:9000 minioadmin minioadmin && mc mb -p local/bench"Substitutes when images or ports aren't available:
- Postgres port taken (a host postgres already on 5432): use it —
DATABASE_URL=postgresql://ci:ci@127.0.0.1:5432/ci(the CI creds; thecidatabase works whenCREATE DATABASEis denied). - MinIO image won't pull (
pull access denied/ quay 401):s3rveris a drop-in —npx -y s3rver -p 9000 -a 0.0.0.0 --directory ~/s3data, thenAWS_ACCESS_KEY_ID=S3RVER AWS_SECRET_ACCESS_KEY=S3RVER aws --endpoint-url http://127.0.0.1:9000 s3 mb s3://benchand setTIGRIS_ACCESS_KEY_ID/TIGRIS_SECRET_ACCESS_KEY=S3RVERin.env.local. npm run db:migratesaysDATABASE_URL ... is not set: it doesn't read.env.local— run withset -a; . ./.env.local; set +a; npm run db:migrate.- Minting
/mcpaccess tokens for MCP E2E: insert ajwksrow (EdDSA keypair, both JWKs stored), anoauth_clientrow, and anoauth_consentrow (clientId + userId + referenceId=orgId, non-nullscopes), then sign anat+jwt(kid = the jwks row id) with claimssub=user,client_id,org,scope, iss=$BETTER_AUTH_URL/api/auth, aud=$BETTER_AUTH_URL/mcp— seelib/oauth/tokens.tsverifyAccessTokenfor the exact shape. - Use ClickHouse >= 25.x: 24.8 fails
ch:migratewithTTL expression result column should have DateTime or Date type, but has DateTime64(3,'UTC'). - The importer passes
clickhouse_settings: { date_time_input_format: 'best_effort' }per insert, so a default-configured server works. If you hitCannot parse input: expected '"' before: 'Z"...'(older code), work around it server-side, and remember to REMOVE the override before verifying an importer fix — otherwise the server setting masks it:docker exec ch bash -c 'mkdir -p /etc/clickhouse-server/users.d && printf "<clickhouse><profiles><default><date_time_input_format>best_effort</date_time_input_format></default></profiles></clickhouse>" > /etc/clickhouse-server/users.d/besteffort.xml' docker restart ch # verify which mode is actually active: curl -s "http://127.0.0.1:8123/?user=default&password=chpass" \ --data-binary "SELECT value FROM system.settings WHERE name='date_time_input_format'"
2. benchmarks-platform .env.local
Point TIGRIS_* at MinIO — the events and artifacts routes require S3 or they
return 502 on every batch. region: "auto" + presigned PUT works with MinIO.
DATABASE_URL=postgresql://postgres:postgres@127.0.0.1:5432/bench
DATABASE_URL_UNPOOLED=postgresql://postgres:postgres@127.0.0.1:5432/bench
CLICKHOUSE_URL=http://127.0.0.1:8123
CLICKHOUSE_DATABASE=default
CLICKHOUSE_USER=default
CLICKHOUSE_PASSWORD=chpass
ADMIN_API_KEY=local-admin-key
BETTER_AUTH_SECRET=<openssl rand -hex 32>
BETTER_AUTH_URL=http://localhost:3000
TIGRIS_ACCESS_KEY_ID=minioadmin
TIGRIS_SECRET_ACCESS_KEY=minioadmin
TIGRIS_STORAGE_ENDPOINT=http://127.0.0.1:9000
TIGRIS_BUCKET=benchThen:
npm run db:migrate
CLICKHOUSE_URL=http://127.0.0.1:8123 CLICKHOUSE_USER=default CLICKHOUSE_PASSWORD=chpass npm run ch:migrate
npm run devch:migrate does not read .env.local; without exported vars it silently
prints "CLICKHOUSE_URL not set; skipping".
3. Mandatory seed (otherwise every run creation 500s on an FK)
app/api/v1/benchmarks/[slug]/runs/route.ts hardcodes a default org and user id
for every created run. Insert those exact rows:
INSERT INTO "user" (id,name,email,email_verified)
VALUES ('mxYI5c90QNkRPhvuc2HvNHy5mM7MG7jG','David Tice','david@example.com',true);
INSERT INTO organization (id,name,slug,created_at,owner_id)
VALUES ('zMrSfAyEyVJ2eKIxgoBZtrMvEd6a78ad','ComputeSDK','computesdk',now(),'mxYI5c90QNkRPhvuc2HvNHy5mM7MG7jG');
INSERT INTO member (id,organization_id,user_id,role,created_at)
VALUES ('mem1','zMrSfAyEyVJ2eKIxgoBZtrMvEd6a78ad','mxYI5c90QNkRPhvuc2HvNHy5mM7MG7jG','owner',now());Re-read those constants before seeding — they may change.
As of the org-scoped-auth change (lib/api/api-auth.ts requireApiAuth), this seed is
only needed for the admin-key path: an org-scoped key supplies the run's
organizationId itself and userId from the key's created_by. Admin-key runs with no
organizationId in the body still fall back to those two hardcoded ids.
3b. Minting org-scoped bp_ API keys locally
The org API-key HTTP route is session-scoped and answers {"error":"Unauthenticated"}
to the admin key, so keys can only be created from the dashboard UI or directly. For
scripted multi-tenant tests, mint them with the platform's own generator so the sha256
hash/prefix/lastFour match what verifyApiKey() expects:
// scripts/e2e-seed-orgs.ts (throwaway), run with:
// npx tsx --env-file=.env.local scripts/e2e-seed-orgs.ts
import { generateApiKey } from "@/lib/api-keys";
import { apiKey, member, organization, user } from "@/db/auth-schema";
const g = generateApiKey(); // g.plaintext is the bp_<prefix>_<secret> to send
await db.insert(apiKey).values({
id, organizationId, name, prefix: g.prefix, hashedKey: g.hashedKey,
lastFour: g.lastFour, createdBy: someUserId, createdAt: new Date(),
revokedAt: null, expiresAt: null, // set these to test revoked/expired → 401
});Each org needs user + organization (owner_id) + member rows first. To view an
org's runs in the dashboard, add a member row for your dashboard user in that org —
dashboard auth is session-based and completely separate from API keys.
4. Dashboard access
Sign up via POST /api/auth/sign-up/email (email+password is enabled), then add
a member row for that user in the seeded org, and sign in at
http://localhost:3000/signin. Run pages live at
/{orgSlug}/benchmarks/{benchmarkSlug}/runs/{runId} (+ /workers).
5. Getting results into ClickHouse locally
@vercel/queue.send() fails locally (swallowed as a warning), so nothing
imports automatically. Trigger it by hand — the cron route accepts the admin key
when CRON_SECRET is unset:
curl -s "http://localhost:3000/api/cron/import-clickhouse?limit=50" -H "Authorization: Bearer local-admin-key"Check imported/failed/failureSamples in the response.
6. Running a benchmark with no provider credentials
Build first (packages/*/dist is not committed): pnpm install && pnpm -r --filter "./packages/**" build.
Write a throwaway bench inside the repo (untracked, e.g. e2e-local/local.bench.ts)
so pnpm workspace resolution finds @benchsdk/runner, with a fake participant:
import { defineBenchmarkConfig, defineTask } from '@benchsdk/runner';
export const config = defineBenchmarkConfig({
benchmarkSlug: 'e2e-local', benchmarkName: 'E2E', iterations: 4, concurrency: 1,
participants: [{ name: 'local', requiredEnvVars: [] }],
});
export const task = defineTask(async (ctx) => {
await ctx.step('create', () => new Promise((r) => setTimeout(r, 50)));
ctx.measure({ ok: true });
});Run it via the bench run CLI (under tsx so the .bench.ts module loads without a build):
BENCHMARKS_PLATFORM_URL=http://localhost:3000 BENCHMARKS_PLATFORM_API_KEY=<org bp_ key> \
npx tsx packages/benchsdk-runner/dist/bin.js run e2e-local/local.bench.ts --iterations 4 --concurrency 2BENCHMARKS_PLATFORM_URL is the root URL (the runner appends /api/v1).
The runner authenticates with BENCHMARKS_PLATFORM_API_KEY (an org-scoped bp_ key — mint one
locally per the section below) and pulls the owning org slug from the server, so the "View at:"
URL it prints always points at the run's real org (no BENCHMARKS_PLATFORM_ORG_SLUG needed).
6b. Probing tenant isolation
With benchmarks-platform#47, all app/api/v1/benchmarks/** routes take either the global
ADMIN_API_KEY or an org bp_ key. Expected shapes when a key from org B addresses org A's run:
403 {"error":"API key is not authorized for this organization"} (getScopedRun /
requireRunAccess), 404 for an unknown run/benchmark, 401 Invalid API key for
revoked/expired/malformed tokens, 401 API key required with no header. Listings
(GET /benchmarks/:slug/runs and /results) narrow by organizationScope(auth).
When probing the heartbeat in-process cache, the body must carry currentStep, all four
progressDone/InFlight/Errors/Total fields and concurrency (top-level, not nested)
or the cache is never populated and your "cache poisoning" probe proves nothing. A
cache-served beat is recognisable because the response's worker object is minimal (no
status/benchmarkId columns). Route timing metadata (cacheCoalesced) is only logged
for requests slower than 1000 ms, so don't rely on the dev-server log for this.
7. Useful probes when testing lifecycle behaviour
psqlis not installed on the host — usedocker exec pg psql -U postgres -d bench ....- To catch transient run statuses (
planned→in_progress→completed), poll in a background subshell while the bench runs, thensort -uthe samples:
Note(for i in $(seq 1 60); do docker exec pg psql -U postgres -d bench -tA -c \ "select r.status||'|'||p.status||'|'||w.status from ..."; sleep 0.4; done > /tmp/poll.txt) &exec's&backgrounds the wholecd X && ...chain — passworkdirinstead of a leadingcd, or the foreground command runs in the wrong dir. - To force a failed run, make the harness task throw for one taskIndex; the CLI
calls
failWorkerwhen any task fails (runner.ts), which is what drives the run tofailed. - Worker release / re-claim and oversized event batches are easiest to drive with
raw curl/python against the v1 API using the admin key; the events body needs
{type:'task_results', sequenceNumber, isFinal, attemptId, records:[...]}.
8. Known sharp edges (verify before blaming your setup)
--group-by round:targetConcurrencyis read by the platform as tasks-per-worker, so the runner must send the fullschedule.length. If it sends 1, only one task is planned (worker range 0-0) while every record is still accepted — results look right but progress/ranges do not.benchmark_run_executions.statusshould roll up from worker status (in_progresson first claim,completed/failedon the last worker). If a finished run still readsplanned, the rollup is broken — the dashboard badge derives status separately and will hide it, so check Postgres and/progress.run.statusvs/progress.summary.status, which must agree.- Barriers: custom step names work off the
concurrencysamples the heartbeat sends, butworker.readyandsandbox.liveare derived from the worker'sprogress_in_flightcolumn — for those you must callreporter.setProgress({ inFlight: N, ... })first or the barrier seesactive=0and hangs until timeout. client.releaseWorker()→ 200 puts the worker back topendingand the attempt toreleased; a re-claim then getsattemptNumber: 2. A duplicate-key error onunique(worker_id, attempt_number)means the claim route hardcoded attempt number 1.- Body limits: the events route allows 32 MiB (
MAX_EVENT_BODY_BYTES), all other v1 routes 1 MiB. 5,000 records × 3 steps ≈ 3.15 MiB, so full-size SDK batches should be 202. If you see 413 on events, the raised cap is not wired up. Always checkbenchmark_event_batches.status = 'persisted'too — a 202 only means the body was accepted. - Org-scoped
bp_…keys on benchmark v1 routes arrived with benchmarks-platform#47 (see §3b/§6b). On a platform checkout without it those routes are admin-only and everybp_…key 401s there, even though the same key works on/api/v1/organizations/...— check which behaviour your checkout has before debugging a 401.
Devin Secrets Needed
None. This entire flow runs offline with local docker containers and a
self-chosen ADMIN_API_KEY.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

