E2E Tests

E2E tests run SDK scenarios in subprocesses against a mock Braintrust server. Prefer extending the closest existing scenario over inventing a new pattern.

The goal for provider e2e tests is always to exercise actual models with real, valid input data, either live during recording or through cassettes populated with actual model responses. The mock Braintrust server captures telemetry; provider behavior should come from the real provider.

Read first:

• e2e/README.md
• Closest e2e/scenarios/<name>/scenario.test.ts
• Relevant shared helper in e2e/helpers/
• Relevant assertions.ts file when the scenario family already factors shared checks that way

Workflow

1. Start from the closest existing scenario and keep its structure unless the new case clearly needs a new pattern.
2. Default to module-scope setup with prepareScenarioDir({ scenarioDir: resolveScenarioDir(import.meta.url) }). This copies the scenario into an isolated temp directory and installs any scenario-local dependencies before the test bodies run.
3. Use withScenarioHarness(...) for every scenario test. Pick the runner that matches the real entrypoint:
   • runScenarioDir() for default tsx-driven TypeScript scenarios
   • runNodeScenarioDir() for plain Node entrypoints and hook coverage
   • runDenoScenarioDir() for nested Deno runners
4. Snapshot stable contracts, not raw noise. Normalize before snapshotting and prefer focused summaries over full payload dumps.
5. Run the narrowest test first, then rerun updated scenarios three times before treating snapshots as stable.

Commands

Run workspace scripts from the repo root when you want the standard e2e entrypoints:

pnpm run test:e2e
pnpm run test:e2e:record # re-record provider cassettes and update snapshots

Try not to use specific test narrowing commands unless hunting down a very nasty and specific bug.

Cassettes

Cassettes mock provider HTTP responses (OpenAI, Anthropic, ...) so external-provider scenarios replay in CI without provider keys. The harness starts a local @braintrust/seinfeld cassette server and points provider SDK base URL env vars at that server, so subprocesses and SDK-launched binaries are covered too.

• Record cassettes by calling actual provider models through the real SDK. Do not hand-author model responses, embedding vectors, or usage data, or replace the provider with a synthetic HTTP server or fetch stub to make an e2e scenario pass. Synthetic responses belong in focused unit tests and do not satisfy provider e2e coverage.
• Use real, valid inputs that the model can process. Multimodal scenarios need decodable images, audio, video, and documents; arbitrary bytes labeled with a media MIME type do not qualify.
• If recording is blocked by missing credentials, model access, or provider availability, report the blocker and the remaining recording work. Do not fabricate cassette data or silently substitute synthetic provider responses.
• Default to cheap provider models for real API calls and cassette recordings, unless the user explicitly requests otherwise. Choose a model that supports the behavior under test, and keep prompts, output limits, and media duration small while preserving meaningful coverage.
• External-provider tests should thread runContext: { variantKey: "...", originalScenarioDir } into the scenario runner. In normal replay mode, missing cassette entries fail loudly instead of skipping or falling back to live providers. In record / record-missing mode, they run so new cassettes can be authored.
• Thread runContext: { variantKey: "...", originalScenarioDir } into runScenarioDir/runNodeScenarioDir. Cassettes live at e2e/scenarios/<name>/__cassettes__/<variantKey>.cassette.json (parallel to __snapshots__/). Only set runContext.cassette explicitly for unusual cases, such as cassette: false on a non-provider mode inside an otherwise provider-backed scenario.

• To re-record after changing a scenario:

  ANTHROPIC_API_KEY=... OPENAI_API_KEY=... \
    pnpm --filter=@braintrust/js-e2e-tests run test:e2e:record -- scenarios/<name>/scenario.test.ts

  Then run again in BRAINTRUST_E2E_CASSETTE_MODE=replay with no provider keys to confirm the cassette is sufficient.

• Volatile fields in request bodies (e.g. AI-SDK experimental_generateMessageId) need a per-scenario filter. Add or update <scenario-dir>/cassette-filter.mjs with a named filter export. The cassette server loads that filter through withScenarioHarness(...).
• After recording, always inspect the cassette for leaked API keys before committing. The recorder redacts common patterns (paranoid preset), but confirm that no real keys appear in request headers, response bodies, or any URL query parameters. If a key slips through, remove the cassette, add a custom redact rule, and re-record.

Preferred Patterns

• Keep the expensive setup at module scope with prepareScenarioDir(...). Only call installScenarioDependencies(...) directly when you are testing installer behavior or need a nonstandard setup.
• Run every scenario through withScenarioHarness(...).
• Keep reusable logic in e2e/helpers/. Keep one-off fixtures and scenario-specific files inside the scenario directory.
• Snapshot stable contracts, not raw noise. Use normalizeForSnapshot(...) before inline snapshots and formatJsonFileSnapshot(...) plus file snapshots for larger payloads or version matrices.
• For span-tree snapshots, use matchSpanTreeSnapshot(...). It writes and asserts paired .span-tree.json and .span-tree.txt files from the same normalized span tree. The JSON file is the structural contract that is easiest to parse mechanically; the TXT file is the ASCII tree that is easiest to review by eye. Keep them in sync by updating both through the e2e update/record commands, and never hand-edit only one side of the pair.
• For tracing-instrumentation span-tree snapshots, always capture input, output, metadata, context, error, and metrics. Normalize volatile values within these fields, but do not omit a field or selectively filter its semantic contents to reduce snapshot size or noise.
• When a scenario family already has assertions.ts, keep version- or provider-specific test setup in scenario.test.ts and reuse the shared assertions file.
• Keep the CI e2e summary up to date. If a scenario version matrix or variantKey changes, update e2e/config/pr-comment-scenarios.json in the same change and follow the established pattern used by other versioned scenarios: one summary row per version, not separate wrapped/auto rows unless that pattern already exists for the scenario family.
• Run new or updated scenarios three times in a row before considering snapshots stable.
• Do not add tests directly asserting on the auto instrumentation configs.

Scenario Patterns

• SDK primitive scenarios: use scenario.ts with normal SDK calls and assert on testRunEvents(). See trace-primitives-basic.
• Wrapper scenarios: use events() rather than testRunEvents(), find the root span first, and scope payload snapshots with payloadRowsForRootSpan(...). Pair span and payload snapshots when the wrapper emits merged log rows.
• Provider instrumentation scenarios often split setup and shared assertions. See e2e/scenarios/anthropic-instrumentation/assertions.ts, e2e/scenarios/google-genai-instrumentation/assertions.ts, and similar directories before creating a new pattern.
• Version matrix scenarios: put shared logic in scenario.impl.* or shared assertion helpers, then loop over versions from aliases or helper-generated scenario lists. Do not duplicate the same assertions per version by hand.
• Test runner integration scenarios (deno, vitest, jest, ...): keep the outer e2e suite in scenario.test.ts, the spawned entry in scenario.ts, and nested test files in names like runner.case.ts. Do not name nested runner files *.test.ts.

Scenario-Local Dependencies

• Only add a scenario-local package.json for truly scenario-specific external dependencies.
• Workspace packages belong in e2e/package.json as workspace:^, not in scenario manifests.
• Do not use workspace: specs in scenario-local manifests.
• If a scenario manifest exists, commit its lockfile.

Generate the lockfile with:

pnpm install --dir e2e/scenarios/<name> --ignore-workspace --lockfile-only --strict-peer-dependencies=false

Debugging

• Flaky snapshot: normalize the changing field instead of snapshotting around it.
• Request-flow assertions: grab requestCursor() before running the scenario, then inspect requestsAfter(...).
• If the scenario is external-provider backed, first confirm it has a cassette or run in BRAINTRUST_E2E_CASSETTE_MODE=record with the required provider env var set.
• Deno/browser scenarios may intentionally differ from Node. Assert the real runtime contract instead of copying Node expectations blindly.