dd-instrument-test-optimization
Skill185 starsChanged 38 days agoMITView on GitHub
Sets up Datadog Test Optimization (CI test visibility, per-test traces, flaky test detection) in a repo's CI pipeline for Node.js, Python, Java/JVM, Go, Ruby, .NET, Swift, or JUnit XML. Configures CI auto-instrumentation or manual test tracers, forwards container variables, and checks that test events arrive. Use when asked to add, set up, enable or verify Test Optimization or CI test visibility. Not for triaging flaky tests or unblocking a failing PR; use triage-flaky-test or unblock-pr.
- Reads credentials
In this repository43 other files it gives its agents.
- dd-account-setupSkill
- dd-agent-install-dockerSkill
- dd-agent-install-kubernetesSkill
- dd-agent-install-linuxSkill
- dd-agent-install-macosSkill
- dd-agent-install-windowsSkill
- dd-android-sdkSkill
- dd-anthropic-usage-and-costs-onboardingSkill
- dd-apmSkill
- dd-auditSkill
- dd-aws-ccmSkill
- dd-aws-integrationSkill
- dd-aws-log-forwardingSkill
- dd-aws-storage-managementSkill
- dd-azure-ccmSkill
- dd-azure-integrationSkill
- dd-azure-log-forwardingSkill
- dd-browser-sdkSkill
- dd-confluent-cloud-integrationSkill
- dd-dbm-rds-postgresSkill
- dd-docsSkill
- dd-gcp-ccmSkill
- dd-gcp-integrationSkill
- dd-gcp-log-forwardingSkill
- dd-github-copilot-onboardingSkill
- dd-github-costs-onboardingSkill
- dd-github-onboardingSkill
- dd-instrument-error-trackingSkill
- dd-instrument-llmoSkill
- dd-instrument-product-analyticsSkill
- dd-instrument-rumSkill
- dd-logsSkill
- dd-monitorsSkill
- dd-nebius-integrationSkill
- dd-oci-ccmSkill
- dd-oci-integrationSkill
- dd-openai-onboardingSkill
- dd-orchestratorSkill
- dd-pagerduty-onboardingSkill
- dd-product-recommenderSkill
- dd-profiler-setupSkill
- dd-pupSkill
- dd-slack-integrationSkill
The file
We split this skill into the rules it gives an agent
5 rules. About 0 are common, 2 are distinctive, and 3 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈2
Distinctive · estimate
This author's own choices. Worth a look.
3
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 21
Credentials: Test Optimization uses DD_API_KEY in agentless mode.
- Distinctive: line 37
No test runner or no CI workflow found: stop before editing, report what is missing, and ask which test command or CI provider to configure.
- Very specific: line 49
If upgrading a job from manual or JUnit XML instrumentation to auto-instrumentation, remove the old manual/JUnit steps after wiring …
- Very specific: line 50
When tests_run_in_containers is true (docker run or docker compose), forward the required Datadog, runtime, and CI provider environment …
- Distinctive: line 53
Run local validation: Run the project's test command or configuration check locally (without exposing real keys in command arguments), fix …
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Datadog Test Optimization
Test Optimization collects per-test traces, failure reasons, and git metadata from CI test runs so Datadog can detect flaky tests and run Test Impact Analysis. No application resource has to be created in Datadog beforehand: the test service appears automatically when a test run sends spans with a valid API key and site.
How it works
- What turns Test Optimization on: the CI test job must either run the Datadog Test Optimization CI step (
datadog/test-visibility-github-action@v3, the GitLab script, or the CircleCI orb), run the language tracer in CI Visibility mode (DD_CIVISIBILITY_ENABLED=trueandDD_CIVISIBILITY_AGENTLESS_ENABLED=truewith the test runner hook), or upload JUnit XML reports (datadog-ci junit upload). Havingdd-traceorddtraceinstalled for APM or Code Coverage does not turn Test Optimization on. - GitHub Actions cannot mutate
NODE_OPTIONSacross steps: for JavaScript and TypeScript on GitHub Actions,datadog/test-visibility-github-action@v3exportsDD_TRACE_PACKAGE(andDD_TRACE_ESM_IMPORTfor Vitest), which the test step itself must wire intoNODE_OPTIONS. Without that env line on the test step, no test spans are collected. - Containers cross an execution boundary: when a CI step runs tests through
docker runordocker compose, host-level environment variables do not enter the container automatically. The Datadog variables, the runtime injection variable, and the CI provider's git/pipeline variables must be passed into the container, and prebuilt test images must use manual instrumentation inside the container. - Credentials: Test Optimization uses
DD_API_KEYin agentless mode. Reference it from the CI provider's secret store (${{ secrets.DD_API_KEY }},$DD_API_KEY, or$(DD_API_KEY)); never write a literal key into a workflow or config file.
Set up
Copy these steps into your reply and tick them off as you go.
- Inspect. In a collection of independent projects, ask which one to instrument. Read dependency manifests, test configs, and CI workflow files (
.github/workflows/*.yml,.gitlab-ci.yml,.circleci/config.yml,Jenkinsfile,.buildkite/pipeline.yml,buildkite/pipeline.yml,bitbucket-pipelines.yml,.travis.yml,azure-pipelines.yml) to record:- Runtime and language:
nodejs(JS/TS),python,java(Java/Kotlin/Scala +maven/gradle/sbt),go,ruby,dotnet,swift, orother. - Test framework and version: Jest (>=24.8.0 with
jest-circus), Vitest (>=1.6.0), Mocha (>=5.2.0), Playwright (>=1.18.0), Cypress (>=6.7.0), Cucumber, pytest (>=3.0.0), unittest, JUnit 4 (>=4.10) / JUnit 5 (>=5.3), TestNG, Spock, Scalatest, RSpec (>=3.0.0), Minitest (>=5.0.0),go test, xUnit (>=2.2), NUnit (>=3.0), MSTest v2 (>=14), BenchmarkDotNet, or XCTest. - CI provider and test jobs: which workflow/job runs the tests, and whether the test command crosses into a container (
docker runordocker compose). - Repository name (
<REPO>): normalized lowercasehost/owner/repowithout scheme or.git(fromgit remote get-url origin,package.json,go.mod,pyproject.toml, orpom.xml). - Datadog site (
<SITE>): fromDD_SITEor existing Datadog config (defaultdatadoghq.comwhen unset; if an invalid site is supplied, stop and ask). - Existing Test Optimization setup: whether the test job already runs auto-instrumentation, manual CI Visibility (
DD_CIVISIBILITY_ENABLED/ddtrace-run/--ddtrace/orchestrion go test/dd-trace ci run), ordatadog-ci junit upload.
- Runtime and language:
- Check prerequisites and stop conditions.
- Already instrumented with auto-instrumentation: make no changes and proceed to Verify and view.
- No test runner or no CI workflow found: stop before editing, report what is missing, and ask which test command or CI provider to configure.
- Unsupported test runner (such as Node's built-in
node --testor an older framework version): stop before editing unless the user explicitly requested JUnit XML upload. Explain the two options—migrating to a supported runner (such as Vitest for Node tests) for full tracing and flaky-test management, or uploading JUnit XML reports via references/junit-xml.md—and ask which path to take.
- Choose the instrumentation mode.
- When the Datadog Onboarding MCP server is connected, call
Datadog:test_optimization_onboardingwith the substantiatedruntime,language,test_framework,build_tool(for Java),ci_provider,repo_name,instrumentation(autoby default,manualorjunitxmlwhen required), andtests_run_in_containers. When the MCP server is not connected or the tool fails, follow the bundled reference files below for the exact setup steps. - Whether or not the MCP tool is available, apply the exact rules in the matching reference:
- Auto-instrumentation (recommended for GitHub Actions, GitLab CI, and CircleCI when tests run on the runner or in a container built during the job, except Cypress and BenchmarkDotNet): open references/ci-autoinstrumentation.md.
- Manual instrumentation (required for Buildkite, Bitbucket, Jenkins in automated runs, Azure Pipelines, Travis CI, Cypress, BenchmarkDotNet, Swift, or when tests run inside a prebuilt Docker image where host auto-instrumentation cannot install the tracer): open references/manual-by-runtime.md.
- JUnit XML upload (when requested or chosen for an unsupported runner): open references/junit-xml.md.
- When the Datadog Onboarding MCP server is connected, call
- Wire the CI workflow and test configuration.
- Modify only the existing test job(s) in the CI workflow; preserve unrelated jobs, steps, comments, and YAML formatting.
- If upgrading a job from manual or JUnit XML instrumentation to auto-instrumentation, remove the old manual/JUnit steps after wiring auto-instrumentation (if
datadog-ci junit uploadused--xpath-tag, warn that custom XPath tags are not carried over). - When
tests_run_in_containersis true (docker runordocker compose), forward the required Datadog, runtime, and CI provider environment variables into the container as shown in references/ci-autoinstrumentation.md. - Reference
DD_API_KEYfrom the CI secret store (for example${{ secrets.DD_API_KEY }}in GitHub Actions or$DD_API_KEYin GitLab/Buildkite/CircleCI). In Buildkite, write$DD_API_KEY, never$$DD_API_KEY(which escapes substitution).
- Run local validation. Run the project's test command or configuration check locally (without exposing real keys in command arguments), fix any errors your edits caused, and run it again until it passes.
Verify and view
Record the highest level you actually reached:
- Configuration changed: the CI workflow and any dependency/config files are updated and valid YAML.
- Tests ran: the test suite ran locally or in CI and exited 0.
- Export accepted: in a local run with
DD_CIVISIBILITY_ENABLED=trueandDD_CIVISIBILITY_AGENTLESS_ENABLED=true, or in a CI run withDD_API_KEYconfigured, the tracer ordatadog-ci junit uploadaccepted the test payload without authentication errors. Datadog shows test events:
Datadog:search_datadog_test_events, or the Search test events endpoint (POST /api/v2/ci/tests/events/search) whenDD_API_KEYandDD_APP_KEYare set, finds@git.repository.id_v2:"<REPO>"in the last 15 minutes:SITE=<site from step 1> REPO=<lowercase host/owner/repo> case "$SITE" in datadoghq.com) API_HOST=api.datadoghq.com ;; datadoghq.eu) API_HOST=api.datadoghq.eu ;; us3.datadoghq.com) API_HOST=api.us3.datadoghq.com ;; us5.datadoghq.com) API_HOST=api.us5.datadoghq.com ;; ap1.datadoghq.com) API_HOST=api.ap1.datadoghq.com ;; ap2.datadoghq.com) API_HOST=api.ap2.datadoghq.com ;; uk1.datadoghq.com) API_HOST=api.uk1.datadoghq.com ;; ddog-gov.com) API_HOST=api.ddog-gov.com ;; *) API_HOST= ;; esac if [ -n "$API_HOST" ] && [ -n "$DD_API_KEY" ] && [ -n "$DD_APP_KEY" ]; then keys() { echo "DD-API-KEY: $DD_API_KEY"; echo "DD-APPLICATION-KEY: $DD_APP_KEY"; } BODY=$(printf '{"filter":{"query":"@git.repository.id_v2:\\"%s\\"","from":"now-15m","to":"now"},"page":{"limit":5}}' "$REPO") keys | curl -sS -w ' HTTP %{http_code}' -H @- -H 'Content-Type: application/json' -X POST -d "$BODY" "https://$API_HOST/api/v2/ci/tests/events/search" fi
A local config edit does not trigger CI by itself. Tell the user to add DD_API_KEY to their CI secrets (if not already set) and push a commit to run the workflow.
The app host is app.datadoghq.com for datadoghq.com, app.datadoghq.eu for datadoghq.eu, app.ddog-gov.com for ddog-gov.com, and the site itself for the others, for example us5.datadoghq.com. URL-encode @git.repository.id_v2:"<REPO>" in the query string. End with this summary:
Test Optimization: configured (<auto | manual | junitxml> for <framework> on <ci_provider>) | already configured | waiting for your choice
Changed: <files, or none>
Secret needed: DD_API_KEY in <CI provider secret store> (create key at https://APP_HOST/organization-settings/api-keys)
Evidence: <highest level reached>; not checked: <levels and the step left>
View: https://APP_HOST/ci/test/runs?query=%40git.repository.id_v2%3A%22<URL_ENCODED_REPO>%22Undo
Revert the CI workflow edits and any tracer devDependency added during this run.
Gotchas
| Mistake | Symptom | Fix |
Using datadog/test-optimization-action in GitHub Actions | Workflow fails because that action does not exist | Use datadog/test-visibility-github-action@v3 with api_key |
Omitting NODE_OPTIONS on the GitHub Actions test step for JS/TS | Action runs, but 0 test spans are collected | Add NODE_OPTIONS: -r ${{ env.DD_TRACE_PACKAGE }} (plus --import ${{ env.DD_TRACE_ESM_IMPORT }} for Vitest) |
Setting NODE_OPTIONS (-r dd-trace/ci/init) or RUBYOPT (-rdatadog/ci/auto_instrument) in the env: of a step that also installs dependencies | npm or bundle fails on a clean checkout, because the preloaded library is not installed yet | Export the variable on a line after the install command, or prefix only the test command |
Running tests in docker run without -e flags | Container runs tests without Datadog or CI git metadata | Forward Datadog, runtime, and CI provider env vars into the container; use manual setup for prebuilt images |
Writing $$DD_API_KEY in Buildkite env: | Buildkite escapes $$ to a literal string, causing 403 auth failure | Write $DD_API_KEY |
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

