dd-openai-onboarding
Skill185 starsChanged 38 days agoMITView on GitHub
Connects OpenAI to Datadog so the user can see organization-wide OpenAI API usage metrics and OpenAI costs in Cloud Cost Management (on by default). Guides creating the OpenAI admin key the integration needs (organization owners only), entered in a separate secure prompt where the agent never sees it, and verifies that usage data is actually arriving. Use when the user wants OpenAI API usage and cost data in Datadog, or asks to connect, configure, or repair the Datadog OpenAI integration. Not for tracing OpenAI calls from application code, which LLM Observability covers.
- Pipes a download into a shell
- Installs packages
In this repository43 other files it gives its agents.
- dd-account-setupSkill
- dd-agent-install-dockerSkill
- dd-agent-install-kubernetesSkill
- dd-agent-install-linuxSkill
- dd-agent-install-macosSkill
- dd-agent-install-windowsSkill
- dd-android-sdkSkill
- dd-anthropic-usage-and-costs-onboardingSkill
- dd-apmSkill
- dd-auditSkill
- dd-aws-ccmSkill
- dd-aws-integrationSkill
- dd-aws-log-forwardingSkill
- dd-aws-storage-managementSkill
- dd-azure-ccmSkill
- dd-azure-integrationSkill
- dd-azure-log-forwardingSkill
- dd-browser-sdkSkill
- dd-confluent-cloud-integrationSkill
- dd-dbm-rds-postgresSkill
- dd-docsSkill
- dd-gcp-ccmSkill
- dd-gcp-integrationSkill
- dd-gcp-log-forwardingSkill
- dd-github-copilot-onboardingSkill
- dd-github-costs-onboardingSkill
- dd-github-onboardingSkill
- dd-instrument-error-trackingSkill
- dd-instrument-llmoSkill
- dd-instrument-product-analyticsSkill
- dd-instrument-rumSkill
- dd-instrument-test-optimizationSkill
- dd-logsSkill
- dd-monitorsSkill
- dd-nebius-integrationSkill
- dd-oci-ccmSkill
- dd-oci-integrationSkill
- dd-orchestratorSkill
- dd-pagerduty-onboardingSkill
- dd-product-recommenderSkill
- dd-profiler-setupSkill
- dd-pupSkill
- dd-slack-integrationSkill
The file
Show its rules and text
We split this skill into the rules it gives an agent
17 rules. About 0 are common, 9 are distinctive, and 8 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈9
Distinctive · estimate
This author's own choices. Worth a look.
8
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 15
Use scripts/setup.py relative to this file.
- Very specific: line 32
Datadog for Government is not supported: If pup auth status reports a ddog-gov.com site, stop and tell the user to set up the integration …
- Very specific: line 95
OpenAI admin key: Only organization owners can create one, and the key is shown only once by OpenAI.
- Distinctive: line 106
Before gathering anything else, ask whether to set up the integration for the Datadog org the user is currently logged in to.
- Distinctive: line 130
Gather values: Use one interactive question form for values not already supplied:
- Very specific: line 143
Explain: Before opening anything, tell the user what they will need to do: create an OpenAI admin key (the browser opens on …
- Distinctive: line 149
Ask for approval: Use the interactive question tool once:
- Distinctive: line 152
Question: ask this, with each step on its own line:
- Distinctive: line 165
Run setup: After approval, run:
- Distinctive: line 183
Run the command with the longest command timeout your tool allows (at least 10 minutes).
- Very specific: line 202
Before asking for a key, the script lists existing OpenAI accounts.
- Very specific: line 237
Ask them to create an admin key at https://platform.openai.com/settings/organization/admin-keys (organization owners only) and rerun.
- Distinctive: line 266
If the key had already been entered, inspect the account list as above first.
- Very specific: line 270
After the setup script returns complete, confirm that data is arriving and then show the user what they can do with it, without blocking …
- Very specific: line 273
Start a background subagent with this task, using python instead of python3 on Windows.
- Distinctive: line 296
When the subagent reports, relay the result to the user.
- Distinctive: line 316
If no subagent tool is available, run the same command yourself and report the result the same way.
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Datadog OpenAI Integration
Use scripts/setup.py relative to this file. It owns browser handoff, hidden
key entry in a separate terminal, the OpenAI key check, Pup OAuth validation,
duplicate prevention, one-shot registration, and secret redaction. Do not
reproduce its API reads or writes.
The integration (openai) ingests organization-wide usage as openai.*
metrics and, when cost data ingestion is on, OpenAI costs into Cloud Cost
Management (CCM). Both are configured on one Datadog account: the OpenAI key is
its secret, and ccm_enabled and llm_observability_enabled are its settings,
so the script creates the account and turns on CCM in the same request.
Out of scope: tracing an application's own OpenAI calls. That is LLM
Observability instrumentation (the dd-instrument-llmo skill), not this
integration.
Phase 0: Preflight
Datadog for Government is not supported. If pup auth status reports a
ddog-gov.com site, stop and tell the user to set up the integration from the
Datadog app instead. The script refuses Government sites too.
Pup. Installed and authenticated. The setup script performs every Datadog
request through pup api, which owns authentication selection, OAuth
refresh, site, and organization selection, and it validates Pup itself before
touching anything:
pup auth statusauthenticated: true→ continue.pupis missing → ask before installing it:brew tap datadog-labs/pack && brew install datadog-labs/pack/pup(macOS or Linux with Homebrew),curl -fsSL https://raw.githubusercontent.com/DataDog/pup/main/install.sh | sh, ornpm install -g @datadog/pup(Windows or anywhere with npm); see https://github.com/DataDog/pup. Then runpup auth login.authenticated: false→ the setup script reportspup_not_authenticatedwith the exactpup auth logincommand. Ask before running it, and do not ask the user to create or export Datadog API or application keys.
Pup version. The setup script uses Pup v2 CLI flags (such as
--no-envelope); Pup 1.x fails with an opaque pup_request_failed. Check:
pup --versionA major version of 2 or later is required. If it is older, ask before
upgrading. When the environment allows it, upgrade with the official
installer or npm as above. Some workspaces block piping a remote script to a
shell (curl | sh), and /usr/local/bin may not be user-writable; there,
install from the release archive in a fresh empty directory instead (replace
2.4.3 and the platform suffix with the latest release):
VERSION=2.4.3
curl -fsSLO "https://github.com/DataDog/pup/releases/download/v${VERSION}/pup_${VERSION}_Linux_x86_64.tar.gz" &&
curl -fsSLO "https://github.com/DataDog/pup/releases/download/v${VERSION}/pup_${VERSION}_checksums.txt" &&
grep "pup_${VERSION}_Linux_x86_64.tar.gz$" "pup_${VERSION}_checksums.txt" | sha256sum -c - &&
tar -xzf "pup_${VERSION}_Linux_x86_64.tar.gz" &&
sudo install -m 0755 ./pup /usr/local/bin/pup &&
pup auth loginPython. The flow runs scripts/setup.py, so confirm python3 is on
PATH before starting:
command -v python3 || echo "MISSING python3"Desktop session. Key entry needs a local graphical terminal: Terminal.app
on macOS; gnome-terminal, konsole, xterm or x-terminal-emulator with DISPLAY
or WAYLAND_DISPLAY set on Linux; a console on Windows or WSL; or a new tmux
window when the agent session runs inside tmux, which also works on headless
hosts. If none is available, such as over SSH without tmux or in a container,
stop before approval and ask the user to run this from a machine with a
desktop session or inside tmux.
OpenAI admin key. Only organization owners can create one, and the key is
shown only once by OpenAI. A project key (sk-proj-...) connects but
collects no usage or cost data, so the script refuses it. If the user is not an
OpenAI organization owner, stop before approval: ask them to have an owner
create the admin key and paste it into the prompt, or run the setup themselves.
Cost data. Appears in Cloud Cost Management only if the Datadog org uses CCM.
Confirm the logged-in Datadog org
Before gathering anything else, ask whether to set up the integration for the
Datadog org the user is currently logged in to. Read the org's name from the
same pup session and site this run uses with pup api /api/v2/current_user
(the name of the orgs entry under included; the site is the one pup auth
status reported). Then ask once with the host's native selector:
Do you want to set up this integration for the Datadog org
<org name>on<site>?
- Yes → continue.
- No → stop. Have the user sign pup into the right org (see the pup step above), then ask again. Never continue on an org the user did not confirm.
Workflow
- [ ] Values gathered (name, cost data, LLM Observability, custom tags)
- [ ] User told about the admin key page and the separate terminal
- [ ] Approval received
- [ ] setup.py ran; status recorded
- [ ] Result handled per Results and recovery
- [ ] Background verification started and reportedGather values. Use one interactive question form for values not already supplied:
- Datadog integration display name: recommend a name that identifies
the OpenAI organization, such as
openai-production. - Cost data: whether to ingest OpenAI costs into Cloud Cost Management (on by default).
- LLM Observability: whether LLM Observability may also use this key to run evaluations (off by default; those calls are billed by OpenAI).
- Custom tags (optional):
key:valuetags to add to this account's OpenAI metrics and logs, such asteam:ml; no spaces or commas. Leave empty for none.
- Datadog integration display name: recommend a name that identifies
the OpenAI organization, such as
- Explain. Before opening anything, tell the user what they will need to
do: create an OpenAI admin key (the browser opens on
https://platform.openai.com/settings/organization/admin-keys) and paste it into a separate terminal window that the script opens. Explain the prerequisites in Phase 0: Preflight. Ask for approval. Use the interactive question tool once:
- Title:
Approve OpenAI integration setup Question: ask this, with each step on its own line:
May I run the interactive setup? It will:
- Open the OpenAI admin keys page.
- Ask you to paste an OpenAI admin key into a hidden prompt in a new terminal.
- Check the key with OpenAI.
- Register the named Datadog integration account, with cost data ingestion on.
The key is never visible to the agent, and will not be printed, passed as a command argument, stored in a file, or returned in output.
- Approve option:
Approve and open the secure key prompt - Cancel option:
Cancel without making changes
- Title:
Run setup. After approval, run:
python3 <SKILL_DIR>/scripts/setup.py --account-name "<DATADOG_INTEGRATION_NAME>"On Windows, use
pythoninstead ofpython3. If the user declined cost data, append--disable-ccm; if they opted into LLM Observability, append--enable-llm-observability; append--custom-tags <key:value>once per custom tag. Pup uses its default session, and picks the Datadog site fromDD_SITE(defaultdatadoghq.com): if the user's Pup session is on another site, run the command withDD_SITEset to it (e.g.datadoghq.eu); if they selected a named Pup organization session, also set theDD_ORGenvironment variable to that session's name. Set them the way the shell expects: aNAME=valueprefix on POSIX shells,$env:NAME="value";in PowerShell, orset "NAME=value" &&in cmd.exe. Do not detach the process or redirect its input.Run the command with the longest command timeout your tool allows (at least 10 minutes). The user may take up to 5 minutes to enter the key, and Datadog requests have no client-side timeout. Killing the command during registration leaves the outcome unknown.
- Hand off to the terminal. Tell the user to switch to the new terminal
when it appears — a tmux window named
openai-tokenwhen the session runs inside tmux. The script always opens a new graphical terminal (or tmux window) for key entry and fails instead of prompting in place. Never ask the user to paste the key into chat, and never pass it through a command argument, environment variable, shell pipe, redirected input, or file. - Handle the result per Results and recovery. If
statusis notcompleteoralready_configured, do not start verification. - Verify in the background per Getting the Most Out of Your Integration.
Before asking for a key, the script lists existing OpenAI accounts. One account
with the same display name returns already_configured without requesting a
key, and reports its ccm_enabled and llm_observability_enabled plus any
that differ from the request in settings_differ. Multiple
accounts with the same name fail closed. The script checks the key with OpenAI
before anything reaches Datadog, never POSTs twice, and keeps the key in memory
only.
Results and recovery
Report the script's status and warnings without exposing credentials. On
failure, the script prints status: error and an error.reason.
complete→ continue with Getting the Most Out of Your Integration.already_configured→ the account exists and was left unchanged. Ifsettings_differis present, tell the user which settings differ from what they asked for (currentvsrequested). If they wanted cost data andaccount.ccm_enabledisfalse, ask whether to turn it on with--turn-on-ccm; if yes, run:python3 <SKILL_DIR>/scripts/setup.py --account-name "<DATADOG_INTEGRATION_NAME>" --turn-on-ccmIt needs no key: it changes only
ccm_enabledon that account, sending back every other setting (LLM Observability, custom tags) as it was, and confirms the change (ccm_enabled, orccm_already_enabled). If the user is repairing an account whose key was revoked or replaced, the script does not ask for a new key: send them to the Configuration tab of the OpenAI integration tile in Datadog to update the key on that account.pup_cli_outdated→ the installed Pup is older than v2 and cannot run this setup. Ask before upgrading it (see the Pup version preflight above), then rerun once the upgrade is done.admin_key_required→ the user pasted a project key; nothing was written. Ask them to create an admin key athttps://platform.openai.com/settings/organization/admin-keys(organization owners only) and rerun.invalid_openai_keyoropenai_key_rejected→ nothing was written to Datadog. Ask the user to create a fresh admin key and rerun.openai_check_failed→ OpenAI could not be reached, or answered with a temporary error, so the key was neither accepted nor rejected; nothing was written. Rerun shortly (and make sure the network allowsapi.openai.com).credential_entry_timeout,terminal_helper_not_connectedorcredential_entry_cancelled→ nothing was written. Ask whether the user is ready, then rerun the same command.terminal_launch_failedorinteractive_tty_required→ nothing was written. Follow the Desktop session item in Phase 0.datadog_create_failed→ Datadog rejected the request and nothing was written. Reportdetails.pup_error.http_status, and fix the cause before rerunning.duplicate_datadog_accounts→ several accounts share the name. Ask the user which one to keep, and do not create another.ambiguous_datadog_post→ do not rerun. Inspect the list withpup api --no-envelope --output json --method GET /api/v2/web-integrations/openai/accounts, which is read-only and the one exception to the rule at the top. If the name is listed, the account was created. If not, do not rerun: the original write may still land or become visible later. Send the user to the OpenAI integration tile in Datadog to check the account.datadog_account_not_visible→ the POST succeeded, but the account is not listed yet. Wait a minute and inspect the list as above. Do not rerun.- Pup authentication failures → run
pup auth refresh; if that fails, runpup auth loginwith the reported missing scopes. - Any other reason → report
error.messageand fix the cause before rerunning. If the key had already been entered, inspect the account list as above first.
Getting the Most Out of Your Integration
After the setup script returns complete, confirm that data is arriving and
then show the user what they can do with it, without blocking the conversation:
Start a background subagent with this task, using
pythoninstead ofpython3on Windows. The subagent runs in a fresh shell, so give it the sameDD_SITEandDD_ORGsettings setup used; otherwise it falls back to the default site and could check the wrong one:Run
python3 <SKILL_DIR>/scripts/verify_data.py --account-name "<DATADOG_INTEGRATION_NAME>"with the sameDD_SITEandDD_ORGsetup used (omit any it did not set), passed through the execution tool's environment settings or set in the same shell invocation: POSIXDD_SITE=<site> DD_ORG=<session> python3 ...; PowerShell$env:DD_SITE="<site>"; $env:DD_ORG="<session>"; python ...; cmd.exeset "DD_SITE=<site>" && set "DD_ORG=<session>" && python .... Run it with a command timeout of at least 16 minutes. It polls Datadog for up to 15 minutes and prints JSON. Do not retry it or change anything. Report back itsstatus(succeeded,failed, orerror),checks,links,ccm_enabled,message, and anyerrorobject.Tell the user, in these words:
"Your OpenAI integration is set up! I'm now checking in the background that your usage metrics are arriving in Datadog. This can take up to 15 minutes, and I'll let you know as soon as they show up. Feel free to keep working in the meantime."
- When the subagent reports, relay the result to the user.
succeeded: celebrate. Give a short, enthusiastic headline that starts with
🎉, such as "🎉 Your OpenAI usage metrics are now live in Datadog!", the
links.metrics link (an openai.* metric from this account in the Metrics
Explorer; usage points are stamped with when the usage happened, so tell the
user to widen the time range if the graph looks empty), and two or three
one-sentence next steps, for example: chart token usage by model and
project_name; set a monitor on token usage to catch spikes; and, when
ccm_enabled is true, that OpenAI costs will appear in Cloud Cost Management
(links.costs) within about 24 hours. Only describe what the script confirmed.
failed: stay encouraging and relay the script's message. Metrics only
exist for API usage, OpenAI reports usage with a delay, and the first data can
take longer to appear; suggest checking openai.* metrics later before treating it as
broken.
error: report the reason. For datadog_read_not_permitted, ask before
running the reported pup auth login --extra-scopes ... command.
If no subagent tool is available, run the same command yourself and report the result the same way.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

