anomalydetection-run-gensim-scenario
Skill3.8k starsChanged 11 days agoApache-2.0View on GitHub
Record gensim episodes with the anomaly-detection recorder on either an AWS EKS cluster (--backend eks) or a remote Linux VM (--backend vm) and fetch validated Parquet recordings for the Agent Anomaly Detection team's testbench evals. Use for "record a gensim scenario", "record a gensim scenario on a VM", "record gensim episodes on EKS", "set up the gensim VM", or "record a batch of episodes and fetch the parquet". Not a general-purpose gensim or Agent benchmarking workflow.
- Reads credentials
Adopted by
Steady: 0 new projects took it in the last 4 weeks, none the 4 weeks before.thin data
Who's talking about DataDog/datadog-agent
- aws/aws-sdk-go-v2“[SOLUTION IN THREAD] "not found, ResolveEndpointV2" service modules before 11/15/23 are incompatible …”GitHub · Nov 16, 2023 · 45 reactions
- micrometer-metrics/micrometer“Unix Domain Socket support (StatsD)”GitHub · Aug 17, 2018 · 22 reactions
- golang/go“proposal: x/tools/go/analysis/passes/appends: check for incorrect slice length initialization ”GitHub · Oct 14, 2024 · 18 reactions
- airbytehq/airbyte“[STORAGE-S3] Add Support for Custom S3 Endpoint”GitHub · Aug 16, 2024 · 14 reactions
- prometheus/blackbox_exporter“List of metrics provided in a human-readable format”GitHub · Dec 15, 2021 · 13 reactions
The biggest 5 of 54 mentions linking to the repository that our crawl found. Each line is their own headline.
See all 54
Talked about on GitHub and Stack Overflow (54 times).
On GitHub · 46
- Nov 16, 2023[SOLUTION IN THREAD] "not found, ResolveEndpointV2" service modules before 11/15/23 are incompatible against newer root modules — aws/aws-sdk-go-v245 reactions · 33 comments
- Aug 17, 2018Unix Domain Socket support (StatsD) — micrometer-metrics/micrometer22 reactions · 23 comments
- Oct 14, 2024proposal: x/tools/go/analysis/passes/appends: check for incorrect slice length initialization — golang/go18 reactions · 28 comments
- Aug 16, 2024[STORAGE-S3] Add Support for Custom S3 Endpoint — airbytehq/airbyte14 reactions · 21 comments
- Dec 15, 2021List of metrics provided in a human-readable format — prometheus/blackbox_exporter13 reactions · 5 comments
41 more on GitHub
- Feb 26, 2026cmd/go: decide and document official recommendation for typical 'go' version in go.mod for open source libraries — golang/go9 reactions · 13 comments
- Dec 26, 2024[BUG] go.mod tidy fails due to unresolved module in dd-trace-go — DataDog/dd-trace-go9 reactions · 8 comments
- Oct 31, 2018[disk] Blacklist certain partitions by default — DataDog/integrations-core9 reactions · 6 comments
- Mar 27, 2024datadog-agent: panic: must be called from a bundle — NixOS/nixpkgs8 reactions · 5 comments
- Feb 3, 2026cmd/go: go list computes build info for test packages when not needed — golang/go7 reactions · 4 comments
- Jan 28, 2022v1.35.0 breaks any program trying to use a `-v` flag — DataDog/dd-trace-go6 reactions · 5 comments
- May 10, 2021Validator reports false duplicates — dependabot/dependabot-core6 reactions · 2 comments
- Feb 17, 2025[receiver/sqlserver] Support query-level log collection [Top Query] — open-telemetry/opentelemetry-collector-contrib4 reactions · 11 comments
- Jul 20, 2026chore(ci): update Datadog Agent version to 7.81.1 — DataDog/saluki3 reactions · 3 comments
- Jun 20, 2025datadog-agent: 7.56.2 -> 7.70.2 — NixOS/nixpkgs3 reactions · 3 comments
- May 17, 2024Service V2 collector — prometheus-community/windows_exporter3 reactions
- Nov 13, 2026Publish signed images from registry.ddbuild.io — DataDog/datadog-operator2 reactions · 4 comments
- Nov 13, 2026[Backport v1.32] Remove Autopilot protocol classification workaround — DataDog/datadog-operator2 reactions · 2 comments
- Nov 12, 2026Update dd-pkg to 0.10.0 — DataDog/datadog-agent-buildimages2 reactions · 1 comment
- Nov 11, 2026feat(openfeature): send DD-Service on agentless configuration requests — DataDog/dd-trace-go2 reactions · 3 comments
- Nov 9, 2026chore(ci): update Datadog Agent version to 7.85.2 — DataDog/saluki2 reactions · 4 comments
- Nov 6, 2026chore(ci): render SMP regression reports locally — vectordotdev/vector2 reactions · 4 comments
- Nov 4, 2026CONTINT-5659: adds kubernetes_state.pod.unready metric to metadata.csv — DataDog/integrations-core2 reactions · 3 comments
- Nov 2, 2026[Backport v1.32] Update default Agent to `7.85.0` — DataDog/datadog-operator2 reactions · 2 comments
- Nov 2, 2026Update default Agent to `7.85.0` — DataDog/datadog-operator2 reactions · 2 comments
- Oct 26, 2026chore(ci): update Datadog Agent version to 7.84.6 — DataDog/saluki2 reactions · 3 comments
- Oct 21, 2026feat(nodeagent): add DD_POD_NAME as env var — DataDog/datadog-operator2 reactions · 4 comments
- Oct 19, 2026chore(ci): update Datadog Agent version to 7.84.3 — DataDog/saluki2 reactions · 3 comments
- Oct 12, 2026chore(ci): update Datadog Agent version to 7.84.2 — DataDog/saluki2 reactions · 3 comments
- Oct 9, 2026Revert Python version upgrade on 7.84.x — DataDog/integrations-core2 reactions · 6 comments
- Oct 5, 2026chore(ci): update Datadog Agent version to 7.84.1 — DataDog/saluki2 reactions · 2 comments
- Sep 29, 2026Install DotSlash in the standard build images — DataDog/datadog-agent-buildimages2 reactions · 2 comments
- Sep 28, 2026chore(ci): update Datadog Agent version to 7.83.3 — DataDog/saluki2 reactions · 2 comments
- Sep 28, 2026chore(deps): bump the docker-compose-dev group across 2 directories with 14 updates — DataDog/dd-trace-go2 reactions · 2 comments
- Sep 28, 2026Update go dependencies — DataDog/datadog-operator2 reactions · 3 comments
- Sep 21, 2026chore(ci): update Datadog Agent version to 7.83.2 — DataDog/saluki2 reactions · 2 comments
- Sep 15, 2026fix(datadog): configure Core Agent for PAR split mode — DataDog/helm-charts2 reactions
- Sep 4, 2026feat(orchestrator): grant KubeAI collection RBAC — DataDog/datadog-operator2 reactions · 1 comment
- Sep 3, 2026chore(ci): update Datadog Agent version to 7.83.0 — DataDog/saluki2 reactions · 2 comments
- Sep 3, 2026fix(metrics): normalize metric names prior to filtering — DataDog/saluki2 reactions · 4 comments
- Aug 26, 2026chore(ci): update Datadog Agent version to 7.82.3 — DataDog/saluki2 reactions · 2 comments
- Aug 24, 2026chore(ci): update Datadog Agent version to 7.82.2 — DataDog/saluki2 reactions · 2 comments
- Aug 21, 2026Revert "Revert "Make sure we keep the testing params when extra params does not override minor version"" — DataDog/agent-linux-install-scriptsays it tested it2 reactions · 4 comments
- Aug 18, 2026feat(datadog-agent-commons): add minimal IPC TLS feature — DataDog/saluki2 reactions · 2 comments
- Jul 29, 2026[ACIX-1887] Remove legacy devcontainer image — DataDog/datadog-agent-buildimages2 reactions · 1 comment
- Oct 5, 2024Update ecs_fargate docs to note DD_DOGSTATSD_TAG_CARDINALITY setting — DataDog/integrations-core2 reactions · 2 comments
On Stack Overflow · 8
- Mar 13, 2020How do you pass file_system_blacklist arg to Datadog Docker Agent run command?3 votes
- Dec 11, 2019Containers - Failed to get host IPs3 votes
- May 13, 2020Is there a serverless kubernetes datadog agent?2 votes
- Jun 29, 2018How to set docker labels on ECS agent container?2 votes · 1 answer
- Apr 28, 2022Issues with setting up Autodiscovery Auto-Configuration for Kubernetes integration in Datadog1 vote
3 more on Stack Overflow
- Nov 4, 2020How to configure hostname in ECS Fargate task definition1 vote
- Jan 26, 2023datadog local check setup same as check_mk local check for monitoring0 votes
- Mar 31, 2021How does the serverless datadog forwarder encrypt/encode their logs?0 votes
In this repository44 other files it gives its agents.
- AGENTS.mdAGENTS.md
- bazel/AGENTS.mdAGENTS.md
- omnibus/AGENTS.mdAGENTS.md
- packages/AGENTS.mdAGENTS.md
- tasks/AGENTS.mdAGENTS.md
- CLAUDE.mdCLAUDE.md
- .cursor/rules/agent_corecheck.mdcCursor rule
- .cursor/rules/bazel-buildifier.mdcCursor rule
- .cursor/rules/e2e_tests.mdcCursor rule
- .cursor/rules/general-guidelines.mdcCursor rule
- .cursor/rules/system_probe_modules.mdcCursor rule
- agent-supply-chain-newsletterSkill
- alliumSkill
- auto-jiraSkill
- create-componentSkill
- create-config-fieldSkill
- create-core-checkSkill
- create-epic-recapSkill
- create-go-moduleSkill
- create-invoke-taskSkill
- create-prSkill
- create-release-noteSkill
- create-runtime-settingSkill
- create-status-providerSkill
- create-subcommandSkill
- cws-btfhub-syncSkill
- cws-iouring-coverageSkill
- describe-functional-changesSkill
- e2e-auditSkill
- explain-lading-configSkill
- follow-prSkill
- gpu-live-metric-validationSkill
- handle-pr-ci-failureSkill
- injector-devSkill
- locate-config-settingSkill
- quality-gate-size-analysisSkill
- review-pr-commentsSkill
- run-e2eSkill
- run-jiraSkill
- run-windows-e2eSkill
- triage-ci-failureSkill
- update-3rd-party-libsSkill
- update-otel-depsSkill
- write-e2eSkill
The file
We split this skill into the rules it gives an agent
10 rules. About 0 are common, 3 are distinctive, and 7 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈3
Distinctive · estimate
This author's own choices. Worth a look.
7
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 20
Run the bundled helpers one at a time from the datadog-agent worktree and check their exit status (avoid chaining cd <dir> && ..., as user …
- Very specific: line 33
Ask for missing inputs in a single round trip before changing resources (AskUserQuestion requires >= 2 options per question):
- Very specific: line 63
Use at least 4 vCPU / 16 GiB and about 20 GB free disk.
- Very specific: line 78
Create or reuse the gadget-dev kind cluster (assets/kind.yaml uses one worker so signals stay in one pod) and deploy the recorder …
- Very specific: line 86
If gadget-dev already exists on the authorized disposable VM and its nodes are Ready, run bash "$SKILL/scripts/vm.sh" reuse-cluster …
- Very specific: line 123
When launching a batch from a shell loop, stagger the starts (e.g.
- Distinctive: line 203
After all episodes in the batch have been uploaded, fetched, and validated:
- Very specific: line 205
Clean
gensim-episodesgenerated files: run git -C <gensim-repo> status on the repo used for submit (the clean clone, or the user's … - Distinctive: line 215
Use this final-response template per episode:
- Distinctive: line 244
Read only for a failed or unfamiliar step:
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Record gensim episodes (VM or EKS)
Owner: Agent Anomaly Detection (@DataDog/agent-anomaly-detection).
Accept --backend vm|eks. If the user does not specify a backend, ask them to choose between eks and vm:
eksis for benchmark recordings on a disposable AWS sandbox stack and automatically uploads the recorded archive to S3.vmis for interactive debugging or lightweight single-node experiments.
Run the bundled helpers one at a time from the datadog-agent worktree and check their exit status (avoid chaining
cd <dir> && ..., as user shell cd hooks can return non-zero in non-interactive shells; use GENSIM_REPO_PATH or
git -C <repo> instead). They do not automatically tear down infrastructure so you can reuse a healthy VM or EKS
cluster across a batch. Do not claim a live AWS or VM run unless the user explicitly authorized one and you actually
performed it.
1. Collect inputs, authorization, and backend choice
| Backend | When to choose | Execution model |
eks | Benchmark recordings and batches on a disposable AWS sandbox stack; automatically uploads recordings to S3 | dda inv aws.eks.gensim.{submit,status,fetch,stop-all,destroy} |
vm | Interactive debugging, single-worker kind inspection, quick shortened trials | SSH helpers in scripts/vm.sh, scripts/scenario.sh, scripts/fetch.sh |
Ask for missing inputs in a single round trip before changing resources (AskUserQuestion requires >= 2 options per
question):
- Backend:
eks(benchmark recordings on a disposable AWS sandbox stack; uploads to S3) orvm(interactive debugging) - Recorder image tag:
datadog/agent-dev:<tag>fromdev_branch-recorder-full - Target & S3 bucket: EKS stack (default
gensim-eks) + S3 bucket (defaultqbranch-gensim-recordings), or VMuser@host/ stack - Credentials env file: local file defining
DD_API_KEYandDD_APP_KEYfor dddev - Authorization & clean clone: live-run approval, plus approval to clone
gensim-episodesto/tmp/gensim-episodes-cleanif the local checkout is dirty
| Input | Requirement |
| Backend & target | Ask if not provided: --backend eks (benchmark recordings on a disposable AWS sandbox stack, automatically uploads to S3; authorized EKS stack name, default gensim-eks, plus S3 bucket) or --backend vm (disposable user@host or authorized new AWS VM) |
| Scenario | Directory in the gensim-episodes repo containing play-episode.sh and chart/; episode from episodes/*.yaml. A path under observer-scenarios* is an old recording output (episode.json + parquet/); read its episode.json to map back to <episode-dir>:<scenario> in gensim-episodes |
| Credentials | For vm, local env file defining DD_API_KEY and DD_APP_KEY for dddev; for eks, configured e2e AWS sandbox profile and dddev keys. Do not read, echo, or print secret values |
| Recorder image | datadog/agent-dev:<tag> from CI job dev_branch-recorder-full (confirm pipeline and amd64 platform). If no tag is known, point the user at the CI Visibility Explorer (ci_level:job env:prod @git.repository.id:gitlab.ddbuild.io/DataDog/datadog-agent @ci.pipeline.name:DataDog/datadog-agent @ci.provider.instance:gitlab-ci @ci.job.name:dev_branch-recorder-full @ci.status:success): https://app.datadoghq.com/ci/ci-cd/explorer/executions?query=ci_level%3Ajob%20env%3Aprod%20%40git.repository.id%3Agitlab.ddbuild.io%2FDataDog%2Fdatadog-agent%20%40ci.pipeline.name%3ADataDog%2Fdatadog-agent%20%40ci.provider.instance%3Agitlab-ci%20%40ci.job.name%3Adev_branch-recorder-full%20%40ci.status%3Asuccess&agg_m=count&agg_m_source=base&agg_t=count&ci_cd_explorer_tab=pipelines&fromUser=true&refresh_mode=sliding&paused=false |
| Run settings | FLUSH (default 1m, FLUSH_SECONDS=60), RETENTION (default 300m), RECORDING_ONLY (true/false, default true for recording), RUN_MODE (full or shortened on VM), or EKS_MODE (record-parquet, live-anomaly-detection, live-and-record) |
| Output | Any fresh nonexistent local absolute path per episode (e.g. /tmp/scenarios/kafka, /tmp/vm-scenarios/<episode>, or /tmp/eks-scenarios/<episode>); never overwrite an existing directory. --episode/--scenario on fetch still use the canonical gensim-episodes names |
| Batch / teardown | Episodes to run and whether to keep the EKS stack for reuse; on EKS the default is to destroy each stack once its recording is fetched and validated, retain it only if the user asks to reuse it |
Retention must exceed the episode duration plus the final flush wait (and the full batch on VM). Do not operate on
production machines or clusters: these steps deploy workloads on disposable AWS sandbox stacks or VMs and create dddev
monitors/events. Confirm disposable host/stack authorization and dddev credentials before mutating resources. The VM
helpers require --disposable-vm to record that authorization; the flag is not a substitute for obtaining it.
Set SKILL to this skill's absolute directory. Local prerequisites before execution:
- Shared (validation): Bash, Python 3, and PyArrow (
uv run --no-project --with pyarrow ...). - VM (
--backend vm): localssh,scp, andrsyncwith access to the disposable VM (plusdda inv e2e.setupwhen provisioning a new VM). - EKS (
--backend eks):dda, configured e2e AWS sandbox profile (aws-vault/dda inv e2e.setup),kubectl,helm, and read/write access to the selected S3 bucket.
2A. VM happy path (--backend vm)
Use at least 4 vCPU / 16 GiB and about 20 GB free disk. Set VM to the confirmed user@host (and SSH_PORT if
non-default), plus FLUSH (default 1m), FLUSH_SECONDS (matching FLUSH in seconds, default 60), RETENTION
(default 300m), RECORDING_ONLY (default true), and RUN_MODE (full or shortened).
- Provision or probe the VM (run
dda inv e2e.setup --team=<team>first if AWS e2e setup is missing):
bash "$SKILL/scripts/vm.sh" create-vm --disposable-vm "$STACK" t3.2xlarge
bash "$SKILL/scripts/vm.sh" probe "$VM"Require x86_64, passwordless sudo, Docker without sudo, kind, kubectl, Helm, Python 3 with PyYAML, and inotify limits of
at least 524288 watches / 1024 instances. Load the bootstrap reference only when probe
reports missing tools, Docker permissions, or low inotify limits.
- Create or reuse the
gadget-devkind cluster (assets/kind.yamluses one worker so signals stay in one pod) and deploy the recorder (assets/recorder-values.yaml):
bash "$SKILL/scripts/vm.sh" create-cluster --disposable-vm "$VM"
bash "$SKILL/scripts/vm.sh" deploy-recorder --disposable-vm "$VM" "$ENV_FILE" "$TAG" "$CLUSTER_NAME" "$FLUSH" "$RETENTION" "$RECORDING_ONLY"If gadget-dev already exists on the authorized disposable VM and its nodes are Ready, run
bash "$SKILL/scripts/vm.sh" reuse-cluster --disposable-vm "$VM" instead of create-cluster. Helpers use
~/gensim/kubeconfig and require kind-gadget-dev.
Expected outcome: the recorder runs in the intended cluster with the configured values
and a mode-600 env file. Credentials must not appear in command arguments or output. Confirm one Ready recorder Agent
pod with zero restarts and metrics/log Parquet files after a flush interval.
- Prepare the scenario, run the episode, and fetch/validate into a fresh local directory:
bash "$SKILL/scripts/scenario.sh" prepare --disposable-vm "$VM" "$SCENARIO_ID" "$SCENARIO_DIR"
bash "$SKILL/scripts/scenario.sh" run --disposable-vm "$VM" "$SCENARIO_ID" "$EPISODE" "$RELEASE" "$RUN_MODE" qbranch_gensim
uv run --no-project --with pyarrow bash "$SKILL/scripts/fetch.sh" "$VM" "$SCENARIO_ID" "$EPISODE" "$OUT" "$FLUSH_SECONDS"Expected outcome from prepare: a fresh scenario copy ready to run on the VM, without a competing Agent or stale
results. Reuse the prepared copy across episodes of the same scenario; uninstall the previous scenario release before
switching scenarios.
Expected outcome from run: results/<episode>-1.json, results/<episode>.log, and Agent status snapshots for
recording validation. Retain the tool session and wait for completion; episodes survive SSH disconnections, so inspect
the log and running workloads before retrying. Use shortened instead of full only when requested (recopy before a
full run).
2B. EKS happy path (--backend eks)
EKS orchestrates episodes in-cluster via job/gensim-orchestrator, collects from every recorder Agent pod after waiting
one full flush interval past end_time, and uploads a zip archive to S3. Load the EKS reference
only when:
- the local
gensim-episodescheckout is dirty ore2e.setupfails on a missing local SSH keypair, - a previous Pulumi submit was interrupted or the cluster reports busy,
- multi-pod collection/upload fails or local validation needs to be re-run against an already downloaded
$OUT, or - inspecting VM vs EKS Helm differences or running the live acceptance checklist.
- Authenticate to AWS once, before any other step (especially before launching several
submitcalls in parallel). If the SSO session is expired, everydda/aws-vaultcall opens its own browser login and hangs. Ask the user to run! aws sso login --profile sso-agent-sandbox-account-admin-8h, then verify withaws-vault exec sso-agent-sandbox-account-admin-8h -- aws sts get-caller-identity --query Account --output text. When launching a batch from a shell loop, stagger the starts (e.g.sleep 15) and redirect eachddacall's stdin from/dev/null, otherwise it consumes the loop's input and only the first stack launches. - Submit the episode or batch in the background (first-time cluster provisioning takes >10 minutes; never run under a
short foreground timeout).
--episodesis<dir under postmortems/ or synthetics/>:<episodes/*.yaml name without extension>, e.g.703_shopify_bfcm_incident:kafka-partition-saturation. Source"$ENV_FILE"(set -a; . "$ENV_FILE"; set +a) in the same shell without printing secrets, and append--recording-only=falsewhenRECORDING_ONLY=false:
GENSIM_REPO_PATH="${GENSIM_REPO_PATH:-}" dda inv aws.eks.gensim.submit \
--stack-name="$STACK" \
--image="docker.io/datadog/agent-dev:$TAG" \
--episodes="$SCENARIO_ID:$EPISODE" \
--s3-bucket="$S3_BUCKET" \
--mode="${EKS_MODE:-record-parquet}" \
--flush-interval="$FLUSH" \
--retention="$RETENTION" > /tmp/gensim-submit-"$STACK".log 2>&1 # run in background- Monitor in-cluster execution until
collection=succeededandupload=succeeded:
dda inv aws.eks.gensim.status --stack-name="$STACK"status separates monitor outcome (monitor), multi-pod Parquet collection (collection), S3 upload (upload), and
local validation (validation). An uploaded S3 artifact (upload=succeeded, validation=pending) is not yet a
validated benchmark recording.
- Fetch the uploaded artifact into a fresh nonexistent local directory (
$OUT, e.g./tmp/scenarios/kafkaor/tmp/eks-scenarios/<episode>) and run the shared PyArrow validator (fetchautomatically invokesuv run --no-project --with pyarrow python3when the activepython3lackspyarrow, and infers the flush interval fromrecording-manifest.jsonunless--flush-interval="$FLUSH"is passed):
dda inv aws.eks.gensim.fetch \
--stack-name="$STACK" \
--run-id="$RUN_ID" \
--episode="$SCENARIO_ID" \
--scenario="$EPISODE" \
--s3-bucket="$S3_BUCKET" \
--flush-interval="$FLUSH" \
--output-dir="$OUT"3. Shared validation contract and testbench replay
Both scripts/fetch.sh (VM) and dda inv aws.eks.gensim.fetch (EKS) run
scripts/recording.py validate:
- Fetch helpers create
$OUTexclusively so existing data is never overwritten;recording.py validatecan be re-run idempotently on$OUTbefore the archive is cleaned up. - Verifies safe archive paths, Agent pod readiness and zero restarts (
agent-before.json/agent-after.json), and manifest SHA-256 checksums. Legacy VM recordings without before/after snapshots remain fetchable withagent_history_verified: false. - Preserves whole-file trimming: retains
.parquetfiles whoseTimemin/max overlapsstart_time..end_timeacross seconds, milliseconds, microseconds, and nanoseconds. - Requires nonempty in-window metrics (
observer-metrics-*.parquet) and logs (observer-logs-*.parquet) from every expected recorder pod. Sparse timestamps producecoverage_warnings, not automatic failure; inspect warnings before claiming complete coverage. Monitor"success": falseinepisode.jsondoes not invalidate a recording. - Log filtering (VM and EKS): the observer's defaults drop logs below
warnand rate-limitinfologs (anomaly_detection.logs.{containers,internal,kubelet}.min_severity=warnand.max_rate_medium_priority=100, defined inpkg/config/setup/all_settings.go). Left alone, a recording keeps only thewarn/errorrows and loses the workloads'infoaccess logs, while metrics look normal and validation still passes (at most acoverage_warningsentry).assets/recorder-values.yaml(VM) andagent-values.yaml.tmpl(EKS) therefore setDD_ANOMALY_DETECTION_LOGS_{CONTAINERS,INTERNAL,KUBELET}_MIN_SEVERITY=infoand..._MAX_RATE_{MEDIUM,LOW}_PRIORITY=-1. Keep both files in sync, and if a custom values file is used, keep these overrides. Sanity check after fetching: countStatusvalues inparquet/observer-logs-*.parquet; onlyerror/warnrows from workloads that normally log atinfomean the filter is active. - Writes
episode.json,validation.json,recording-manifest.json, andparquet/(preserving untrimmed sidecars such asadvances.jsonlanddetect_digests.jsonl). On failure, retains partial output in$OUTfor diagnosis.
Replay only after local validation succeeds (usable: true in validation.json and states.validation: "passed" in
recording-manifest.json):
dda inv anomalydetection.launch-testbench --scenarios-dir "$(dirname "$OUT")" --build4. Teardown and report
After all episodes in the batch have been uploaded, fetched, and validated:
- Clean
gensim-episodesgenerated files: rungit -C <gensim-repo> statuson the repo used forsubmit(the clean clone, or the user's checkout if used directly) to ensure no generated files (results/*.json, deleted chart templates) remain. In the clean clone, restore generated changes withgit -C <clone> checkout -- <generated-files>. - VM: uninstall selected Helm releases (
KUBECONFIG="$HOME/gensim/kubeconfig" helm --kube-context kind-gadget-dev --namespace default uninstall <release>), delete kind only when requested (kind delete cluster --name gadget-dev), and destroy the VM only when requested (dda inv aws.destroy-vm --stack-name=<stack>). - EKS: destroy the stack by default (
dda inv aws.eks.gensim.destroy --stack-name=<stack>, ~13m) — including failed or abandoned stacks — and confirm resources are deleted; usedda inv aws.eks.gensim.stop-all --stack-name=<stack>only when the user asks to keep it for reuse. Each stack attaches agensim-ecr-pushinline policy to the sharedec2InstanceRole(10,240 B IAM cap), so un-destroyed stacks eventually block all new stacks withLimitExceeded(never deleteec2InstanceRolepolicies manually).
Use this final-response template per episode:
| Field | Report |
| Episode | Backend (vm or eks), scenario id, episode, run ID, mode |
| Target / image | VM SSH target or EKS stack/namespace, recorder pod count, exact image tag and observed image IDs |
| Output / artifact | Local $OUT path (and S3 artifact key on EKS), kept metrics/log file and row counts, removed file count |
| Window | Start/end, flush interval, observed coverage from validation.json |
| Quality / states | episode_outcome, collection, upload, validation (passed/failed), sample gaps, restarts; distinguish unknown from verified |
| Teardown | Releases/workloads/cluster/VM removed vs retained (for EKS, whether destroy completed); gensim-episodes repo left clean |
Example 1 (VM): "Record a shortened kafka-partition-saturation episode on ubuntu@10.0.1.23 (--backend vm) with
datadog/agent-dev:my-branch-recorder-full, use ~/dddev.env, save under /tmp/vm-scenarios, and keep the VM."
Result: backend=vm, reused ubuntu@10.0.1.23 (1 recorder pod), output /tmp/vm-scenarios/kafka-partition-saturation,
4 metrics files (1000 rows) and 4 log files (200 rows) retained, 2 out-of-window files removed, validation=passed,
agent_history_verified=true, VM and kind cluster retained.
Example 2 (EKS): "Record authcore-pgbouncer:pool-saturation with --backend eks on stack gensim-eks using
docker.io/datadog/agent-dev:my-branch-recorder-full, upload to qbranch-gensim-recordings, fetch to
/tmp/eks-scenarios/pool-saturation, and keep the cluster."
Result: backend=eks, run_id=eval-20261015T120000Z-a1b2c3d-9f8e7d, 2 recorder pods on gensim-eks, S3 artifact
gensim-results-eval-20261015T120000Z-a1b2c3d-9f8e7d-authcore-pgbouncer-pool-saturation.zip (collection=succeeded,
upload=succeeded), fetched to /tmp/eks-scenarios/pool-saturation with 8 metrics files and 8 log files retained across
both pods, SHA-256 checksums verified, validation=passed, gensim-episodes repo verified clean, EKS cluster retained
for the next batch.
Fill these fields from real observations, not the examples.
Troubleshooting references
Read only for a failed or unfamiliar step:
- Recorder configuration: https://datadoghq.atlassian.net/wiki/spaces/agent/pages/7272235685/Using+the+recorder+to+record+gensim+staging+scenarios
- Lima/kind setup: https://datadoghq.atlassian.net/wiki/spaces/agent/pages/6238700466/Gensim+Hands+On
- VM Ubuntu bootstrap — load only when
vm.sh probefails on missing tools or sysctl limits. - EKS multi-pod recording & live acceptance checklist — load only for EKS recovery, troubleshooting, or live acceptance handoff.
- Docker permission denied after bootstrap: close the multiplexed SSH connection and reconnect to pick up group membership.
- Exec format errors: confirm an amd64 image for the x86_64 VM.
- Docker 29 image-load errors: use a single-platform image archive instead of
kind load docker-image. - Multiple recorder pods (VM only, where
kinduses one worker) or restarting containers (VM or EKS): stop, fix topology/resources, and start a new recording.
To validate helper changes without provisioning resources, run
uv run --no-project --with pyarrow --with pyyaml bash "$SKILL/scripts/test.sh" and ShellCheck on scripts/*.sh.
The tests mock SSH/SCP/dda/kubectl/helm/aws and use generated Parquet fixtures; the user performs live AWS testing.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

