agentleFS
open-thoughtsopen-thoughts/OpenThoughts-Agent/.agents/skills/datagen-reduce-dataset-snapshots/SKILL.md

datagen-reduce-dataset-snapshots

Skill309 starsChanged 4 months agoApache-2.0View on GitHub

Reduce the Daytona snapshot (unique-environment) count of a Harbor task dataset below the cap by editing its patcher's environment-build logic, without breaking task quality. Use when a dataset is flagged "SnapshotCapExceeded" / "N unique environments" with N over the threshold (target < 10), e.g. swegym at 906. The loop: set snapshot+oracle thresholds → count → diagnose the env-hash driver → group/unionize Dockerfiles in the patcher → regenerate + upload → re-count → TWO-TIER quality gate (harbor infra smoke + `--stages oracle` gold-patch yield) → navigate the snapshot↔fidelity tradeoff within a bounded iteration budget → record. Runs LOCALLY on the Mac + Daytona (no GPU).

  • Installs packages

The file

We split this skill into the rules it gives an agent

5 rules. About 0 are common, 0 are distinctive, and 5 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈0

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 5

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    Keep a get_specs(repo, version) map so each repo still gets its correct Python

    line 91
  2. Very specific:

    Regenerate the full dataset + upload to a NEW repo: Never overwrite the validated artifact.

    line 95
  3. Very specific:

    Read infra_rate. Ignore this script's solve_rate — it's the *agent's* task-solve rate, low by design, NOT a measure of task well-formedness.

    line 112
  4. Very specific:

    If oracle pass < threshold, the grouping broke some repos' installs — inspect oracle_failures/ + traces/, fix the per-repo install in …

    line 130
  5. Very specific:

    Record in the tracker: Add the new repo to notes/RL/a3/a3_rl_tracker.md (and, for datagen rows, the MiniMax tracker …

    line 134

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

datagen-reduce-dataset-snapshots

Harbor's Daytona backend builds one container snapshot per unique environment directory, keyed by a content hash of environment/ — which for our patchers is just environment/Dockerfile (solution/tests/metadata live in sibling dirs and don't affect the hash). Daytona enforces a HARD org cap (40) and a per-launch max_new_snapshots (10). A dataset whose tasks each render a distinct Dockerfile explodes to ~1 snapshot/task and is unlaunchable. Fix: make the patcher render a small shared set of Dockerfiles (grouped by a coarse key like Python version); clone the repo + run repo-specific install at agent/verifier runtime instead of image-build time, so thousands of tasks collapse onto a handful of environments.

Thresholds — set BEFORE regenerating (step 0, write in the log)

Snapshot reduction is lossy: fewer envs → less each task's env is tailored → some repos' installs stop reproducing the gold patch → oracle yield drops. You are choosing an operating point on the snapshot↔fidelity curve, not "fixing a bug" — decide these up front to avoid an unbounded chase:

  • Snapshot ceiling: hard < 10 (ideally ≤ 8). Non-negotiable — it's the cap. (20 is the "skip the dataset" line; this skill pulls a dataset back under it.)
  • Oracle-yield floor: a number set up front (e.g. ≥ 80% for a dataset destined for RL/datagen verification; lower only with explicit reason). Without a pre-set floor every result looks "one more round will help" and you over-fit get_specs to the 40-task sample.
  • Iteration budget: max regenerate→oracle rounds (e.g. 2–3). Each round is a full regenerate + upload + oracle sample (slow + Daytona builds); diminishing returns set in fast once the easy repo-family fixes are in.
  • Sample size for the yield estimate: 40 is a usable read; don't re-sample endlessly chasing a ±few-point wobble — that is not signal.

Authoritative count tool

scripts/harbor/count_snapshots_from_tasks.py computes the exact content-hash dedup count Daytona's auto_snapshot path uses (get_task_environment_hash / analyze_task_dockerfiles). Run it on a local task dir (post-extraction or post-generation), not a live HF id, to skip the registry round-trip:

PY=/Users/benjaminfeuer/miniconda3/envs/otagent/bin/python
$PY -m scripts.harbor.count_snapshots_from_tasks --local-dataset <tasks_dir>
# read the "UNIQUE ENVIRONMENTS (SNAPSHOTS): N" line

For an uploaded HF dataset, extract first:

$PY -m scripts.datagen.extract_tasks_from_parquet \
  --parquet <hf-id> --output_dir /tmp/snapcount-<slug> --on_exist overwrite
$PY -m scripts.harbor.count_snapshots_from_tasks --local-dataset /tmp/snapcount-<slug>

The cycle

  1. Count the current artifact. Extract the flagged HF dataset → count. Confirm it's genuinely over threshold — use the tool above, not row counts.
  2. Diagnose the env-hash driver. Find the patcher. Most live in the shared data/patchers/ dir (patch_<name>_tasks.py, patch_exp_rpt_*_tasks.py, patch_mix_h*_tasks.py, patch_code_contests_tasks.py, …); only a few datasets keep a per-source-subdir patcher (data/<name>/generate_patched.py, e.g. swegym, swesmith). ls data/patchers/ | grep -i <name>; ls -d data/<name> 2>/dev/null. The unique-env count == number of distinct rendered Dockerfile strings. The explosion almost always comes from per-task interpolation into the Dockerfile: repo@commit in a build-time git clone/RUN, a per-instance base image, or per-task apt pins. Generate a small sample with the current patcher and count it — if the uploaded artifact is high but a fresh sample is low, the artifact is just stale (made by an older patcher) and step 4 is a pure regenerate+reupload.
  3. Rewrite the env logic to group (only if step 2's fresh sample is still high). Make the Dockerfile depend on a coarse grouping key (e.g. Python version), not the task. The swegym pattern:
    • Base image + a union of system apt deps per group; nothing task-specific in the Dockerfile.
    • Move git clone <repo>@<commit> and repo-specific pip install/make into instruction.md (agent setup), solution/solve.sh, tests/test.sh — they run at trial time, against the shared image.
    • Keep a get_specs(repo, version) map so each repo still gets its correct Python
      • install command; only the Dockerfile-affecting part is coarsened. Re-generate a sample → count → iterate the grouping until < 10.
  4. Regenerate the full dataset + upload to a NEW repo. Never overwrite the validated artifact; bump the version suffix (...-validated-v2 → ...-v3, or ...-snap-reduced). Run the patcher with --limit <=0> (no limit), --target-repo laion/<new-name> (public per feedback_hf_public_default). Then re-extract + re-count the uploaded repo to confirm < 10 end-to-end.
  5. Quality gate — TWO-TIER (DO NOT SKIP). Snapshot reduction is valid only if the tasks still build AND stay verifiable. Two distinct signals; conflating them is the classic mistake:

    • Tier 1 — infra (harbor smoke): does the env build + the agent run without crashing? Run:
      echo "laion/<new-name>" > /tmp/snap_check.md
      FORCE_COLOR=1 SAMPLE_SIZE=200 ./scripts/daytona/batch_validate_from_md.sh /tmp/snap_check.md
      # summary TSV: /Users/benjaminfeuer/Documents/agent-traces-analysis/summary.tsv
      Read infra_rate. Ignore this script's solve_rate — it's the agent's task-solve rate, low by design, NOT a measure of task well-formedness.
    • Tier 2 — oracle correctness (THE real quality gate): does the gold patch still make tests pass? batch_validate does NOT run this; run it explicitly:
      $PY scripts/daytona/validate_and_upload_from_hf.py \
        --repo_id laion/<new-name> --extract_dir <cache> \
        --stages oracle --sample_size 40 --sample_seed 42 --skip_upload \
        --keep_failed_dir <dir>/oracle_failures
      # prints "Success: S  Fail: F  Missing: M" → oracle pass = S/(S+F)
      A low oracle rate means the env/install/test harness no longer reproduces the conditions the patch needs — the env-collapse broke repo-specific installs.
    • CAP-SAFETY: do NOT re-validate the OLD high-env artifact. Sampling 200 (or 40) tasks from a 906-env dataset tries to build that many snapshots and blasts the org cap. The new low-env artifact is cheap to validate; for a baseline use the old artifact's recorded validation number (tracker / a prior summary.tsv), or judge against the floor set in step 0 above.
    • If oracle pass < threshold, the grouping broke some repos' installs — inspect oracle_failures/ + traces/, fix the per-repo install in get_specs, regenerate, re-test (within the budget).
  6. Record in the tracker. Add the new repo to notes/RL/a3/a3_rl_tracker.md (and, for datagen rows, the MiniMax tracker experiments/active/datagen/minimax-m2.7-tt/tracker.md): the new HF id, before→after snapshot count, and smoke-test infra/solve rates. Write a dated log to /Users/benjaminfeuer/Documents/agent_logs/.

Decision at each round (don't iterate reflexively)

  1. Oracle ≥ floor at < 10 snapshots → SHIP (record both numbers; done).
  2. Oracle < floor, failures cluster on a few repo-families with identifiable missing install steps → one targeted get_specs round (add the apt pkg / pip constraint / install command for those families), regenerate, re-oracle. Spend a budget slot.
  3. Oracle < floor, failures spread evenly (systemic — the shared env can't satisfy the repo diversity) → the inherent tradeoff, not a bug. Do NOT keep tweaking get_specs. Pick a coarser-but-larger grouping that still fits the cap (e.g. group by py-version × repo-family → maybe 8–15 envs instead of 5) and re-measure the (snapshots, oracle) point. Present the tradeoff curve.
  4. Budget exhausted and still < floor at any ≤-cap grouping → STOP and surface to the user with the curve (e.g. "5 envs → 48% oracle; 12 → 74%; 30 → 91% but busts the 10-cap"). Shipping a lower-fidelity dataset, raising the floor, or shelving is the user's call, not an infinite loop's.

Worked example — swegym (#31, 906 → 5 snapshots, a FAILED reduction)

laion/swegym-tasks-patched-validated-v2: 989 tasks → 906 unique envs (≈1:1). The patcher data/swegym/generate_patched.py already groups Dockerfiles by Python version (get_specs() interpolates only {python_version} + {extra_packages} from a fixed apt_map, clones the repo at runtime), so v2 was a stale artifact from an older per-task patcher. Regenerating with the current patcher → laion/swegym-tasks-patched-validated-v3 → 906 → 5 snapshots (no patcher code change). But the quality gate exposed the tradeoff: Tier-1 harbor smoke = 100% infra (looks great, would be mistaken for success); Tier-2 oracle = 19/40 = 47.5% — 5 shared py-version envs can't satisfy every repo's install. v3 is snapshots-green but oracle-red = a FAILED reduction, not shippable.

Guardrails

  • Never ship below the oracle floor just because snapshots are green. A tiny-snapshot dataset whose gold patches don't verify is worse than useless for RL/datagen (the reward signal is broken). Snapshots-green + oracle-red is a failed reduction, recorded as such.
  • Never raise/bypass the Daytona cap (max_new_snapshots, max_org_snapshots) or convert SnapshotCapExceeded to a warning — reduce the real count (feedback_daytona_snapshot_caps_hard_limit).
  • Never overwrite the existing validated artifact — always a new versioned repo.
  • Uploads are PUBLIC by default to laion/; enable_db_registration stays off (these are task datasets, not models).
  • On the Mac, run everything with the otagent python (/Users/benjaminfeuer/miniconda3/envs/otagent/bin/python); source "${DC_AGENT_SECRET_ENV:?set DC_AGENT_SECRET_ENV first}" first (.agents/secret.md).

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.