agentleFS
open-thoughtsopen-thoughts/OpenThoughts-Agent/.agents/skills/analyze-datagen-campaign-summary/SKILL.md

analyze-datagen-campaign-summary

Skill309 starsChanged 4 months agoApache-2.0View on GitHub

Build a clean per-dataset summary table/CSV for a datagen (trajectory-generation) campaign — one row per task source with Status (COMPLETED / FAILED / RUNNING / NOT STARTED), N Trials Completed, Mean Turns/Trace, Mean Tok/Trace, Mean Reward, and the HF trace-repo link. Use when asked to "summarize the campaign", "which datasets did we complete + their rewards/trials", "build a completion table/CSV", or to reconcile a prose tracker into auditable per-dataset metrics. Computes metrics by STREAMING each uploaded HF trace dataset (disk-bounded) and reusing the canonical OT-Agent analysis tools (scripts/analysis/utils.py: extract_conversation_text / count_turns / extract_reward) + the Qwen3-8B tokenizer; HF-ground-truths Status by probing each repo. Related skills: analyze-dataset-token-length (token method), analyze-job-history-iris (harbor Mean / trials from logs).

The file

We split this skill into the rules it gives an agent

1 rule. About 0 are common, 0 are distinctive, and 1 is very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈0

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 1

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    NOT the row's model field.

    line 91

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

analyze-datagen-campaign-summary

Turn a datagen campaign's prose tracker (e.g. ~/Documents/experiments/{active,complete}/<campaign>/tracker.md, whose per-dataset status lives in sentences, not columns) into a clean, auditable per-dataset table/CSV with computed metrics. Built for the qwen3.5-122b-tt 32k campaign but campaign-agnostic — swap the dataset list.

Target columns: Datagen Model | Task Source | Status | N Trials Completed | Mean Turns / Trace | Mean Tok / Trace | Mean Reward | HF Repo Link.

Why this skill exists (the two traps)

  1. The tracker rarely has reward / turns / tokens. Trackers record throughput (gen tok/s) + row counts in prose; mean reward, mean turns, and mean tokens are almost never written down. They must be COMPUTED from the uploaded HF trace datasets.
  2. Status in prose is stale/ambiguous (a "RUNNING" row that actually finished; a "rescued" row with no clean repo name). Ground-truth Status against HF: if the trace repo exists with rows → COMPLETED (and its row count IS N Trials Completed); otherwise fall back to the tracker's status hint.

The data model (uploaded OT-Agent trace dataset)

Each row of penfever/<slug>-<model>-traces is one trial/trace: {conversations: [{role,content},…], agent, model, model_provider, date, task, episode, run_id, trial_name, result, verifier_output}.

  • N Trials Completed = row count of the dataset (one row = one completed trace).
  • Mean Turns / Trace = mean count_turns(row) = mean number of conversation messages (canonical definition in scripts/analysis/utils.py; total messages, not just assistant turns — state it in the notes).
  • Mean Tok / Trace = mean Qwen3-8B token length of the whole conversation — plain method from the analyze-dataset-token-length skill: tokenizer(extract_conversation_text(row), add_special_tokens=False). Tokenizer is always Qwen/Qwen3-8B for these datasets (their trace-dataset convention), regardless of the served model name (model field is hosted_vllm/<numeric-id>, not a usable tokenizer).
  • Mean Reward = Harbor-flat mean of result via mean_reward_per_trial semantics: extract_reward each row (parses the result string, e.g. "0.0" → 0.0), missing/non-numeric counts as 0.0. This matches harbor's <done>/<total> Mean: accuracy exactly — do NOT drop nulls or the number won't reconcile.

Reuse the canonical tools (do NOT reinvent)

/Users/benjaminfeuer/Documents/OpenThoughts-Agent/scripts/analysis/utils.py:

  • extract_conversation_text(record) — conversation → full text to tokenize (handles messages/conversations).
  • count_turns(record) — turns.
  • extract_reward(record) — parses result → float|None. mean_reward_per_trial(rows) — Harbor-flat mean.
  • load_hf_trace_dataset(repo_id) — non-streaming loader (fine for small repos; see disk note for large ones).

Token-length details (methods, tokenizer, the metadata-confound trap) → the analyze-dataset-token-length skill. If you'd rather source Mean Reward + trials from the job logs instead of the HF dataset (e.g. the repo was never uploaded), the analyze-job-history-iris skill's analyze_iris_harbor_job.py sidecar carries the harbor Mean: + non_empty_trials per job — but the uploaded dataset is the more reliable ground truth for a COMPLETED row.

⚠ Handling the LARGE trace datasets (disk + bandwidth)

Some campaign datasets are big (tens of thousands of rows / hundreds of MB / dozens of shards). Full load_dataset caches the whole parquet to ~/.cache/huggingface → can blow local disk (a full disk bricks the supervisor — see the disk-health rule in supervisor-init). So:

  • STREAM (load_dataset(repo, split="train", streaming=True)) and accumulate in ONE pass — disk stays bounded (shards read on the fly, not cached whole).
  • Point HF_HOME / HF_DATASETS_CACHE at the scratchpad and df -h / before launching; bandwidth is unavoidable (the conversations column is the bulk, needed for both turns and tokens) but streaming avoids the disk blowup.
  • Batch the tokenizer (e.g. 128 texts) rather than per-row; TOKENIZERS_PARALLELISM=false to avoid the fork-after-tokenizer deadlock when parallelizing.
  • Parallelize across datasets with a ProcessPoolExecutor (≈5 workers) — CPU-bound tokenization scales well; each worker streams its own datasets. Checkpoint per-dataset to JSONL so a crash/interrupt resumes instead of recomputing the expensive large ones.

Procedure

  1. Build the dataset list from the campaign tracker. One entry per task source: (idx, task_source, candidate_hf_repo_or_None, status_hint, note). candidate_hf_repo = the exact penfever/<slug>-…-traces slug the tracker names (the slug transform is IRREGULAR — copy the stated repo, don't derive it). status_hint ∈ {COMPLETED, FAILED, RUNNING, NOT STARTED} (pending → NOT STARTED & repo=None; killed-not-rescued / blocked-skipped → FAILED & repo=None).
  2. Per dataset: probe HF (HfApi().dataset_info(repo)); on 404 keep the hint + NULL metrics. Else stream, compute n_trials, mean_turns, mean_tok (Qwen3-8B), mean_reward (Harbor-flat), set Status=COMPLETED and HF Repo Link = https://huggingface.co/datasets/<repo>.
  3. Write the CSV sorted by idx; NULL metrics render as empty cells, missing repo as NULL.
  4. VERIFY before delivering (the user asked for it to be correct): spot-check that computed n_trials matches the tracker's stated row counts on a few datasets, and that a KNOWN-degenerate dataset reconciles (e.g. qwen3.5-122b-tt codenet-python-v2 mean reward ≈ 0.017 ↔ the tracker's "~2% pass-rate"). Mean tokens should sit under the campaign's context window (32k here) for the vast majority.

Definitions to state alongside the table (so it's auditable)

  • Datagen Model = the trajectory-generation model (constant per campaign; e.g. Qwen3.5-122B-A10B-FP8), NOT the row's model field.
  • Mean Turns/Trace = mean total conversation messages (count_turns).
  • Mean Tok/Trace = mean Qwen3-8B plain token count of the full conversation.
  • Mean Reward = Harbor-flat trial mean (missing/error = 0.0) — reconciles with the harbor Mean: line.
  • N Trials Completed = uploaded productive rows (may be < tasks for partial/rescued jobs; note it).

Cross-reference

  • analyze-dataset-token-length — token-length method, Qwen3-8B convention, the metadata-confound trap.
  • analyze-job-history-iris — harbor Mean: + productive-trial counts from job logs (alt metric source).
  • datagen-launch-iris — how the trace datasets are produced/rescued/uploaded (upstream of this table).
  • scripts/analysis/utils.py — the canonical extract/count/reward helpers this skill reuses.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.