Complete a Task

This skill brings a single task page (e.g. /tasks/motion-generation) up to a "complete" state: the major benchmarks exist and are linked to the task, their source papers are indexed and enriched, the top papers have evaluation rows, and every leaderboard has been verified apples-to-apples.

It is an orchestrator. The real work is done by four existing skills, each run in its own subagent so the context stays small and the work parallelizes:

The legacy Papers With Code dump (data/legacy_pwc_data/evaluation-tables.json) is the source of truth for which benchmarks and papers matter — it already encodes years of community curation about what the real leaderboards are. We mine it for the top benchmarks and their top papers, then re-create that data properly in the live database via the sub-skills.

Why delegate to subagents (and always on Composer 2.5)

Each sub-skill is long and reads a lot of source material (papers, model cards, READMEs, legacy tables). Running them inline would blow up your context and make the orchestration unmanageable. Spawning one subagent per unit of work (per paper, per benchmark) keeps each agent focused on a small, well-scoped job and lets independent work run in parallel.

Always pass model: "composer-2.5-fast" when launching these subagents. Composer 2.5 is cheap and plenty capable for these mechanical, well-specified sub-tasks, and the user has asked that all delegated work in this pipeline use it. The orchestration (this skill) stays on whatever model the user is driving.

When launching a subagent, give it everything it needs to run autonomously: the exact sub-skill path to read first, the task slug, the specific paper/benchmark it owns, the relevant legacy rows, and what to report back. Subagents don't see this conversation, so be explicit.

Pre-flight

Run everything from the repo root with the project environment:

uv run --env-file backend/keys.env ...

Confirm the backend is up (start it without asking if not, per AGENTS.md):

curl -s -o /dev/null -w "%{http_code}\n" http://localhost:8000/api/v1/papers/recent?limit=1

Resolve the task row from the slug or name the user gave:

uv run --env-file backend/keys.env python - <<'PY'
from sqlalchemy import text
from backend.database.connection import get_engine
slug = "motion-generation"
with get_engine().connect() as conn:
    row = conn.execute(text(
        "SELECT id, name, slug FROM tasks WHERE slug = :s OR name ILIKE :n"
    ), {"s": slug, "n": f"%{slug.replace('-', ' ')}%"}).mappings().first()
    print(dict(row) if row else "TASK NOT FOUND")
PY

If the task itself doesn't exist, this is the wrong skill — use add-task first, then come back.

Use a TodoList to track the phases below; this is a long pipeline and it's easy to lose a benchmark or paper partway through.

Phase 1 — Mine the legacy dump for benchmarks + papers

The legacy file is ~250 MB. Never load it whole on this VPS. Use the bundled streaming miner.

Site task slugs rarely match legacy task names (the site's motion-generation lives under the legacy task Motion Synthesis), so discover the legacy name first:

uv run --env-file backend/keys.env python \
  .agents/skills/complete-task/scripts/mine_legacy_task.py \
  --list-tasks "motion|dance|gesture"

Pick the legacy task name(s) that genuinely correspond to the site task — sometimes more than one legacy task maps to one site task (e.g. Motion Synthesis + Motion Captioning). Then aggregate:

uv run --env-file backend/keys.env python \
  .agents/skills/complete-task/scripts/mine_legacy_task.py \
  --task "Motion Synthesis" --max-benchmarks 5 --max-papers 10 \
  --out /tmp/complete-<slug>.json

This returns, per benchmark (ranked by number of leaderboard rows, capped at the top 5):

• benchmark, row_count, unique_papers, the legacy metrics columns, and
• top_papers: up to 10 deduped papers with model_name, metrics, paper_url, paper_title, paper_date.

Keep the caps the user asked for — at most 5 benchmarks, at most 10 papers per benchmark — so the pipeline stays bounded. The union of all paper_urls across benchmarks is your paper worklist; the set of benchmark names is your benchmark worklist.

Sanity-check the picks against the live task page before committing to them — the legacy metric columns tell you what leaderboard metrics the benchmark needs, and a benchmark with very few rows or an obviously off-topic name (legacy regex matches are loose) should be dropped.

Phase 2 — Take stock of what already exists

The user explicitly wants graceful degradation: benchmarks may already exist with many evals, and top papers may already be indexed and enriched. Do not redo finished work — verify it instead. Check the current state before launching any write subagents:

uv run --env-file backend/keys.env python - <<'PY'
from sqlalchemy import text
from backend.database.connection import get_engine

task_id = <task_id>
benchmark_names = ["HumanML3D", "KIT Motion-Language", "AIST++", ...]
arxiv_ids = ["2312.00063", ...]  # parsed from legacy paper_url where possible

with get_engine().connect() as conn:
    print("== datasets / task link / metrics / eval count ==")
    for name in benchmark_names:
        rows = conn.execute(text("""
            SELECT d.id, d.name, d.slug,
                   bool_or(td.task_id = :tid) AS linked_to_task,
                   count(DISTINCT tdm.metric_id) FILTER (WHERE tdm.task_id = :tid) AS metrics_on_task,
                   count(DISTINCT e.id) FILTER (WHERE e.task_id = :tid) AS evals_on_task
            FROM datasets d
            LEFT JOIN task_datasets td ON td.dataset_id = d.id
            LEFT JOIN task_dataset_metrics tdm ON tdm.dataset_id = d.id
            LEFT JOIN evaluations e ON e.dataset_id = d.id
            WHERE d.name ILIKE :n OR d.slug ILIKE :s
            GROUP BY d.id, d.name, d.slug
        """), {"tid": task_id, "n": name, "s": name.lower().replace(" ", "-")}).mappings().all()
        print(name, "=>", [dict(r) for r in rows] or "MISSING")

    print("== papers indexed? ==")
    for aid in arxiv_ids:
        r = conn.execute(text(
            "SELECT id, arxiv_id, urls_extracted FROM papers WHERE arxiv_id = :a "
            "ORDER BY is_latest_version DESC NULLS LAST LIMIT 1"
        ), {"a": aid}).mappings().first()
        print(aid, "=>", dict(r) if r else "NOT INDEXED")
PY

From this, bucket each item:

• Benchmark: missing → Phase 3 (add-benchmark); exists but not linked to this task or missing the needed metric → Phase 3 (add-benchmark handles linking + metrics); fully wired with evals → skip to Phase 6 (verify-evals).
• Paper: not indexed → Phase 4 index step; indexed but sparse (no repo/tags) → Phase 4 enrich-paper; already enriched with evals on these benchmarks → leave it, just verify in Phase 6.

Phases 3 and 4 are independent and can run concurrently. Phase 5 (add-evals) needs the benchmarks from Phase 3 to exist first (so the leaderboard + metric are configured), and Phase 6 (verify-evals) runs last, once rows are in place.

Phase 3 — Create missing benchmarks (one subagent per benchmark)

For each benchmark that is missing, unlinked, or missing its leaderboard metric, launch a subagent that reads and follows add-benchmark. Divide the work one benchmark per subagent so they run in parallel.

Subagent prompt template (launch with model: "composer-2.5-fast", run_in_background: true):

Read and follow the skill at .agents/skills/add-benchmark/SKILL.md.

Add/repair the benchmark "<benchmark name>" and link it to task "<task-slug>" (task_id <id>).
Source metrics seen in legacy PwC for this benchmark: <metrics list from the miner>.
Candidate source paper (introduced the benchmark): <paper_title> <paper_url>.

Goals: ensure a datasets row exists; resolve and attach the benchmark's *introducing* source paper
(native paperswithcode.co paper URL, indexing/creating it if needed per the skill); ensure the
(task_id, dataset_id) link exists; ensure the leaderboard metric(s) the legacy table uses are linked.
Report the dataset id, slug, source paper URL, and metrics linked.

Be careful (the skill stresses this): the benchmark's source paper is the paper that introduced the dataset, not a model paper that merely scores on it. Pass the miner's paper_title/paper_url only as a candidate; let the subagent verify provenance.

Phase 4 — Index + enrich the top papers (one subagent per paper)

Papers from the legacy dump may not be in the live DB yet. For each paper on the worklist:

1. Index it if missing. For an arXiv paper, the admin endpoint is the simplest path:

   source backend/keys.env
   curl -s -X POST "http://localhost:8000/api/v1/papers/index/$ARXIV_ID" \
     -H "X-Admin-Token: $PWC_ADMIN_API_TOKEN" -o /dev/null -w "%{http_code}\n"
   # 200 = indexed, 409 = already present (both fine), 404 = not on arXiv

   Parse the arXiv id from the legacy paper_url (e.g. https://arxiv.org/abs/2312.00063v1 → 2312.00063). If a top paper has no arXiv id (CVPR/proceedings/blog), it must be created via add-external-paper; fold that into the subagent prompt so the worker creates the external row before enriching.

2. Enrich it with a subagent that reads and follows enrich-paper. Launch with model: "composer-2.5-fast", run_in_background: true:

   Read and follow the skill at .agents/skills/enrich-paper/SKILL.md.
   
   Enrich the paper "<paper_title>" (arxiv <arxiv_id> / paperswithcode.co/paper/<id>).
   It is a top paper on the "<task-slug>" task's benchmark "<benchmark>".
   Find its own official GitHub repo + project page (authenticity over recall) and add high-confidence
   task/method tags from the existing taxonomy, MERGING with — never clobbering — existing tags.
   If the paper row does not exist and it has no arXiv id, first create it via add-external-paper.
   Report the paper URL, links saved, and tags added/removed.

Papers that are already indexed and enriched (from Phase 2) don't need a subagent here — skip them.

Phase 5 — Import evaluations (one subagent per paper)

Once the benchmarks exist (Phase 3 complete) and papers are indexed (Phase 4), import the leaderboard rows. One subagent per paper, reading and following add-evals. Launch with model: "composer-2.5-fast", run_in_background: true:

Read and follow the skill at .agents/skills/add-evals/SKILL.md.

Add evaluation rows for the paper "<paper_title>" (arxiv <arxiv_id> / paperswithcode.co/paper/<id>),
which is a top paper for the "<task-slug>" task. Target benchmarks for this paper (from legacy PwC):
<benchmark names + the metric columns + the model_name(s) and scores the miner found for this paper>.

Follow the skill's rules exactly: only import rows on benchmarks already wired up with the configured
leaderboard metric; attach rows to THIS paper's id (no external/comparator evals); set is_open,
num_parameters, harness per the skill. Use the legacy numbers as a hint of which table to find in the
source, but re-source each score from the paper/README/model card — do not trust legacy values blindly.
Report rows added, benchmarks skipped + why, and the paper URL.

add-evals is conservative by design: it skips benchmarks that aren't fully configured. That's why Phase 3 runs first. If a subagent reports it skipped a benchmark because the metric wasn't configured, that's a signal Phase 3 missed something — fix the benchmark, then re-run that paper's eval subagent.

The legacy metrics columns are a strong hint, but the leaderboard contract is whatever metric is actually configured on the live (task_id, dataset_id). Let add-evals pick the best eligible row per benchmark; don't force every legacy column.

Phase 6 — Verify each benchmark leaderboard (one subagent per benchmark)

After evals are in, audit each benchmark for apples-to-apples comparability. One subagent per benchmark, reading and following verify-evals. Launch with model: "composer-2.5-fast", run_in_background: true:

Read and follow the skill at .agents/skills/verify-evals/SKILL.md.

Audit the benchmark "<benchmark>" (dataset slug <slug>) on task "<task-slug>" (task_id <id>).
Confirm every evaluation row is apples-to-apples: same metric, same dataset split, same evaluation setup,
and a score faithful to its source paper. Establish the leaderboard contract first (split/metric/setup),
then classify each row (PASS / WRONG_SPLIT / WRONG_SETUP / WRONG_METRIC / WRONG_PAPER / DUPLICATE /
UNVERIFIABLE). Fix or delete per the skill's remediation rules. Report the contract, a row-by-row verdict
table, and the benchmark URL.

This is also the path for the "already complete" case: if Phase 2 showed a benchmark already has many evals, you can skip straight here and just verify what exists rather than re-importing.

Orchestration tips

• Batch independent subagents in one turn. Within a phase, launch all the per-paper or per-benchmark subagents together so they run concurrently. Across phases, respect the ordering (benchmarks → evals → verify); enrichment (Phase 4) overlaps freely with benchmark creation (Phase 3).
• Reconcile in the parent. As background subagents report back, keep your TodoList current and watch for cross-cutting issues (a benchmark a subagent couldn't source, a metric mismatch surfaced by add-evals, a paper with no arXiv id). Resolve those at the orchestrator level so workers don't duplicate writes.
• Idempotency. Every sub-skill is written to be safe to re-run. If a subagent fails midway, re-launch it for the same unit rather than trying to hand-patch the DB.
• Stay memory-aware throughout. Beyond the miner, never bulk-load papers-with-abstracts.json or the full eval tables. Targeted SQL, the miner, gh api .../readme, and per-paper PDFs in /tmp are enough.

Reporting back

End with a concise completion report for the task:

• The task page URL: https://paperswithcode.co/tasks/<slug>.
• Benchmarks: for each of the (≤5) benchmarks — created / already-existed / repaired, dataset slug, source paper URL, metrics linked, and the benchmark URL https://paperswithcode.co/benchmark/<slug>.
• Papers: which top papers were indexed, which were enriched (repos/tags added), and which were already complete.
• Evaluations: rows added per paper/benchmark, and benchmarks skipped by add-evals with the reason.
• Verification: per-benchmark verdict summary (PASS counts and any rows fixed/deleted), and any leaderboard left with an open question for the user.
• Anything that needs a human decision (ambiguous benchmark source paper, a legacy benchmark with no clear live equivalent, a paper with no resolvable source).

When NOT to use this skill

• The user wants to add one benchmark → add-benchmark. One paper's evals → add-evals. Enrich one paper → enrich-paper. Verify one leaderboard → verify-evals. This skill is for whole-task buildout.
• The task page itself doesn't exist yet → add-task first.
• The user wants evaluation rows from a specific paper or source rather than whole-task buildout → add-evals.