agentleFS
opendatadiscoveryopendatadiscovery/odd-team/.claude/skills/next-batch/SKILL.md

next-batch

Skill0 starsChanged 2 months agoApache-2.0View on GitHub

Autonomous batch driver for the rev-2 agentic-ontology sprint. Picks the next pending theme from state/sprint-themes.yaml, spawns 5 file-analyser subagents in parallel, then 5 reducers in parallel, merges deltas, refreshes coverage, appends the investigator-log entry, commits, pushes to remote, marks the theme done. Designed to be fired by `/loop` overnight without maintainer input.

  • Deletes or force-pushes
  • Commits and pushes

The file

Show its rules and text

We split this skill into the rules it gives an agent

20 rules. About 0 are common, 7 are distinctive, and 13 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈7

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 13

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  • could be automatic: a setting could enforce it
  1. Very specific:

    Resolve sprint branch: Read state/sprint-themes.yaml and extract policy.push_target_branch.

    line 25
  2. Very specific:

    Theme pick: Read state/sprint-themes.yaml.

    line 31
  3. Very specific:

    Failure handling: If 1-2 of 5 file-analysers fail (timeout / error / refusal) → continue with the 3-4 that succeeded.

    line 62
  4. Distinctive:

    Verify all expected sidecar paths exist on disk before proceeding (Glob + ls).

    line 66
  5. Very specific:

    Pre-step — warm the graph (rev 7.1): Before spawning the reducers, run ONCE: lineage/_extractor/.venv/bin/lineage-extractor graph-build …

    line 68
  6. Very specific:

    Failure handling: If 1-2 reducers fail → still commit + push the partial state with those reducers' artefacts marked …

    line 83
  7. Distinctive:

    Run these Bash commands in sequence:

    line 89
  8. Distinctive:

    If push fails (non-fast-forward — unlikely in single-session mode):

    line 135
  9. Very specific:

    If push still fails → mark theme blocked with blocked_reason: push-conflict-after-rebase and exit.

    line 140
  10. Distinctive:

    No force-push needed — this is a fresh commit appended to the sprint branch.

    line 150
  11. Very specific:

    Commit + push this single-file change with message [next-batch] theme {theme_id} done.

    line 159
  12. Distinctive:

    If at any point a non-recoverable failure occurred (≥3 file-analysers failed, ≥3 reducers failed, or push hard-failed):

    line 163
  13. Very specific:

    NEVER run git push --force, git reset --hard, git checkout main, rm -rf, git branch -D autonomously.

    line 179
  14. Very specific:

    NEVER write to source repos at <REPO_ROOT>, <DOCS_REPO>, etc.

    line 180
  15. Very specific:

    NEVER edit adrs/drafts/* — those are maintainer-authored design docs.

    line 181
  16. Distinctive:

    NEVER edit CLAUDE.md — it is the governance surface.

    line 182
  17. Very specific:

    If a tool call hits a permission prompt → that's a settings.local.json gap.

    line 184
  18. Distinctive:

    When the maintainer resumes after a halt:

    line 188
  19. Very specific:

    Read state/sprint-themes.yaml to see the last-attempted theme + its status.

    line 189
  20. Very specific:

    Read state/batch-{theme_id}-trace.yaml (if present) for the per-agent outcomes.

    line 190

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

/next-batch — autonomous rev-2 batch driver

This skill is the inner-body of the overnight autonomous loop. One invocation = one full batch end-to-end. /loop 75m /next-batch fires this every 75 minutes; each invocation runs to completion synchronously within a single agent turn.

Path resolution (run BEFORE pre-flight)

The agent-prompt templates in this SKILL use <WORKSPACE_ROOT>, <REPO_ROOT>, and <DOCS_REPO> as placeholders so the SKILL does not embed any maintainer-specific absolute path. The orchestrator (you, the agent invoking the SKILL) MUST substitute them at runtime before passing prompts to subagents. Resolve them like this:

WORKSPACE_ROOT=$(git rev-parse --show-toplevel)     # current repo root
REPO_ROOT=$(realpath "$WORKSPACE_ROOT/../odd-platform")
DOCS_REPO=$(realpath "$WORKSPACE_ROOT/../documentation")

Substitute these into every Agent prompt below. Never hardcode an absolute path into a committed file (memory: feedback_no_hardcoded_absolute_paths).

Pre-flight (FIRST 5 actions in order — abort if any fails)

  1. Resolve sprint branch. Read state/sprint-themes.yaml and extract policy.push_target_branch. This is the long-lived sprint branch every batch commits to. Substitute <SPRINT_BRANCH> everywhere below with that value. If the key is absent or empty → ABORT with "sprint-themes.yaml policy.push_target_branch unset".
  2. Branch check. git rev-parse --abbrev-ref HEAD must equal <SPRINT_BRANCH>. If it doesn't, ABORT — surface "wrong branch, halting loop" and exit. Never run on main.
  3. Clean-tree check. git status -s should show no uncommitted unrelated changes. The only acceptable state is "clean working tree" OR pending changes confined to lineage/ / state/ / .claude/ from a prior interrupted batch. If foreign changes exist, ABORT.
  4. Theme pick. Read state/sprint-themes.yaml. Find the FIRST entry with status: pending. If none → exit cleanly with "no pending themes — autonomous loop completed all queued work" and SUGGEST to the maintainer that they extend the queue or stop the loop.
  5. Consecutive-failure halt. Walk batch_history from the tail. If the last 3 entries all have status: blocked → ABORT with "3 consecutive failed batches, halting loop for maintainer review" and exit. Don't loop forever.
  6. Theme lock. Edit state/sprint-themes.yaml — change the picked theme's status from pending to in_progress + add started_at: <ISO timestamp>. Stage + commit ONLY this file with message [next-batch] theme {theme_id} in_progress. Push. (This commit lets parallel sessions see the lock if you ever add parallelism later.)

Phase 1 — 5 file-analyser subagents IN PARALLEL (foreground)

In ONE assistant message, fire 5 Agent tool calls with subagent_type: file-analyser and run_in_background: false. Each prompt follows this template (substitute per target_node from the picked theme):

You are the file-analyser subagent. Enrich ONE node end-to-end per your system prompt's non-negotiable rules. This is batch {THEME_ID} of the ODD agentic-ontology rev-2 sprint (autonomous overnight run on <SPRINT_BRANCH>).

WORKSPACE_ROOT_ABS: <WORKSPACE_ROOT>
REPO_ROOT_ABS: <REPO_ROOT>
SCHEMA: v0.3.0 (rev 2) — sidecar MUST include `upstream_callers` and `downstream_side_effects` blocks; every `uncovered_behaviour` needs `test_class`.

TARGET NODE
- node_id: {synthetic_node_id from theme entry}
- source file: {source_file from theme entry}
- output sidecar: <WORKSPACE_ROOT>/lineage/odd-platform/understanding/{slugified-node-id}.md

CONTEXT: Theme rationale: {rationale from theme entry}. Cross-reference existing sidecars where material (use Glob/Grep against lineage/odd-platform/understanding/). Live doc WebFetch on `docs.opendatadiscovery.org` for any doc-link claim; if WebFetch is denied this session, inherit verifications from neighbour sidecars at status 200 within the last 11 days per the stale-probe cadence (established pattern across batches D/E/F/G).

OUTPUT: reply with (1) `Wrote: <repo-relative path>` (2) `Summary: <N counts + headline finding>`.

The assistant turn BLOCKS until all 5 file-analysers complete (parallel-foreground). Expect 5-15 min wall-clock for the slowest.

Slug rule for the output filename: replace spaces and special chars with __ between axis segments, single _ within identifiers (mirroring the existing convention — see existing files in lineage/odd-platform/understanding/ for shape).

Failure handling. If 1-2 of 5 file-analysers fail (timeout / error / refusal) → continue with the 3-4 that succeeded. If 3+ fail → mark the theme blocked (see Phase 4 failure path) and exit. Log per-agent outcomes in a per-batch state/batch-{theme_id}-trace.yaml file.

Phase 2 — 5 reducers IN PARALLEL (foreground)

Verify all expected sidecar paths exist on disk before proceeding (Glob + ls). For any missing → log + continue with the present subset.

Pre-step — warm the graph (rev 7.1). Before spawning the reducers, run ONCE: lineage/_extractor/.venv/bin/lineage-extractor graph-build odd-platform. This rebuilds the ephemeral graph + vector index from the canonical detail/ files (cache-checked) so every reducer's dedup graph-search hits a current, warm index reflecting all committed prior findings — and the 5 reducers do not each cold-build in parallel. Per adrs/drafts/agentic-graph-retriever.md.

In ONE assistant message, fire 5 Agent tool calls IN PARALLEL with run_in_background: false. Each reducer dedups via semantic graph-search per playbooks/registry-search-spawn.md (rev 7.1 — superseding the registry-search grep). Subagent types:

  • concept-merger — write/refresh concepts/detail/ files; dedup graph-search --label Concept.
  • adr-archaeologist — write/refresh implicit-adrs/detail/ + refactoring-scopes/detail/ files; dedup graph-search --label ImplicitADR / --label RefactoringScope. Do NOT emit index-batch-{theme_id}-append.md directives — index files are decommissioned; detail/ is canonical.
  • doc-gap-finder — write/refresh doc-gaps/detail/ files; dedup graph-search --label DocGap. WebFetch if available; otherwise inherit from neighbour sidecars per stale-probe cadence. Do NOT emit index-batch-append.md.
  • test-coverage-mapper — write/refresh test-map/detail/ files; dedup graph-search --label TestGap. Do NOT emit test-map/index.delta.yaml.
  • feature-flow-builder — write/refresh feature-flows/detail/ files; dedup graph-search --label Feature. Detail-file writes must use YAML-safe scalars: never emit a bare scalar containing : or starting with @ — use |- block scalar. test_matrix MUST be objects with state/covered/uncovered shape (not plain strings); terminal_side_effect MUST be object with side_effect_class/description.

Per-agent prompts follow the same shape as batch H's reducer prompts (see lineage/odd-platform/investigator-log.md batch H entry for the canonical structure). The 5 new sidecar paths are the only sidecars to read in full; PROCESSED_NODE_IDS for the prior batches inherits from the existing index frontmatters.

The assistant turn BLOCKS until all 5 reducers complete. Expect 15-45 min for the slowest (typically concept-merger or adr-archaeologist).

Failure handling. If 1-2 reducers fail → still commit + push the partial state with those reducers' artefacts marked stale_after_batch_{theme_id} in the manifest. If 3+ reducers fail → mark theme blocked and exit.

Phase 3 — YAML autofix + coverage + investigator-log + commit + push

Note — index maintenance dropped (2026-05-26). Post rev-7.1 the graph retriever reads embeddings + detail/ files directly; the index files (concepts/index.yaml, test-map/index.yaml, feature-flows/index.yaml, the markdown index.md aggregates) are denormalized aggregates that no longer serve any retrieval-time consumer. They had become drift-generating decoration. The previous Phase 3 steps that rebuilt indexes + merged index-batch-{theme_id}-append.md directives are removed. Reducer agents should no longer be asked to emit append directives. Detail files remain the source of truth.

Run these Bash commands in sequence:

  1. YAML safety pass — catches broken scalars before they pollute the detail/ corpus:
    python3 lineage/_extractor/registry-shard/yaml_safe_fix.py 2>&1 | tail -20
    Reports fixed: N, unfixable: M. The unfixable files are quarantined to .broken-yaml-pending-fix (data preserved in .broken-yaml-backup). If M > 0 — surface in investigator-log under "Follow-ups" but do NOT block the batch; the data is recoverable next batch when the reducer prompt's YAML-safe rule fires.
  2. (removed — markdown-index appends are no longer maintained)
  3. (removed — rebuild_indexes.py is no longer run; detail/ is canonical, graph-search reads it directly via embeddings)

3.5. Coherence sweep (LSN-018 — pre-commit anomaly detector):

python3 lineage/_extractor/registry-shard/coherence_sweep.py --batch {THEME_ID} 2>&1 | tail -5

Sweeps the new artefacts emitted this batch (and pre-existing ones) for cross-registry anchor overlaps where one side asserts a NEGATION about an entity another side positively names. Output: state/coherence-sweep-batch-{THEME_ID}.md. Empty → batch commits as usual. Non-empty top-tier (no-existing-back-link) candidates → READ the report, decide for each top candidate whether to: (a) supersede the older claim per Rule 6 of the reducer prompts, (b) accept the older claim and rewrite the new artefact, or (c) accept both as legitimately-different facts and add back-links. The sweep does not block the batch by itself — but each unresolved anomaly becomes a follow-up captured in the investigator-log "Follow-ups" section.

  1. Cleanup leftover delta + append files (no longer needed; nothing consumes them):
    find lineage/odd-platform -name 'index.delta.yaml' -delete
    find lineage/odd-platform -name 'index.delta.*.yaml' -delete
    find lineage/odd-platform -name 'concepts.delta.batch-*.yaml' -delete
    find lineage/odd-platform -name 'index-batch-*-append.md' -delete
    find lineage/odd-platform -name 'batch-*-delta.yaml' -delete
  2. Refresh coverage:
    python3 lineage/_extractor/registry-shard/coverage.py --write-manifest
  3. Append investigator-log entry. Construct a multi-section batch entry (sidecars added, reducer diffs, cumulative state, next-batch notes if any). Pattern: read batch H's + batch I's investigator-log entries as template. Use cat >> investigator-log.md << HEREDOC. Always include a "Follow-ups" section listing any .broken-yaml-pending-fix files this batch produced + any markdown-index detail-without-index discrepancies.
  4. Stage + commit + push:

    git add lineage/odd-platform/understanding/{new-sidecars-glob} \
            lineage/odd-platform/{concepts,implicit-adrs,refactoring-scopes,doc-gaps,test-map,feature-flows}/ \
            lineage/odd-platform/manifest.yaml \
            lineage/odd-platform/investigator-log.md \
            state/sprint-themes.yaml \
            state/batch-{THEME_ID}-trace.yaml
    git commit -m "batch {THEME_ID} (autonomous) — {THEME_NAME}; coverage {direct%} direct / {effective%} effective"
    git push origin <SPRINT_BRANCH>

    If push fails (non-fast-forward — unlikely in single-session mode):

    git pull --rebase origin <SPRINT_BRANCH>
    git push origin <SPRINT_BRANCH>

    If push still fails → mark theme blocked with blocked_reason: push-conflict-after-rebase and exit.

  5. Preserve .broken-yaml-pending-fix files (if any) in a follow-up commit so they reach the remote:
    if ls lineage/odd-platform/**/*.broken-yaml-pending-fix 2>/dev/null | grep -q .; then
      git add lineage/odd-platform/**/*.broken-yaml-pending-fix lineage/odd-platform/**/*.broken-yaml-backup 2>/dev/null
      git commit -m "[next-batch] theme {THEME_ID} — preserve broken-yaml-pending-fix files for next-batch recovery"
      git push origin <SPRINT_BRANCH>
    fi
    No force-push needed — this is a fresh commit appended to the sprint branch.

Phase 4 — Mark theme done (success path)

Edit state/sprint-themes.yaml:

  • Change the picked theme's status from in_progress to done
  • Add completed_at: <ISO>
  • Append to top-level batch_history: {batch_id: {THEME_ID}, status: done, completed_at: ..., sidecars_added: <N>, delta_summary: "..."}

Commit + push this single-file change with message [next-batch] theme {theme_id} done.

Phase 4 (failure path) — Mark theme blocked

If at any point a non-recoverable failure occurred (≥3 file-analysers failed, ≥3 reducers failed, or push hard-failed):

  • Edit state/sprint-themes.yaml — change status to blocked, add blocked_at + blocked_reason: "<one-line>".
  • Commit + push: [next-batch] theme {theme_id} blocked — {short-reason}.
  • Exit (do NOT halt the loop unless 3 consecutive blocks).

Exit message format

Reply with EXACTLY ONE of these formats (the loop driver's status surface):

  • Success: BATCH {THEME_ID} DONE — sidecars +{N}, direct {direct%}, effective {effective%}, features {total_features} ({new_features_count} new), {total_test_gaps} test-gaps ({critical_count} CRITICAL).
  • Blocked: BATCH {THEME_ID} BLOCKED — {blocked_reason}. Loop continues; mark for maintainer review.
  • Halt: LOOP HALTING — {reason}. {THEME_ID or N/A} state preserved; resume manually after investigation.
  • Queue empty: QUEUE EMPTY — all themes done or blocked. Loop has nothing further; safe to stop.

Safety rails (universal)

  • NEVER run git push --force, git reset --hard, git checkout main, rm -rf, git branch -D autonomously.
  • NEVER write to source repos at <REPO_ROOT>, <DOCS_REPO>, etc. — these are READ-ONLY from this skill's perspective. Only the workspace at <WORKSPACE_ROOT> is writeable.
  • NEVER edit adrs/drafts/* — those are maintainer-authored design docs.
  • NEVER edit CLAUDE.md — it is the governance surface; the maintainer hand-authors changes.
  • APPROACH.md is the deliverable — update it when methodology insights land (new failure modes, new rules, new layers, new probe classes, new bootstrapping signals). Do NOT rewrite the methodology unilaterally; do NOT delete sections; preserve revision history. Substantive additions land alongside an LSN entry that triggered them.
  • If a tool call hits a permission prompt → that's a settings.local.json gap; halt the loop (don't try to bypass).

Resumption after a halt

When the maintainer resumes after a halt:

  1. Read state/sprint-themes.yaml to see the last-attempted theme + its status.
  2. Read state/batch-{theme_id}-trace.yaml (if present) for the per-agent outcomes.
  3. Manually fix the root cause (broken sidecar, conflicted yaml, etc.).
  4. Either reset the theme to pending (retry) or leave as blocked and skip to the next.
  5. Resume /loop 75m /next-batch.

Cross-references

  • adrs/drafts/feature-anchored-ontology.md rev 2 — the methodology this skill implements at scale.
  • state/sprint-themes.yaml — the priority queue this skill consumes.
  • lineage/_extractor/registry-shard/shard.py — canonical sidecar/index shapes.
  • lineage/_extractor/registry-shard/coverage.py — coverage refresh.
  • lineage/_extractor/registry-shard/merge_deltas.py — uniform delta-merge helper (referenced from this skill; created by slice-9 follow-up).
  • lineage/odd-platform/investigator-log.md (batch H entry) — template for the per-batch log entry this skill appends.
  • .claude/agents/file-analyser.md, .claude/agents/{concept-merger,adr-archaeologist,doc-gap-finder,test-coverage-mapper,feature-flow-builder}.md — the subagent contracts.
  • playbooks/registry-search-spawn.md — the dedup protocol the reducers follow.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.