agentleFS
AgentJDrewAgentJDrew/agent-team-factory/.claude/skills/build-pipeline/SKILL.md

build-pipeline

Skill0 starsChanged 4 months agoMITView on GitHub

Build a multi-stage pipeline with gates, kill conditions, and state persistence. Use when building a pipeline, creating a multi-stage workflow, setting up a validation or review pipeline, or designing a stage-gated process with sub-stage gates.

2 other projects use a version of this.

Adopted by

Most recently taken up by

  • Other versions2 projects use a version of this.

    Word for word · 0

    None found.

    Edited · 0

    None found.

    Wrote their own version · 2

    1. tabler/tabler★ 42ksame name
    2. link7373/agentic-bi-team★ 4same name

    How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.

  • In this repository5 other files it gives its agents.

The file

We split this skill into the rules it gives an agent

10 rules. About 0 are common, 8 are distinctive, and 2 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈8

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 2

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Every production pipeline has these components:

    line 13
  2. Distinctive:

    Every stage must define when a candidate DIES:

    line 25
  3. Distinctive:

    Write state after EVERY stage (not just at the end)

    line 32
  4. Very specific:

    Read research/agent-architecture-playbook.md section 1.3 (Pipeline pattern) and section 5 (Quality Control)

    line 50
  5. Very specific:

    Read research/case-study-compendium.md for pipeline failure modes from production (stage skipping, single-point dependency, cost modeling …

    line 51
  6. Distinctive:

    After the first substantive work stage (catch early errors)

    line 97
  7. Distinctive:

    Before any irreversible action (deployment, publication, external delivery)

    line 98
  8. Distinctive:

    No state persistence (crash = restart from scratch)

    line 134
  9. Distinctive:

    No backward propagation (failures don't flow upstream)

    line 137
  10. Distinctive:

    No batch retrospective (never learning from pipeline runs)

    line 138

≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Build Pipeline

Trigger

"Build a pipeline", "create a multi-stage workflow", "I need a validation pipeline", "set up a stage-gated process".

What a Pipeline Needs

Every production pipeline has these components:

1. Stage Definitions

Each stage needs:

  • Name and number (sequential, with sub-stages like 1.5 for intermediate gates)
  • Owner (which agent or function runs this stage)
  • Input (what data enters this stage)
  • Process (what happens)
  • Gate (binary PASS/KILL condition with explicit threshold)
  • Output (what leaves this stage for the next)

2. Kill Conditions (Per Stage)

Every stage must define when a candidate DIES:

  • Hard kills: automatic, no judgment needed (threshold breach)
  • Soft kills: spawn agent for judgment, but bias toward kill
  • Advisory flags: noted but don't block (inform downstream)

3. State Persistence

Pipeline state must survive crashes:

  • Write state after EVERY stage (not just at the end)
  • Atomic writes: tmp file -> rename (never corrupt the state file)
  • Include: current stage, strategy status (ACTIVE/KILLED), metrics per stage

4. Risk Gates

Second-pass check at every stage:

  • Inline logic handles the fast path
  • risk_gates.run_gates() catches what inline logic misses
  • Worse action wins (KILL > VETO > HOLD > ALERT > PASS)

5. Supabase Tracking

Log every event:

  • log_stage() on entry and completion
  • kill_strategy() with root_cause_tag on death
  • Batch retrospective on pipeline completion

Research (Read Before Building)

  1. Read research/agent-architecture-playbook.md section 1.3 (Pipeline pattern) and section 5 (Quality Control)
  2. Read research/case-study-compendium.md for pipeline failure modes from production (stage skipping, single-point dependency, cost modeling errors)
  3. standards/floor-spec.md §8 requires every staged pipeline to have a named review gate whose holder carries BLOCK or VETO authority, with verifier ≠ author — design the review points below to satisfy that.

Build Process

Step 1: Define Stages

Ask user:

  1. What are the stages in your process?
  2. What kills a candidate at each stage? (specific threshold)
  3. What data flows between stages?
  4. Where do agents need to review (not just automate)?

Step 2: Generate Stage Functions

For each stage, generate a function following this pattern:

def stage_N_name(candidates: list[dict], run_id: str) -> tuple[list[dict], list[str]]:
    """[Stage N]: [Description]. Gate: [threshold]."""
    survivors, killed = [], []
    for item in candidates:
        metrics = compute_stage_metrics(item)
        if kill_condition(metrics):
            log_kill(item["name"], run_id, stage="N", reason="...", metrics=metrics)
            killed.append(item["name"])
        else:
            log_pass(item["name"], run_id, stage="N", metrics=metrics)
            survivors.append(item)
    return survivors, killed

Adapt compute_stage_metrics, kill_condition, log_kill, and log_pass to the domain. For database-backed teams, add write-through to the persistence layer.

Step 3: Register Stages and Review Points

STAGES = {"1": stage_1, "2": stage_2, "2.5": stage_2_5, ...}
STAGE_ORDER = ["1", "2", "2.5", "3", ...]
REVIEW_POINTS = {
    "2": "Spawn peer review agents for quality check",
    "4": "Spawn domain experts for deep review",
    "N": "Spawn risk/compliance for final gate"
}

Review points are stages where the orchestrator pauses automation and spawns agent teams for human-judgment review. Define these for:

  • After the first substantive work stage (catch early errors)
  • Before any irreversible action (deployment, publication, external delivery)
  • At the final gate (risk/compliance veto)

Step 4: Build Orchestrator

def run_pipeline(candidates: list[dict], run_id: str | None = None) -> dict:
    rid = run_id or generate_run_id()
    alive, killed = list(candidates), []
    for stage in STAGE_ORDER:
        if not alive:
            break
        if stage in REVIEW_POINTS:
            # Pause: spawn review agents, require unanimous APPROVE
            review_result = run_review(alive, rid, stage, REVIEW_POINTS[stage])
            alive = [c for c in alive if review_result[c["name"]] == "APPROVE"]
            killed.extend([c["name"] for c in alive if review_result.get(c["name"]) != "APPROVE"])
        alive, dead = STAGES[stage](alive, rid)
        killed.extend(dead)
        persist_state(rid, stage, alive, killed)  # After EVERY stage
    return {"passed": alive, "killed": killed, "run_id": rid}

Step 5: State Persistence

Write state after every stage (not just at the end):

def persist_state(run_id: str, stage: str, alive: list, killed: list) -> None:
    state = {"run_id": run_id, "stage": stage, "alive": alive, "killed": killed, "timestamp": now()}
    write_atomic(state, path=f"pipeline-state-{run_id}.json")  # tmp -> rename

Pipeline Anti-Patterns (from production pipeline post-mortems)

  1. Skipping stages without explicit approval
  2. Using one-size-fits-all thresholds (profile-aware is better)
  3. Checking only the final output, not intermediate stages
  4. No state persistence (crash = restart from scratch)
  5. Advisory flags instead of binary gates (ambiguity = missed kills)
  6. Risk gates exist but are never called (orphaned enforcement)
  7. No backward propagation (failures don't flow upstream)
  8. No batch retrospective (never learning from pipeline runs)

Parallel Fan-Out Note

For any multi-item batch (N ≥ 2 posts, audits, reviews, analyses, cleanups), spawn one top-level agent per item, running in parallel, rather than routing the batch through a single orchestrator that processes items serially or via internal sub-agents invisible to the user.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.