build-pipeline
Skill0 starsChanged 4 months agoMITView on GitHub
Build a multi-stage pipeline with gates, kill conditions, and state persistence. Use when building a pipeline, creating a multi-stage workflow, setting up a validation or review pipeline, or designing a stage-gated process with sub-stage gates.
2 other projects use a version of this.
Adopted by
Most recently taken up by
- link7373/agentic-bi-teamtheir own version, first seen Nov 10, 2026
- tabler/tablertheir own version, first seen Oct 8, 2026
Other versions2 projects use a version of this.
Word for word · 0
None found.
Edited · 0
None found.
Wrote their own version · 2
- tabler/tabler★ 42ksame name
- link7373/agentic-bi-team★ 4same name
How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.
In this repository5 other files it gives its agents.
- CLAUDE.mdCLAUDE.md
- audit-teamSkill
- build-agentSkill
- build-skillSkill
- build-teamSkill
The file
We split this skill into the rules it gives an agent
10 rules. About 0 are common, 8 are distinctive, and 2 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈8
Distinctive · estimate
This author's own choices. Worth a look.
2
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 13
Every production pipeline has these components:
- Distinctive: line 25
Every stage must define when a candidate DIES:
- Distinctive: line 32
Write state after EVERY stage (not just at the end)
- Very specific: line 50
Read research/agent-architecture-playbook.md section 1.3 (Pipeline pattern) and section 5 (Quality Control)
- Very specific: line 51
Read research/case-study-compendium.md for pipeline failure modes from production (stage skipping, single-point dependency, cost modeling …
- Distinctive: line 97
After the first substantive work stage (catch early errors)
- Distinctive: line 98
Before any irreversible action (deployment, publication, external delivery)
- Distinctive: line 134
No state persistence (crash = restart from scratch)
- Distinctive: line 137
No backward propagation (failures don't flow upstream)
- Distinctive: line 138
No batch retrospective (never learning from pipeline runs)
≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Build Pipeline
Trigger
"Build a pipeline", "create a multi-stage workflow", "I need a validation pipeline", "set up a stage-gated process".
What a Pipeline Needs
Every production pipeline has these components:
1. Stage Definitions
Each stage needs:
- Name and number (sequential, with sub-stages like 1.5 for intermediate gates)
- Owner (which agent or function runs this stage)
- Input (what data enters this stage)
- Process (what happens)
- Gate (binary PASS/KILL condition with explicit threshold)
- Output (what leaves this stage for the next)
2. Kill Conditions (Per Stage)
Every stage must define when a candidate DIES:
- Hard kills: automatic, no judgment needed (threshold breach)
- Soft kills: spawn agent for judgment, but bias toward kill
- Advisory flags: noted but don't block (inform downstream)
3. State Persistence
Pipeline state must survive crashes:
- Write state after EVERY stage (not just at the end)
- Atomic writes: tmp file -> rename (never corrupt the state file)
- Include: current stage, strategy status (ACTIVE/KILLED), metrics per stage
4. Risk Gates
Second-pass check at every stage:
- Inline logic handles the fast path
- risk_gates.run_gates() catches what inline logic misses
- Worse action wins (KILL > VETO > HOLD > ALERT > PASS)
5. Supabase Tracking
Log every event:
- log_stage() on entry and completion
- kill_strategy() with root_cause_tag on death
- Batch retrospective on pipeline completion
Research (Read Before Building)
- Read
research/agent-architecture-playbook.mdsection 1.3 (Pipeline pattern) and section 5 (Quality Control) - Read
research/case-study-compendium.mdfor pipeline failure modes from production (stage skipping, single-point dependency, cost modeling errors) standards/floor-spec.md§8 requires every staged pipeline to have a named review gate whose holder carries BLOCK or VETO authority, with verifier ≠ author — design the review points below to satisfy that.
Build Process
Step 1: Define Stages
Ask user:
- What are the stages in your process?
- What kills a candidate at each stage? (specific threshold)
- What data flows between stages?
- Where do agents need to review (not just automate)?
Step 2: Generate Stage Functions
For each stage, generate a function following this pattern:
def stage_N_name(candidates: list[dict], run_id: str) -> tuple[list[dict], list[str]]:
"""[Stage N]: [Description]. Gate: [threshold]."""
survivors, killed = [], []
for item in candidates:
metrics = compute_stage_metrics(item)
if kill_condition(metrics):
log_kill(item["name"], run_id, stage="N", reason="...", metrics=metrics)
killed.append(item["name"])
else:
log_pass(item["name"], run_id, stage="N", metrics=metrics)
survivors.append(item)
return survivors, killedAdapt compute_stage_metrics, kill_condition, log_kill, and log_pass to the domain. For database-backed teams, add write-through to the persistence layer.
Step 3: Register Stages and Review Points
STAGES = {"1": stage_1, "2": stage_2, "2.5": stage_2_5, ...}
STAGE_ORDER = ["1", "2", "2.5", "3", ...]
REVIEW_POINTS = {
"2": "Spawn peer review agents for quality check",
"4": "Spawn domain experts for deep review",
"N": "Spawn risk/compliance for final gate"
}Review points are stages where the orchestrator pauses automation and spawns agent teams for human-judgment review. Define these for:
- After the first substantive work stage (catch early errors)
- Before any irreversible action (deployment, publication, external delivery)
- At the final gate (risk/compliance veto)
Step 4: Build Orchestrator
def run_pipeline(candidates: list[dict], run_id: str | None = None) -> dict:
rid = run_id or generate_run_id()
alive, killed = list(candidates), []
for stage in STAGE_ORDER:
if not alive:
break
if stage in REVIEW_POINTS:
# Pause: spawn review agents, require unanimous APPROVE
review_result = run_review(alive, rid, stage, REVIEW_POINTS[stage])
alive = [c for c in alive if review_result[c["name"]] == "APPROVE"]
killed.extend([c["name"] for c in alive if review_result.get(c["name"]) != "APPROVE"])
alive, dead = STAGES[stage](alive, rid)
killed.extend(dead)
persist_state(rid, stage, alive, killed) # After EVERY stage
return {"passed": alive, "killed": killed, "run_id": rid}Step 5: State Persistence
Write state after every stage (not just at the end):
def persist_state(run_id: str, stage: str, alive: list, killed: list) -> None:
state = {"run_id": run_id, "stage": stage, "alive": alive, "killed": killed, "timestamp": now()}
write_atomic(state, path=f"pipeline-state-{run_id}.json") # tmp -> renamePipeline Anti-Patterns (from production pipeline post-mortems)
- Skipping stages without explicit approval
- Using one-size-fits-all thresholds (profile-aware is better)
- Checking only the final output, not intermediate stages
- No state persistence (crash = restart from scratch)
- Advisory flags instead of binary gates (ambiguity = missed kills)
- Risk gates exist but are never called (orphaned enforcement)
- No backward propagation (failures don't flow upstream)
- No batch retrospective (never learning from pipeline runs)
Parallel Fan-Out Note
For any multi-item batch (N ≥ 2 posts, audits, reviews, analyses, cleanups), spawn one top-level agent per item, running in parallel, rather than routing the batch through a single orchestrator that processes items serially or via internal sub-agents invisible to the user.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

