build-pipeline
Skill4 starsChanged 7 days agoMITView on GitHub
Design and build an ETL/ELT pipeline or a summary/aggregate table — from source identification through layered transformations, data-quality gates, scheduling, and documentation. Use for new data ingestion or taming huge datasets into fast queryable tables.
2 other projects use a version of this.
Adopted by
Most recently taken up by
- tabler/tablertheir own version, first seen Oct 8, 2026
- AgentJDrew/agent-team-factorytheir own version, first seen Jul 6, 2026
Other versions2 projects use a version of this.
Word for word · 0
None found.
Edited · 0
None found.
Wrote their own version · 2
- tabler/tabler★ 42ksame name
- AgentJDrew/agent-team-factory★ 0same name
How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.
In this repository21 other files it gives its agents.
- CLAUDE.mdCLAUDE.md
- analyzeSkill
- build-dashboardSkill
- build-modelSkill
- causal-analysisSkill
- connect-dataSkill
- define-kpisSkill
- evalSkill
- experimentSkill
- forecastSkill
- health-checkSkill
- investigate-metricSkill
- make-deliverableSkill
- powerbiSkill
- privacy-reviewSkill
- research-domainSkill
- scorecardSkill
- setup-teamSkill
- tableauSkill
- triageSkill
- upgradeSkill
The file
We split this skill into the rules it gives an agent
1 rule. About 0 are common, 0 are distinctive, and 1 is very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈0
Distinctive · estimate
This author's own choices. Worth a look.
1
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 14
Check for existing work: Search pipelines/ and the warehouse for tables that already cover this.
≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Build Pipeline / Summary Table
Coordinates data-engineer (ingestion, raw→staging) and analytics-engineer (marts, summary tables). Args may name the source or desired table, e.g. /build-pipeline daily revenue summary from billing DB.
Procedure
- Scope. Establish: What data? From where ({{SOURCE_SYSTEMS}} — see
knowledge/data-sources.md)? Who consumes the result and at what grain/freshness? Incremental history needed or current-state only? One clarifying round max; default to daily incremental refresh at the finest grain consumers need. - Check for existing work. Search
pipelines/and the warehouse for tables that already cover this. Extending beats duplicating. - Design doc first. Write
pipelines/<name>/DESIGN.md(half a page): source → layers → target table(s) with grain, refresh strategy (full/incremental + watermark), schedule, quality checks, estimated volume/cost. When the target is a mart, design it with the four steps (standards/data-modeling-standards.md): name the business process → declare the grain ("one row per ___") → dimensions → facts; and pick the fact type (transaction / periodic snapshot / accumulating snapshot / factless) that matches the process. For large or costly builds, confirm the design with the user before implementing. - Implement, layer by layer (per
standards/sql-and-data-standards.md):raw: verbatim landing, load metadata columns (_loaded_at,_source).staging: typed, renamed to convention, deduplicated, one view/table per source entity.marts/agg: the consumer-facing table. Header comment documents purpose, grain, refresh, owner. Use surrogate keys and conformed dimensions (reuse the shareddim_customer/dim_date, don't fork a new one). For dimensions whose history matters, set the SCD strategy per attribute — Type 2 (effective-dated rows) where history is needed, Type 1 (overwrite) where it isn't — and note it in the header.- Idempotent writes only (MERGE / delete-insert by partition). Test on a limited date range first.
- Quality gates. End-of-pipeline checks that fail loudly: source-vs-target row reconciliation, key uniqueness at declared grain, null thresholds on critical columns, freshness check. Save as runnable queries in
pipelines/<name>/checks/. - Backfill historical data if required — chunked by partition, verifying counts per chunk.
- Schedule on {{ORCHESTRATOR}}; if none, document the exact command and recommended cron in the design doc.
- Document & hand off. Update
pipelines/README.md(inventory + runbook) andknowledge/data-sources.md(new tables, grains, quirks discovered). In the same commit: add or update the table's section in its domain reference doc underknowledge/domains/(grain, required filters, when not to use it, gotchas), deprecate any near-duplicate table this one replaces, and havegovernance-officerclassify any new column holding personal data inknowledge/governance-policy.mdbefore the table is announced. A pipeline whose documentation arrives later is a pipeline the team will query wrongly in the meantime. Tell the user: what was built, sample of the output, refresh schedule, and cost/runtime observed.
Created by Colin Beck — https://www.linkedin.com/in/beckcolin/
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

