agentleFS
link7373link7373/agentic-bi-team/.claude/skills/build-pipeline/SKILL.md

build-pipeline

Skill4 starsChanged 7 days agoMITView on GitHub

Design and build an ETL/ELT pipeline or a summary/aggregate table — from source identification through layered transformations, data-quality gates, scheduling, and documentation. Use for new data ingestion or taming huge datasets into fast queryable tables.

2 other projects use a version of this.

Adopted by

Most recently taken up by

The file

We split this skill into the rules it gives an agent

1 rule. About 0 are common, 0 are distinctive, and 1 is very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈0

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 1

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    Check for existing work: Search pipelines/ and the warehouse for tables that already cover this.

    line 14

≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Build Pipeline / Summary Table

Coordinates data-engineer (ingestion, raw→staging) and analytics-engineer (marts, summary tables). Args may name the source or desired table, e.g. /build-pipeline daily revenue summary from billing DB.

Procedure

  1. Scope. Establish: What data? From where ({{SOURCE_SYSTEMS}} — see knowledge/data-sources.md)? Who consumes the result and at what grain/freshness? Incremental history needed or current-state only? One clarifying round max; default to daily incremental refresh at the finest grain consumers need.
  2. Check for existing work. Search pipelines/ and the warehouse for tables that already cover this. Extending beats duplicating.
  3. Design doc first. Write pipelines/<name>/DESIGN.md (half a page): source → layers → target table(s) with grain, refresh strategy (full/incremental + watermark), schedule, quality checks, estimated volume/cost. When the target is a mart, design it with the four steps (standards/data-modeling-standards.md): name the business process → declare the grain ("one row per ___") → dimensions → facts; and pick the fact type (transaction / periodic snapshot / accumulating snapshot / factless) that matches the process. For large or costly builds, confirm the design with the user before implementing.
  4. Implement, layer by layer (per standards/sql-and-data-standards.md):
    • raw: verbatim landing, load metadata columns (_loaded_at, _source).
    • staging: typed, renamed to convention, deduplicated, one view/table per source entity.
    • marts/agg: the consumer-facing table. Header comment documents purpose, grain, refresh, owner. Use surrogate keys and conformed dimensions (reuse the shared dim_customer/dim_date, don't fork a new one). For dimensions whose history matters, set the SCD strategy per attribute — Type 2 (effective-dated rows) where history is needed, Type 1 (overwrite) where it isn't — and note it in the header.
    • Idempotent writes only (MERGE / delete-insert by partition). Test on a limited date range first.
  5. Quality gates. End-of-pipeline checks that fail loudly: source-vs-target row reconciliation, key uniqueness at declared grain, null thresholds on critical columns, freshness check. Save as runnable queries in pipelines/<name>/checks/.
  6. Backfill historical data if required — chunked by partition, verifying counts per chunk.
  7. Schedule on {{ORCHESTRATOR}}; if none, document the exact command and recommended cron in the design doc.
  8. Document & hand off. Update pipelines/README.md (inventory + runbook) and knowledge/data-sources.md (new tables, grains, quirks discovered). In the same commit: add or update the table's section in its domain reference doc under knowledge/domains/ (grain, required filters, when not to use it, gotchas), deprecate any near-duplicate table this one replaces, and have governance-officer classify any new column holding personal data in knowledge/governance-policy.md before the table is announced. A pipeline whose documentation arrives later is a pipeline the team will query wrongly in the meantime. Tell the user: what was built, sample of the output, refresh schedule, and cost/runtime observed.

Created by Colin Beck — https://www.linkedin.com/in/beckcolin/

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.