building-data-pipelines
Skill1 starsChanged 16 days agoMITView on GitHub
Use when building an ingestion or ETL/ELT pipeline that moves data between systems — keep extraction, transformation, and loading separate, and make runs idempotent and backfillable. Triggers include "build a pipeline", "ingest data from X", "sync X to the warehouse", "ETL/ELT", or a scheduled job that extracts then transforms then loads.
In this repository44 other files it gives its agents.
- CLAUDE.mdCLAUDE.md
- bigquery-superpowersSkill
- data-quality-checksSkill
- dbt-incremental-modelsSkill
- dimensional-modelingSkill
- reviewing-sqlSkill
- writing-dbt-modelsSkill
- arenaSkill
- authority-boundarySkill
- beadflowSkill
- blast-radiusSkill
- control-uiSkill
- create-prSkill
- create-verification-skillSkill
- deslopSkill
- diagnosing-bugsSkill
- feedback-to-infrastructureSkill
- finding-concurrency-bugsSkill
- fixing-and-merging-prsSkill
- foundational-thinkingSkill
- grilling-frontend-prototypingSkill
- improve-harnessSkill
- maintain-verification-skillSkill
- make-operations-idempotentSkill
- metamorphic-testingSkill
- minimize-reader-loadSkill
- preflightSkill
- prototypeSkill
- recon-before-actionSkill
- redesign-from-first-principlesSkill
- resolve-merge-conflictsSkill
- tddSkill
- whySkill
- writing-great-skillsSkill
- catch-me-upSkill
- cross-critiqueSkill
- definition-of-doneSkill
- experience-firstSkill
- grillingSkill
- grill-with-docsSkill
- handoffSkill
- never-block-on-the-humanSkill
- show-me-your-workSkill
- unslopSkill
The file
We split this skill into the rules it gives an agent
3 rules. About 0 are common, 2 are distinctive, and 1 is very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈2
Distinctive · estimate
This author's own choices. Worth a look.
1
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 50
Land raw, immutable: Don't overwrite raw extracts.
- Distinctive: line 52
Validate at the boundary: Run data-quality-checks on landing and before promotion.
- Very specific: line 60
Cache what worked: After solving a non-trivial integration, append the method/endpoint/auth/pagination/gotchas to a PATTERNS.md so the next …
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Building Data Pipelines
A pipeline you can re-run without fear, backfill cleanly, and reason about one stage at a time is worth ten clever scripts. Two principles do most of the work: separate the stages, and make every run idempotent.
Separate Extract → Land → Transform → Load
The source system is for extraction only. Get data out fast, process it elsewhere. Never run transformation logic inside the extraction loop.
- Extract — pull raw data using the highest-leverage approach available
(bulk export/dump > cached state > bulk API > per-record). See
recon-before-action for the
speed hierarchy; don't hand-roll pagination when a
bq extractor vendor dump exists. - Land — write the raw extract to durable storage (object store, file, or a raw warehouse table) before transforming. Never hold a large dataset only in memory. Landed raw data is your replay buffer: you can re-transform without re-hitting the source.
- Transform — filter, dedupe, enrich, validate in your processing layer (SQL/dbt/pandas) reading from the landed data, not from the source.
- Load — write the finished result to its destination.
Keeping these separate means a transform bug doesn't force a re-extract, and a source outage doesn't corrupt your marts.
Make every run idempotent
Re-running the pipeline (after a failure, or on overlapping windows) must produce the same result, not duplicates. This is a metamorphic relation — running twice equals running once.
- Upsert/merge on a key, or replace-by-partition — never blind-append.
- Use a watermark (high-water mark on a load timestamp / sequence) to pull only new data, with a lookback window so late-arriving rows aren't missed. Combine with key-based upsert so re-seen rows update instead of duplicate (same discipline as dbt-incremental-models).
- Make backfills first-class. Parameterize the run by date/window so you can reprocess history with the same code path as the daily run. A pipeline that can't backfill is a pipeline that can't recover.
Operational must-haves
- Land raw, immutable. Don't overwrite raw extracts; partition them by load date so you can always replay.
- Validate at the boundary. Run data-quality-checks on landing and before promotion; fail loud, quarantine bad batches.
- Right-size the machinery. Per recon-before-action's complexity gate: a one-off pull is a script, not a Prefect flow. Don't build orchestration for work that runs twice.
- Parallelize per-record work in bounded groups (
asyncio.gather/Promise.all), never sequentially — and if you do, check finding-concurrency-bugs. - Cache what worked. After solving a non-trivial integration, append the
method/endpoint/auth/pagination/gotchas to a
PATTERNS.mdso the next pipeline skips the recon.
Red flags — STOP
- "I'll transform the rows as I page through the API" → no. Land first, transform from the landing zone.
- "It appends, so a re-run will just add the new rows" → you'll duplicate. Upsert on a key or replace the partition.
- "I'll backfill by editing the script for the old dates" → parameterize it so backfill and daily share one code path.
Original skill — distilled from Kevin's Extract→Land→Transform→Load protocol.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

