data-pipeline-architect
Skill17 starsChanged 5 months agoApache-2.0Archived repositoryView on GitHub
Designs ETL/ELT data pipelines with proper extraction, transformation, and loading patterns, including orchestration, error handling, and data quality validation.
5 other projects use a version of this.
Adopted by
Most recently taken up by
- jaskirat1616/grok-skillstheir own version, first seen Nov 1, 2026
- JayRHa/agent-skillstheir own version, first seen Oct 15, 2026
- poorvith-mp/skills-developertheir own version, first seen Aug 19, 2026
- JayRHa/AgentSkillstheir own version, first seen Jun 9, 2026
Other versions5 projects use a version of this.
Word for word · 1
- organvm-iv-taxis/a-i--skills★ 17100% same
Edited · 0
None found.
Wrote their own version · 4
- jaskirat1616/grok-skills★ 22same name
- JayRHa/agent-skills★ 12same name
- JayRHa/AgentSkills★ 12same name
- poorvith-mp/skills-developer★ 0same name
How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.
In this repository170 other files it gives its agents.
- AGENTS.mdAGENTS.md
- docs/AGENTS.mdAGENTS.md
- CLAUDE.mdCLAUDE.md
- .github/copilot-instructions.mdCopilot instructions
- algorithmic-artSkill
- audio-engineering-patternsSkill
- canvas-designSkill
- creative-writing-craftSkill
- generative-art-algorithmsSkill
- generative-art-deploymentSkill
- generative-music-composerSkill
- interactive-theatre-designerSkill
- modular-synthesis-philosophySkill
- movement-notation-systemsSkill
- narratological-algorithmsSkill
- natural-center-extractionSkill
- react-three-fiber-patternsSkill
- reality-tv-narrative-analyzerSkill
- script-analysis-dramaturgicalSkill
- theme-factorySkill
- three-js-interactive-builderSkill
- corpus-persona-extractionSkill
- data-ingestion-pipelineSkill
- data-storytelling-analystSkill
- market-gap-analysisSkill
- ml-experiment-trackerSkill
- prompt-atom-formalizationSkill
- sql-query-optimizerSkill
- systemic-product-analystSkill
- time-series-analystSkill
- accessibility-patternsSkill
- agent-testing-patternsSkill
- api-design-patternsSkill
- artifacts-builderSkill
- backend-implementation-patternsSkill
- browser-extension-patternsSkill
- cli-tool-designSkill
- code-refactoring-patternsSkill
- code-review-checklistSkill
- coding-standards-enforcerSkill
- community-platform-patternsSkill
- configuration-managementSkill
- continuous-learning-agentSkill
- data-backup-patternsSkill
- database-migration-patternsSkill
- deployment-cicdSkill
- docker-containerizationSkill
- dotfile-systems-architectSkill
- error-handling-logging-patternsSkill
- fastapi-patternsSkill
- feature-workflow-orchestratorSkill
- frontend-design-systemsSkill
- fullstack-starter-packSkill
- gcp-resource-optimizerSkill
- htmx-interaction-patternsSkill
- iterative-code-explorationSkill
- json-schema-designSkill
- mcp-builderSkill
- mcp-server-orchestratorSkill
- mobile-platform-architectSkill
- The first 60 shown.
The file
We split this skill into the rules it gives an agent
6 rules. 0 are common, 6 are distinctive, and 0 are very specific to this project.
0
Common
Most of the 3 edited versions we read keep these.
6
Distinctive
This author's own choices. Worth a look.
0
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 54
When target has limited compute
- Distinctive: line 55
When transformation reduces data volume significantly
- Distinctive: line 56
When sensitive data must be masked before landing
- Distinctive: line 61
When raw data preservation is needed
- Distinctive: line 62
When transformations change frequently
- Distinctive: line 141
Type 2: Add row with validity dates
“Common” is measured: kept by at least 6 in 10 of the 3 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Data Pipeline Architect
This skill provides guidance for designing robust, scalable data pipelines that move data reliably from sources to destinations.
Core Competencies
- ETL vs ELT: Traditional Extract-Transform-Load vs modern Extract-Load-Transform patterns
- Orchestration: Airflow, Dagster, Prefect, dbt for workflow management
- Data Quality: Validation, monitoring, lineage tracking
- Scalability: Batch vs streaming, partitioning, parallelization
Pipeline Design Process
1. Requirements Analysis
To begin pipeline design, gather:
- Source systems and data formats (APIs, databases, files, streams)
- Target destinations (data warehouse, lake, lakehouse)
- Freshness requirements (real-time, hourly, daily)
- Data volume and velocity estimates
- Quality and compliance requirements
2. Architecture Selection
Batch Pipelines - For periodic bulk processing:
- Schedule-driven (hourly, daily, weekly)
- Higher latency tolerance
- Simpler error recovery (re-run entire batch)
- Tools: Airflow, dbt, Spark
Streaming Pipelines - For real-time requirements:
- Event-driven processing
- Sub-second to minute latency
- Complex state management
- Tools: Kafka, Flink, Spark Streaming
Hybrid Approaches - Lambda or Kappa architecture:
- Batch layer for completeness
- Speed layer for low latency
- Serving layer for queries
3. ETL vs ELT Decision
ETL (Transform before Load):
- When target has limited compute
- When transformation reduces data volume significantly
- When sensitive data must be masked before landing
- Legacy data warehouse patterns
ELT (Transform after Load):
- Modern cloud warehouses with cheap compute
- When raw data preservation is needed
- When transformations change frequently
- dbt-style transformations in warehouse
4. Pipeline Components
Extraction Layer:
- Full extraction vs incremental (CDC, timestamp-based)
- API pagination and rate limiting
- Connection pooling and retry logic
- Schema detection and drift handling
Transformation Layer:
- Data cleansing and standardization
- Business logic application
- Aggregation and denormalization
- Type casting and null handling
Loading Layer:
- Upsert strategies (merge, delete+insert)
- Partitioning schemes (time, hash, range)
- Index management
- Transaction boundaries
5. Error Handling Patterns
┌─────────────────────────────────────────────────────────┐
│ Pipeline Execution │
├─────────────────────────────────────────────────────────┤
│ ┌─────────┐ ┌───────────┐ ┌──────────┐ │
│ │ Extract │───▶│ Transform │───▶│ Load │ │
│ └────┬────┘ └─────┬─────┘ └────┬─────┘ │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ ┌─────────┐ ┌───────────┐ ┌──────────┐ │
│ │ Retry │ │ Dead Letter│ │ Rollback │ │
│ │ w/Backoff│ │ Queue │ │ Checkpoint│ │
│ └─────────┘ └───────────┘ └──────────┘ │
└─────────────────────────────────────────────────────────┘- Retry with backoff: Transient failures (network, rate limits)
- Dead letter queues: Poison messages that can't be processed
- Checkpointing: Resume from last successful point
- Idempotency: Safe to re-run without duplicates
6. Data Quality Framework
Implement checks at each stage:
| Stage | Check Type | Example |
| Extract | Completeness | Row count matches source |
| Extract | Freshness | Data timestamp within SLA |
| Transform | Validity | Values in expected ranges |
| Transform | Uniqueness | Primary keys unique |
| Load | Reconciliation | Target matches source totals |
| Load | Integrity | Foreign keys valid |
7. Monitoring and Observability
Essential metrics to track:
- Pipeline duration and trends
- Row counts at each stage
- Error rates and types
- Data freshness (time since last successful run)
- Resource utilization
Alert on:
- SLA breaches (data not fresh)
- Anomalous row counts (±20% from baseline)
- Schema changes in sources
- Repeated failures
Common Patterns
Slowly Changing Dimensions (SCD)
- Type 1: Overwrite (no history)
- Type 2: Add row with validity dates
- Type 3: Previous value column
- Type 4: History table
Incremental Processing
-- Timestamp-based incremental
SELECT * FROM source
WHERE updated_at > {{ last_run_timestamp }}
-- CDC-based (Change Data Capture)
-- Captures inserts, updates, deletes from transaction logIdempotent Loads
-- Delete + Insert pattern
DELETE FROM target WHERE date_partition = '2024-01-15';
INSERT INTO target SELECT * FROM staging WHERE date_partition = '2024-01-15';
-- Merge/Upsert pattern
MERGE INTO target t
USING staging s ON t.id = s.id
WHEN MATCHED THEN UPDATE SET ...
WHEN NOT MATCHED THEN INSERT ...References
references/orchestration-patterns.md- Airflow, Dagster, Prefect patternsreferences/data-quality-checks.md- Validation frameworks and rulesreferences/pipeline-templates.md- Common pipeline architectures
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

