agentleFS
mikeparcewskimikeparcewski/wicked-garden/skills/data-engineer/SKILL.md

wicked-garden-data-engineer

Skill9 starsChanged 7 months agoMITView on GitHub

ETL pipeline design, data quality assessment, schema validation, and performance optimization as a delegated worker skill. Use when: designing or reviewing ETL/ELT pipelines, assessing dataset quality (completeness, uniqueness, validity, consistency, timeliness), validating data against schemas, optimizing data-processing performance, or recording data-engineering findings on an active task. For inline (non-delegated) data work, use the wicked-garden-data skill's sub-actions instead.

  • Installs packages

The file

We split this skill into the rules it gives an agent

8 rules. About 0 are common, 8 are distinctive, and 0 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈8

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 0

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    If none of these is available, or Python 3 is missing, skip the script-backed step, say so, and follow the manual alternative where one is …

    line 14
  2. Distinctive:

    Before manual work, leverage available tools:

    line 19
  3. Distinctive:

    When designing ETL/ELT pipelines:

    line 30
  4. Distinctive:

    Use the schema validator script:

    line 80
  5. Distinctive:

    Document breaking vs non-breaking changes

    line 92
  6. Distinctive:

    When reviewing existing pipelines:

    line 177
  7. Distinctive:

    Always prioritize actionable insights:

    line 188
  8. Distinctive:

    Schema-first: Always define schemas before processing

    line 216

≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Data Engineer

You design and review data pipelines with a focus on quality, performance, and maintainability.

Runtime

Script-backed steps use the wicked-garden launcher: wicked-garden run <plugin-root-relative path, e.g. scripts/…> [args]. It is on PATH after npm i -g wicked-garden; otherwise use npx wicked-garden run …; inside a wicked-crew run it is "$WICKED_GARDEN_ROOT/scripts/wicked-garden". If none of these is available, or Python 3 is missing, skip the script-backed step, say so, and follow the manual alternative where one is given next to it — never invent the script's output. Relative paths in this skill are relative to the directory that contains this SKILL.md.

First Strategy: Use wicked-* Ecosystem

Before manual work, leverage available tools:

  • wicked-garden-data (analyze sub-action): For data profiling and SQL queries via DuckDB
  • wicked-garden-search: Find existing pipeline code
  • Native tasks: TaskCreate/TaskUpdate with metadata={event_type, chain_id, source_agent, phase} track data quality issues
  • wicked-garden-mem (recall action): Recall past pipeline patterns

Core Responsibilities

1. Pipeline Design

When designing ETL/ELT pipelines:

Check existing patterns:

Grep "pipeline|etl|transform" {target}

Design checklist:

  • Data sources identified and accessible
  • Schema evolution strategy defined
  • Error handling and retry logic
  • Idempotency and reprocessing support
  • Monitoring and alerting plan
  • Data quality checks embedded
  • Performance optimization strategy
  • Cost estimation completed

Output format:

## Pipeline Design: {name}

### Architecture
- **Pattern**: [Batch/Streaming/Hybrid]
- **Orchestration**: [Airflow/Dagster/Prefect/Other]
- **Storage**: [Data Lake/Warehouse/Lakehouse]

### Data Flow
1. **Source**: {description}
2. **Extract**: {method and frequency}
3. **Transform**: {key transformations}
4. **Load**: {destination and format}

### Quality Gates
- **Source validation**: {checks}
- **Transform validation**: {checks}
- **Load validation**: {checks}

### Performance
- **Expected volume**: {records/day}
- **Processing time**: {estimate}
- **Cost estimate**: {$/month}

### Risk Assessment
- **High**: {critical risks}
- **Medium**: {moderate risks}
- **Mitigation**: {strategies}

2. Schema Validation

Use the schema validator script:

wicked-garden run scripts/data/schema_validator.py \
  --schema schemas/expected.json \
  --data data/actual.csv

Schema design principles:

  • Explicit types (avoid variant unless necessary)
  • Clear nullability contracts
  • Consistent naming conventions
  • Version schemas explicitly
  • Document breaking vs non-breaking changes

3. Data Quality Assessment

Profile datasets using:

wicked-garden run scripts/data/data_profiler.py \
  --input data/sample.csv \
  --output profile.json

Quality dimensions:

  • Completeness: Null rates per column
  • Uniqueness: Duplicate detection
  • Validity: Type conformance and constraints
  • Consistency: Cross-field validation
  • Timeliness: Freshness metrics

Report format:

## Data Quality Report

**Dataset**: {name}
**Rows**: {count}
**Columns**: {count}

### Quality Metrics
| Dimension | Score | Issues |
|-----------|-------|--------|
| Completeness | {%} | {null columns} |
| Uniqueness | {%} | {duplicate rate} |
| Validity | {%} | {constraint violations} |

### Critical Issues
- {Issue with severity and impact}

### Recommendations
1. {Prioritized action items}

4. Performance Optimization

Review checklist:

  • Appropriate partitioning strategy
  • Efficient file formats (Parquet > CSV)
  • Pushdown predicates to sources
  • Incremental processing where possible
  • Proper indexing on lookup columns
  • Batch size optimization
  • Parallelism configuration

Profiling queries:

-- Find largest tables
SELECT table_name, row_count, size_bytes
FROM information_schema.tables
ORDER BY size_bytes DESC;

-- Identify slow queries
SELECT query_text, execution_time
FROM query_history
WHERE execution_time > 60
ORDER BY execution_time DESC;

5. Record findings on the active task

Append the findings to the current task's description — the harness's task list where it has one (the wicked-garden-workflow skill's refs/integration.md carries the field list and the harness-specific Hand-off), else your working notes:

[data-engineer] Pipeline Review

**Architecture**: {summary}
**Quality Score**: {score}/100

### Critical Findings
- {finding}

### Recommendations
1. {action item with priority}

**Confidence**: {HIGH|MEDIUM|LOW}

Pipeline Review Guidelines

When reviewing existing pipelines:

  1. Read pipeline code: Understand orchestration logic
  2. Check error handling: How failures are managed
  3. Validate schemas: Are schemas versioned and enforced?
  4. Assess monitoring: What metrics are tracked?
  5. Review testing: Are there data quality tests?
  6. Check documentation: Is the pipeline well-documented?

Output Structure

Always prioritize actionable insights:

## Data Engineering Assessment

**Target**: {what was reviewed}
**Type**: [Pipeline Design|Schema Review|Quality Assessment]

### Summary
{2-3 sentence overview}

### Findings
| Priority | Finding | Impact | Effort |
|----------|---------|--------|--------|
| P1 | {critical} | HIGH | {S/M/L} |

### Recommendations
1. **{Action}** - {rationale and expected outcome}

### Next Steps
- {Immediate action}
- {Follow-up work}

**Confidence**: {HIGH|MEDIUM|LOW}

Quality Standards

  • Schema-first: Always define schemas before processing
  • Fail fast: Validate early, fail loudly
  • Idempotent: Pipelines should be rerunnable
  • Observable: Emit metrics and logs at every stage
  • Tested: Data quality tests are non-negotiable

Dispatch

Worker skill, reached by its name — wicked-garden-data-engineer — through a Hand-off from the wicked-garden-data router or any caller. The pre-v12.25 subagent-delegation form is retired with the fork frontmatter.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.