agentleFS
Sign inStart free
roedyrustamroedyrustam/vibes-plug/skills/data-pipeline-etl-expert/SKILL.md

data-pipeline-etl-expert

Skill75 starsChanged 39 days agoMITView on GitHub

Expert guide for Data Pipelines, ETL/ELT, and Analytics Engineering. Covers dbt, Apache Airflow, Dagster, BigQuery, ClickHouse, and DuckDB / Panduan ahli untuk Data Pipelines, ETL/ELT. Mencakup dbt, Airflow, Dagster, BigQuery, ClickHouse, dan DuckDB.

The file

We split this skill into the rules it gives an agent

9 rules. About 0 are common, 8 are distinctive, and 1 is very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈8

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 1

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    When designing reporting dashboards or analytics infrastructure for a SaaS.

    line 21
  2. Distinctive:

    When moving large volumes of data from transactional databases (PostgreSQL/MySQL) to a data warehouse.

    line 22
  3. Distinctive:

    When the user asks about "dbt", "Airflow", "ELT", or "Analytics Engineering".

    line 23
  4. Distinctive:

    When building local or edge analytics using DuckDB.

    line 24
  5. Distinctive:

    Prefer Extract-Load-Transform (ELT) over traditional ETL.

    line 29
  6. Distinctive:

    Extract & Load: Use tools like Airbyte or Fivetran to dump raw data directly into the Data Warehouse.

    line 30
  7. Distinctive:

    Treat SQL like software engineering.

    line 34
  8. Very specific:

    Use dbt (Data Build Tool) to version control your SQL transformations.

    line 35
  9. Distinctive:

    Use Jinja templating in dbt to DRY up complex SQL queries.

    line 37

≈ “Common” is an estimate: at least 1 in 5 of the 350,495 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Data Pipeline & ETL Expert

English | Bahasa Indonesia

<a name="english"></a>

English

Description

A specialized skill for building robust data architectures, Analytics Engineering, and ETL (Extract, Transform, Load) or ELT pipelines. It covers modern data stack orchestration (Airflow, Dagster), transformation tools (dbt), and high-performance OLAP databases (BigQuery, Snowflake, ClickHouse, DuckDB).

Trigger Conditions

  • When designing reporting dashboards or analytics infrastructure for a SaaS.
  • When moving large volumes of data from transactional databases (PostgreSQL/MySQL) to a data warehouse.
  • When the user asks about "dbt", "Airflow", "ELT", or "Analytics Engineering".
  • When building local or edge analytics using DuckDB.

Core Architectural Guidelines

1. ELT over ETL

Prefer Extract-Load-Transform (ELT) over traditional ETL.

  • Extract & Load: Use tools like Airbyte or Fivetran to dump raw data directly into the Data Warehouse.
  • Transform: Perform transformations inside the Data Warehouse using SQL (via dbt) to leverage the warehouse's massive compute power.

2. Analytics Engineering with dbt

Treat SQL like software engineering.

  • Use dbt (Data Build Tool) to version control your SQL transformations.
  • Implement tests (not_null, unique) on critical tables.
  • Use Jinja templating in dbt to DRY up complex SQL queries.

3. Data Orchestration (Airflow vs Dagster)

  • Apache Airflow: The industry standard for scheduling and monitoring complex DAGs (Directed Acyclic Graphs). Best for Python-heavy teams.
  • Dagster: A modern alternative focused on data assets rather than just tasks. Use Dagster when you want better local testing and asset-driven lineage.

4. OLAP Database Selection

  • BigQuery / Snowflake: Best for massive scale, fully managed cloud data warehousing.
  • ClickHouse: Best for real-time, sub-second analytical queries on massive event streams.
  • DuckDB: Best for local analytics, embedded analytical pipelines, or processing parquets in edge environments (Node.js/Python).

Orchestration & Integration

  • Pairs with data-telemetry-expert to process the raw telemetry events captured by PostHog/OpenTelemetry.
  • Complements python-programming-expert as Python is the lingua franca of data engineering.
  • Works with cron-scheduler-expert when simpler, non-DAG cron jobs are sufficient for small ETL tasks.

<a name="bahasa-indonesia"></a>

Bahasa Indonesia

Deskripsi

Panduan khusus untuk membangun arsitektur data yang kuat, Analytics Engineering, dan pipeline ETL/ELT. Mencakup orkestrasi (Airflow, Dagster), alat transformasi (dbt), dan database OLAP berkinerja tinggi (BigQuery, Snowflake, ClickHouse, DuckDB).

Kondisi Pemicu

  • Saat merancang infrastruktur analitik atau dashboard pelaporan untuk SaaS.
  • Saat memindahkan data bervolume besar dari database transaksional ke Data Warehouse.
  • Saat membangun analitik lokal yang cepat menggunakan DuckDB.

Panduan Arsitektur Inti

1. ELT lebih disukai daripada ETL

  • Extract & Load: Pindahkan data mentah (raw data) langsung ke Data Warehouse (menggunakan Airbyte/Fivetran).
  • Transform: Lakukan transformasi data di dalam Data Warehouse menggunakan SQL (dbt) untuk memanfaatkan kekuatan komputasi gudang data yang masif.

2. Analytics Engineering dengan dbt (Data Build Tool)

Perlakukan transformasi data (SQL) layaknya rekayasa perangkat lunak. Gunakan dbt untuk version control, pengujian otomatis (not_null, unique), dan dokumentasi skema data Anda.

3. Orkestrasi Data (DAG)

Gunakan Apache Airflow atau Dagster untuk menjadwalkan dan memonitor alur kerja data yang kompleks (DAG). Dagster sangat direkomendasikan untuk pendekatan modern yang berpusat pada aset data (asset-driven orchestration).

4. Pemilihan Database OLAP

  • BigQuery / Snowflake: Gudang data cloud fully-managed untuk analitik skala masif.
  • ClickHouse: Sangat cepat untuk kueri analitik real-time. Cocok untuk data event stream/log.
  • DuckDB: SQLite untuk analitik. Sangat cepat untuk memproses file Parquet atau CSV secara lokal maupun di lingkungan edge/serverless (via Python/Node.js).

Integrasi Orkestrasi

  • Bekerja sama dengan data-telemetry-expert untuk memproses data mentah yang dikumpulkan.
  • Melengkapi python-programming-expert dalam menulis skrip orkestrasi Airflow/Dagster.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

Check your setup, free