GCP Data Engineering Skill

Overview

Guides architectural decisions for data pipelines, Medallion lakehouses, and analytical storage on Google Cloud Platform (GCP). Balances data volume, SLA freshness, operational simplicity, and cloud cost—prioritizing warehouse-native ELT (Dataform, BigQuery) over custom streaming (Dataflow/Beam) unless sub-minute SLAs apply, while enforcing capability-gated artifacts and fail-honest CLI verification.

When to Use

Trigger Scenarios

• Designing GCP lakehouse platforms (Cloud Storage, BigQuery, Datastream, Pub/Sub, Dataform, Composer).
• Selecting batch vs. log-based CDC vs. streaming ingestion and Dataform vs. Composer vs. Dataflow orchestration.
• Designing BigQuery partitioning, clustering, and Medallion layers (Bronze → Silver → Gold).

When NOT to Use

• CDC & SCD Type 2 merge patterns: Route to cdc-scd-patterns.
• Slow/expensive BigQuery SQL plan tuning: Route to bigquery-query-optimization.
• Single-node RDBMS SQL tuning: Route to sql-query-optimization.
• Application backend code: Route to python-expert.

Process

Phase 1: Capability-Gated Planning & Target AGENTS.md

1. Maintain root AGENTS.md (<= 90 lines) linking dataform/bq commands and context/memory/MEMORY.md.
2. Author PLAN.md (<= 50 lines) gating artifacts individually:
   • docs/<feature-name>/data-contracts.md (<= 60 lines): ONLY IF new/modified schemas or SLAs are introduced; SKIP if schemas exist.
   • docs/<feature-name>/pipeline-lineage.md (<= 45 lines): ONLY IF multi-layer Bronze/Silver/Gold flows change; SKIP for single-mart edits.
   • Delegate sql-query-optimizer ONLY IF large partitioned scans or multi-table JOIN/window shuffle risks exist; invoke Tier 3 Data Challenger ONLY IF CDC/SCD-2 merges, late-arriving dedup, or destructive/backfill logic changes.

Phase 2: Medallion Storage, Ingestion & Orchestration

1. Storage Layers: GCS immutable raw landing zone + BigQuery Medallion tables: Bronze (append-only raw/CDC), Silver (deduplicated, typed, SCD-2), Gold (BI star marts).
2. Ingestion (Default Batch): Scheduled loads (GCS/Dataform/Composer); Datastream for operational DB CDC; Pub/Sub → BigQuery (or Dataflow only when windowed stream joins/non-SQL code are required).
3. Orchestration: Dataform for warehouse-native SQLX & assertions; Cloud Composer only for cross-service DAGs.

Phase 3: BigQuery Cost Governance & Fail-Honest Verification

1. Partition & Cluster: Partition by date/timestamp or ingestion time (require_partition_filter = true on large tables); cluster up to 4 selective filter/join columns; prune columns (never SELECT *).
2. Fail-Honest CLI Gate: Run dataform compile and bq query --dry_run. If CLI binaries or GCP ADC credentials fail/missing, STOP and report BLOCKED: Not executed — <exact CLI error>—never fabricate byte counts or compile success.
3. Standardized Test Matrix (docs/<feature-name>/data-test-matrix.md <= 75 lines): Record IEEE/ISTQB test cases (TC-01...) with ID, Ref, Level, Objective, Steps, Expected Result, Assertion / Query, and Status.

Usage

# Estimate scanned bytes via BigQuery dry-run
bq query --use_legacy_sql=false --dry_run "SELECT id, created_at FROM \`my_project.analytics.events\` WHERE event_date = '2026-09-01'"

# Compile Dataform dependency graph (scoped or full)
dataform compile --actions silver_orders
dataform compile

• Host Execution: Run bq and dataform via shell (Bash in Claude Code; run_command in Antigravity) and persist contracts/SLAs to context/memory/MEMORY.md.

Red Flags

• Fabricating bq query --dry_run byte counts or pretending dataform compile ran when CLI/ADC credentials fail instead of reporting BLOCKED: Not executed — <exact CLI error>.
• Using Dataflow/Kafka when batch/Datastream meets SLAs, or Composer for SQL-only transforms Dataform handles natively.
• Querying SELECT * or raw CDC tables without partition filters or a Silver deduplication layer.

Verification

• Root AGENTS.md (<= 90 lines), PLAN.md (<= 50 lines), and capability-gated docs/<feature-name>/ artifacts respect line caps.
• Tables define explicit partitioning (require_partition_filter = true) and clustering keys across Bronze/Silver/Gold.
• dataform compile and bq query --dry_run exit 0, OR execution halts honestly with BLOCKED: Not executed — <exact CLI error>.
• docs/<feature-name>/data-test-matrix.md (<= 75 lines) documents TC-01... with ID, Ref, Level, Objective, Steps, Expected Result, Assertion / Query, and Status.

References

• GCP Data Stack Reference