data-modeler
Skill0 starsChanged 10 months agoMITView on GitHub
Design and implement data pipelines using Medallion architecture, Data Vault 2.0, and Spark optimization. Use when building ETL/ELT processes, designing data schemas, optimizing Spark performance, implementing data governance, or architecting lakehouse solutions.
1 other project uses a version of this.
Adopted by
Most recently taken up by
- emtcmca/guildproof-skillstheir own version, first seen Sep 19, 2026
Other versions1 project uses a version of this.
Word for word · 0
None found.
Edited · 0
None found.
Wrote their own version · 1
- emtcmca/guildproof-skills★ 0same name
How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.
In this repository33 other files it gives its agents.
- architectSkill
- brainstormingSkill
- concise-planningSkill
- context-engineerSkill
- data-deprecation-analysisSkill
- data-pipeline-engineerSkill
- data-quality-frameworksSkill
- dbt-transformation-patternsSkill
- github-workflow-automationSkill
- guardianSkill
- implementerSkill
- json-to-pydanticSkill
- llm-app-patternsSkill
- navigatorSkill
- ops-managerSkill
- orchestratorSkill
- principal-data-engineerSkill
- product-manager-toolkitSkill
- proj-memorySkill
- prompt-librarySkill
- qa-testerSkill
- researcherSkill
- senior-data-engineerSkill
- software-architectureSkill
- spark-optimizationSkill
- sql-optimization-patternsSkill
- state-managerSkill
- subagent-driven-developmentSkill
- thinkerSkill
- top-web-vulnerabilitiesSkill
- verification-before-completionSkill
- web-design-guidelinesSkill
- workflow-managerSkill
The file
We split this skill into the rules it gives an agent
7 rules. About 0 are common, 5 are distinctive, and 2 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈5
Distinctive · estimate
This author's own choices. Worth a look.
2
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 30
PySpark: Avoid UDFs. Use explicit imports and aliases (F, T).
- Distinctive: line 31
Idempotency: All pipelines must be safe to re-run and support backfilling.
- Distinctive: line 33
Data Quality: Use Great Expectations or Chispa for transformation validation.
- Very specific: line 57
Technical Constraints: Do not deploy infrastructure.
- Distinctive: line 59
Governance Constraints: All models must include data quality tests and backfill strategy.
- Distinctive: line 64
Missing Backfill Strategy: Not planning how to reload historical data leads to operational nightmares.
- Distinctive: line 67
No SCD (Slowly Changing Dimensions): Not tracking how attributes change over time breaks historical analysis.
≈ “Common” is an estimate: at least 1 in 5 of the 350,495 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Data Modeler Skill - Lakehouse & Data Engineering
Overview
The Data Modeler skill manages the lifecycle of data within a Lakehouse environment. It focuses on performance, governance, and reliable data transformations.
Core Capabilities
- Medallion Architecture: Implementation of Bronze (Raw), Silver (Clean/Vault), and Gold (Business) layers.
- Data Vault 2.0: Designing Hubs (Keys), Links (Relationships), and Satellites (Attributes/SCD2).
- Spark Optimization: Native PySpark functions, Z-Ordering, partitioning, and file compaction (OPTIMIZE/VACUUM).
- Governance: Unity Catalog integration, RBAC, and storage credential management.
- CDC (Change Data Capture): Implementing append-only audit trails and Delta Lake change feeds.
Standards
- PySpark: Avoid UDFs. Use explicit imports and aliases (
F,T). Use.transform()for modularity. - Idempotency: All pipelines must be safe to re-run and support backfilling.
- Partitioning: Date-based for time-series; 100-1000 partitions ideal.
- Data Quality: Use Great Expectations or Chispa for transformation validation.
When to Use
- Designing database schemas or lakehouse layouts.
- Building ETL/ELT pipelines.
- Optimizing slow queries or high-cost data processing.
- Implementing data privacy and governance controls.
Constraints
- NO implementation of application business logic.
- NO infrastructure deployment (route to ops-manager).
- NO code without data quality checks.
Outputs & Deliverables
- Primary Output: Data model designs, schema definitions, and pipeline specs (e.g.,
specs/<pipeline>.md) - Secondary Output: Example pipeline code snippets and data quality checks
- Success Criteria: Models include schema, partitioning strategy, and validation rules
- Quality Gate: Data model reviewed by
implementerandops-managerbefore production run
Additional Constraints
- Technical Constraints: Do not deploy infrastructure; hand off IaC to
ops-manager. - Scope Constraints: In scope: model design and pipeline specs. Out of scope: production deployment and cluster configuration.
- Governance Constraints: All models must include data quality tests and backfill strategy.
Common Pitfalls
- Over-Partitioning: Excessive partitions increase query latency and metadata overhead. 100-1000 is ideal; validate before deployment.
- Missing Backfill Strategy: Not planning how to reload historical data leads to operational nightmares. Every model needs a backfill procedure.
- Ignoring Data Quality: Garbage in = garbage out. Define validation rules before implementation, not after bugs appear.
- Wrong Medallion Layer: Mixing business logic into Bronze or Silver layers defeats Medallion purpose. Keep layers clean and separated.
- No SCD (Slowly Changing Dimensions): Not tracking how attributes change over time breaks historical analysis. Use SCD2 for audit trails.
- Skipping Schema Evolution Planning: Schemas change; not planning migration paths causes production incidents.
Integration Points
| Phase | Input From | Output To | Context |
| Design | architect, requirements | Schema specification | Understand business primitives and data flows |
| Pipeline Dev | Schema specs | implementer | Hand off for ETL/ELT implementation |
| Optimization | Performance metrics | ops-manager | Request infrastructure scaling if needed |
| Quality | Data rules | Monitoring/alerts | Set up data quality checks and validations |
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.