data-lake-platform
Skill6 starsChanged 2 days agoMITView on GitHub
Padrões de data lake e lakehouse — ingestão, CDC, Iceberg/Delta/Hudi, Trino/DuckDB, orquestração e governança, self-hosted ou em nuvem. Use ao desenhar ou operar uma plataforma de dados.
- Installs packages
1 other project uses a version of this.
Adopted by
Most recently taken up by
- vasilyu1983/AI-Agents-publictheir own version, first seen Oct 13, 2026
Other versions1 project uses a version of this.
Word for word · 0
None found.
Edited · 0
None found.
Wrote their own version · 1
- vasilyu1983/AI-Agents-public★ 94same name
How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.
In this repository134 other files it gives its agents.
- CLAUDE.mdCLAUDE.md
- accessibilitySkill
- ai-ml-data-scienceSkill
- ai-ml-timeseriesSkill
- brand-platformSkill
- brand-researchSkill
- brand-rolloutSkill
- brand-trackingSkill
- brand-verbal-identitySkill
- brand-visual-systemSkill
- copy-humanizeSkill
- core-web-vitalsSkill
- data-analytics-engineeringSkill
- data-sql-optimizationSkill
- data-supabase-patternsSkill
- deep-researchSkill
- dev-api-designSkill
- dev-database-patternsSkill
- dev-debuggingSkill
- dev-defuddleSkill
- dev-error-handlingSkill
- dev-git-workflowSkill
- dev-security-patternsSkill
- dev-technical-writingSkill
- dev-testing-strategySkill
- dev-typescript-patternsSkill
- doc-to-markdownSkill
- finance-bookkeepingSkill
- finance-cash-flowSkill
- finance-receivables-payablesSkill
- finance-reportingSkill
- finance-tax-complianceSkill
- instagram-dm-flowsSkill
- instagram-publishingSkill
- instagram-strategySkill
- legal-clause-librarySkill
- legal-compliance-lgpdSkill
- legal-contract-draftingSkill
- legal-contract-lifecycleSkill
- legal-researchSkill
- linkedin-publishingSkill
- linkedin-strategySkill
- negotiationSkill
- nextjs-react-best-practicesSkill
- presentation-designSkill
- pricingSkill
- robotics-bringup-testingSkill
- robotics-component-researchSkill
- robotics-electrical-safetySkill
- robotics-firmware-patternsSkill
- robotics-iot-connectivitySkill
- robotics-mechanical-buildSkill
- robotics-motion-controlSkill
- robotics-power-batterySkill
- robotics-project-planningSkill
- robotics-ros2-autonomySkill
- sala-de-controleSkill
- sales-deck-productionSkill
- sales-discovery-intakeSkill
- sales-enablementSkill
- The first 60 shown.
The file
We split this skill into the rules it gives an agent
7 rules. About 0 are common, 6 are distinctive, and 1 is very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈6
Distinctive · estimate
This author's own choices. Worth a look.
1
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 21
Add governance, lineage, data quality, and cost controls
- Very specific: line 47
Add governance, lineage, and quality gates: references/governance-catalog.md (use assets/cross-platform/template-data-quality-governance.md …
- Distinctive: line 79
Add quality gates (freshness, volume, schema, distribution) per tier
- Distinctive: line 81
Treat access control and audit logging as first-class requirements
- Distinctive: line 131
Use web search/web fetch to verify current external facts, versions, pricing, deadlines, regulations, or platform behavior before final …
- Distinctive: line 132
Prefer primary sources; report source links and dates for volatile information.
- Distinctive: line 133
If web access is unavailable, state the limitation and mark guidance as unverified.
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Data Lake Platform
Build and operate production data lakes and lakehouses: ingest, transform, store in open formats, and serve analytics reliably.
When to Use
- Design data lake/lakehouse architecture
- Set up ingestion pipelines (batch, incremental, CDC)
- Build SQL transformation layers (SQLMesh, dbt)
- Choose table formats and catalogs (Iceberg, Delta, Hudi)
- Deploy query/serving engines (Trino, ClickHouse, DuckDB)
- Implement streaming pipelines (Kafka, Flink)
- Set up orchestration (Dagster, Airflow, Prefect)
- Add governance, lineage, data quality, and cost controls
Triage Questions
- Batch, streaming, or hybrid? What is the freshness SLO?
- Append-only vs upserts/deletes (CDC)? Is time travel required?
- Primary query pattern: BI dashboards (high concurrency), ad-hoc joins, embedded analytics?
- PII/compliance: row/column-level access, retention, audit logging?
- Platform constraints: self-hosted vs cloud, preferred engines, team strengths?
Default Baseline (Good Starting Point)
- Storage: object storage + open table format (usually Iceberg)
- Catalog: REST/Hive/Glue/Nessie/Unity (match your platform)
- Transforms: SQLMesh or dbt (pick one and standardize)
- Lake query: Trino (or Spark for heavy compute/ML workloads)
- Serving (optional): ClickHouse/StarRocks/Doris for low-latency BI
- Governance: DataHub/OpenMetadata + OpenLineage
- Orchestration: Dagster/Airflow/Prefect
Workflow
- Pick table format + catalog:
references/storage-formats.md(useassets/cross-platform/template-schema-evolution.mdandassets/cross-platform/template-partitioning-strategy.md) - Design ingestion (batch/incremental/CDC):
references/ingestion-patterns.md(useassets/cross-platform/template-ingestion-governance-checklist.mdandassets/cross-platform/template-incremental-loading.md) - Design transformations (bronze/silver/gold or data products):
references/transformation-patterns.md(useassets/cross-platform/template-data-pipeline.md) - Choose lake query vs serving engines:
references/query-engine-patterns.md - Add governance, lineage, and quality gates:
references/governance-catalog.md(useassets/cross-platform/template-data-quality-governance.mdandassets/cross-platform/template-data-quality.md) - Plan operations + cost controls:
references/operational-playbook.mdandreferences/cost-optimization.md(useassets/cross-platform/template-data-quality-backfill-runbook.mdandassets/cross-platform/template-cost-optimization.md)
Architecture Patterns
- Medallion (bronze/silver/gold):
references/architecture-patterns.md - Data mesh (domain-owned data products):
references/architecture-patterns.md - Streaming-first (Kappa):
references/streaming-patterns.md
Quick Start
dlt + ClickHouse
pip install "dlt[clickhouse]"
dlt init rest_api clickhouse
python pipeline.pySQLMesh + DuckDB
pip install sqlmesh
sqlmesh init duckdb
sqlmesh plan && sqlmesh runReliability and Safety
Do
- Define data contracts and owners up front
- Add quality gates (freshness, volume, schema, distribution) per tier
- Make every pipeline idempotent and re-runnable (backfills are normal)
- Treat access control and audit logging as first-class requirements
Avoid
- Skipping validation to "move fast"
- Storing PII without access controls
- Pipelines that can't be re-run safely
- Manual schema changes without version control
Resources
| Resource | Purpose |
| references/architecture-patterns.md | Medallion, data mesh |
| references/ingestion-patterns.md | dlt vs Airbyte, CDC |
| references/transformation-patterns.md | SQLMesh vs dbt |
| references/storage-formats.md | Iceberg vs Delta |
| references/query-engine-patterns.md | ClickHouse, DuckDB |
| references/streaming-patterns.md | Kafka, Flink |
| references/orchestration-patterns.md | Dagster, Airflow |
| references/bi-visualization-patterns.md | Metabase, Superset |
| references/cost-optimization.md | Cost levers and maintenance |
| references/operational-playbook.md | Monitoring and incident response |
| references/governance-catalog.md | Catalog, lineage, access control |
| references/data-mesh-patterns.md | Domain ownership, data products, federated governance |
| references/data-quality-patterns.md | Quality gates, validation frameworks, SLOs, anomaly detection |
| references/security-access-patterns.md | Row/column security, encryption, audit logging, compliance |
Templates
| Template | Purpose |
| assets/cross-platform/template-medallion-architecture.md | Baseline bronze/silver/gold plan |
| assets/cross-platform/template-data-pipeline.md | End-to-end pipeline skeleton |
| assets/cross-platform/template-ingestion-governance-checklist.md | Source onboarding checklist |
| assets/cross-platform/template-incremental-loading.md | Incremental + backfill plan |
| assets/cross-platform/template-schema-evolution.md | Schema change rules |
| assets/cross-platform/template-cost-optimization.md | Cost control checklist |
| assets/cross-platform/template-data-quality-governance.md | Quality contracts + SLOs |
| assets/cross-platform/template-data-quality-backfill-runbook.md | Backfill incident/runbook |
Related Skills
| Skill | Purpose |
| ai-ml-data-science | Feature engineering |
| data-sql-optimization | OLTP optimization |
Fact-Checking
- Use web search/web fetch to verify current external facts, versions, pricing, deadlines, regulations, or platform behavior before final answers.
- Prefer primary sources; report source links and dates for volatile information.
- If web access is unavailable, state the limitation and mark guidance as unverified.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

