etl / instructions
Copilot instructions160 starsChanged 6 months agoMITView on GitHub
This repository contains the ETL (Extract, Transform, Load) system for Our World in Data. It processes, versions, and maintains datasets from various sources. Use these guidelines when suggesting code for this project.
We split this copilot-instructions.md into the rules it gives an agent
13 rules. About 0 are common, 10 are distinctive, and 3 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈10
Distinctive · estimate
This author's own choices. Worth a look.
3
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 10
NEVER query MCP servers with custom python code, just ask the MCP server directly.
- Distinctive: line 18
Follow PEP 8 Python style guidelines with a line length of 120 characters
- Distinctive: line 19
Use type hints for all function definitions
- Distinctive: line 21
Always handle country name harmonization using the ETL's built-in functions
- Distinctive: line 22
Document functions and classes with docstrings
- Distinctive: line 23
Add appropriate metadata to datasets according to our metadata schema
- Distinctive: line 24
Write unit tests for new functionality
- Distinctive: line 29
Create a snapshot of the raw data
- Distinctive: line 32
Create grapher configuration if needed
- Very specific: line 36
Run make test to run tests
- Very specific: line 37
Run make format to format code
- Very specific: line 38
Run make etl to process data
- Distinctive: line 49
Use these guidelines when suggesting code for this project.
≈ “Common” is an estimate: at least 1 in 5 of the 3,669 Copilot instructions files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
OWID ETL Copilot Instructions
This repository contains the ETL (Extract, Transform, Load) system for Our World in Data. It processes, versions, and maintains datasets from various sources.
Custom instructions
- NEVER query MCP servers with custom python code, just ask the MCP server directly. If it is not running, let me know.
Architecture
- We use a DAG-based workflow with YAML configuration files in the
dag/directory - Processing happens in channels: Snapshot → Meadow → Garden → Grapher
- Data processing steps are defined with URIs like
data://garden/happiness/2025-03-28/happiness
Development Guidelines
- Follow PEP 8 Python style guidelines with a line length of 120 characters
- Use type hints for all function definitions
- For large datasets, prefer pandas operations that work on entire dataframes over row-by-row operations
- Always handle country name harmonization using the ETL's built-in functions
- Document functions and classes with docstrings
- Add appropriate metadata to datasets according to our metadata schema
- Write unit tests for new functionality
- Imports should be placed at the top of the file
- Let the errors propagate without try-catch block, don't catch them and re-raise with a generic message
Process for Creating ETL Steps
- Create a snapshot of the raw data
- Process to meadow format (clean CSV)
- Harmonize to garden format (standardized entities and variables)
- Create grapher configuration if needed
Tools We Use
- We use
makefor common development tasks - Run
make testto run tests - Run
make formatto format code - Run
make etlto process data - We use
pytestfor testing
File Structure Conventions
- Steps for building datasets are in the
etl/stepsdirectory - Core ETL code is in the
etl/directory - Snapshots are stored in
snapshots/<producer>/<date>/directories - Python processing code for snapshots is in
.pyfiles with the same name - DVC files (
.dvc) track the large data files - Metadata for datasets is in
.meta.ymlfiles
Use these guidelines when suggesting code for this project.
Where files are headingNew Copilot instructions files have more rules about branches and pull requests, and fewer about code style.
Of the Copilot instructions files we read, the 2,297 first seen in the last six months, against the older ones: the kind of rule that gained most share, and the kind that lost most. About files like this in general, not this one, and an estimate, since a kind is a pattern in the words.
In this repository39 other files it gives its agents.
- AGENTS.mdAGENTS.md
- CLAUDE.mdCLAUDE.md
- add-gdp-scatterSkill
- add-ivs-indicatorsSkill
- add-provider-regionsSkill
- babysit-prSkill
- check-chart-previewSkill
- check-empty-entitiesSkill
- check-hardcoded-yearsSkill
- check-metadata-styleSkill
- check-metadata-typosSkill
- check-outdated-practicesSkill
- create-chartSkill
- create-datasetSkill
- create-etl-stepsSkill
- create-explorerSkill
- create-snapshotSkill
- create-static-vizSkill
- develop-vscode-extensionSkill
- develop-wizard-appSkill
- draft-data-update-slack-postSkill
- edit-faust-metadataSkill
- fact-check-datasetSkill
- find-chart-referencesSkill
- fix-dependabotSkill
- generate-metadataSkill
- map-charts-to-mdimSkill
- map-explorer-to-mdimSkill
- migrate-explorer-to-etlSkill
- profile-etl-stepSkill
- query-grapher-dbSkill
- remap-ghost-variablesSkill
- report-indicator-changesSkill
- review-data-prSkill
- sync-grapher-schemaSkill
- tool-owlSkill
- update-climate-dataSkill
- update-datasetSkill
- update-wildfires-dataSkill
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
Reviews can't be read right now.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.