data-product-validation
Skill5 starsChanged 2 months agoMITView on GitHub
Score whether a data product idea is worth building before committing resources. Validation scorecard, experiment design, and go/kill decisions. Use when evaluating feasibility, making go/no-go decisions, validating demand, sizing bets, or when someone asks "is this worth building?" or "should we invest in this?"
In this repository24 other files it gives its agents.
- CLAUDE.mdCLAUDE.md
- analystSkill
- arbitrage-audit-dataSkill
- dashboards-to-decisionsSkill
- databricks-data-product-reviewSkill
- data-consumer-discoverySkill
- data-model-designSkill
- data-pipeline-qualitySkill
- data-product-operatorSkill
- data-product-thinkingSkill
- data-quality-assessmentSkill
- data-storytellingSkill
- data-team-operating-modelSkill
- data-team-positioningSkill
- duckdb-data-profilingSkill
- ethical-risk-assessmentSkill
- grill-me-dataSkill
- healthcare-cohort-designSkill
- healthcare-data-domainSkill
- healthcare-data-readiness-debriefSkill
- healthcare-data-reconciliationSkill
- metrics-definitionSkill
- research-synthesis-dataSkill
- stakeholder-alignmentSkill
The file
We split this skill into the rules it gives an agent
3 rules. About 0 are common, 3 are distinctive, and 0 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈3
Distinctive · estimate
This author's own choices. Worth a look.
0
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 90
When the scorecard says "Investigate," pick the cheapest experiment that addresses the weakest dimension:
- Distinctive: line 92
Sample Query (1 day): Write the SQL. Run it against production data.
- Distinctive: line 100
Not all data product decisions are equal:
≈ “Common” is an estimate: at least 1 in 5 of the 350,495 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Validation Scorecard
Score each dimension 1-5. Multiply all five for a composite score (max 3,125).
1. Demand Frequency
How often do consumers need this answer?
| Score | Frequency | Example |
| 5 | Daily or real-time | ICU readmission risk scores |
| 4 | Weekly | Regional performance reports |
| 3 | Monthly | Quarterly business reviews |
| 2 | Quarterly | Annual planning data |
| 1 | Rarely or once | Ad-hoc executive request |
2. Decision Impact
What happens when consumers don't have this data?
| Score | Impact | Example |
| 5 | Critical decision blocked | Can't discharge patients without risk assessment |
| 4 | Significant delay or cost | Team spends 20+ hours/week on manual workaround |
| 3 | Moderate inconvenience | Report takes 3 hours instead of 5 minutes |
| 2 | Minor friction | Slightly slower process, acceptable workaround exists |
| 1 | Mild inconvenience | Nice to have, nobody changes behavior without it |
3. Workaround Effort
What are consumers doing today instead?
| Score | Effort | Example |
| 5 | Custom tooling built | Analyst maintains a 47-tab Excel model updated daily |
| 4 | Significant manual process | 3 people spend 2 days/week compiling reports |
| 3 | Partial automation | Script exists but breaks frequently, needs babysitting |
| 2 | Simple workaround | Quick Excel export, takes 30 minutes |
| 1 | Haven't tried | Nobody has attempted to solve this yet |
4. Data Feasibility
Can we actually build this with available data?
| Score | Feasibility | Example |
| 5 | Data exists, quality verified | Source audited, passes data-quality-assessment checks |
| 4 | Data exists, quality unknown | Source available but no quality baseline established |
| 3 | Data exists, known issues | Source has gaps or accuracy problems that need fixing first |
| 2 | Data partially exists | Need to combine 3+ sources, some missing |
| 1 | Data doesn't exist | Would need new data collection or acquisition |
Cross-reference data-quality-assessment for the 5-dimension scoring. A Data Feasibility score of 1 kills the idea regardless of other scores.
5. Schema Risk
How locked-in are consumers once we ship?
| Score | Risk | Example |
| 5 | Easily re-modeled | Internal API, few consumers, versioning supported |
| 4 | Moderate coupling | Dashboard consumed by 5-10 users, change is disruptive but manageable |
| 3 | Significant coupling | 20+ downstream queries depend on current schema |
| 2 | High coupling | External consumers or contractual SLAs on schema shape |
| 1 | Breaking change impossible | Regulated output, schema changes require compliance review |
Thresholds
| Composite Score | Decision | Action |
| 250+ | Build | Shape a pitch, allocate a cycle |
| 100-249 | Investigate | Run a 1-week experiment to de-risk the lowest-scoring dimension |
| <100 | Kill | Document why and move on. Revisit only if new evidence surfaces |
A Data Feasibility score of 1 kills the idea at any composite score. You can't build a data product without data.
Experiment Types
When the scorecard says "Investigate," pick the cheapest experiment that addresses the weakest dimension:
Sample Query (1 day): Write the SQL. Run it against production data. Does the result answer the consumer's question? Tests Data Feasibility and reveals quality issues fast. If the query returns garbage, kill early.
Manual Pipeline (1 week): Build the transformation by hand for one consumer. Deliver a spreadsheet or flat file. Measure: did they use it? Did it change a decision? Tests Demand Frequency and Decision Impact with real behavior.
Schema Prototype (2-3 weeks): Build a minimal star schema with one fact table and 2-3 dimensions. Load a month of data. Let 2-3 consumers query it. Tests Schema Risk and reveals modeling assumptions before they calcify.
Type 1 vs Type 2 Decisions
Not all data product decisions are equal:
Type 1 (irreversible): Schema design decisions. Data source commitments. Terminology mappings published to external consumers. SCD strategies once historical data accumulates. These deserve full validation and the scorecard process.
Type 2 (reversible): Adding a metric to an existing dashboard. New filters on an existing report. Feature additions to established products. These don't need a scorecard. Just build and measure.
Discovery budget should match decision type. Spend 2 weeks validating a Type 1. Spend 2 hours on a Type 2.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.