data-profiling
Skill2 starsChanged 12 days agoMITView on GitHub
Data Profiling. Explore and profile a dataset before modeling - EDA (exploratory data analysis), distributions, skew, outliers, missingness (MCAR, MAR, MNAR), duplicates, cardinality, class balance, label quality, leakage risk, bias and representativeness, drift between periods. Use when someone says explore, profile, EDA, data quality, understand the data, check the dataset, or before any machine learning, data mining or statistical analysis. CRISP-DM Data Understanding.
3 other projects use a version of this.
Adopted by
Most recently taken up by
- okou-ai/okou-skillstheir own version, first seen Nov 4, 2026
- vm0-ai/vm0-skillstheir own version, first seen Oct 8, 2026
- ai-analyst-lab/ai-analysttheir own version, first seen Sep 10, 2026
Other versions3 projects use a version of this.
Word for word · 0
None found.
Edited · 0
None found.
Wrote their own version · 3
- ai-analyst-lab/ai-analyst★ 304same name
- okou-ai/okou-skills★ 81same name
- vm0-ai/vm0-skills★ 81same name
How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.
In this repository16 other files it gives its agents.
- ai-prediction-explainerSkill
- data-mining-machine-learningSkill
- data-miningSkill
- data-preparationSkill
- data-storytellingSkill
- feature-engineeringSkill
- hyperparameter-tuningSkill
- ml-quality-guardSkill
- ml-web-applicationSkill
- model-deploymentSkill
- model-evaluationSkill
- model-explainabilitySkill
- model-trainingSkill
- notebook-auditSkill
- problem-framingSkill
- statistical-analysisSkill
The file
We split this skill into the rules it gives an agent
2 rules. 0 are common, 2 are distinctive, and 0 are very specific to this project.
0
Common
Most of the 3 edited versions we read keep these.
2
Distinctive
This author's own choices. Worth a look.
0
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 21
Drift: If data spans periods, compare early and late slices (KS test, PSI).
- Distinctive: line 39
Every claim about the data cites the cell that computed it.
“Common” is measured: kept by at least 6 in 10 of the 3 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Data Profiling
Explore before inferring. Profiling ends with a short list of facts that change modeling choices, not a gallery of charts.
Procedure
- Grain and keys. State what one row is. Assert key uniqueness (
df[key].is_unique). Count rows per entity; repeated entities force group-aware splits. - Schema.
df.shape,df.dtypes,df.head(). Assert expected columns and types. Parse dates once. Note units per numeric column. - Ranges.
describe()on numerics; flag impossible values (negative ages, future dates, totals that do not add up). Decide error versus signal before touching outliers. - Missingness. Share per column, share by target class, co-missing patterns. Classify each as MCAR, MAR or MNAR with a reason. Missing-by-target differences are signal and a leakage hint.
- Distributions. Skew and tails, multimodality, zero inflation, bounded proportions. One chart per finding, axes labeled with units.
- Target. Prevalence with the positive count, label definition, label delay, label noise sample check (read 20 raw records per class).
- Relationships. Target rate by key categories, correlation matrix (Spearman for skew), VIF for linear models, Simpson's paradox check on the main segment.
- Leakage scan. For each column: when is it recorded relative to the prediction time? Columns created after the event are removed now. Suspiciously strong single predictors get investigated.
- Representativeness and bias. Compare sample with the population the model will score: time window, regions, channels, protected groups. Note selection and survivorship bias.
- Drift. If data spans periods, compare early and late slices (KS test, PSI). Drift decides between random and forward-chaining validation.
Output
A profile summary in the notebook, under findings headings, plus these lines for ANALYSIS_MANIFEST.md:
Grain: <one row = ...>; entities repeat: <yes/no, max rows per entity>
Target: <definition>; prevalence <p> (<k> positives of <n>)
Missingness: <column: rate, mechanism, plan>
Leakage removed: <columns and why>
Scenarios detected: <IDs from references/scenarios.md>
Split implication: <stratified | group | forward-chaining>Rules
- Print summaries, never full frames. Clear large outputs before saving.
- Every claim about the data cites the cell that computed it.
- Profile on training data only once a split exists; the test set is not explored.
- Data conditions and their remedies:
references/scenarios.md.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.