agentleFS
Sign inStart free
Scientific-ToolingScientific-Tooling/structured-intelligence/skills/stat-assess-data-quality/SKILL.md

stat-assess-data-quality

Skill4 starsChanged 12 days agoMITView on GitHub

Assess data quality by reporting missing values, outliers, sample size, and variance structure for each variable in a table.

The file

We split this skill into the rules it gives an agent

3 rules. About 0 are common, 3 are distinctive, and 0 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈3

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 0

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Read the data file; record total rows (n) and total columns.

    line 27
  2. Distinctive:

    Write a quality report (TSV) and missingness plot (PDF) to output directory.

    line 33
  3. Distinctive:

    Does not impute or remove data — assessment only.

    line 47

≈ “Common” is an estimate: at least 1 in 5 of the 350,495 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Skill: Statistical Data Quality Assessment

Use When

  • User wants a quick data quality report before any statistical analysis
  • User needs to identify columns with high missingness or extreme outliers
  • User wants to understand variance structure and flag zero-variance or near-zero-variance columns
  • User is preparing data for modeling and wants a pre-flight check

Inputs

  • Required:
    • Data table (CSV or TSV)
  • Optional:
    • Outlier detection method: iqr (default) or zscore
    • IQR multiplier for outlier threshold (default: 1.5)
    • Z-score threshold (default: 3.0)
    • Output directory (default: ./qc_output)

Workflow

  1. Read the data file; record total rows (n) and total columns.
  2. For each column, report: data type, non-missing count, missing count, missing rate (%).
  3. For numeric columns: compute mean, SD, min, max; detect outliers using IQR fence or Z-score; report outlier count and outlier rate (%).
  4. Flag columns with: missing rate > 20%, outlier rate > 5%, zero variance, near-zero variance (coefficient of variation < 0.01).
  5. Report sample size adequacy: flag if n < 30 (small sample warning).
  6. Produce a missingness heatmap (rows = samples, columns = variables) if any missing values exist.
  7. Write a quality report (TSV) and missingness plot (PDF) to output directory.
  8. Summarize flags: list columns requiring attention and recommended actions (imputation, removal, transformation).
  9. Save the executable analysis script (analysis_code.py or analysis_code.R) and software version manifest (session_info.txt) used to produce results.

Output Contract

  • Quality report table (TSV): column, dtype, n_non_missing, n_missing, missing_rate, n_outliers, outlier_rate, mean, sd, min, max, flags
  • Missingness heatmap (PDF, only if missing values present)
  • Summary of flagged columns (printed to stdout)
  • Reproducibility artifacts: analysis_code.py or analysis_code.R, plus session_info.txt

Limits

  • IQR-based outlier detection assumes roughly unimodal distributions; may flag many values in skewed data.
  • Does not impute or remove data — assessment only.
  • Very large datasets (> 1M rows) may be slow; consider sampling first.
  • Common failure: mixed-type columns (e.g., numeric with "N/A" strings) require preprocessing.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

Check your setup, free