agentleFS
TrecekTrecek/useful-claude-skills/.claude/skills/exp-lens-pipeline-integrity/SKILL.md

exp-lens-pipeline-integrity

Skill1 starsChanged 8 months agoMITView on GitHub

Create Pipeline Integrity experimental design diagram showing data splits, leakage points, preprocessing order, and label contamination. Integrity lens answering "Could data handling create optimistic bias?

The file

We split this skill into the rules it gives an agent

10 rules. About 0 are common, 8 are distinctive, and 2 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈8

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 2

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Do not litter the codebase with useless comments, TODO markers, or explanatory annotations — the skill output and diagram speak for …

    line 30
  2. Distinctive:

    Document pipeline invariants that guard against leakage

    line 36
  3. Very specific:

    BEFORE creating any diagram, LOAD the /mermaid skill using the Skill tool - this is MANDATORY

    line 37
  4. Distinctive:

    Look for: load, read, fetch, dataset, csv, parquet, download

    line 49
  5. Distinctive:

    Look for: transform, normalize, scale, encode, impute, clean, preprocess

    line 53
  6. Distinctive:

    Look for: split, train_test, fold, cross_val, stratify, group

    line 57
  7. Distinctive:

    Look for: feature, extract, select, engineer, embed, vectorize

    line 61
  8. Distinctive:

    Look for: fit, train, predict, evaluate, score, metric, loss

    line 65
  9. Very specific:

    Write the diagram to: temp/exp-lens-pipeline-integrity/exp_diag_pipeline_integrity_{YYYY-MM-DD_HHMMSS}.md

    line 112
  10. Distinctive:

    Before creating the diagram, verify:

    line 224

≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Pipeline Integrity Experimental Design Lens

Philosophical Mode: Integrity Primary Question: "Could data handling create optimistic bias?" Focus: Data Splits, Leakage Points, Preprocessing Order, Label Contamination, Pipeline Invariants

When to Use

  • ML pipeline with train/test splits
  • Preprocessing before or after splitting is ambiguous
  • Feature engineering touching labels
  • User invokes /exp-lens-pipeline-integrity or /make-experiment-diag pipeline

Critical Constraints

NEVER:

  • Modify any source code files
  • Do not litter the codebase with useless comments, TODO markers, or explanatory annotations — the skill output and diagram speak for themselves

ALWAYS:

  • Classify every pipeline stage as pre-split or post-split
  • Trace whether transforms are fitted on full data or train-only
  • Flag all label-touching feature engineering steps
  • Document pipeline invariants that guard against leakage
  • BEFORE creating any diagram, LOAD the /mermaid skill using the Skill tool - this is MANDATORY

Analysis Workflow

Step 1: Launch Parallel Exploration Subagents

Spawn Explore subagents to investigate:

Data Loading & Sources

  • Find data ingestion code, raw data paths
  • Look for: load, read, fetch, dataset, csv, parquet, download

Preprocessing & Transforms

  • Find normalization, encoding, imputation steps
  • Look for: transform, normalize, scale, encode, impute, clean, preprocess

Split Logic

  • Find train/test/validation split code
  • Look for: split, train_test, fold, cross_val, stratify, group

Feature Engineering

  • Find feature creation, selection, extraction
  • Look for: feature, extract, select, engineer, embed, vectorize

Model Training & Evaluation

  • Find training loops and evaluation metrics
  • Look for: fit, train, predict, evaluate, score, metric, loss

Step 2: Map the Complete Pipeline

Map the full pipeline from raw data to reported metrics. For each stage, determine:

  • What information flows in?
  • What information flows out?
  • Could any downstream information leak upstream?
  • Classify each stage as pre-split or post-split.

Step 3: Identify Leakage Risks

CRITICAL — Analyze Leakage Direction: For every data transformation:

  • Does it use information from the full dataset (leakage risk) or only from the training partition?
  • Is normalization fitted on train-only or all data?
  • Are features derived from labels?

Assign a severity level (High/Medium/Low) to each leakage risk based on whether it would invalidate reported metrics.

Step 4: Create the Diagram

Use flowchart with:

Direction: LR (data flows left to right)

Subgraphs:

  • RAW DATA
  • PREPROCESSING
  • SPLIT POINT
  • TRAIN PATH
  • TEST PATH
  • EVALUATION

Node Styling:

  • cli class: Data sources
  • handler class: Transforms
  • detector class: Split point and validation gates
  • stateNode class: Data stores
  • gap class: Leakage risks
  • output class: Metrics and results
  • phase class: Model training

Edge Labels: full data, train only, test only, LEAKAGE RISK

Step 5: Write Output

Write the diagram to: temp/exp-lens-pipeline-integrity/exp_diag_pipeline_integrity_{YYYY-MM-DD_HHMMSS}.md


Output Template

# Pipeline Integrity Diagram: {Experiment Name}

**Lens:** Pipeline Integrity (Integrity)
**Question:** Could data handling create optimistic bias?
**Date:** {YYYY-MM-DD}
**Scope:** {What was analyzed}

## Pipeline Stages

| Stage | Input | Output | Pre/Post Split | Leakage Risk? |
|-------|-------|--------|----------------|---------------|
| {stage} | {input} | {output} | {Pre/Post} | {Yes/No} |

## Pipeline Diagram

```mermaid
%%{init: {'flowchart': {'nodeSpacing': 50, 'rankSpacing': 60, 'curve': 'basis'}}}%%
flowchart LR
    %% CLASS DEFINITIONS %%
    classDef cli fill:#1a237e,stroke:#7986cb,stroke-width:2px,color:#fff;
    classDef stateNode fill:#004d40,stroke:#4db6ac,stroke-width:2px,color:#fff;
    classDef handler fill:#e65100,stroke:#ffb74d,stroke-width:2px,color:#fff;
    classDef phase fill:#6a1b9a,stroke:#ba68c8,stroke-width:2px,color:#fff;
    classDef newComponent fill:#2e7d32,stroke:#81c784,stroke-width:2px,color:#fff;
    classDef output fill:#00695c,stroke:#4db6ac,stroke-width:2px,color:#fff;
    classDef detector fill:#b71c1c,stroke:#ef5350,stroke-width:2px,color:#fff;
    classDef gap fill:#ff6f00,stroke:#ffa726,stroke-width:2px,color:#000;
    classDef integration fill:#c62828,stroke:#ef9a9a,stroke-width:2px,color:#fff;

    subgraph Raw ["RAW DATA"]
        SRC["Raw Dataset<br/>━━━━━━━━━━<br/>Source path<br/>N samples"]
    end

    subgraph Preprocessing ["PREPROCESSING"]
        PREP["Normalization / Encoding<br/>━━━━━━━━━━<br/>Fitted on: full/train?"]
        LEAK["Leaky Transform<br/>━━━━━━━━━━<br/>Uses full dataset"]
    end

    subgraph SplitPoint ["SPLIT POINT"]
        SPLIT["Train/Test Split<br/>━━━━━━━━━━<br/>Stratified? Ratio?"]
    end

    subgraph TrainPath ["TRAIN PATH"]
        TRAIN_DATA["Train Set<br/>━━━━━━━━━━<br/>N_train samples"]
        MODEL["Model Training<br/>━━━━━━━━━━<br/>fit()"]
    end

    subgraph TestPath ["TEST PATH"]
        TEST_DATA["Test Set<br/>━━━━━━━━━━<br/>N_test samples"]
    end

    subgraph Evaluation ["EVALUATION"]
        METRIC["Reported Metric<br/>━━━━━━━━━━<br/>score / loss"]
    end

    %% PIPELINE FLOWS %%
    SRC -->|"full data"| PREP
    PREP -->|"full data"| LEAK
    LEAK -.->|"LEAKAGE RISK"| METRIC
    PREP -->|"full data"| SPLIT
    SPLIT -->|"train only"| TRAIN_DATA
    SPLIT -->|"test only"| TEST_DATA
    TRAIN_DATA -->|"fit"| MODEL
    MODEL -->|"predict"| TEST_DATA
    TEST_DATA -->|"evaluate"| METRIC

    %% CLASS ASSIGNMENTS %%
    class SRC cli;
    class PREP handler;
    class LEAK gap;
    class SPLIT detector;
    class TRAIN_DATA,TEST_DATA stateNode;
    class MODEL phase;
    class METRIC output;

Color Legend:

ColorCategoryDescription
Dark BlueData SourceRaw input datasets
OrangeTransformPreprocessing and feature engineering steps
RedSplit / GateSplit point and validation gates
TealData StorePartitioned data stores (train/test)
PurpleTrainingModel training stages
Dark TealOutputReported metrics and results
AmberLeakage RiskTransforms using full-dataset information

Leakage Assessment

RiskStageMechanismSeverity
{risk name}{stage}{how leakage occurs}{High/Medium/Low}

Pipeline Invariants

  • All scalers/encoders fitted on train partition only
  • Feature selection criteria computed from train partition only
  • No label information used in feature construction
  • Test set never seen by any fitting step

---

## Pre-Diagram Checklist

Before creating the diagram, verify:

- [ ] LOADED `/mermaid` skill using the Skill tool
- [ ] Using ONLY classDef styles from the mermaid skill (no invented colors)
- [ ] Diagram will include a color legend table

---

## Related Skills

- `/make-experiment-diag` - Parent skill for lens selection
- `/mermaid` - MUST BE LOADED before creating diagram
- `/exp-lens-reproducibility-artifacts` - For artifact completeness audit
- `/exp-lens-measurement-validity` - For outcome measurement validity

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.