exp-lens-comparator-construction
Skill1 starsChanged 8 months agoMITView on GitHub
Create Comparator Construction experimental design analysis assessing whether baselines and controls are fair and relevant. Counterfactual lens answering "Is the comparator fair and relevant?
In this repository59 other files it gives its agents.
- CLAUDE.mdCLAUDE.md
- arch-lens-c4-containerSkill
- arch-lens-concurrencySkill
- arch-lens-data-lineageSkill
- arch-lens-deploymentSkill
- arch-lens-developmentSkill
- arch-lens-error-resilienceSkill
- arch-lens-module-dependencySkill
- arch-lens-operationalSkill
- arch-lens-process-flowSkill
- arch-lens-repository-accessSkill
- arch-lens-scenariosSkill
- arch-lens-securitySkill
- arch-lens-state-lifecycleSkill
- audit-archSkill
- audit-bugsSkill
- audit-cohesionSkill
- audit-defense-standardsSkill
- audit-frictionSkill
- audit-implSkill
- audit-testsSkill
- design-guardsSkill
- dry-walkthroughSkill
- elaborate-phaseSkill
- exp-lens-benchmark-representativenessSkill
- exp-lens-causal-assumptionsSkill
- exp-lens-error-budgetSkill
- exp-lens-estimand-claritySkill
- exp-lens-exploratory-confirmatorySkill
- exp-lens-fair-comparisonSkill
- exp-lens-governance-riskSkill
- exp-lens-iterative-learningSkill
- exp-lens-measurement-validitySkill
- exp-lens-pipeline-integritySkill
- exp-lens-randomization-blockingSkill
- exp-lens-reproducibility-artifactsSkill
- exp-lens-sensitivity-robustnessSkill
- exp-lens-severity-testingSkill
- exp-lens-unit-interferenceSkill
- exp-lens-validity-threatsSkill
- exp-lens-variance-stabilitySkill
- id-slopSkill
- implement-worktreeSkill
- investigateSkill
- make-arch-diagSkill
- make-experiment-diagSkill
- make-groupsSkill
- make-planSkill
- make-reqSkill
- make-scenariosSkill
- mermaidSkill
- plan-arcSkill
- rectifySkill
- review-approachSkill
- update-architectureSkill
- update-reqsSkill
- update-specsSkill
- verify-diagSkill
- verify-proposed-diagSkill
The file
We split this skill into the rules it gives an agent
11 rules. About 0 are common, 9 are distinctive, and 2 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈9
Distinctive · estimate
This author's own choices. Worth a look.
2
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 30
Do not litter the codebase with useless comments, TODO markers, or explanatory annotations — the skill output and diagram speak for …
- Distinctive: line 35
Check for confounding differences in implementation, tuning, data access, and compute
- Very specific: line 38
BEFORE creating any diagram, LOAD the /mermaid skill using the Skill tool - this is MANDATORY
- Distinctive: line 50
Look for: baseline, control, comparison, prior, state-of-the-art, vanilla, default, reference
- Distinctive: line 54
Look for: reproduce, reimplement, original, paper, author, tuned, optimized, hyperparameter
- Distinctive: line 58
Look for: version, library, framework, gpu, hardware, environment, checkpoint
- Distinctive: line 62
Look for: tune, search, grid, optuna, sweep, budget, trials, epochs
- Distinctive: line 66
Look for: date, published, year, updated, latest, deprecated, legacy
- Distinctive: line 88
If a diagram adds value, create a simplified flowchart.
- Very specific: line 104
Write the analysis to: temp/exp-lens-comparator-construction/exp_diag_comparator_construction_{YYYY-MM-DD_HHMMSS}.md
- Distinctive: line 204
Before creating the diagram, verify:
≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Comparator Construction Experimental Design Lens
Philosophical Mode: Counterfactual Primary Question: "Is the comparator fair and relevant?" Focus: Baseline Choice, Control Realism, Version Matching, Effort Symmetry, Baseline Drift
When to Use
- Benchmark comparisons where baseline quality is questioned
- Ablation studies needing fair controls
- Claims of improvement over prior work
- User invokes
/exp-lens-comparator-constructionor/make-experiment-diag comparator
Critical Constraints
NEVER:
- Modify any source code or experiment files
- Do not litter the codebase with useless comments, TODO markers, or explanatory annotations — the skill output and diagram speak for themselves
- Accept at face value that baselines received symmetric treatment
ALWAYS:
- Build a fairness matrix covering all treatment-vs-comparator pairs
- Check for confounding differences in implementation, tuning, data access, and compute
- Assess whether each comparator is the best available alternative at the time of the experiment
- Identify temporal drift in baseline relevance
- BEFORE creating any diagram, LOAD the
/mermaidskill using the Skill tool - this is MANDATORY
Analysis Workflow
Step 1: Launch Parallel Exploration Subagents
Spawn Explore subagents to investigate:
Baseline/Control Definitions
- Find what the proposed method is compared against
- Look for: baseline, control, comparison, prior, state-of-the-art, vanilla, default, reference
Implementation Parity
- Find whether baselines get equal engineering effort
- Look for: reproduce, reimplement, original, paper, author, tuned, optimized, hyperparameter
Version & Environment Match
- Find whether baselines use the same software/hardware environment
- Look for: version, library, framework, gpu, hardware, environment, checkpoint
Tuning Protocol Symmetry
- Find whether hyperparameter tuning is symmetric
- Look for: tune, search, grid, optuna, sweep, budget, trials, epochs
Temporal Baseline Drift
- Find whether baselines have been updated or are stale
- Look for: date, published, year, updated, latest, deprecated, legacy
Step 2: Build the Comparator Inventory
For each comparator, assess:
- Is it the best available alternative?
- Is it given equal engineering effort?
- Is it run in the same environment?
- Is the tuning budget symmetric?
- Has it drifted since originally published?
Step 3: Construct the Fairness Matrix
CRITICAL — Analyze Counterfactual Quality: For each treatment-vs-comparator pair:
- Does the comparison isolate the intended factor?
- Are there confounding differences in implementation, tuning, data access, or compute?
Build a fairness matrix with rows = comparators, columns = fairness dimensions.
Step 4: Create the Optional Comparison Diagram
If a diagram adds value, create a simplified flowchart. This is OPTIONAL for this hybrid lens — the tables are the primary output.
Direction: LR (treatment and comparator flow in parallel toward evaluation)
Subgraphs: "PROPOSED METHOD", "COMPARATOR(S)", "SHARED EVALUATION"
Node Styling:
cliclass: proposed method nodesphaseclass: comparator method nodeshandlerclass: shared evaluation pipeline nodesoutputclass: results nodesgapclass: asymmetries flaggeddetectorclass: parity checks
Step 5: Write Output
Write the analysis to: temp/exp-lens-comparator-construction/exp_diag_comparator_construction_{YYYY-MM-DD_HHMMSS}.md
Output Template
# Comparator Construction Analysis: {Experiment Name}
**Lens:** Comparator Construction (Counterfactual)
**Question:** Is the comparator fair and relevant?
**Date:** {YYYY-MM-DD}
**Scope:** {What was analyzed}
## Comparator Inventory
| Comparator | Source | Reimplemented? | Same Environment? | Same Tuning Budget? |
|------------|--------|---------------|-------------------|---------------------|
| {name} | {paper/repo} | Yes / No / Partial | Yes / No | Yes / No / Unknown |
## Fairness Matrix
| Comparator | Best Available? | Equal Effort? | Same Env? | Symmetric Tuning? | Temporally Current? |
|------------|----------------|--------------|-----------|-------------------|---------------------|
| {name} | Yes / No | Yes / No | Yes / No | Yes / No | Yes / No |
## Comparison Diagram (Optional)
```mermaid
%%{init: {'flowchart': {'nodeSpacing': 50, 'rankSpacing': 60, 'curve': 'basis'}}}%%
flowchart LR
%% CLASS DEFINITIONS %%
classDef cli fill:#1a237e,stroke:#7986cb,stroke-width:2px,color:#fff;
classDef stateNode fill:#004d40,stroke:#4db6ac,stroke-width:2px,color:#fff;
classDef handler fill:#e65100,stroke:#ffb74d,stroke-width:2px,color:#fff;
classDef phase fill:#6a1b9a,stroke:#ba68c8,stroke-width:2px,color:#fff;
classDef newComponent fill:#2e7d32,stroke:#81c784,stroke-width:2px,color:#fff;
classDef output fill:#00695c,stroke:#4db6ac,stroke-width:2px,color:#fff;
classDef detector fill:#b71c1c,stroke:#ef5350,stroke-width:2px,color:#fff;
classDef gap fill:#ff6f00,stroke:#ffa726,stroke-width:2px,color:#000;
classDef integration fill:#c62828,stroke:#ef9a9a,stroke-width:2px,color:#fff;
subgraph Proposed ["PROPOSED METHOD"]
METHOD["Proposed Method<br/>━━━━━━━━━━<br/>{method name}"]
end
subgraph Comparators ["COMPARATOR(S)"]
COMP1["Comparator 1<br/>━━━━━━━━━━<br/>{name}"]
COMP2["Comparator 2<br/>━━━━━━━━━━<br/>{name}"]
end
subgraph Evaluation ["SHARED EVALUATION"]
EVAL["Evaluation Pipeline<br/>━━━━━━━━━━<br/>{dataset/benchmark}"]
RESULTS["Results<br/>━━━━━━━━━━<br/>{metrics reported}"]
PARITY["Parity Check<br/>━━━━━━━━━━<br/>{asymmetry found}"]
ASYM["Asymmetry<br/>━━━━━━━━━━<br/>{description}"]
end
METHOD -->|"evaluated on"| EVAL
COMP1 -->|"evaluated on"| EVAL
COMP2 -->|"evaluated on"| EVAL
EVAL --> RESULTS
RESULTS --> PARITY
PARITY -.->|"flagged"| ASYM
class METHOD cli;
class COMP1,COMP2 phase;
class EVAL handler;
class RESULTS output;
class PARITY detector;
class ASYM gap;Color Legend:
| Color | Category | Description |
| Dark Blue | Proposed Method | The method being evaluated |
| Purple | Comparators | Baselines and controls |
| Orange | Evaluation | Shared evaluation pipeline |
| Dark Teal | Results | Reported outcomes |
| Red | Parity Checks | Fairness verification points |
| Yellow | Asymmetries | Flagged unfair differences |
Asymmetry Register
| # | Asymmetry | Affects | Impact Assessment | Remediation |
| 1 | {description} | {comparator(s)} | High / Medium / Low | {how to fix} |
Recommendations
- {Most critical fairness fix — e.g., retune baseline with same budget}
- {Version alignment or environment standardization needed}
- {Additional comparator that should be included}
---
## Pre-Diagram Checklist
Before creating the diagram, verify:
- [ ] LOADED `/mermaid` skill using the Skill tool
- [ ] Using ONLY classDef styles from the mermaid skill (no invented colors)
- [ ] Diagram will include a color legend table
---
## Related Skills
- `/make-experiment-diag` - Parent skill for lens selection
- `/mermaid` - MUST BE LOADED before creating diagram
- `/exp-lens-estimand-clarity` - For clarifying what the comparison is measuring
- `/exp-lens-fair-comparison` - For deeper analysis of evaluation protocol fairnessDoes it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

