design-experiment

Purpose

Translate a falsifiable hypothesis into a statistically defensible experiment. Factorial, ablation, comparison, scaling, and robustness are modes.

When to use / not applicable

Use after a hypothesis has an operational construct and discriminating prediction. Not applicable when the hypothesis or measurable outcome is absent.

Input contract

mode_contracts:
  factorial: &experiment_input
    required: [falsifiable_hypothesis, outcome, factors, constraints]
    optional: [baseline, candidate_models, budget, robustness_axes]
    constraints: [analysis_plan_must_be_preregistered]
  ablation: *experiment_input
  comparison: *experiment_input
  scaling: *experiment_input
  robustness: *experiment_input

Execution protocol

Do not perform called SOP operations inline; each loaded SOP owns its contract and thresholds.

1. You MUST load skill identify-variables to operationalize the outcome, factors, controls, and their functional roles. You MUST load skill enumerate-dimension-values to define admissible levels or perturbation values. If the resource envelope or feasibility constraints cannot support an executable design, consider analyze-constraints-readiness before committing to a mode.
2. Choose factorial, ablation, comparison, scaling, or robustness mode.
3. You MUST load skill specify-metrics to preregister metrics, estimands, directionality, and decision thresholds. You MUST load skill estimate-sample-size to derive the sample or repetition requirement. You MUST load skill select-statistical-method to select the inference or estimation method before observing outcomes; specify power and the stopping rule. If the proposed metric or validator may share artifacts, labels, or assumptions with the system under test, consider audit-validator-independence before freezing the analysis plan.
4. You MUST load skill construct-design-matrix to construct the runnable matrix for the selected mode. You MUST load skill design-randomness-protocol to define seeds, repetitions, and propagation rules. You MUST load skill specify-execution-environment to capture interpretation-relevant hardware, software, data, configuration, and versions. You MUST load skill specify-reproducibility-protocol to define and test the intended reproduction level. You MUST load skill optimize-design-under-budget to select a feasible information-efficient design under the declared resource envelope; include resource and failure checks. Once the design has produced observations and the task changes from planning to inference, analyze-experiment-results may be the better next tactic.

Mode branches

For comparison or robustness, You MUST load skill select-experimental-baseline to choose a controlled baseline matched to the claim.

• factorial: vary multiple factors in a structured design so main effects and interactions are estimable within the declared resource envelope.
• ablation: remove or replace components systematically to attribute the outcome to individual parts and suspected interactions. You MUST load skill map-ablation-components to define ablatable units, dependencies, and legal removal or replacement operations.
• comparison: evaluate a target against controlled baselines with matched confounds, compute, tuning effort, and preregistered tests.
• scaling: instantiate geometric or otherwise justified scale points to test how the outcome changes across the declared regime.
• robustness: perturb relevant conditions or inputs and measure whether the claimed effect survives the defined stress space.

Output contract

mode_contracts:
  factorial:
    produces: [factor_level_matrix, factor_level_catalog, estimands_main_effects_interactions, metric_significance_plan, sample_power_plan]
    delta_fields: [findings, decisions, uncertainties, open_questions]
  ablation:
    produces: [ablation_matrix, baseline_anchors_full_minimal, attribution_contrasts, component_interaction_plan]
    delta_fields: [findings, decisions, uncertainties, open_questions]
  comparison:
    produces: [controlled_baseline_comparison, matched_confound_controls, seed_environment_protocol, statistical_comparison_plan, reproducibility_protocol]
    delta_fields: [findings, decisions, uncertainties, open_questions]
  scaling:
    produces: [scaling_axes, geometric_scale_points, scaling_experiment_grid, curve_fit_plan, scale_budget_plan]
    delta_fields: [findings, decisions, uncertainties, open_questions]
  robustness:
    produces: [perturbation_stress_matrix, severity_axes, baseline_comparison, degradation_metrics, survival_criteria]
    delta_fields: [findings, decisions, uncertainties, open_questions]

Thresholds and quality gates

• experiment-design HARD-GATE and Budget Gate remain mandatory; no exit before declared minimum yield.
• Resource and run-scale gates are relative to the declared resource envelope and eligible factor/condition space; record numerator, denominator, batch increment, stopping reason, and source references.
• Scale, ablation, comparison, and robustness coverage must reach a justified relative floor over the declared design space; do not substitute an unreasoned fixed run count.
• Factor levels, comparison baseline, statistical test, significance threshold, sample-size rationale, and stopping rule must all be explicit.
• Significance threshold must be pre-registered, not chosen post-hoc.
• Budget-constrained design must report at least one feasible design under the stated resource envelope.
• Scaling mode retains the source criterion of a geometric progression, typically 4-8 points, while evaluating coverage relative to the declared scale domain. Relative audit: declared universe = eligible scale domain; numerator = scale points instantiated; batch increment = points added per design pass; stopping reason = predeclared coverage or saturation rule; source references = scaling-design ledger and design records; direction/threshold rationale = geometric spacing preserves regime sensitivity, with 4-8 retained as the source band.

Failure and counterexamples

Reject post-hoc factor selection, outcome leakage, missing control, unpowered comparison, or a design whose claimed inference exceeds its measured outcome.

Provenance map

experiment-design, factor-level-design, ablation-design, comparison-design, scaling-design, robustness-design, statistical-method-selection, reproducibility-protocol, budget-constrained-design; mode-specific steps retained, agent dispatch removed.

Legacy context checkpoint / Delta notes

Record hypothesis ID, design mode, factors/levels, analysis plan, budget, preregistration status, and unresolved threats.

Preserved source criteria ledger

Context checkpoint / Delta notes

Record hypothesis ID, design mode, factors/levels, analysis plan, budget, preregistration status, and unresolved threats.