MiroSwarm Experiment Protocol

Every experiment must be reproducible from git checkout + one command.

Required structure for every experiment script

# experiments/<exp_name>.py

from src.simulator import Simulator
from src.config import default_config
import json, csv
from pathlib import Path
import argparse

def run(seed: int, condition: str, out_dir: Path):
    cfg = default_config()
    cfg.condition = condition  # 'hybrid' | 'gradient' | 'homogeneous' | 'no_memory'
    cfg.seed = seed
    sim = Simulator(cfg)
    metrics = sim.run()
    # write metrics to CSV
    ...

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("--seeds", type=int, default=20)
    parser.add_argument("--out", type=str, default="data/<exp_name>")
    args = parser.parse_args()
    out = Path(args.out)
    out.mkdir(parents=True, exist_ok=True)

    for condition in ["hybrid", "gradient", "homogeneous", "no_memory"]:
        for seed in range(args.seeds):
            run(seed, condition, out)

    # write meta.json with full config + git SHA + timestamp
    ...

Mandatory rules

1. Minimum 20 seeds per condition. Statistical claims need replicates.
2. One CSV per (condition, seed) with columns: t, mean_dist_to_source, success, connectivity, ....
3. meta.json at the experiment root containing: full config dict, git SHA (git rev-parse HEAD), Python version, package versions (pip freeze), timestamp, command line.
4. Never overwrite data. If the output dir exists and is non-empty, refuse to run unless --force is passed.
5. Print progress. Use tqdm over the seed loop so the user sees live progress.
6. Set the random seed at every level: numpy, Python random, and any LLM calls (where supported via the API's seed parameter).

Reporting results

After an experiment completes, generate a paired plot under figures/<exp_name>/ showing mean ± std per condition, plus a small results table in reports/<exp_name>.md with columns: condition, mean final-distance, success rate, mean wall-clock, n.

Statistical comparison

When comparing conditions, always:

• Report mean ± std (not just mean).
• Use a non-parametric test (Mann-Whitney U) for pairwise comparisons.
• State sample size n explicitly in the figure caption.
• Never claim "X is better than Y" without citing the test result.

LLM cost discipline

Hybrid runs cost real money. Before running a sweep:

• Estimate cost: n_seeds × steps_per_seed × LLM_calls_per_step × cost_per_call.
• If estimate > $5, ask the user before running.
• Cache LLM responses by (persona, memory_hash, neighbor_summary_hash) so identical reasoning contexts don't re-bill.