agentleFS
elodin-syselodin-sys/elodin/.cursor/skills/elodin-monte-carlo/SKILL.md

elodin-monte-carlo

Skill554 starsChanged 5 months agoApache-2.0View on GitHub

Develop and calibrate simulations against experimental truth data using elodin monte-carlo. Use when vendoring real telemetry as a reference profile, adding a truth-replay ghost entity, writing campaign specs/hooks and run scoring, reconstructing missing data channels from physics, or iterating on guidance and physics models with campaign feedback.

The file

Show its rules and text

We split this skill into the rules it gives an agent

17 rules. About 0 are common, 8 are distinctive, and 9 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈8

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 9

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Run the campaign after every meaningful change.

    line 26
  2. Very specific:

    Keep the verbatim source file untouched in data/ next to a derived, SI-unit version the sim actually loads.

    line 32
  3. Very specific:

    Write a sanity_check() in the reference module that re-derives the cleaned data from the raw file and asserts agreement, plus checks …

    line 34
  4. Very specific:

    Put all truth handling in one stdlib-only Python module (see examples/apollo-lander/reference.py) shared by the sim and any external …

    line 51
  5. Very specific:

    Document the uncertainty (~±10% is typical) and assert anchors in sanity_check().

    line 78
  6. Very specific:

    Never give the ghost an el.Body: If physics systems match it, gravity integrates its velocity while replay snaps its position — sawtooth …

    line 105
  7. Very specific:

    Do not drive the ghost from pre_step DB writes: explicit writes land on their own clock and interleave with telemetry commits, producing …

    line 108
  8. Distinctive:

    Put sim-vs-truth pairs on every KDL graph, and include raw measurement curves (e.g.

    line 115
  9. Very specific:

    Keep the LHS seed fixed while iterating so campaign-to-campaign deltas reflect your changes, not resampling.

    line 136
  10. Very specific:

    Use [[build]] steps in campaign.toml to compile external FSW (and generate configs) once before workers start.

    line 143
  11. Distinctive:

    Verify exports are clean: row counts equal across entities, no empty cells, no physically impossible values.

    line 174
  12. Distinctive:

    Use the editor for visual regressions (attitude behavior, ghost overlap, terrain seating).

    line 176
  13. Distinctive:

    When a controller tracks the truth profile, the campaign will find every weakness.

    line 182
  14. Distinctive:

    If the feed-forward is *not* flyable (demands more than the actuator), the tracker oscillates no matter the gains — fix the reference …

    line 189
  15. Distinctive:

    Prefer one unified tracking law with state-dependent limits over discrete phase switches.

    line 192
  16. Distinctive:

    Add terminal logic: near the end, stop chasing the time-indexed reference and null residual errors — event timing shifts with small …

    line 196
  17. Very specific:

    Stale processes hold UDP ports: Prefer direct world.recipe() sidecars plus s10 readiness probes.

    line 205

≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Truth-Data-Driven Simulation with Elodin Monte Carlo

The most effective way to build a credible simulation is to anchor it to experimental truth data and use elodin monte-carlo as the test harness: every model change is judged by a 30-run campaign against recorded reality, in under a minute. This skill codifies that workflow. The canonical worked example is examples/apollo-lander/ (with the full methodology in its WHITEPAPER.md).

The development loop

vendor truth data -> build reference module -> sim + truth ghost + graphs
      ^                                                    |
      |                                                    v
 narrow spec.toml <- read report <- run campaign <- score runs vs truth
                         |
                         v
              export worst run CSV -> diagnose time series -> fix model

Run the campaign after every meaningful change. Distribution deltas (RMSE, success rate, margins, dispersion) tell you immediately whether a change helped, broke something, or just moved noise.

1. Vendor truth data with provenance

  • Keep the verbatim source file untouched in data/ next to a derived, SI-unit version the sim actually loads. Cite the source URL in README/docs.
  • Write a sanity_check() in the reference module that re-derives the cleaned data from the raw file and asserts agreement, plus checks documented anchor values. Make python reference.py print the profile and the check.
  • Distrust every column. Real-world traps hit in practice: a "meters" column converted at x0.3405 instead of x0.3048 (recompute conversions from the original-unit column); digitized charts with out-of-order and duplicate timestamps (sort, average duplicates); head samples from a different measurement frame (a state-vector update step) that must be trimmed and back-extrapolated; a sensor channel that is not the state you want (radar slant range is not altitude).
  • Cross-validate datasets against each other and against documented events: mission reports, transcripts, and timelines give anchor points (event times, velocity callouts, fuel consumed) that catch unit and frame errors no amount of smoothing will.

2. Build a dependency-free reference module

Put all truth handling in one stdlib-only Python module (see examples/apollo-lander/reference.py) shared by the sim and any external controller:

  • Resample onto a uniform grid, despike with a median filter, smooth with a moving average. Only enforce monotonicity if the physics demands it — real profiles wiggle, and the wiggles are history worth keeping.
  • Expose interpolators (altitude(t), descent_rate(t), ...) plus the raw arrays. In sim.py, convert once to jnp.asarray and close over them in systems: they become JIT-time constants (large baked constants are interned and shared across campaign workers).

3. Reconstruct missing channels from physics

Truth datasets are rarely complete. When a needed channel is missing (e.g. horizontal velocity existed only as a chart image), reconstruct it:

  • Integrate the vehicle dynamics along the channels you do have, using documented schedules (throttle history, event times) for the rest.
  • Allocate consistently: if total thrust is known and the vertical share is implied by the recorded altitude, the horizontal share is the remainder — this makes the reconstructed profile flyable by construction, which matters when a controller later tracks it (see section 7).
  • Calibrate segment-by-segment through documented anchors so the profile passes through known values exactly; integrate the best-documented segment unscaled and let it yield the unknown initial condition.
  • Iterate fixed-point if channels couple (two passes usually converge).
  • Document the uncertainty (~±10% is typical) and assert anchors in sanity_check().

4. Truth ghost: replay reality inside the sim

Render the recorded vehicle next to the simulated one. The robust pattern:

# Kinematic ghost: NO el.Body — gravity/integrator/telemetry systems never match it.
world.spawn([
    StaticSceneObject(el.WorldPos(...)),           # world_pos only
    el.C(TruthMarker, jnp.array([1.0])),           # scopes the playback query
    el.C(Altitude, ...), el.C(VerticalSpeed, ...), # graph channels
], name="lander_truth")

@el.system
def truth_playback(
    tick: el.Query[el.SimulationTick], truth: el.Query[TruthMarker]
) -> el.Query[el.WorldPos, Altitude, VerticalSpeed]:
    t_s = tick[0] * SIM_TIME_STEP
    alt = jnp.interp(t_s, ref_time, ref_altitude)
    ...
    return truth.map((el.WorldPos, Altitude, VerticalSpeed), lambda _: (...))

Hard-won rules:

  • Never give the ghost an el.Body. If physics systems match it, gravity integrates its velocity while replay snaps its position — sawtooth motion, runaway velocities in the graphs, and corrupted exports.
  • Do not drive the ghost from pre_step DB writes: explicit writes land on their own clock and interleave with telemetry commits, producing CSV exports with holes. The in-sim el.SimulationTick playback keeps truth on exactly the simulated vehicle's commit clock, so exports line up row-for-row.
  • A marker component is the cheapest way to scope a playback query to one entity.

Put sim-vs-truth pairs on every KDL graph, and include raw measurement curves (e.g. slant range next to true altitude) — visible divergence between a sensor and the state teaches more than hiding it.

5. Campaign as the test harness

  • el.monte_carlo.params_spec(...) defaults drive single runs; spec.toml drives campaigns. Keep ranges mirrored, and anchor ranges in data (a fuel chart pins the propellant load; navigation accuracy pins IC dispersions). When control authority is saturated (e.g. a full-throttle braking burn), keep IC ranges tight — the real system could not recover big errors either, and neither can yours.
  • Score every run against truth in post_step and emit el.monte_carlo.result(traj_rmse=..., pitch_rmse=..., miss_distance=..., soft_landing=...). Fit metrics (RMSE vs truth) are what turn a campaign from a stress test into a calibration engine: report the best-fit run's params, narrow spec.toml around them, repeat (or automate the loop, see examples/apollo-lander/calibrate.py).
  • Latch event metrics in-sim, at the event. A touchdown-speed metric read one tick after contact reads the zeroed post-contact state; capture it in the same system that detects the event, before state is clobbered.
  • Keep the LHS seed fixed while iterating so campaign-to-campaign deltas reflect your changes, not resampling. If you edit spec.toml, re-run elodin monte-carlo sample (the runner warns when a --plan CSV is older than its sibling spec).
  • Control concurrency with --workers N / workers = N in campaign.toml (exactly N runs at once). S10_MAX_INFLIGHT is only the low-level process-count escape hatch.
  • Use [[build]] steps in campaign.toml to compile external FSW (and generate configs) once before workers start; [env] sets campaign-wide process environment. Put per-worker resources under [resources.ports] — numeric bases give a validated static plan, "auto" allocates per run — and read them with el.monte_carlo.port("name", default) or ELODIN_MC_PORT_<NAME>.
  • At scale: scratch_dir = "auto" moves per-run DB IO to tmpfs when the artifact disk can't sustain workers x write IOPS; [retention] prune_on_pass/prune_on_fail globs bound disk without hook-side cleanup; export_db(..., pattern="rocket.*") exports only scored components; and a [quality] max_behind_deadline_frac = 0.05 gate marks load-degraded real-time runs degraded instead of silently ingesting skewed samples. Failed runs carry a one-line failure_reason in results.csv.

6. Diagnose with the data, not by staring at code

  • Export any run and interrogate it with quick Python:

    elodin-db export dbs/<campaign>/runs/run_0000012/db \
        --format csv --join --flatten -o /tmp/run12

    Print a time-series table of the suspect channels (state, reference, command, actuator) every N seconds — most control/physics bugs are obvious in 20 rows.

  • Rank runs by the failing metric (post_run_result.json per run) and export the worst one, not the best.
  • Treat "too perfect" as a bug. Metrics that are identically zero or pegged to a clamp value usually mean the measurement is wrong (read after state-clobber, trivially satisfied criterion), not that the system is great.
  • Verify exports are clean: row counts equal across entities, no empty cells, no physically impossible values. Data quality bugs masquerade as physics.
  • Use the editor for visual regressions (attitude behavior, ghost overlap, terrain seating); confirm emergent event times against the historical record.

7. Tracking recorded profiles: control lessons

When a controller tracks the truth profile, the campaign will find every weakness. Patterns that survived:

  • Feed-forward the reconstructed profile; bound the feedback. Clamp each feedback channel's authority (e.g. ±0.8 m/s²) around the flyable feed-forward. Unbounded feedback lets channels fight over a saturated actuator and limit-cycle.
  • If the feed-forward is not flyable (demands more than the actuator), the tracker oscillates no matter the gains — fix the reference (section 3), not the gains.
  • Prefer one unified tracking law with state-dependent limits over discrete phase switches; let phases emerge (a speed-dependent tilt budget reproduces a pitchover; a demand-dependent throttle latch reproduces throttle-down).
  • Add terminal logic: near the end, stop chasing the time-indexed reference and null residual errors — event timing shifts with small tracking errors, and arriving early must not mean arriving with the reference's residual velocity.
  • Capped, slowly-fading position trim erodes IC offsets without destabilizing the velocity loop.

8. Operational pitfalls

  • Stale processes hold UDP ports. Prefer direct world.recipe() sidecars plus s10 readiness probes. Linux campaigns use cgroup teardown to reap reparented daemons; --keep-existing opts out of scoped pre-reaping.
  • Return valid = False from post_run hooks for infrastructure failures so they are reported separately from scored misses.
  • CI hygiene per repo rules: ruff format && ruff check --fix for the Python, cargo fmt/cargo clippy -- -Dwarnings for external controllers.

Reference example

examples/apollo-lander/ exercises every pattern above: vendored NASA telemetry with unit-bug handling and anchor checks, a physics-based horizontal-profile reconstruction, a kinematic truth ghost on el.SimulationTick, an external Rust LGC tracking the profile through a SITL UDP bridge, campaign hooks producing fit metrics and landing-ellipse statistics, and a calibration loop. Its WHITEPAPER.md documents the full methodology and the honest caveats.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.