simulation-regression
Skill177 starsChanged 7 days agoApache-2.0View on GitHub
Reproduce autonomous-driving behavior with the right replay or closed-loop harness, explicit assertions, frozen fixtures, and regression evidence. Use for replay reproduction, closed-loop simulation, scenario regression, first-failure-frame analysis, and converting issues into regressions.
In this repository19 other files it gives its agents.
- AGENTS.mdAGENTS.md
- .github/copilot-instructions.mdCopilot instructions
- agent-evaluationSkill
- algorithm-developmentSkill
- buildSkill
- change-reviewSkill
- container-buildSkill
- engineering-solution-designSkill
- interface-migrationSkill
- issue-localizationSkill
- lintSkill
- long-horizon-developmentSkill
- model-artifact-integrationSkill
- multi-repo-devSkill
- performance-optimizationSkill
- repository-agent-setupSkill
- test-impact-analysisSkill
- unified-core-runtimeSkill
- vendorSkill
The file
We split this skill into the rules it gives an agent
9 rules. About 0 are common, 8 are distinctive, and 1 is very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈8
Distinctive · estimate
This author's own choices. Worth a look.
1
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 10
Use this skill to reproduce existing issues and validate scenarios affected by algorithm or engineering changes.
- Distinctive: line 47
Use the harness's isolation mechanism.
- Distinctive: line 60
Verify fixtures and runtime prerequisites first so a scenario that never triggered cannot be reported as “passing.”
- Distinctive: line 71
Use original evidence to reconstruct the first failing frame, decision, or state, and compare relevant inputs and outputs across versions.
- Distinctive: line 82
Use numerical tolerances permitted by the contract.
- Distinctive: line 93
After minimization, verify that the scenario still triggers the same mechanism.
- Very specific: line 100
If using the existing Planning sim-agent, first read ad_agents/sim_agent/skills/planning-simulation/SKILL.md, and preserve its Plan …
- Distinctive: line 106
Follow the authorization requirements of other existing harnesses.
- Distinctive: line 121
When the environment or fixture is invalid, stop behavioral evaluation and diagnose the environment first.
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Simulation, Replay, and Scenario Regression
Scope and prerequisites
Use this skill to reproduce existing issues and validate scenarios affected by
algorithm or engineering changes. Do not launch the full stack for every local
change.
First define the behavior to prove and its scope, then read the owning
module's contract and existing scenarios.
For commands, runtime environments, and evidence scope, see the authoritative
registries in the skill index. Before execution, verify the
actual harness and its help; do not invent scenario IDs, parameters,
assertions, or artifact paths.
For an unknown root cause, use issue-localization first; do not use extensive
simulation as a substitute for diagnosis.
Workflow
1. Select an execution model for the problem
| --- | --- | --- |
| Model | Can demonstrate | Cannot demonstrate alone |
| Local unit or component test | Mathematical properties, state transitions, and local interfaces | Whole-vehicle interactions and real sensor behavior |
| Open-loop log replay | First-failure reconstruction and module-output comparison for recorded inputs | Closed-loop outcomes after new decisions change the ego vehicle or environment |
| Closed-loop simulation | Behavior under decision, control, and environment feedback | Real-world safety outside the simulation model's coverage |
| HIL or road vehicle | Specific behavior coupled to hardware and the actual environment | General correctness across all scenarios |
Choose the smallest model that supports the current conclusion. Confirm which modules the harness actually runs, which outputs come from the recording, and which are recomputed; do not let old outputs mask the change under test. Road tests and hardware operations require explicit authorization and established safety procedures; ordinary regression must not escalate to them automatically.
2. Freeze reproducibility conditions
Record code and build versions, configuration, map, route, vehicle parameters, input package, model version, module set, random seed, and clock/replay rate, trimming the list to the scenario. Specify initial state, cleanup between runs, and recording-package integrity. Use the harness's isolation mechanism; do not manually clear shared directories or interrupt another person's process. Reuse existing scenarios first. If a new scenario is necessary, retain only the essential trigger conditions and use legitimate data sources.
3. Define decidable assertions before execution
Express expected behavior using input preconditions, observations, time windows, thresholds, and evaluation methods. Include proof that trigger conditions occurred, outputs are valid, and behavioral constraints related to the change are met. Check collision, feasibility, tracking error, completion conditions, or deadlines as relevant; do not invent criteria. Verify fixtures and runtime prerequisites first so a scenario that never triggered cannot be reported as “passing.”
Distinguish environment or startup failures, invalid scenarios, failed behavioral assertions, skips, and passes. Missing required topics, probes, or metrics cannot count as a pass; a successful process exit does not mean assertions passed. Visualization can aid analysis but does not replace computable assertions.
4. Execute and analyze the first failure
Use original evidence to reconstruct the first failing frame, decision, or
state, and compare relevant inputs and outputs across versions.
Closed-loop trajectories may diverge quickly. Prioritize checking divergence
preconditions and the first causal difference; do not force alignment of
subsequent frames that have already diverged.
For performance claims, hand off to performance-optimization.
Preserve failure evidence first; do not rerun without bounds until a pass appears. If instability is suspected, repeat a bounded number of times under fixed conditions and record every result, seed, and failure rate. Choose the number of repetitions based on the problem and cost. Use numerical tolerances permitted by the contract. Do not require bit-for-bit consistency from nondeterministic systems or hide instability by loosening assertions.
5. Turn a stable reproduction into a regression
Once trigger conditions and correct outcomes are clear, reduce the real issue to a minimal scenario executable by an existing harness. Preserve necessary timing and interactions; do not copy the entire incident package. A regression for a fix should detect the original defect and pass after the fix; avoid snapshots of current output alone. After minimization, verify that the scenario still triggers the same mechanism; similar symptoms do not prove the mechanisms are the same. Maintain scenario assertions, fixture versions, and sources. Archive a separate knowledge document only when requested by the user.
Planning sim-agent boundary
If using the existing Planning sim-agent, first read
ad_agents/sim_agent/skills/planning-simulation/SKILL.md,
and preserve its Plan validation, exact-plan approval, Worker, and
deterministic-evaluation boundaries.
This skill does not approve plans, replace that executor, or automatically
start an agent workflow.
Follow the authorization requirements of other existing harnesses; do not
invent a generic approval tool.
Deliverables and acceptance
Report the behavioral question, execution model and limitations, frozen conditions, scenario and assertions, actual execution status, first-failure or comparison findings, artifact locations, and uncovered risks. Link results to the relevant code and fixtures. Preserve raw logs, structured results, and available JUnit/probe artifacts; use the actual harness outputs as evidence and do not fabricate missing files. A passing software simulation does not establish road or hardware validation.
Failure handling and basis
When the environment or fixture is invalid, stop behavioral evaluation and diagnose the environment first. Ask before changing configuration, dependencies, data, or authorization boundaries. Report unavailable hardware or data as evidence gaps. Reuse the execution models and first-failure-frame methods from context, as well as deterministic assertions from the existing Planning skill. See the skill index for evidence acceptance principles; do not copy external execution or deployment procedures.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

