fprime-system-test-design
Skill12k starsChanged 21 months agoApache-2.0View on GitHub
Use when designing a system-level or hardware-in-the-loop test plan for an F Prime component, driver, or subsystem that will be run and audited by agents (stage 2 of the system test process, or the test-design phase of component development). Produces a requirements-traced plan (`plan.json`, rendered to `plan.md` by `fth render-plan`) with a one-sentence purpose and a structured, machine-evaluable pass rule per test, planted-fault-disjoint redundancy per requirement, numeric metrics, a fault-injection menu, and an honest not-covered list. Keywords: F Prime, system test, HIL, test plan, traceability, pass rule, oracle, fault injection, qualification, fth, plan.json.
Trending
More talk lately.about the whole repository
Who's talking about nasa/fprime
- Vibernets Streaming“Linux on Mars - NASA Mars Helicopter (අඟහරු ගිය Linux)”YouTube · Mar 3, 2021 · 12k views
- Suck Less | Tym Rabchuk“You Can Actually Build a Spacecraft With This #nasa #space #diy”YouTube · Jun 30, 2026 · 1.7k views
- こぶたNEWS“宇宙で初めての火星ヘリコプター”YouTube · Apr 10, 2021 · 741 views
- Marcia en Desplegando Cloud“🇪🇸#5.17 - Python en Acción (Podcast Charlas Técnicas de AWS)”YouTube · Nov 5, 2024 · 281 views
- Gourcer“Gource visualisation: nasa/fprime on 2021-04-22”YouTube · Apr 22, 2021 · 179 views
The biggest 5 of 71 mentions linking to the repository that our crawl found. Each line is their own headline.
See all 71
Talked about on GitHub, YouTube, Hacker News and the web (71 times).
On GitHub · 50
- Aug 19, 2026Add FPP (F Prime Prime) language — github-linguist/linguist15 reactions · 4 comments
- Sep 21, 2026Add New Darkmode UI Option — nasa/fprime-gds2 reactions
- Jul 31, 2025Deprecate serialize/deserialize in favor of [de]serialize[From/To] — nasa/fpp2 reactions · 1 comment
- Mar 19, 2026Add ConsoleNoWait — fprime-community/fprime-vxworks1 reaction
- Nov 22, 2025Merge release/v3.1.0a12 into release/v3.1.0 — nasa/fpp1 reaction
45 more on GitHub
- Oct 13, 2025Add ComAggregator for nasa/fprime#4264 — fprime-community/fprime-workshop-led-blinker1 reaction
- Dec 20, 2021FPP Visualization Automation — nasa/fprime-tools1 reaction · 5 comments
- Nov 10, 2026Svc/DpWriter: add processing override commands to force data product processing types on or off — lestarch-autobot/fprime0 reactions · 1 comment
- Oct 29, 2026Update for nasa/fprime#6174: add ComQueue drop-counter telemetry to packets — fprime-community/fprime-yamcs-reference0 reactions
- Oct 29, 2026Update for nasa/fprime#6174: add ComQueue drop-counter telemetry to packets — fprime-community/fprime-zephyr-reference0 reactions
- Oct 29, 2026Update for nasa/fprime#6174: add ComQueue drop-counter telemetry to packets — nasa/fprime-examples0 reactions
- Oct 28, 2026Update for nasa/fprime#6278: define FwTlmPacketizeOffsetType in TlmPacketizerCfg.fpp — fprime-community/fprime-zephyr-reference0 reactions
- Oct 28, 2026Restrict checkTimers and tlmWrite to having a single message on the q… — FireflySpace/fprime0 reactions
- Oct 24, 2026Upgrade to F Prime v4.4.1 — Open-Source-Space-Foundation/proves-core-reference0 reactions
- Oct 23, 2026TlmPacketizer: address lestarch-autobot review must-fix items (nasa/fprime#6144) — FireflySpace/fprime0 reactions
- Oct 16, 2026Include annotation for packets from FPP in dictionary — nasa/fpp0 reactions · 3 comments
- Oct 14, 2026Manual merge from nasa devel #5996 (pktType) — FireflySpace/fprime0 reactions
- Oct 12, 2026Svc/GenericHub: drop message with event and counter on allocation failure instead of asserting — JPL-Devin/fprime0 reactions · 1 comment
- Sep 30, 2026CI: fold workflow_dispatch runner/jobs inputs into the Ref and Framework matrices — JPL-Devin/fprime0 reactions · 1 comment
- Sep 22, 2026docs: import LED Blinker timeliness.md into website build — JPL-Devin/fprime0 reactions · 1 comment
- Sep 17, 2026Check C++ formatting — fprime-community/fprime-baremetal0 reactions
- Sep 14, 2026WasmSequencer: give the I32 host resume its own pause-check choice — JPL-Devin/fprime0 reactions · 1 comment
- Sep 11, 2026Fw/Types: serialize bool via U8 path and static-assert pointer size — JPL-Devin/fprime0 reactions · 1 comment
- Sep 1, 2026Rename LinuxUartDriver to PosixUartDriver with deprecated compatibility layer — JPL-Devin/fprime0 reactions · 2 comments
- Aug 22, 2026Feature: DFA-based structural health monitor app for telemetry channels — nasa/cFS0 reactions · 9 comments
- Jul 17, 2026Add TypeDemo component — fprime-community/fprime-yamcs-reference0 reactions
- Jun 22, 2026Add README.md and How-To guide for developing cFS F Prime applications — JPL-Devin/fprime_cfs_reference0 reactions · 2 comments
- Jun 16, 2026Use pr-xxxx pattern for GDS across Integration Test CI — JPL-Devin/fprime0 reactions · 3 comments
- May 29, 2026[#5159] ROSES - Student Review of Rule-Based Testing — lestarch-autobot/fprimesays it tested it0 reactions
- May 29, 2026[#5115] ROSES - Document how to decide between Active/Passive/Queued component - Student Review — lestarch-autobot/fprime0 reactions
- May 29, 2026[#5111] CCSDS Improvements — lestarch-autobot/fprime0 reactions
- May 29, 2026[#5099] DpCatalog Discards Status When Handling StateFile Database — lestarch-autobot/fprime0 reactions
- May 29, 2026[#5051] Add CI run exercising non-default FwType config values — lestarch-autobot/fprime0 reactions
- May 29, 2026[#5018] Replace hand-coded buffer and string classes with instantiated templates — lestarch-autobot/fprime0 reactions
- May 29, 2026[#4846] Add `fprime-util new --config-module` — lestarch-autobot/fprime0 reactions
- May 29, 2026[#4841] DpWriter always deallocates buffers to the same BufferManager — lestarch-autobot/fprime0 reactions
- May 29, 2026[#4806] Change fprime-gds serializable array type representation — lestarch-autobot/fprime0 reactions
- May 29, 2026[#4792] Consolidate Parameters ground scripts — lestarch-autobot/fprime0 reactions
- May 29, 2026[#4745] Update Core components to use FPP constants for string arguments — lestarch-autobot/fprime0 reactions
- May 29, 2026[#4711] fprime_test_api: Sometimes fprime_test_api does not connect to currently running fprime-gds session — lestarch-autobot/fprime0 reactions
- May 29, 2026[#4697] ROSES - Student Review of ISR pattern — lestarch-autobot/fprime0 reactions
- May 29, 2026[#4535] Configurable CPU Affinity for Subtopologies — lestarch-autobot/fprime0 reactions
- May 29, 2026[#4508] ROSES - Student review - Memory management — lestarch-autobot/fprime0 reactions
- May 29, 2026[#4454] VxWorks CI To Build All — lestarch-autobot/fprime0 reactions
- May 29, 2026[#4425] PrmDb filename in subtopology should be configurable — lestarch-autobot/fprime0 reactions
- May 29, 2026[#4324] Switch `FW_PACKET_COMMAND` enumerated value to non-zero value — lestarch-autobot/fprime0 reactions
- May 29, 2026[#4290] Clean up setup — lestarch-autobot/fprime0 reactions
- May 29, 2026[#4099] ROSES - Subtopologies (Working With) - Student Review — lestarch-autobot/fprime0 reactions · 1 comment
- May 29, 2026[#3662] ROSES - Add integration tests for fprime-examples repository — lestarch-autobot/fprime0 reactions
- May 29, 2026[#3565] Add local script to preview website rendering — lestarch-autobot/fprime0 reactions
On YouTube · 13
- Mar 3, 2021Linux on Mars - NASA Mars Helicopter (අඟහරු ගිය Linux) — Vibernets Streaming12k views · 50 comments
- Jun 30, 2026You Can Actually Build a Spacecraft With This #nasa #space #diy — Suck Less | Tym Rabchuk1.7k views · 2 comments
- Apr 10, 2021宇宙で初めての火星ヘリコプター — こぶたNEWS741 views · 3 comments
- Nov 5, 2024🇪🇸#5.17 - Python en Acción (Podcast Charlas Técnicas de AWS) — Marcia en Desplegando Cloud281 views · 2 comments
- Apr 22, 2021Gource visualisation: nasa/fprime on 2021-04-22 — Gourcer179 views
8 more on YouTube
- Feb 21, 2021Gource visualisation: nasa/fprime on 2021-02-21 — Gourcer157 views
- May 2, 2021Ferhell News 3 - Winget, WSL, Banimento da Universidade de Minessota e o Linux no Espaço! — Ferhell Linux & Techs117 views · 14 comments
- Jul 12, 2026GitHub - nasa/fprime: F´ - A flight software and embedded systems framework — GitHub Daily Trend AI Podcast95 views
- Feb 22, 2021Gource visualisation: nasa/fprime on 2021-02-22 — Gourcer70 views
- Oct 10, 2026NASA’s F Prime Turns Flight Software Into Components — Cochran Labs59 views
- Jul 2, 2026NASA fprime - A flight software — RandomTechAce37 views
- Jul 12, 2026nasa/fprime - Gource visualisation — Gourcer36 views
- Jul 4, 2026Почему эти 8 проектов уже изменили будущее? #Shorts — GitHub Находки3 views
On Hacker News · 6
- Feb 18, 2021The open source flight code that the NASA Mars drone is running68 points · 3 comments
- Feb 19, 2021Vue.js is powering NASA flight software4 points
- Jan 31, 2024NASA/fprime: F´ – A flight software and embedded systems framework2 points
- Jan 26, 2024F Prime – Flight software framework by NASA2 points
- Jan 21, 2024F': Ingenuity's Open-Source Flight Software Framework2 points
1 more on Hacker News
- Feb 1, 2024Fprime – A flight software and embedded systems framework by NASA1 point
On the web · 2
- Jul 12, 2026nasa/fprime: F´ - A f… - GitHub Daily Trend - Apple 팟캐스트 — podcasts.apple.com
- Jul 12, 2026nasa/fprime: F´ - A f... - GitHub Daily Trend — podcasts.apple.com
In this repository41 other files it gives its agents.
- AGENTS.mdAGENTS.md
- cmake/AGENTS.mdAGENTS.md
- docs/AGENTS.mdAGENTS.md
- CLAUDE.mdCLAUDE.md
- .github/copilot-instructions.mdCopilot instructions
- agent-skill-authoringSkill
- ai-session-reportSkill
- ai-session-summarySkill
- ci-test-runtime-policySkill
- fprime-checklist-closeoutSkill
- fprime-checklist-designSkill
- fprime-checklist-implementationSkill
- fprime-cmake-build-systemSkill
- fprime-component-checklist-reviewSkill
- fprime-component-design-fppSkill
- fprime-component-developmentSkill
- fprime-component-implementationSkill
- fprime-component-integration-testSkill
- fprime-component-requirementsSkill
- fprime-component-unit-testSkill
- fprime-cpp-designSkill
- fprime-fpp-navigationSkill
- fprime-ground-input-tracingSkill
- fprime-hardware-input-tracingSkill
- fprime-issue-risk-evaluationSkill
- fprime-iterative-developmentSkill
- fprime-maintenanceSkill
- fprime-requirements-reviewSkill
- fprime-run-for-recordSkill
- fprime-system-test-harnessSkill
- fprime-test-oracleSkill
- fprime-topology-developmentSkill
- fprime-unit-testingSkill
- jpl-design-principlesSkill
- maintainer-lookupSkill
- post-inline-reviewSkill
- pr-diff-scopingSkill
- prompt-injection-precheckSkill
- re-review-stateSkill
- triage-classifierSkill
- write-system-functional-docSkill
The file
We split this skill into the rules it gives an agent
6 rules. About 0 are common, 3 are distinctive, and 3 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈3
Distinctive · estimate
This author's own choices. Worth a look.
3
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 19
Write the plan a human approves before any test code is built, and that the runner, auditor, and human all read later.
- Very specific: line 154
Every value a rule deliberately ignores or masks (a status bit, a boot-cumulative counter judged only by Δ, a tolerance) has a rationale …
- Very specific: line 164
Redundancy: Every requirement with system-test coverage is covered by ≥ 2 tests whose planted fault sets are disjoint, so no drawn planted …
- Distinctive: line 171
Prefer EVR sequences a human can check off (start → progress → complete with counts).
- Distinctive: line 180
Every expected constant cites its source (datasheet section, ICD).
- Very specific: line 260
Only a uut-state planted fault changes what the independent final-state dump should find.
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Skill: F Prime system test design
Write the plan a human approves before any test code is built, and
that the runner, auditor, and human all read later. Design from the
requirements, SDD and FPP model; open the code — if any exists — last.
Governing rules are in
.github/agents/_shared/test-process-contract.md.
1. Inputs
- Human goals for the qualification (one paragraph, quoted verbatim).
- Requirements list (REQ-IDs) and
docs/sdd.mdof the UUT. - The deployment the tests run in. Default: the project's full
system deployment on the complete testbed (contract §0). Run the
L0 feasibility check first — the plan's marker (
NO_OP_STRING, or a testerMARK(U32)where the deployment has no string command) must round-trip byte-exact through the archive and the plan's event rate must fit the downlink; if either fails on the system deployment, present a one-line test-deployment decision with that constraint, and wait for the human. When the plan is written before the code exists (test-design phase), do the check from the deployment's configuration and ground-system documentation and write the plan's first line asL0: desk-checked; bench confirmation pending— the stage-2 refresh (§7) confirms it on the bench. A project with no deployment is the same human call (contract §2): ask first; without one, test in a temporary uncommitted deployment. - The FPP model: ports, commands, events, telemetry and parameters are the observables and the fault-injection surface (planted faults are harness configuration, §4).
- UUT code, when it exists, read after the plan's test entries (§3)
exist, to locate observable counters. In the development lifecycle the
plan is written before implementation (
phase-review-loop.md, test-design phase) and this input is absent. - The device datasheet named in the requirements Introduction, with the bench identification reply beside it. If the bench device is not the datasheet's device, the plan is not written: the designer sends the mismatch back as a requirements reopen.
2. Plan structure
The plan is plan.json; plan.md is rendered from it (fth
render-plan, byte-stable) and never edited by hand. Keys, predicate
grammar and the test entry are defined in the tooling's
docs/plan-format.md (pinned in test-process-contract.md §8);
fth verify-plan must pass before a plan is handed to anyone — an
unknown helper, planted fault or requirement or an unparsable predicate is a
load error, never a verdict.
Coverage targets are part of the plan and are set here, in the design
phase, then confirmed at design review (design-review): "coverage":
{"ut": {"lines_min": N, "branches_min"?: N}, "system": {"lines_min": N,
"branches_min"?: N}, "not_covered_ack": [...], "rationale": "..."}.
Baselines: UT lines_min ≥ 95 (100 where practical — the host build can
force every error branch); on-target lines_min 75 (system-caused error
branches cannot be forced without breaking test-as-you-fly). The planner
may move either with the reason stated in rationale. Branches are
always reported, thresholded only when branches_min is set (C++ inserts
hidden branches; the on-target gcovr run excludes throw/unreachable
branches). fth verify-plan refuses a plan without the block; fth
deliver fails on it; the dev campaign's reports are judged against it
and enter the RfR package as data inputs.
0 Scope UUT, deployment, bench, ground system, out of scope
1 Entry gates what must be true before a run (SHAs, marker round-trip, downlink
budget, bench recipe); stage-1 close (unit and component integration
tests pass) is an entry gate, not an RfR test
2 Test levels L0 provenance and feasibility · L2 functional · L3 persistence/reset
L4 endurance/performance · L5 fault paths · L6 negative controls
L7 final state for independent verification
3 Test entries one block per test (§3)
4 Traceability `requirements`: REQ-ID → {tests, verification, note}; redundancy rule below, or one row in the SDD's "Out-of-scope failure modes" table (no Waivers section)
Helpers `helpers`: [{instance, component, src, repo}] — every harness-only component of the
test deployment (e.g. FramTester); the package renders a succinct SDD page per helper
5 Metrics numeric thresholds with units, decided here, never relaxed later
6 Fault menu planted faults the harness must support to plant failures (§4)
7 Not covered what this plan cannot prove on this bench, and why
8 Decision log numbered rulings with who decided and whenThe SDD's ## Tests table is DERIVED, never hand-edited: fth render-sdd-tests plan.json
docs/sdd.md rewrites it (and every requirement's Verification cell that names a system
test) between <!-- fth:tests:begin --> / <!-- fth:tests:end --> markers from the plan's
purpose and covers, in natural test-id order; commit the result on the UUT branch with
the plan. The package renderer (render-ql --sdd) fails when the SDD's test set differs
from the plan's, so the SDD and the VnV matrix cannot drift. What the quick-look package
must contain is the "QL package contract" in fprime-run-for-record §5a — design to it.
Where it lives: plan.json, plan_helpers.py and the stimulus tests are
committed in the deployment branch beside the test-helper component
(fprime-system-test-harness "Layout rule") — never in the tooling repo.
3. One test entry
plan.json tests[] entry (rendered form in parentheses):
id L2-6 # unique, ≤ 8 chars; the manifest's test list
title Full-device write/verify sweep
purpose One sentence: what a PASS proves, in the requirement's words.
covers ["REQ-FRAM-003", "REQ-FRAM-007"]
setup <state the UUT must be in>
procedure ["MARK(60)", "bracket (2 samples)", "SWEEP(seed=0x5005, 4096, 0x5005)",
"action: OpenOCD reset run", "MARK(61)"] # NAME(args), bracket, wait, action:
windows [{"name": "W", "start_id": 60, "end_id": 61, "reset": false}]
metrics ["<predicate with units>"] # judged beside the test's checks of the window
# no `complete` / `all_of` since phase 1: the named checks are declared IN the integration test
# (`check("<predicate>")`, `require("STD")` inside `with window(...)`; `fprime-system-test-harness` §3)
rationale [{"pred": "<predicate>", "text": "why this value is ignored / masked"}]
knobs ["verify_wrong_seed"] # planted faults that may be drawn against this test (key name kept)
evidence <which EVR sequence or telemetry strip a human should see>Rules:
purposeis one sentence per test; the QL page and the SDD test table print it, so it names the observable outcome, not the procedure.- One set of named checks per test, declared in its test function; no
"verify visually", no "as expected". Infrastructure preconditions
(
require) and UUT facts (check) are separate (fprime-test-oracle§1): a precondition that does not hold makes the testUNVERIFIED(Incomplete); a failed check ormetricspredicate isFAIL. Captain's rule: unverified only when the absence itself is not indicative of a UUT failure assuming a working system — never put a UUT fact inrequire(a hung UUT would read as a bench problem). - Predicates are structured text from the grammar of
docs/plan-format.md(event X(args) count == n,event A precedes event B,delta/first/last(ch),last(ch) & mask == v,commands_sent == n,t(B) - t(CMD sent) <= N s, the completeness forms,<packet> samples >= n) or a reference to a registered@helper({"helper", "text", "args"}). Nothing else: prose, regexes and inline code are load errors. The predicate text is the key the verdict files, the planted faultfailslists andfth unsealmatch byte-for-byte. - Every value a rule deliberately ignores or masks (a status bit, a
boot-cumulative counter judged only by Δ, a tolerance) has a
rationaleline naming the predicate and the source of the value (datasheet section, bench reading confirmed at stage 2). - A window that spans a reboot carries
"reset": true; the plan's top-levelreboot_min_backstep_sis the FSW-clock fall-back the exporter counts as a new boot epoch (default 1 s), and a reset window must hold exactly one epoch boundary. Judge the two boots withsegment-aware helpers or per-boot predicates;durationis undefined across a reboot. - Redundancy. Every requirement with system-test coverage is
covered by ≥ 2 tests whose planted fault sets are disjoint, so no drawn planted fault
can remove the prime-path verification of a requirement (
fth drawrefuses a draw that would). A requirement that cannot get a second planted-fault-disjoint test carries an explicit waiver in itsrequirementsentry:verificationtext naming no test (e.g.UT only,Inspection) and anotesaying why. - Prefer EVR sequences a human can check off (start → progress → complete with counts); telemetry strips when counters are the evidence; never prose from the runner.
- Negative controls are real tests: a request that must be rejected has a pass rule that the rejection event appears and nothing else happened (counters unchanged, no bus traffic).
- Include at least one test whose evidence lies outside the UUT's own reporting path (independent dump, external instrument, datasheet constant) so a UUT that lies about itself is caught.
- Every expected constant cites its source (datasheet section, ICD).
- L0 includes an image-identity test: its procedure sends
version.VERSION ALL; its pass rule is that theSvc.Versionpayloads in the window match the manifest'srepos[].describestrings:ProjectVersionandFrameworkVersionequal the project and framework describe; eachLibraryVersionsentry is<library>@<describe>(F Prime bakesgit describe --tags --always --dirty --brokenof each repository into the image; a tagged commit describes as the bare tag; payloads are 40-character strings, so the harness repo and every library must describe in ≤ 40 characters or the comparison is on the truncated prefix and the plan says so). A deployment withoutSvc.Versionis a stage-2 finding. - A final test leaves the UUT in a state the auditor can verify independently (e.g. a known pseudo-random image whose generator and seed are handed over).
Observability rules for pass-rule predicates (found by running the checker):
Svc.TlmChanreports one value per channel per flush, so a false→true→false→true within one tick coalesces. Judge transitions by events, channels by last value; never predicate on a channel sequence across a tick.- Count commands with
cmdDisp.OpCodeDispatchedevents from the START marker row to the END event, not withCommandsDispatcheddeltas (the END-marker sample lands on either side of the window with flush phase). - A write-on-change counter never emitted since boot has delta 0; a boot-zero counter first seen inside a window has baseline 0. Say which channels are which in the plan.
- Telemetry you predicate on must be scheduled: a
BufferManagerwithoutschedInwired emits nothing. - A setup/re-apply that re-acknowledges settings doubles acknowledgement counts — bound counts "before t(SETUP.sent)" when the procedure ends with a restoring setup.
4. Fault menu
The menu is generated from the @planted_fault declarations of the plan's
helpers module (fth planted-faults-menu plan.json, fprime-test-oracle §5);
the designer writes the declarations, never the JSON. Planted faults are
command rewrites or insertions applied by the harness to the drawn test
only, so a planted fault is invisible to the checkers and absent from
harness output. A uut-state planted fault's disturbing command does appear in
commands.jsonl; the auditor is forbidden to reason from it (contract
§5). Generated shape:
{ "max_count": 2,
"knobs": [
{ "name": "verify_wrong_seed", "class": "harness-expectation",
"applies_to": ["L2-6", "L7-1"],
"fails": ["event SweepComplete(kind=VERIFY, pages=2048, errors=0) count == 1"],
"mechanism": "harness verifies against seed+1" },
{ "name": "corrupt_page", "class": "uut-state",
"applies_to": ["L2-6", "L3-1"], "ripples_to": ["L7-1"],
"fails": ["event SweepComplete(kind=VERIFY, pages=2048, errors=0) count == 1"],
"mechanism": "harness overwrites one page between WRITE and VERIFY" },
{ "name": "short_write", "class": "stimulus",
"applies_to": ["L2-2"],
"fails": ["delta(framTester.BytesVerified) == 4096"],
"mechanism": "harness sends length-1 and still expects the full length" }
] }max_countis the most tests the stage-3 draw may plant; the menu must cover at least2 × max_counttests, and the draw picks a hidden number in[1, max_count].class:harness-expectation(the checker's expectation is wrong),uut-state(device content or state is disturbed),stimulus(the drive differs from the plan). At least one planted fault isuut-state: expectation-only planted faults prove the checker reads the harness's error event, not that the test catches a defect.fails: the pass-rule predicates, verbatim, that the planted fault breaks. At unseal any other failed predicate on a planted test is an unexplained failure and a human trigger.ripples_to: tests that legitimately fail when this planted fault is planted (a disturbed device stays disturbed), as a list or as a map{"L7-1": ["<predicate>"]}naming the predicates that fail there (map form enables the unexplained-failure check on ripples). Verified in the dry run; the draw expectsFAIL-RIPPLEfor them.- Only a
uut-stateplanted fault changes what the independent final-state dump should find; aharness-expectationorstimulusplanted fault drawn on the final test leaves the device correct and the dump is expected to PASS. - Planted faults that need a different build, a different test ID, or a visible flag are not acceptable.
- Together with §3's redundancy rule: a planted fault may apply to several tests, but for every requirement at least one covering test must carry no planted fault that any other covering test also carries.
5. Hand-off
Two audiences, two artifacts:
- The human reads the SDD only — one document, never a separate
pack or zip. Add a Tests section to
docs/sdd.mdthat is one table and nothing else — no preamble, no paragraphs — with three columns:Test(name or ID),Description(one sentence, ≤ 15 words, on what it proves),Requirements(REQ-IDs verified). It covers the system tests of this plan and the unit tests (fprime-component-unit-test), one row each; no procedure, stimulus, pass-rule, metric, menu or decision-log columns. A REQ-ID with no test gets one row whose Description names the reason. That table is the test-design gate's quick-look. - The builder and the reviewer read the plan. Deliver
plan.json(§2 structure, one entry per test;fth verify-planpasses), its renderedplan.md, the helpers module with the@planted_faultdeclarations and the generated fault menu, outside the component tree; the plan also declares the development-evidence coverage thresholds,coverage: {"lines_min": %, "branches_min": %, "not_covered_ack": ["why each uncovered path is acceptable (waiver id, UT covers it, not plantable on the bench)", …]}— read byfth dev-campaign(harness skill §3a), never by the RfR commands; an uncovered path that no acknowledgement names is a finding; it is the baseline the harness implements and both checkers evaluate, and it is not pasted into the SDD or into chat. Reviewer findings are recorded per the loop's record rule (phase-review-loop.md, "Record and gate"); the dispositions and the disagreement log are appended to §8 of the plan. The plan goes to the human only when the loop's stop rule is met (phase-review-loop.md, Bound and Rounds).
6. Stage-2 exit gate
Stage 2 closes only when all three hold, on the bench, with the frozen plan (the third is development evidence — a review input, not an RfR input):
- Mutation vectors (
fth vectors plan.json vectors/): one clean full-plan record that PASSes every test, and one single-knob record per planted fault that FAILs exactly that planted fault'sfailsset on its target (andripples_to, FAIL not UNVERIFIED) while every other test passes. Records live undervectors/<name>/withvector.json(kind: clean | single-knob | regression (Kept data names, not vocabulary: the"knobs"plan/menu key, thesingle-knobvector kind,faults-menu.json,sealed/faults.json, and the plan.md labelsPlanted faults:—fth import-planalso accepts the olderFault knobs:/Fault planted faults:. Prose and code identifiers say *planted fault*.)) and run in CI; a vector that no longer judges as declared blocks the manifest. - Stage-3 rehearsal, end to end with a throwaway seed on a
scratch manifest:
fth draw→fth run→fth audit→fth unsealwithout a tooling error, the harness reading that realsealed/faults.json(fram-r1 was voided at run start because the harness had only ever read a hand-written dry-run fault file). - Development evidence (
fth dev-campaign, harness skill §3a) exits 0: UT coverage and on-target coverage at or above the plan'scoveragethresholds with the uncovered lines acknowledged, and the pure run on the RfR image PASSes every test with engine = checker.
7. Refresh at stage 2
A plan approved in stage 1 (before code) is refreshed when stage 2
starts on the UUT SHA: confirm L0 on the bench and replace the
desk-checked line; read the UUT code — last — for observable counters
and update §3 evidence paths where a counter is better than a report;
re-check every fault-menu planted fault is still applicable. The reviewer sees
the diff once (system-test-review Mode A on the changed entries, one
review; an empty diff needs none), then the human re-approves it
(contract §2, plan change); an empty diff is reported as such. No test
is added, removed or weakened here without the human's decision.
Anti-patterns
- Designing from the code: tests that restate what the implementation does rather than what the requirement demands.
- Metrics without thresholds, or thresholds chosen after seeing data.
- "Adversary re-runs the tests" — the auditor reads the record; it does not generate a second one.
- Reduced deployments chosen for convenience rather than by the human.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

