fprime-run-for-record
Skill12k starsChanged 21 months agoApache-2.0View on GitHub
Use when executing or auditing a run-for-record (stage 3 of the system test process): freezing the manifest, drawing the human-seeded planted faults, running the approved plan once, hashing the record, auditing it blind into quick-looks, and producing the unseal table the human reviews. The mechanics are the `fth` CLI of fprime-testing-harness (manifest, verify, draw, run, audit, hash-record, verify-record, unseal). Keywords: F Prime, run for record, RfR, seal, manifest, fault injection, quick-look, audit, unseal, fth.
- Installs packages
Trending
More talk lately.about the whole repository
Who's talking about nasa/fprime
- Vibernets Streaming“Linux on Mars - NASA Mars Helicopter (අඟහරු ගිය Linux)”YouTube · Mar 3, 2021 · 12k views
- Suck Less | Tym Rabchuk“You Can Actually Build a Spacecraft With This #nasa #space #diy”YouTube · Jun 30, 2026 · 1.7k views
- こぶたNEWS“宇宙で初めての火星ヘリコプター”YouTube · Apr 10, 2021 · 741 views
- Marcia en Desplegando Cloud“🇪🇸#5.17 - Python en Acción (Podcast Charlas Técnicas de AWS)”YouTube · Nov 5, 2024 · 281 views
- Gourcer“Gource visualisation: nasa/fprime on 2021-04-22”YouTube · Apr 22, 2021 · 179 views
The biggest 5 of 71 mentions linking to the repository that our crawl found. Each line is their own headline.
See all 71
Talked about on GitHub, YouTube, Hacker News and the web (71 times).
On GitHub · 50
- Aug 19, 2026Add FPP (F Prime Prime) language — github-linguist/linguist15 reactions · 4 comments
- Sep 21, 2026Add New Darkmode UI Option — nasa/fprime-gds2 reactions
- Jul 31, 2025Deprecate serialize/deserialize in favor of [de]serialize[From/To] — nasa/fpp2 reactions · 1 comment
- Mar 19, 2026Add ConsoleNoWait — fprime-community/fprime-vxworks1 reaction
- Nov 22, 2025Merge release/v3.1.0a12 into release/v3.1.0 — nasa/fpp1 reaction
45 more on GitHub
- Oct 13, 2025Add ComAggregator for nasa/fprime#4264 — fprime-community/fprime-workshop-led-blinker1 reaction
- Dec 20, 2021FPP Visualization Automation — nasa/fprime-tools1 reaction · 5 comments
- Nov 10, 2026Svc/DpWriter: add processing override commands to force data product processing types on or off — lestarch-autobot/fprime0 reactions · 1 comment
- Oct 29, 2026Update for nasa/fprime#6174: add ComQueue drop-counter telemetry to packets — fprime-community/fprime-yamcs-reference0 reactions
- Oct 29, 2026Update for nasa/fprime#6174: add ComQueue drop-counter telemetry to packets — fprime-community/fprime-zephyr-reference0 reactions
- Oct 29, 2026Update for nasa/fprime#6174: add ComQueue drop-counter telemetry to packets — nasa/fprime-examples0 reactions
- Oct 28, 2026Update for nasa/fprime#6278: define FwTlmPacketizeOffsetType in TlmPacketizerCfg.fpp — fprime-community/fprime-zephyr-reference0 reactions
- Oct 28, 2026Restrict checkTimers and tlmWrite to having a single message on the q… — FireflySpace/fprime0 reactions
- Oct 24, 2026Upgrade to F Prime v4.4.1 — Open-Source-Space-Foundation/proves-core-reference0 reactions
- Oct 23, 2026TlmPacketizer: address lestarch-autobot review must-fix items (nasa/fprime#6144) — FireflySpace/fprime0 reactions
- Oct 16, 2026Include annotation for packets from FPP in dictionary — nasa/fpp0 reactions · 3 comments
- Oct 14, 2026Manual merge from nasa devel #5996 (pktType) — FireflySpace/fprime0 reactions
- Oct 12, 2026Svc/GenericHub: drop message with event and counter on allocation failure instead of asserting — JPL-Devin/fprime0 reactions · 1 comment
- Sep 30, 2026CI: fold workflow_dispatch runner/jobs inputs into the Ref and Framework matrices — JPL-Devin/fprime0 reactions · 1 comment
- Sep 22, 2026docs: import LED Blinker timeliness.md into website build — JPL-Devin/fprime0 reactions · 1 comment
- Sep 17, 2026Check C++ formatting — fprime-community/fprime-baremetal0 reactions
- Sep 14, 2026WasmSequencer: give the I32 host resume its own pause-check choice — JPL-Devin/fprime0 reactions · 1 comment
- Sep 11, 2026Fw/Types: serialize bool via U8 path and static-assert pointer size — JPL-Devin/fprime0 reactions · 1 comment
- Sep 1, 2026Rename LinuxUartDriver to PosixUartDriver with deprecated compatibility layer — JPL-Devin/fprime0 reactions · 2 comments
- Aug 22, 2026Feature: DFA-based structural health monitor app for telemetry channels — nasa/cFS0 reactions · 9 comments
- Jul 17, 2026Add TypeDemo component — fprime-community/fprime-yamcs-reference0 reactions
- Jun 22, 2026Add README.md and How-To guide for developing cFS F Prime applications — JPL-Devin/fprime_cfs_reference0 reactions · 2 comments
- Jun 16, 2026Use pr-xxxx pattern for GDS across Integration Test CI — JPL-Devin/fprime0 reactions · 3 comments
- May 29, 2026[#5159] ROSES - Student Review of Rule-Based Testing — lestarch-autobot/fprimesays it tested it0 reactions
- May 29, 2026[#5115] ROSES - Document how to decide between Active/Passive/Queued component - Student Review — lestarch-autobot/fprime0 reactions
- May 29, 2026[#5111] CCSDS Improvements — lestarch-autobot/fprime0 reactions
- May 29, 2026[#5099] DpCatalog Discards Status When Handling StateFile Database — lestarch-autobot/fprime0 reactions
- May 29, 2026[#5051] Add CI run exercising non-default FwType config values — lestarch-autobot/fprime0 reactions
- May 29, 2026[#5018] Replace hand-coded buffer and string classes with instantiated templates — lestarch-autobot/fprime0 reactions
- May 29, 2026[#4846] Add `fprime-util new --config-module` — lestarch-autobot/fprime0 reactions
- May 29, 2026[#4841] DpWriter always deallocates buffers to the same BufferManager — lestarch-autobot/fprime0 reactions
- May 29, 2026[#4806] Change fprime-gds serializable array type representation — lestarch-autobot/fprime0 reactions
- May 29, 2026[#4792] Consolidate Parameters ground scripts — lestarch-autobot/fprime0 reactions
- May 29, 2026[#4745] Update Core components to use FPP constants for string arguments — lestarch-autobot/fprime0 reactions
- May 29, 2026[#4711] fprime_test_api: Sometimes fprime_test_api does not connect to currently running fprime-gds session — lestarch-autobot/fprime0 reactions
- May 29, 2026[#4697] ROSES - Student Review of ISR pattern — lestarch-autobot/fprime0 reactions
- May 29, 2026[#4535] Configurable CPU Affinity for Subtopologies — lestarch-autobot/fprime0 reactions
- May 29, 2026[#4508] ROSES - Student review - Memory management — lestarch-autobot/fprime0 reactions
- May 29, 2026[#4454] VxWorks CI To Build All — lestarch-autobot/fprime0 reactions
- May 29, 2026[#4425] PrmDb filename in subtopology should be configurable — lestarch-autobot/fprime0 reactions
- May 29, 2026[#4324] Switch `FW_PACKET_COMMAND` enumerated value to non-zero value — lestarch-autobot/fprime0 reactions
- May 29, 2026[#4290] Clean up setup — lestarch-autobot/fprime0 reactions
- May 29, 2026[#4099] ROSES - Subtopologies (Working With) - Student Review — lestarch-autobot/fprime0 reactions · 1 comment
- May 29, 2026[#3662] ROSES - Add integration tests for fprime-examples repository — lestarch-autobot/fprime0 reactions
- May 29, 2026[#3565] Add local script to preview website rendering — lestarch-autobot/fprime0 reactions
On YouTube · 13
- Mar 3, 2021Linux on Mars - NASA Mars Helicopter (අඟහරු ගිය Linux) — Vibernets Streaming12k views · 50 comments
- Jun 30, 2026You Can Actually Build a Spacecraft With This #nasa #space #diy — Suck Less | Tym Rabchuk1.7k views · 2 comments
- Apr 10, 2021宇宙で初めての火星ヘリコプター — こぶたNEWS741 views · 3 comments
- Nov 5, 2024🇪🇸#5.17 - Python en Acción (Podcast Charlas Técnicas de AWS) — Marcia en Desplegando Cloud281 views · 2 comments
- Apr 22, 2021Gource visualisation: nasa/fprime on 2021-04-22 — Gourcer179 views
8 more on YouTube
- Feb 21, 2021Gource visualisation: nasa/fprime on 2021-02-21 — Gourcer157 views
- May 2, 2021Ferhell News 3 - Winget, WSL, Banimento da Universidade de Minessota e o Linux no Espaço! — Ferhell Linux & Techs117 views · 14 comments
- Jul 12, 2026GitHub - nasa/fprime: F´ - A flight software and embedded systems framework — GitHub Daily Trend AI Podcast95 views
- Feb 22, 2021Gource visualisation: nasa/fprime on 2021-02-22 — Gourcer70 views
- Oct 10, 2026NASA’s F Prime Turns Flight Software Into Components — Cochran Labs59 views
- Jul 2, 2026NASA fprime - A flight software — RandomTechAce37 views
- Jul 12, 2026nasa/fprime - Gource visualisation — Gourcer36 views
- Jul 4, 2026Почему эти 8 проектов уже изменили будущее? #Shorts — GitHub Находки3 views
On Hacker News · 6
- Feb 18, 2021The open source flight code that the NASA Mars drone is running68 points · 3 comments
- Feb 19, 2021Vue.js is powering NASA flight software4 points
- Jan 31, 2024NASA/fprime: F´ – A flight software and embedded systems framework2 points
- Jan 26, 2024F Prime – Flight software framework by NASA2 points
- Jan 21, 2024F': Ingenuity's Open-Source Flight Software Framework2 points
1 more on Hacker News
- Feb 1, 2024Fprime – A flight software and embedded systems framework by NASA1 point
On the web · 2
- Jul 12, 2026nasa/fprime: F´ - A f… - GitHub Daily Trend - Apple 팟캐스트 — podcasts.apple.com
- Jul 12, 2026nasa/fprime: F´ - A f... - GitHub Daily Trend — podcasts.apple.com
In this repository41 other files it gives its agents.
- AGENTS.mdAGENTS.md
- cmake/AGENTS.mdAGENTS.md
- docs/AGENTS.mdAGENTS.md
- CLAUDE.mdCLAUDE.md
- .github/copilot-instructions.mdCopilot instructions
- agent-skill-authoringSkill
- ai-session-reportSkill
- ai-session-summarySkill
- ci-test-runtime-policySkill
- fprime-checklist-closeoutSkill
- fprime-checklist-designSkill
- fprime-checklist-implementationSkill
- fprime-cmake-build-systemSkill
- fprime-component-checklist-reviewSkill
- fprime-component-design-fppSkill
- fprime-component-developmentSkill
- fprime-component-implementationSkill
- fprime-component-integration-testSkill
- fprime-component-requirementsSkill
- fprime-component-unit-testSkill
- fprime-cpp-designSkill
- fprime-fpp-navigationSkill
- fprime-ground-input-tracingSkill
- fprime-hardware-input-tracingSkill
- fprime-issue-risk-evaluationSkill
- fprime-iterative-developmentSkill
- fprime-maintenanceSkill
- fprime-requirements-reviewSkill
- fprime-system-test-designSkill
- fprime-system-test-harnessSkill
- fprime-test-oracleSkill
- fprime-topology-developmentSkill
- fprime-unit-testingSkill
- jpl-design-principlesSkill
- maintainer-lookupSkill
- post-inline-reviewSkill
- pr-diff-scopingSkill
- prompt-injection-precheckSkill
- re-review-stateSkill
- triage-classifierSkill
- write-system-functional-docSkill
The file
We split this skill into the rules it gives an agent
8 rules. About 0 are common, 2 are distinctive, and 6 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈2
Distinctive · estimate
This author's own choices. Worth a look.
6
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 191
Before it, on a shared bench, the runner checks for foreign processes on the serial port and the debugger (pgrep -fa …
- Distinctive: line 201
Never power-cycle a shared USB hub to clear a port.
- Very specific: line 212
Start the ground system and exactly one recorder from bench.json (gds, recorder), fresh archive under record/raw/ for this run ID.
- Very specific: line 221
Run the plan's stimulus tests once, in plan order, through fth.pytest_plugin (window, send, check, require, bench fixtures.
- Very specific: line 225
Stop the ground system; fth export the raw logs into record/{events,channels,commands}.jsonl + export-counts.json (dedup of the recorder's …
- Very specific: line 318
Every item is program-generated from the plan, the record, the F´ dictionary and the FPP model — no LLM prose anywhere in the package.
- Distinctive: line 369
START marker, reboot rows marked.
- Very specific: line 400
Update the watch ledger: append one evidence row per harness feature the campaign exercised to docs/upstream-watch.md in …
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Skill: F Prime run-for-record
Stage 3 of .github/agents/_shared/test-process-contract.md. Nothing
here designs or fixes; it executes an approved plan against a frozen
UUT and harness and makes the result inspectable. Every artifact
lives in the run directory rfr/<run-id>/ outside both repositories;
<run-id> ≤ 8 characters.
0. Tooling
The mechanics live in the fth CLI of
fprime-testing-harness
(stage-3 subset: fth manifest | verify | draw | seal-id | run | audit | hash-record |
verify-record | unseal | render-ql | vectors | selftest), pinned by version in
.github/agents/_shared/test-process-contract.md §8 — no script ships
beside this skill. Install the pinned commit on every host that runs a
stage-3 step (bench host included):
# fprime-testing-harness is a private repository: `pip install git+https://…` has no credentials and
# fails. Clone it ONCE as `harness` — the same checkout `fth manifest --harness harness` pins — with the
# credentials the host already has (gh auth or ssh; a token URL would land in .git/config, so prefer
# those), check out the pinned commit, install editable FROM that checkout:
git clone https://github.com/LeStarch/fprime-testing-harness harness # or: git@github.com:LeStarch/fprime-testing-harness.git
git -C harness checkout --detach v0.1.2 # annotated tag = 855dd4f (fth frozen: refusal fixes only until the next trial closes)
pip install -e "harness[test]"
python -c "import fth, os; assert os.path.realpath(fth.__file__).startswith(os.path.realpath('harness')), fth.__file__"
fth selftest # exercises every seal subcommand in a temp dirThe installed fth and the pinned harness are therefore one tree: the manifest records its sha, and the
assert above (repeat it in row 1) is what ties the tool that runs to the tree that was pinned.
Stage-3 rows never build the image, so the bench host needs only a fprime-gds compatible with the
UUT's lib/fprime (the version its requirements.txt names) for fth run; fprime-fpp and
fprime-tools are build-host prerequisites (required.cmake refuses older ones) and are NOT required
in the stage-3 venv. Check pip show fprime-gds in row 1, before the bench is touched.
Requirements the tooling must keep meeting (its own test suite carries
them; a tool that drops one is not pinned): a planted fault's fails is a
per-test dict ({"<test>": [predicate…]}; a legacy list expands to
every applies_to test) and unseal matches failed predicates against
that test's entry only; draw reads the menu from the manifest;
unseal needs the menu file at its manifest-relative path.
1. Deliver (closes stage 2) — developer side, host only
The developer runs ONE tool; it fails with a findings list instead of producing a package
when anything is missing, unpinned, dirty, stale or under threshold (Captain's rule: input
validation runs as the deliverable is produced). The runner receives rfr-input.zip +
rfr-input.zip.sha256, runs sha256sum -c and fth verify-package rfr-input.zip (every
manifest-pinned file present with its sha256), unzips it to rfr/rfr-input/ and uses
rfr/rfr-input/manifest.json in every later row — the runner never runs fth manifest.
fth deliver --uut <path> --harness <path> --repo project=<path> --repo fprime=<path> [--repo <lib>=<path> …] \
--plan <plan.json> --menu <faults-menu.json> --bench <bench.json> --tests <test/int dir> \
--image <Deployment>=<built>.bin:"<build command>" --rebuild-cmd "<build command>" --toolchain-json <toolchain.json> \
--sdd <uut>/…/docs/sdd.md --dictionary <dict.json> \
--ut-coverage dev-evidence/ut --system-coverage dev-evidence/system --dev-evidence dev-evidence \
[--version-check-waiver version-check-waiver.txt] # only for images without Svc.Version; the reason is the human trigger
-o rfr-input # -> rfr-input/{manifest.json,inputs/,deliver-report.md}, rfr-input.zip, .sha256Inside it, the manifest (fth manifest, for reference — not a runner step):
fth manifest --uut <path> --harness <path> \
--repo project=<path> --repo fprime=<path> [--repo <lib>=<path> …] \
--plan <plan.json> [--plan-revision <rev>] --tests <test/int dir> \
[--checker reviewer=checks/<name>.py] # legacy plan-owned plans only
--image <Deployment>=<flashed>.bin:"<build command>" \
--menu <faults-menu.json> --bench <bench.json> \
--toolchain-json <toolchain.json> \
--ut-coverage <dev-evidence/ut> --system-coverage <dev-evidence/system> \
--coverage-evidence <dev-evidence/evidence.json> # coverage REPORTS pinned as inputs (image stays pure)
# -> rfr/manifest-<uut sha8>.jsonRecords, for the UUT, the harness (both listed in repos
automatically — tester components and test deployments compile into
the image) and every other repository the image is built from, the
SHA and git describe --tags --always --dirty --broken string (dirty
or broken trees are refused); file hashes of the plan, the generated
fault menu, bench.json and (legacy plans) the reviewer's independent checker; the
plan's test list (tests[].id, unique, ≤ 8 characters); the
toolchain block from --toolchain-json (compiler package and
version as the builder installed them — without it a faithful rebuild
can only be reported REBUILD DIFFERS (toolchain unpinned)); and the
exact raw binaries that will be flashed — each must contain every
repository's describe string, which is what Svc.Version reports, or
--no-version-check <reason> is recorded on the manifest and printed
at unseal (policy: required where the deployment has Svc.Version; the SAMD21 waiver is the plan's
image_version: {"policy": "waived", "reason": …} block, which fth deliver takes as the waiver). ELF, hex and UF2 files are refused (flash the .bin and
read back the range with picotool save -r or OpenOCD dump_image).
The checks live in the integration tests; the manifest pins them as stimulus
(--tests: repo, sha, per-file sha256) and the auditor replays exactly those. A
reviewer's checker is a manifest file only for legacy plan-owned plans. The manifest lives beside
the runs, outside both repositories; fth verify <manifest> must pass
before every later step. Before announcing the draw, stage on the bench
host (where the human draws and unseals) the pinned fth, the
manifest, and every manifest-listed non-repo file at its recorded
relative path (plan, fault menu, bench, checker, the helpers module)
— unseal reads the menu by that path and fails with "missing fault
menu" otherwise; bench.json uses relative paths only for the same
reason. The plan's independent final-state tool (§4, final-state paragraph) must have been
exercised once in the stage-2 rehearsal; an improvised read on the day
yields UNVERIFIED.
2. Draw (human action)
The human runs this in a shell of their own, on any machine holding the
pinned fth and the manifest (the fault menu is embedded in the
manifest; no repository or other file is needed — draw checks only
the manifest file, the full verify is the runner's and auditor's
step). The seed comes from an un-echoed prompt (preferred) or a
chmod 600 file that draw deletes after reading — never a chat
message or a command-line argument — and is kept by the human.
fth draw --manifest rfr/manifest-<sha8>.json --run-dir rfr/<run-id> [--seed-file <file>]Writes rfr/<run-id>/sealed/{faults.json, expectations.json,
sealed.sha256} and rfr/<run-id>/seal-id.txt. The number of planted
tests is drawn in [1, max_count] from the approved menu (no
command-line override); expectations are FAIL for drawn tests,
FAIL-RIPPLE for their planted faults' ripples_to, PASS otherwise. The
draw refuses to uncover a requirement: if a candidate draw leaves
any requirement without an unaltered covering test it re-draws from a
derived sub-seed, deterministically, up to --max-tries (16), then
fails loudly — a plan that trips this has a redundancy defect
(fprime-system-test-design §3) and goes back to stage 2. Only
seal-id.txt (and the sha8 the command prints) is readable by runner
and auditor.
3. Runner
Stage 3 is ONE command, from the campaign root on the bench host, with the delivered package as its only required input (README stage-3 row 1):
fth rfr rfr-input.zip --dry-run [--evidence dev-evidence/] [--ledger ledger.md] [--name <campaign>-<date>] # REHEARSAL GATE first: seed-free, bench-free
umask 077; head -c 32 /dev/urandom | base64 > rfr/seed.txt # the seed: the human's, kept nowhere else
fth rfr rfr-input.zip --seed-file rfr/seed.txt [--out rfr] [--evidence dev-evidence/] [--ledger ledger.md] [--name <campaign>-<date>]The rehearsal (--dry-run) runs rows 1–2 for real (unpack, every preflight check, the real verify), then renders every
command line rows 3–8 would execute — flash/dump, GDS, recorder, run-start marker send, the stimulus pytest argv, audit,
unseal, package — into rfr/<run-id>-rehearsal/rfr.log and refuses on any build-artifacts path, .., absolute host path
(devices excepted) or campaign-root path that does not exist. Its log (and sha256) is the gate evidence before the seed is
requested; the real run id stays free. The harness's own CI runs this rehearsal over fresh git clones of a UUT and of the
harness at the pins (no build outputs) plus a delivered zip (tests/test_rfr_clean_clone.py), and proves the old
uut/build-artifacts/... manifest shape is refused by name.
It runs the manual rows 1–8 in order with the arguments the rows name —
verify-package + unpack, the package's manifest, sealed draw, run with
the pure image, audit + replay with the manifest-pinned coverage reports
rendered, quick-look, unseal, package → rfr/<name>.zip + rfr/<name>.zip.sha256 (sha256sum -c format; the row-8 log line names it) — and refuses
with a NAMED check before the bench when anything a row needs is not in
the package or the campaign root (uut//harness/ at the pins, the
running fth is the pinned harness, bench device present, no foreign
holder). rfr/<run>/rfr.log is the per-row record (start, end, exit,
sha256). The first failing row stops it; nothing is retried; the bench is
not re-contacted. No --unsealed/--dev/--plant, no coverage flags exist
on rfr. The runner types nothing else; the seed is never printed.
The manual expansion (for the record, and when a row must be re-issued by order of the human — e.g. a read-only re-audit), run once per run ID:
fth run --run-dir rfr/<run-id> --manifest rfr/rfr-input/manifest.json \
--plan <plan.json> --bench <bench.json> --tests <deployment>/test/intBefore it, on a shared bench, the runner checks for foreign processes
on the serial port and the debugger (pgrep -fa "fprime-gds|fprime-cli|
openocd|comm_bridge|pytest", plus who holds the target's /dev/tty*)
and writes the result — timestamp, the processes found, which port each
holds — into the run log (rfr/<run-id>-run.log / runner/notes.md)
before launching. A process on another port is noted, not a blocker;
a process on the target's port stops the launch. If a foreign process
kills the ground system mid-run, the run is void as an external
outage (recorded as such in runner/notes.md), not a verdict, and is
rerun once under a new run ID; a second outage is reported to the human.
Never power-cycle a shared USB hub to clear a port.
It performs, in order, and stops at the first failure:
fth verifyof the manifest (repos at their SHAs, clean trees, file hashes, pinnedfth).- Flash exactly the manifest image with the recipe's
flashline, read back the same flash range (dump), and writerecord/raw/flash.logwith the debugger USB serial, the image sha256 and the read-back sha256; a missing serial or a differing read-back stops the run. - Start the ground system and exactly one recorder from
bench.json(gds,recorder), fresh archive underrecord/raw/for this run ID; the command sender logs underrecord/raw/cmd/(the only source ofcommands.jsonl). - Send the run-start markers
MARK(0xFFFF0001),MARK(manifest sha8),MARK(0xFFFF0002),MARK(seal-id sha8),MARK(0xFFFF0003),MARK(crc32(<run-id>))(ids ≥ 0xFFFF0000 are reserved; a deployment withNO_OP_STRINGmay instead send"RFR <run> M <sha8>"/"RFR <run> S <sha8>"— the plan'smarkersblock decides). - Run the plan's stimulus tests once, in plan order, through
fth.pytest_plugin(window,send,check,require,benchfixtures; no fixture exposes the draw); the plugin validatessealed/faults.jsonand applies the drawn planted fault to the drawn test only; stdout and pytest output go torunner/. - Stop the ground system;
fth exportthe raw logs intorecord/{events,channels,commands}.jsonl+export-counts.json(dedup of the recorder's double-written rows happens here, per file per boot segment, exact raw line only);fth hash-record. - Derive
runner/verdicts.jsonfrom pytest's junit (fth.junit; the junit is copied intorecord/before hashing),runner/report.md.
The runner agent adds nothing to the record: every anomaly, operator
action and timing it noticed goes to runner/notes.md. It does not
read sealed/, fix, tune or retry; a mid-run break is recorded and
reported, and a rerun is the human's call with a new run ID. Hand over
the run directory by path; the auditor reads record/,
record.sha256, seal-id.txt and nothing under runner/ or
sealed/.
4. Auditor
fth audit --run-dir rfr/<run-id> --manifest rfr/manifest-<sha8>.json \
--plan <plan.json> --tests <test/int> --bench <bench.json> \
--ut-coverage <dev-evidence/ut> --system-coverage <dev-evidence/system> \
[--coverage-evidence <dev-evidence/evidence.json>] # same dirs the manifest pinned; unpinned ones are stamped post-run review inputs
[--rebuild --toolchain-json <toolchain.json>] # legacy plans: --independent checks/<name>.pyEach step is a line in auditor/report.md; a failed hash or marker step
ends the audit with RECORD INVALID:
fth verifyof the manifest andfth verify-record(every export file present, row counts equalexport-counts.json); the run-start markers inrecord/events.jsonlmatch this manifest, this run ID andseal-id.txt;record/raw/flash.logcarries the debugger serial and a read-back sha equal to a manifest image sha (RECORD INVALID (image)otherwise).- With
--rebuild: rebuild from the manifest SHAs with the recordedbuild_cmdand toolchain and reportREBUILD MATCHorREBUILD DIFFERS (<exactly which pinned item differs>)— cross-compiled images often embed paths; a differing rebuild is reported, notRECORD INVALID. - Re-export from a copy of
record/raw/and compare withrecord/*.jsonl(the independent count); replay never touches the archive in place. - Record integrity findings (one START..END per window, no command
outside a window, monotonic time, export counts, junit hashed, no
duplicate test), then replay the same test functions over the
record (
--tests <test/int>) →auditor/junit.xml→auditor/verdicts.jsonand diff verdict andfailedlist per test againstrunner/verdicts.json→auditor/diff-runner-replay.md; anyDIVERGENCEor integrity finding is a human trigger. Legacy plans with plan-owned pass rules: engine [+--independentchecker] →auditor/diff.mdas before. - Render the quick-look pages,
index.md(metadata, VnV matrix) andql.zipintoauditor/ql/— by program only (fth render-qlinsideaudit). No agent writes, edits or summarises a QL page, ever; the auditor's words go toauditor/report.md.
Then, outside the tool: the plan's independent final-state check after
the runner has released the bench and before any reflash — UUT powered
down or halted, a non-F´ tool — written to auditor/final-state.md
(narrative) and auditor/final-state.json: {"test": "L7-1",
"verdict": "PASS|FAIL", "tool", "version", "cmdline",
"expected_sha256", "observed_sha256"}; unseal adds it as its own row
and a missing file is a human trigger. auditor/report.md closes with
every UNVERIFIED window, every divergence, the final-state result,
the rebuild result and the blindness statement (contract §4). Do not
open runner/ or sealed/; do not guess which failures were planted.
fth audit --skip-repo-check exists for hosts without the repos
(tooling CI, vectors) and is development-only: it prints
repo_check: skipped in the QL index metadata and a ⚠ line under every
page header, so a campaign audit that used it is visible to the human
on every page.
5. Quick-look pages
auditor/ql/<id>.md, rendered from the hashed record and the plan,
byte-identical across renders: title and purpose; checker verdict;
covers; the pass rule with each predicate's result and the record
lines (L/C/K = events/channels/commands jsonl line numbers) that
decided it; completeness (commands sent vs dispatched, drops);
rationale lines; hex beside decimal where the rule uses hex; a "vs
plan" column that is an observation, never attributed to a planted fault. The
pass rule is on the page so the human checks the evidence against the
rule, not against anyone's word. index.md carries the run metadata
(manifest, seal id, repo_check) and the requirement × test VnV
matrix.
Simulator evidence is labelled SIMULATED in the manifest and on the
pages and is never a record; the record requires the device to pass the
bench entry gate (answers a probe at the configured baud) first.
5a. QL package contract (acceptance checklist)
Every item is program-generated from the plan, the record, the F´ dictionary and the FPP
model — no LLM prose anywhere in the package. The tooling's tests/test_ql_contract.py
asserts each item; a package missing one is not deliverable.
Package files: index.md; one <test id>.md per plan test; sdd.md (the UUT's SDD with
<a id="REQ-…"></a> anchors, its Tests table derived from the plan — the renderer fails on
drift); commands.md (every command the plan uses: component.mnemonic, opcode hex and
decimal, each argument with type, enum values and meanings, FPP doc-comment; the events
whose doc-comment names the command; dictionary sha256); helpers/<Component>.md for every
plan helpers entry (FPP description, port diagram, commands → commands.md, events with
id/severity/format/args, channels with id/type, parameters, source files with sha256 linked
at the manifest's project sha); an HTML twin of every page (index.html entry point,
<page>.html beside each .md, one offline style.css; .md links rewritten to .html,
heading and REQ-… anchors preserved; ```mermaid blocks pre-rendered to inline SVG by
mmdc when the host has it, else the vendored assets/mermaid.min.js renders them from
file:// — no CDN, no plugin, no timestamps; the .md stays the diffable source); inputs/
(the package is self-contained: manifest.json, plan.json, faults-menu.json, bench.json,
the pinned independent checker and the sibling modules it imports (legacy plans only), the image(s) the
manifest names with its pinned sha256 and the flash read-back sha, the F´ dictionary, and
inputs/README.md listing each file, its sha256, the manifest pin and the stage-3 row that
consumes it — and, as DATA-ONLY rows (repo@sha path sha256, "pinned, not shipped"), the integration tests,
their checks and the planted-fault module the manifest pins as stimulus (fth manifest --tests): the auditor
replays exactly those sources or FAILS; they are never copied into the package — a sha256 that disagrees with a pin is a loud failure; fth unseal adds unseal.md
and seed.txt (verified to reproduce the draw) — the seed enters the package only after the
unseal, never before; tests, stimulus and tester sources stay in the pinned repositories); ql.zip of
all of it; stage-3 row 8 "Package" is fth package --run-dir rfr/<run> --manifest rfr/<package>/manifest.json --name <campaign>-<date> [--evidence dev-evidence/] [--ledger ledger.md] -o <campaign>-<date>.zip — the FINAL delivery is assembled and verified by the tool from the unsealed run (root index.md/index.html with the unseal-attributed headline, pins, coverage, VnV matrix, every test page, unseal.md, dev-evidence, reports, watch rows; ql/ beneath; refuses a sealed run or a missing link; byte-reproducible — run it twice and compare the sha256; the index names the packaging harness beside the audited-under manifest pin). A deviation ledger is a human/LLM document passed in with --ledger; nothing else in the delivery is hand-assembled. The image stays pure, the coverage REPORTS are inputs (Captain's ruling, rfr1 close-out): an instrumented image is never the qualified image and fth run never takes one, but the development campaign's coverage reports (dev-evidence/ut, dev-evidence/system, gcovr data files only) enter the RfR package — pinned by fth manifest --ut-coverage --system-coverage --coverage-evidence dev-evidence/evidence.json (row 2) or, failing that, recorded by sha256 at fth audit --ut-coverage --system-coverage [--coverage-evidence] (row 5) — as inputs/coverage/ rows and a coverage.md page (thresholds from the plan's coverage:, provenance block: coverage image sha256, compiler, dev-campaign run ids — provenance, not equivalence). --dev stamps development runs only; it no longer gates coverage.
index.md: run metadata (manifest sha8, record sha256, seal sha8, dictionary sha256, SDD
sha256, repo-check state); the per-test table (test, title, checker verdict, unseal
expectation, unseal verdict, page); ONE VnV matrix requirement × test with cells
P / E / F / U / · and an "Overall" column (any F → F; any P and no F → P; only E/U → U;
waiver quoted when no test covers) and the legend stating the cell meanings and the
roll-up rule; a "Test helper components" section; tests natural-sorted by id everywhere
(execution order is the plan's business, not the index's).
Each test page, in this order: ← back to [index](index.md) under the H1 and as the last
line; header table — checker verdict, Unseal expectation, Unseal verdict (stamped by
fth unseal), "What this test does" (the plan's purpose), Covers (each REQ linked to
sdd.md#REQ-…), Harness (link to each helper page), Window start / Window end / Window
duration as separate rows per window with epoch; §1 Pass rule — one check per line
(#, window, group, check, plain-words description) with a Rationale line for every value a
rule deliberately ignores; §2 Procedure — plan steps aligned to sent commands with K lines,
every command name linked to commands.md, non-command actions explained ("not a
command — reboot visible as FSW clock restart …"); §3 "Possible Injected Faults" — sealed
draw line and the planted fault | drawn | checks-it-may-break table (a — none — row when no planted fault
applies, never an empty table, no "acted" column) filled by fth unseal, the italic note
says so; §4 Rule verdict (#, window, check, value seen, holds), the verdict line, the
unseal callout (⚠ when more checks failed than the drawn planted fault allows, a planted test
PASSED or an unplanted test FAILED; otherwise "consistent with draw"), the footnote that
counters are boot-cumulative and only Δ is judged, the caption that a handler's event
precedes its OpCodeCompleted; §5 Events — complete, nothing elided, Δ from the window
START marker, reboot rows marked; §6 Telemetry — one row per channel (samples, first/last
with C line and FSW, "all = v" / "changed: a → b" / "monotonic a → b (n distinct)");
§7 Commands last, with a "vs plan" column flagging arg differences and commands not in the
plan as OBSERVATIONS only ("Observation: command K14 differs …" singular when one). Hex
beside decimal wherever a rule uses hex (0x3FFC0 vs 262080).
6. Unseal (human action)
fth unseal --run-dir rfr/<run-id> --manifest rfr/manifest-<sha8>.json \
--seed-file <the Captain's seed file> -o rfr/<run-id>/unseal.md # the seed enters the package here, verified to reproduce the drawVerifies the integrity chain (contract §6), then writes the check-off
table ☐ | test | expected | runner | auditor | quick-look | outcome
for every test on the manifest plus one <test>/dump row per entry in
auditor/final-state.json (expected from the drawn planted fault's class, not
the test's expectation), flags planted or ripple tests whose failed
predicates exceed the planted fault's fails entry for that test, shows any
skipped image version check, and lists the human triggers first with
the seal id to compare with the value printed by draw. It also stamps
every QL page with Unseal expectation / Unseal verdict, inserts the
§3 planted fault table (name, class, kind, mechanism; — none — when nothing
applies) and a callout (⚠ WARNING for uncovered checks, a planted PASS
or an unplanted FAIL; else "Consistent with draw") into index.md, and
rebuilds ql.zip. The human reads trigger rows, spot-checks two or
three nominal rows against the raw record, ticks the boxes, and records
the decision in the plan's decision log.
7. Campaign close (human + agent, after unseal)
- Update the watch ledger: append one evidence row per harness feature the campaign exercised to
docs/upstream-watch.mdin fprime-testing-harness (<campaign> <run id> (<date>): <finding>; movestatustoearnedwhen the row's criterion is met). The QL index already links the ledger rows this run relied on (watch ledgerfield) at the harness sha the manifest pins — a ledger edit lands on main and is picked up by the next manifest, never by re-rendering a sealed run. - Archive the
fth packagezip (+ its sha256) with the campaign record; note anyDIVERGENCE, integrity finding orUNEXPLAINED FAILas a tracked item.
Anti-patterns
- A second run "to confirm" without a new run ID and a stated reason.
- Quick-looks written, edited or summarised by an agent, or built from the runner's report or harness output.
fth audit --skip-repo-checkon a campaign run.- An auditor that announces which failures it believes were planted.
- A seed typed into a chat, a prompt, or a command line.
- Replaying the archive in place.
- Running pytest, the exporter or the engine by hand in stage 3 instead
of
fth run/fth audit.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

