fprime-test-oracle
Skill12k starsChanged 21 months agoApache-2.0View on GitHub
Use when writing or implementing the success condition of a system test as code over a recorded run: the pass-rule format, the exported record it reads, the completeness-versus-UUT split, the live-vs-replay rule (the same test functions, two executions), and the verdict and divergence formats shared by the test designer, reviewer, builder, runner, and auditor; the `@planted_fault` fault-injection declarations and the sealed faults file. Keywords: F Prime, test oracle, pass rule, checker, verdict, record, planted fault, faults.json, fth, divergence.
Trending
More talk lately.about the whole repository
Who's talking about nasa/fprime
- Vibernets Streaming“Linux on Mars - NASA Mars Helicopter (අඟහරු ගිය Linux)”YouTube · Mar 3, 2021 · 12k views
- Suck Less | Tym Rabchuk“You Can Actually Build a Spacecraft With This #nasa #space #diy”YouTube · Jun 30, 2026 · 1.7k views
- こぶたNEWS“宇宙で初めての火星ヘリコプター”YouTube · Apr 10, 2021 · 741 views
- Marcia en Desplegando Cloud“🇪🇸#5.17 - Python en Acción (Podcast Charlas Técnicas de AWS)”YouTube · Nov 5, 2024 · 281 views
- Gourcer“Gource visualisation: nasa/fprime on 2021-04-22”YouTube · Apr 22, 2021 · 179 views
The biggest 5 of 71 mentions linking to the repository that our crawl found. Each line is their own headline.
See all 71
Talked about on GitHub, YouTube, Hacker News and the web (71 times).
On GitHub · 50
- Aug 19, 2026Add FPP (F Prime Prime) language — github-linguist/linguist15 reactions · 4 comments
- Sep 21, 2026Add New Darkmode UI Option — nasa/fprime-gds2 reactions
- Jul 31, 2025Deprecate serialize/deserialize in favor of [de]serialize[From/To] — nasa/fpp2 reactions · 1 comment
- Mar 19, 2026Add ConsoleNoWait — fprime-community/fprime-vxworks1 reaction
- Nov 22, 2025Merge release/v3.1.0a12 into release/v3.1.0 — nasa/fpp1 reaction
45 more on GitHub
- Oct 13, 2025Add ComAggregator for nasa/fprime#4264 — fprime-community/fprime-workshop-led-blinker1 reaction
- Dec 20, 2021FPP Visualization Automation — nasa/fprime-tools1 reaction · 5 comments
- Nov 10, 2026Svc/DpWriter: add processing override commands to force data product processing types on or off — lestarch-autobot/fprime0 reactions · 1 comment
- Oct 29, 2026Update for nasa/fprime#6174: add ComQueue drop-counter telemetry to packets — fprime-community/fprime-yamcs-reference0 reactions
- Oct 29, 2026Update for nasa/fprime#6174: add ComQueue drop-counter telemetry to packets — fprime-community/fprime-zephyr-reference0 reactions
- Oct 29, 2026Update for nasa/fprime#6174: add ComQueue drop-counter telemetry to packets — nasa/fprime-examples0 reactions
- Oct 28, 2026Update for nasa/fprime#6278: define FwTlmPacketizeOffsetType in TlmPacketizerCfg.fpp — fprime-community/fprime-zephyr-reference0 reactions
- Oct 28, 2026Restrict checkTimers and tlmWrite to having a single message on the q… — FireflySpace/fprime0 reactions
- Oct 24, 2026Upgrade to F Prime v4.4.1 — Open-Source-Space-Foundation/proves-core-reference0 reactions
- Oct 23, 2026TlmPacketizer: address lestarch-autobot review must-fix items (nasa/fprime#6144) — FireflySpace/fprime0 reactions
- Oct 16, 2026Include annotation for packets from FPP in dictionary — nasa/fpp0 reactions · 3 comments
- Oct 14, 2026Manual merge from nasa devel #5996 (pktType) — FireflySpace/fprime0 reactions
- Oct 12, 2026Svc/GenericHub: drop message with event and counter on allocation failure instead of asserting — JPL-Devin/fprime0 reactions · 1 comment
- Sep 30, 2026CI: fold workflow_dispatch runner/jobs inputs into the Ref and Framework matrices — JPL-Devin/fprime0 reactions · 1 comment
- Sep 22, 2026docs: import LED Blinker timeliness.md into website build — JPL-Devin/fprime0 reactions · 1 comment
- Sep 17, 2026Check C++ formatting — fprime-community/fprime-baremetal0 reactions
- Sep 14, 2026WasmSequencer: give the I32 host resume its own pause-check choice — JPL-Devin/fprime0 reactions · 1 comment
- Sep 11, 2026Fw/Types: serialize bool via U8 path and static-assert pointer size — JPL-Devin/fprime0 reactions · 1 comment
- Sep 1, 2026Rename LinuxUartDriver to PosixUartDriver with deprecated compatibility layer — JPL-Devin/fprime0 reactions · 2 comments
- Aug 22, 2026Feature: DFA-based structural health monitor app for telemetry channels — nasa/cFS0 reactions · 9 comments
- Jul 17, 2026Add TypeDemo component — fprime-community/fprime-yamcs-reference0 reactions
- Jun 22, 2026Add README.md and How-To guide for developing cFS F Prime applications — JPL-Devin/fprime_cfs_reference0 reactions · 2 comments
- Jun 16, 2026Use pr-xxxx pattern for GDS across Integration Test CI — JPL-Devin/fprime0 reactions · 3 comments
- May 29, 2026[#5159] ROSES - Student Review of Rule-Based Testing — lestarch-autobot/fprimesays it tested it0 reactions
- May 29, 2026[#5115] ROSES - Document how to decide between Active/Passive/Queued component - Student Review — lestarch-autobot/fprime0 reactions
- May 29, 2026[#5111] CCSDS Improvements — lestarch-autobot/fprime0 reactions
- May 29, 2026[#5099] DpCatalog Discards Status When Handling StateFile Database — lestarch-autobot/fprime0 reactions
- May 29, 2026[#5051] Add CI run exercising non-default FwType config values — lestarch-autobot/fprime0 reactions
- May 29, 2026[#5018] Replace hand-coded buffer and string classes with instantiated templates — lestarch-autobot/fprime0 reactions
- May 29, 2026[#4846] Add `fprime-util new --config-module` — lestarch-autobot/fprime0 reactions
- May 29, 2026[#4841] DpWriter always deallocates buffers to the same BufferManager — lestarch-autobot/fprime0 reactions
- May 29, 2026[#4806] Change fprime-gds serializable array type representation — lestarch-autobot/fprime0 reactions
- May 29, 2026[#4792] Consolidate Parameters ground scripts — lestarch-autobot/fprime0 reactions
- May 29, 2026[#4745] Update Core components to use FPP constants for string arguments — lestarch-autobot/fprime0 reactions
- May 29, 2026[#4711] fprime_test_api: Sometimes fprime_test_api does not connect to currently running fprime-gds session — lestarch-autobot/fprime0 reactions
- May 29, 2026[#4697] ROSES - Student Review of ISR pattern — lestarch-autobot/fprime0 reactions
- May 29, 2026[#4535] Configurable CPU Affinity for Subtopologies — lestarch-autobot/fprime0 reactions
- May 29, 2026[#4508] ROSES - Student review - Memory management — lestarch-autobot/fprime0 reactions
- May 29, 2026[#4454] VxWorks CI To Build All — lestarch-autobot/fprime0 reactions
- May 29, 2026[#4425] PrmDb filename in subtopology should be configurable — lestarch-autobot/fprime0 reactions
- May 29, 2026[#4324] Switch `FW_PACKET_COMMAND` enumerated value to non-zero value — lestarch-autobot/fprime0 reactions
- May 29, 2026[#4290] Clean up setup — lestarch-autobot/fprime0 reactions
- May 29, 2026[#4099] ROSES - Subtopologies (Working With) - Student Review — lestarch-autobot/fprime0 reactions · 1 comment
- May 29, 2026[#3662] ROSES - Add integration tests for fprime-examples repository — lestarch-autobot/fprime0 reactions
- May 29, 2026[#3565] Add local script to preview website rendering — lestarch-autobot/fprime0 reactions
On YouTube · 13
- Mar 3, 2021Linux on Mars - NASA Mars Helicopter (අඟහරු ගිය Linux) — Vibernets Streaming12k views · 50 comments
- Jun 30, 2026You Can Actually Build a Spacecraft With This #nasa #space #diy — Suck Less | Tym Rabchuk1.7k views · 2 comments
- Apr 10, 2021宇宙で初めての火星ヘリコプター — こぶたNEWS741 views · 3 comments
- Nov 5, 2024🇪🇸#5.17 - Python en Acción (Podcast Charlas Técnicas de AWS) — Marcia en Desplegando Cloud281 views · 2 comments
- Apr 22, 2021Gource visualisation: nasa/fprime on 2021-04-22 — Gourcer179 views
8 more on YouTube
- Feb 21, 2021Gource visualisation: nasa/fprime on 2021-02-21 — Gourcer157 views
- May 2, 2021Ferhell News 3 - Winget, WSL, Banimento da Universidade de Minessota e o Linux no Espaço! — Ferhell Linux & Techs117 views · 14 comments
- Jul 12, 2026GitHub - nasa/fprime: F´ - A flight software and embedded systems framework — GitHub Daily Trend AI Podcast95 views
- Feb 22, 2021Gource visualisation: nasa/fprime on 2021-02-22 — Gourcer70 views
- Oct 10, 2026NASA’s F Prime Turns Flight Software Into Components — Cochran Labs59 views
- Jul 2, 2026NASA fprime - A flight software — RandomTechAce37 views
- Jul 12, 2026nasa/fprime - Gource visualisation — Gourcer36 views
- Jul 4, 2026Почему эти 8 проектов уже изменили будущее? #Shorts — GitHub Находки3 views
On Hacker News · 6
- Feb 18, 2021The open source flight code that the NASA Mars drone is running68 points · 3 comments
- Feb 19, 2021Vue.js is powering NASA flight software4 points
- Jan 31, 2024NASA/fprime: F´ – A flight software and embedded systems framework2 points
- Jan 26, 2024F Prime – Flight software framework by NASA2 points
- Jan 21, 2024F': Ingenuity's Open-Source Flight Software Framework2 points
1 more on Hacker News
- Feb 1, 2024Fprime – A flight software and embedded systems framework by NASA1 point
On the web · 2
- Jul 12, 2026nasa/fprime: F´ - A f… - GitHub Daily Trend - Apple 팟캐스트 — podcasts.apple.com
- Jul 12, 2026nasa/fprime: F´ - A f... - GitHub Daily Trend — podcasts.apple.com
In this repository41 other files it gives its agents.
- AGENTS.mdAGENTS.md
- cmake/AGENTS.mdAGENTS.md
- docs/AGENTS.mdAGENTS.md
- CLAUDE.mdCLAUDE.md
- .github/copilot-instructions.mdCopilot instructions
- agent-skill-authoringSkill
- ai-session-reportSkill
- ai-session-summarySkill
- ci-test-runtime-policySkill
- fprime-checklist-closeoutSkill
- fprime-checklist-designSkill
- fprime-checklist-implementationSkill
- fprime-cmake-build-systemSkill
- fprime-component-checklist-reviewSkill
- fprime-component-design-fppSkill
- fprime-component-developmentSkill
- fprime-component-implementationSkill
- fprime-component-integration-testSkill
- fprime-component-requirementsSkill
- fprime-component-unit-testSkill
- fprime-cpp-designSkill
- fprime-fpp-navigationSkill
- fprime-ground-input-tracingSkill
- fprime-hardware-input-tracingSkill
- fprime-issue-risk-evaluationSkill
- fprime-iterative-developmentSkill
- fprime-maintenanceSkill
- fprime-requirements-reviewSkill
- fprime-run-for-recordSkill
- fprime-system-test-designSkill
- fprime-system-test-harnessSkill
- fprime-topology-developmentSkill
- fprime-unit-testingSkill
- jpl-design-principlesSkill
- maintainer-lookupSkill
- post-inline-reviewSkill
- pr-diff-scopingSkill
- prompt-injection-precheckSkill
- re-review-stateSkill
- triage-classifierSkill
- write-system-functional-docSkill
The file
We split this skill into the rules it gives an agent
4 rules. About 0 are common, 1 is distinctive, and 3 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈1
Distinctive · estimate
This author's own choices. Worth a look.
3
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 20
Every record is judged twice by the same named checks: live, by the test function under fth run (pytest's junit is the runner's verdict), …
- Distinctive: line 72
START marker sending through the END marker sending, markers included.
- Very specific: line 156
Every test on the manifest gets an entry.
- Very specific: line 235
Run ids named after planted faults leaked the draw into the record → neutral run ids.
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Skill: F Prime test oracle — pass rules and checkers
A test oracle is whatever decides that an observed result is
correct. Here it has two halves with separate names: the pass
rule (the condition, public, part of the approved plan) and the
checker (a program that evaluates pass rules over the record).
Every record is judged twice by the same named checks: live, by the
test function under fth run (pytest's junit is the runner's verdict),
and by the auditor's replay of that same test function over the
sealed record (fth audit --tests, tooling pinned in
test-process-contract.md §8) — so that a disagreement (DIVERGENCE)
is one program disagreeing with itself about one record, which only a
non-deterministic test or a doctored record can produce. The engine
(fth evaluate) and a reviewer's independent checker remain for legacy
plans whose pass rules still live in plan.json.
1. Pass-rule format
(Coverage targets — plan.json coverage: — are the design skill's:
fprime-system-test-design §2; the oracle judges verdicts, not coverage.)
Since phase 1 the named checks are declared in the integration test
(fprime-system-test-harness §3), in the grammar of the tooling's
docs/plan-format.md; plan.json keeps windows and metrics. Checks
refer only to things that exist in the record and are judged per window
over its closed history:
with window(60, 61):
require("STD") # std_complete: markers, dispatcher, drops, packet samples
send("framTester.SWEEP", [0x5005, 4096, 0x5005]); bench.wait_event("framTester.SweepDone", min_new=2)
check("ABS") # absence group from plan.abs
check("event framTester.SweepDone(kind=WRITE, chunks=4096, errors=0) count == 1")
check("event framTester.SweepDone(kind=WRITE) precedes event framTester.SweepDone(kind=VERIFY)")
check("delta(framDriver.SpiErrors) == 0")
check("dump image == xorshift32(seed)", dump_image_matches_xorshift32(0x5A, ...)) # check function
# plan.json: "metrics": ["delta(framTester.BytesVerified) == 262144",
# "t(event framTester.SweepDone(kind=VERIFY)) - t(framTester.SWEEP sent) <= 1200 s"]Rules:
requiremay reference only markers and infrastructure counters (both markers present,commands_sent == count(event cmdDisp.OpCodeDispatched), dispatcher error and drop counters,<packet> samples >= n). UUT events belong incheck. A precondition that does not hold isUNVERIFIED(Incomplete); a failedcheckormetricspredicate in a judgeable window isFAIL. A missing UUT completion event is thereforeFAIL, neverUNVERIFIED(Captain's rule: unverified only when the absence itself is not indicative of a UUT failure assuming a working system). Legacy plans that still carrycomplete/all_ofare judged byfth.enginewith the same rule.- Event arguments are matched from the event text with the plan's
event_patternsregexes (named groups), never from the exporter's numeric scrape (garbage for hex and enums). commands_sentis a record quantity:commands.jsonlrows from the START marker sending through the END marker sending, markers included.- Absence predicates (
count == 0) are valid only on a complete window;deltaneeds ≥ 2 samples andfirst/last≥ 1, elseUNVERIFIED. - Thresholds carry units and come from the plan's metrics section.
t(…)is FSW time;CMD sentis the command'sOpCodeDispatchedrow; a window spanning a reboot (reset: true) has noduration. - A pass rule never references the runner's, harness's, or pytest's own printed verdict.
2. Record export
Checkers read a flat export, not the live ground system, so both can
be re-run on the sealed record. fth export (inside fth run, and
again from a copy inside fth audit) writes it:
record/
events.jsonl one JSON object per event row (FSW time, boot epoch, name, severity, text, args)
channels.jsonl one per telemetry sample
commands.jsonl one per command sending (ground wall-clock, name, args, completion) — from raw/cmd/ only
export-counts.json rows per file plus torn packets, as the exporter saw them
raw/ ground-system archive as stored: GDS logs, recorder logs, cmd/ sender logs, flash.log
record.sha256 one line per file, produced by fth hash-recordThe exporter dedupes exact duplicate recorder lines per file per boot
segment (the recorder writes every row twice), takes commands only from
record/raw/cmd/, and increments the boot epoch when the FSW clock
falls back by more than the plan's reboot_min_backstep_s; t = epoch
× 1e6 + fsw orders rows, archive index breaks ties, file order is
authoritative. fth verify-record fails when any export file is absent
or its row count differs from export-counts.json; the auditor's
re-export from a copy of raw/ is the independent count. Checkers read
the export through the replay API (fth.replay: load_record,
Record.window(start_id, end_id), Window.events/channel/commands_sent/
segment(epoch)), the only read surface the tooling's
docs/replay-api.md promises to keep.
3. Checker contract
Runner and replay (phase 1/2): the test function's check(...)
names are judged at window close; fth run derives
runner/verdicts.json from pytest's junit only (pass → PASS, failed
check → FAIL + the failed names, error/Incomplete → UNVERIFIED), and
fth audit --tests <test/int> re-executes the same functions over the
record (--fth-replay-record: send() must meet the recorded commands
in order, waits return at once) → auditor/junit.xml →
auditor/verdicts.json (checker: fth.replay), diffed per test with the
runner's — verdict and failed — any difference is DIVERGENCE.
Record integrity findings come first (one START..END per window, no
command outside a window, monotonic time, export counts, junit hashed,
no duplicate test). The check grammar and the check functions are unit
tested in the tooling (tests/test_checks.py, tests/test_checks_rules.py);
replay matching and the integrity findings in tests/test_replay_match.py,
tests/test_integrity.py, tests/test_audit_replay.py.
Engine (legacy plans with complete/all_of): fth evaluate
plan.json <record-dir> -o verdicts.json judges every plan predicate and
registered @helper. Output:
verdicts.json
{ "run": "<run id>", "record_sha256": "<sha256 of record.sha256>",
"tests": { "L2-6": { "verdict": "PASS|FAIL|UNVERIFIED",
"failed": ["<predicate text>"],
"evidence": ["events.jsonl:1041-1049"] } } }Independent checker (legacy plans only; optional since phase 2 — no longer part of the contract) — interface frozen 2026-11-02:
python checks/<name>.py <record-dir> <plan.json> <out.json>
-> {"tests": {ID: {"verdict": "PASS|FAIL|UNVERIFIED", "failed": ["<predicate text>"]}}}Positional arguments in that order; <record-dir> holds the three
jsonl files; exit 0 after writing <out.json> (non-zero is
"independent checker failed", reported by the audit, not a verdict);
extra keys are ignored. fth audit --independent diffs verdict and
failed per test byte-for-byte.
- Every test on the manifest gets an entry; a test with no window in
the record is
UNVERIFIEDwithfailed: ["window missing"]. failedholds the plan's predicate strings verbatim — the string is the keyfth unsealmatches against a planted fault'sfails; a paraphrase turns every planted test into an unexplained failure. Transcribe whole sentences including any suffix the plan carries.evidencepoints into the export so a quick-look can be built from it without re-deriving anything.- Both checkers are deterministic and have no bench or network access.
- The independent checker is written by an adversarial reviewer from the plan text alone, without reading the engine, the helpers module or any other checker; the two are compared only by their outputs on the same record (the mutation vectors, then the audit).
4. Divergence report
divergence.md (stage 3: auditor/diff-runner-replay.md — runner junit vs replay)
| Test | Runner (live) | Replay (auditor) | Check(s) in dispute | Record lines |Any row here blocks stage-2 closure (vectors) and is read first by the human at unseal (stage 3).
5. Planted faults and the sealed faults file
A planted fault is a command rewrite declared next to the helpers and
applied by the private sender of fth.pytest_plugin to the drawn
test only (no fixture exposes the draw) — never by name to every test in applies_to:
@planted_fault("verify_wrong_seed", applies_to=["L2-1", "L2-5"], cls="harness-expectation",
fails={"L2-1": ["event framTester.VerifyDone(client=0, offset=0, length=64, status=OK, mismatches=0) count == 1"],
"L2-5": ["event framTester.SweepDone(kind=VERIFY, chunks=4096, errors=0) count == 1"]},
mechanism="VERIFY sent with seed+1", kind="rewrite") # kind: rewrite (default) | insert
def verify_wrong_seed(test_id, command_name, args):
if command_name == "framTester.VERIFY":
return [args[0], args[1], args[2], args[3] + 1] # rewrite: new args
return None # untouched; insert planted faults return [(name, args), …]failsis a per-test dict of plan predicate text, verbatim (a legacy list expands to everyapplies_totest);clsisharness-expectation | uut-state | stimulus;kindis enforced at apply time (arewritethat inserts is aPlanError).fth planted-faults-menu plan.jsongenerates the fault menu from these declarations; it is never hand-written, andfth verify-planrejects a planted fault on an unknown test.- The sealed draw is exactly
{"manifest_sha256": "<sha of the manifest file>", "faults": {"<test>": "<planted_fault>"}}. Anything else — malformed JSON, extra keys, unknown test, unknown or inapplicable planted fault, a stalemanifest_sha256— stops pytest with exit code 2 before any command is sent, with a message that names no planted fault. - Nothing names an active planted fault outside
sealed/: not the run id (it is invertible throughMARK(crc32(run id))), not stdout, not a file underrunner/orrecord/.
6. Lessons that shaped these rules (fram-r1, fram-r2)
- Planted fault applied by name to every
applies_totest → two undrawn tests failed and unseal reported REAL FAILs that were harness scope errors. Fixed by construction: the fixture consults onlyfaults[test_id]. - A draw could remove the only unaltered test covering a requirement →
fth drawrefuses such a draw; plans carry planted-fault-disjoint redundancy. - A hand-written dry-run fault file shape voided fram-r1 at run start
→ the stage-3 rehearsal reads a real
fth drawoutput. - Abbreviated
failedkeys broke unseal matching → predicate text verbatim everywhere, diffed byte-for-byte. - Quick-look pages written by an agent could not be trusted cold → pages, index and VnV matrix are program-generated only.
- Rebuild could only report
DIFFERS (toolchain unpinned)→ the toolchain is pinned in the manifest (--toolchain-json). - The recorder writes every row twice and the sender writes its own
stray logs → dedup is the exporter's job, per file per boot segment,
exact raw line; commands come only from
raw/cmd/. numsscraped from event text were garbage for hex and enums → args are parsed with the plan'sevent_patterns.- Run ids named after planted faults leaked the draw into the record → neutral
run ids; grep every planted fault name over
record/,runner/and the verdicts before shipping.
Anti-patterns
- Pass rules in prose ("sweep completes successfully").
- Checkers that read the harness's own log of what it thinks happened.
- UUT completion events inside
complete— a hung UUT then reads as a bench problem instead of a failure. - Relaxing a predicate because the dry run failed it; that is a plan change (contract §2).
- A planted fault switched on by name, a hand-written fault menu, or a faults file the harness "tolerates".
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

