fidelity-validation
Skill284 starsChanged 6 months agoApache-2.0View on GitHub
Cross-validates sanitized output specs against raw source specs to detect lost behavioral detail, dropped constants, missing features, or diluted precision. Run AFTER sanitization and AFTER contamination audit passes.
In this repository21 other files it gives its agents.
- analysis-pipelineSkill
- autonomous-discoverySkill
- behavioral-spec-writingSkill
- binary-analysisSkill
- community-intelligenceSkill
- container-executionSkill
- contract-detectionSkill
- doc-researchSkill
- ecosystem-analysisSkill
- git-archaeologySkill
- incremental-analysisSkill
- multi-source-synthesisSkill
- provenance-methodologySkill
- runtime-observationSkill
- second-pass-reviewSkill
- source-analysisSkill
- source-completenessSkill
- spec-sanitizationSkill
- test-suite-analysisSkill
- validation-methodologySkill
- visual-explorationSkill
The file
We split this skill into the rules it gives an agent
6 rules. About 0 are common, 3 are distinctive, and 3 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈3
Distinctive · estimate
This author's own choices. Worth a look.
3
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 25
All remediation/rewriting rounds are done
- Very specific: line 47
Write this inventory to workspace/raw/audit/fidelity-inventory.md.
- Distinctive: line 84
Test vectors: raw test vectors vs output test vectors (constants must match)
- Very specific: line 102
Write the full report to workspace/raw/audit/fidelity-report.md.
- Very specific: line 111
Write a summary to workspace/output/audit/fidelity-summary.md:
- Distinctive: line 116
NO details about what was found or what raw specs say — this is an implementer-facing artifact
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Fidelity Validation
Sanitization removes source code identifiers. But aggressive sanitization can also remove behavioral detail that an implementor needs. This skill detects information loss.
Why This Exists
The sanitization pass rewrites raw specs into output specs. Each rewrite risks:
- Dropped behaviors: A feature or edge case described in the raw specs that doesn't appear in the output specs
- Lost constants: A numeric threshold, timeout, limit, or size that was accidentally removed or changed during rewriting
- Diluted precision: A specific behavioral rule replaced with vague language ("the system handles errors" instead of "the system retries with configured backoff — N attempts, base delay, maximum delay, and jitter")
- Missing decision trees: A conditional behavior with multiple branches that was simplified to just the happy path
- Dropped error conditions: Error handling that was described in the raw specs but omitted from the output specs
- Feature gaps: Entire features or sub-features present in the raw specs but missing from the output domain spec
When to Run
Run this AFTER:
- Sanitization (Layer 5) is complete
- Contamination audit (Layer 6) passes — output specs are confirmed free of source identifiers
- All remediation/rewriting rounds are done
This is the final quality gate before handing the output to the implementer.
The Principle
Every behavioral claim in the raw specs must have a corresponding claim in the output specs — with equal or greater precision.
The output specs may use different words (that's the point of sanitization), but it must convey the same behavior. If the raw spec says "retry N times with a specific base, cap, and jitter", the output specs must say the same — not just "the system retries on failure."
Process
Phase 1: Build the Source Claim Inventory
For each raw module spec (workspace/raw/specs/modules/*.md), extract:
- All numeric constants — timeouts, limits, sizes, counts, thresholds, intervals, percentages
- All behavioral rules — "when X happens, the system does Y"
- All error conditions — what errors occur and how they're handled
- All state transitions — state machines, mode changes, lifecycle events
- All decision trees — if/else branches, priority orders, cascades
- All features — distinct capabilities described
Write this inventory to workspace/raw/audit/fidelity-inventory.md.
Phase 2: Cross-Validate Against Output
For each item in the source claim inventory, search the corresponding output domain spec(s) for a matching behavioral claim.
Match criteria:
- The output specs describes the same behavior (possibly with different words)
- Numeric constants match exactly (no rounding, no approximation)
- Decision branches are all present (not just the happy path)
- Error conditions are all present
- State transitions are complete
Report format per item:
### [RAW-SPEC: section-name] Claim: "description of the behavioral claim"
- Constant/Rule/Error/State/Decision/Feature
- Raw value: [exact value from raw specs]
- Output location: [file:section where it appears in the output] or MISSING
- Output value: [exact value from output specs] or N/A
- Status: MATCH | WEAKENED | MISSING | CHANGED
- Notes: [if WEAKENED/MISSING/CHANGED, explain what was lost]Status definitions:
- MATCH: Output spec conveys the same behavior with the same precision
- WEAKENED: Output spec mentions the behavior but with less precision (e.g., "retries on failure" instead of "retries with specific backoff params")
- MISSING: Output spec does not mention this behavior at all
- CHANGED: Output spec describes different behavior for the same scenario
Phase 3: Also Validate Cross-Cutting Artifacts
Repeat Phase 2 for:
- Journey specs: raw journeys vs output journeys
- Contracts: raw contracts vs output contracts
- Protocol specs: raw protocols vs output protocols
- Test vectors: raw test vectors vs output test vectors (constants must match)
- Acceptance criteria: raw ACs vs output ACs (criteria must be present)
Phase 4: Severity Assessment
Categorize all WEAKENED, MISSING, and CHANGED findings:
| Severity | Criteria | Action |
| P0-CRITICAL | A P0 behavior (critical path) is MISSING or CHANGED | Must fix before implementation |
| P0-WEAKENED | A P0 behavior lost precision (constants, branch coverage) | Must fix before implementation |
| P1-MISSING | A P1 behavior is entirely absent from the output | Should fix |
| P1-WEAKENED | A P1 behavior lost precision | Should fix |
| P2-MISSING | A P2/P3 behavior is absent | Advisory — may fix |
| P2-WEAKENED | A P2/P3 behavior lost precision | Advisory — may fix |
Phase 5: Write Report
Write the full report to workspace/raw/audit/fidelity-report.md.
The report SHALL include:
- Summary statistics: total claims checked, MATCH count, WEAKENED count, MISSING count, CHANGED count
- P0-CRITICAL and P0-WEAKENED findings: full details with exact raw vs output text
- P1 findings: full details
- P2 findings: summary only
- Verdict: PASS (zero P0 issues) or FAIL (any P0 issue exists)
Write a summary to workspace/output/audit/fidelity-summary.md:
- Verdict: PASS or FAIL
- Total claims validated
- Match rate percentage
- Number of issues by severity
- NO details about what was found or what raw specs say — this is an implementer-facing artifact
Agent Dispatch Pattern
The fidelity validation is compute-heavy (reading all raw specs AND all output specs). Dispatch as:
- Inventory builders (parallel, one per 4-5 raw module specs): Extract behavioral claims from raw specs →
workspace/raw/audit/fidelity-inventory-batch-N.md - Cross-validators (parallel, one per output domain spec): Read the relevant inventory batches + the output domain spec → produce per-domain findings
- Aggregator (sequential): Merge all findings → severity assessment → final report
Each agent is dispatched as greenfield:analyzer since they need to read both raw/ and output/.
What This Does NOT Check
- Contamination (that's the Layer 6 audit's job)
- Structural organization (output specs may organize differently than raw — that's fine)
- Prose quality (rewording is expected)
- Provenance citations (these are intentionally stripped)
Relationship to Other Skills
- spec-sanitization: Creates the output specs this skill validates
- second-pass-review: Checks for contamination; this skill checks for information loss
- validation-methodology: Defines the acceptance criteria that both raw and output specs must satisfy
- behavioral-spec-writing: Defines what behavioral claims look like
Together, the audit (Layer 6) and fidelity validation form a two-sided check:
- Layer 6 asks: "Is there anything in output/ that shouldn't be?" (contamination)
- Fidelity validation asks: "Is there anything NOT in output/ that should be?" (information loss)
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

