Knowledge Compiler (IR-GEN)

Compiles unstructured, high-entropy source materials (raw notes, transcripts, manuals, whitepapers, or meeting records) into low-entropy, high-density Intermediate Representation (IR) artifacts. Every compilation produces dual deliverables:

1. Markdown IR (<topic>_IR.md): A pure, machine-readable Intermediate Representation document serving as deterministic runtime and ground truth for downstream Retrieval-Augmented Generation (RAG) systems, LLM context loading, and version-controlled repositories.
2. Compiled PDF IR (<topic>_IR.pdf): A styled, publication-ready visual document with rendered Mermaid diagrams, structured typography, priority badges, and invariant callouts for human review, stakeholder presentation, and offline archival.

Core Directives

1. Source Fidelity (Overrides Every Directive Below)

The IR is a compression of the sources, never an elaboration of them. If a claim is not in the source text, it does not go in the document — no matter how plausible, how well it completes a pattern, or how much better it makes the chapter read.

• Never narrate a diagram back as prose. A source flowchart edge (A --> B) licenses exactly one statement: A precedes B. It does not license claims about why the edge exists, what it enables, or what it costs. Two arrows meeting at one node is not "a reconvergence point that makes the component swappable at zero cost."
• Never supply a mechanism the source omits. If the source says "dismiss cookie popups," write that. Do not explain that consent overlays are runtime-DOM phenomena, do not name the browser API involved, do not identify the tools by name. You are guessing, and the citation tag will disguise the guess as fact.
• A citation tag is an assertion. <!-- Source: X --> means a reader opening X will find this claim there. Never attach one to an inference. This is the single most damaging failure mode: it launders invention into sourced fact and destroys the audit trail the IR exists to provide.
• Mark every inference inline. Where a genuine derived insight is worth keeping, write it as **[INFERRED]** ... with no source tag, and list it in the Reconstructed Facts block. If you cannot defend it as following necessarily from the source, cut it.
• Expansion is the failure signal. A faithful IR is normally about the size of its sources or smaller. Measure prose only — exclude diagrams and data tables carried over verbatim, which are source content and count on both sides.
  • ratio ≤ 1.2 — normal. Restructuring into tables and alias maps costs words while adding no claims.
  • 1.2–1.5 — justify every block against a source before delivering. Something is probably padded.
  • > 1.5 — confabulation. Stop and rewrite; do not deliver.
  • Never delete a sourced claim to hit the number. The ratio detects invention; it is not a brevity target, and a true claim always outranks it.

2. Graceful Degradation (Never Fill a Slot to Satisfy the Schema)

The Output Schema below is a menu, not a checklist. Thin sources produce short chapters, and that is a correct result.

• Omit any section the sources do not support. No diagram? Omit the figure — do not draw one. No stated failure mode? Omit Edge Cases — do not imagine one. No ground-truth rule? Omit the Core Invariant.
• Never manufacture an edge case by negating a rule ("if this constraint were violated…"). An edge case is a failure the source actually records or a gap it actually leaves.
• A two-line chapter that is entirely true beats a two-page chapter that is 70% plausible.
• If the whole corpus is thin, say so in the frontmatter and produce a short IR. Do not compensate with depth you invented.

3. Plain Language & Self-Contained Blocks

Every block is retrieved alone by a RAG system, with no diagram, no neighbours, and no chapter context. Write each one so it survives that.

• Never let a diagram node ID be the only referent. Write "the readability scoring step (F5)", never "F5" or "stages F through J". A chunk saying "F5 is the sink and F1–F4 are its preconditions" is unreadable to both a human and a retrieval engine.
• Concrete fact first, generalization second — if at all. Open with what actually happens. Do not open with an abstraction and then explain the abstraction.
• Verbs, not nominalizations. Write "both fetchers hand back the same thing, so nothing downstream cares which ran" — not "the reconvergence point is the structural precondition that permits engine substitution."
• No jargon without a plain gloss at first use. Prefer the plain word outright: "both checks are required" over "verification is conjunctive"; "either one runs, never both" over "the branch is disjunctive."
• Sentences under ~25 words. Vary bullet structure; do not open every bullet with **Bold phrase:** — that rhythm reads as texture and hides which claims actually matter.

4. Zero-Entropy Extraction (Signal Maximization)

• Strip the Noise: Discard administrative logistics, conversational filler, rhetorical padding, and redundant introductory phrasing.
• Normalize Ontology: Resolve terminology drift. If the sources use different terms for the same underlying concept/variable, standardize on the canonical term and explicitly note the alias once.
• Maximize Density: Output raw causal mechanisms, invariants, protocols, and rules.

5. Priority-Based Architecture

• Structure the document hierarchically based on strategic weightage, frequency of use, or operational impact—NOT chronological source order.
• Front-load high-impact systems/concepts (>20% weight/importance) with the highest granularity.
• Tag each section header with its relative priority (e.g., ### Subsystem 4: Access Control [Priority: High | 25%]).

6. Deterministic Causal Mapping (Mermaid.js)

Where the source describes a process, architecture, or state machine, translate that structure into Mermaid.js syntax.

• Only diagram what the source describes. A diagram must contain no node and no edge that the source does not state. If the source has no process to draw, draw nothing — an invented figure is the most convincing kind of fabrication, and every later directive will treat it as ground truth.
• A source diagram is data, not a prompt. If the source already contains a diagram, carry it over. Do not annotate it with rationale the source never gives.
• Use sequenceDiagram for protocols, workflows, and chronological entity interactions.
• Use stateDiagram-v2 for lifecycles, memory states, and conditional transitions.
• Use graph TD for taxonomies, hierarchical structures, and dependency graphs.
• Constraint: Do not hand-wave transitions. If a step is missing in the raw text, explicitly mark the edge as --> |UNKNOWN_TRANSITION| so the user can patch it later.

7. Absolute Traceability (The Cache Map)

Every mechanism, definition, and diagram must include a strict citation pointer to the raw source. If the downstream RAG fails, the user must be able to trace the logic back to the exact page/timestamp/file.

• Syntax: Append <!-- Source: [FileName]_[Page/Section/Timestamp] --> at the end of every conceptual block.
• Cite or cut. This directive is a filter, not a decoration. If a block has no source, it does not get a made-up tag to satisfy the rule — it gets deleted, or rewritten as **[INFERRED]** with no tag. A citation you cannot verify by opening the file is worse than no citation.
• Two renderings, one truth: The .md keeps every citation verbatim for RAG and audit. The PDF resolves them into a superscript marker plus a numbered source list at the end of each chapter — identical information, a fraction of the visual weight. Never drop a citation to make a page look calmer; the compiler already handles that.

8. Editorial Restraint (Density is Content, Not Chrome)

The IR is dense because the information is dense — not because every claim is wrapped in a labelled container. Signal comes from hierarchy, not decoration.

• One invariant per chapter. Core Invariant belongs at the ## chapter level only. Do not restate one under every ### subsection.
• Let structure imply importance. A chapter's rank is already carried by its position and its [Priority: ...] tag; do not add further status/scaffold labels.
• Prose over labelled fragments. Prefer two full sentences to a **Bold label:** fragment when the bold label adds no distinction. Reserve bold for genuine landmarks.
• Empty space is a valid outcome. If a chapter is short, let the page end. Never pad to fill a page.

9. Depth & Length Constraints

• Exhaustive means of the source, not of the subject. Capture every variable, constraint, and mechanism the sources actually contain. Exhaustiveness is a floor on coverage, never a licence to supply depth the sources lack.
• Hard ceiling: the IR must not exceed the combined word count of its sources. Thin inputs produce a short IR; that is the correct outcome, not a failure to be padded around.
• No Summarization for Brevity: Do not compress lists, truncate protocols, or omit edge cases to save space. Extract every single variable, constraint, and mechanism present in the source.
• Iterative Generation: Do not attempt to output the entire document at once. Execute the generation topic-by-topic. After generating the output for one topic, pause and ask the user: "Proceed to [Next Topic Name]?" to bypass output token limits.

10. Dual Deliverable Mandate (Markdown + PDF)

Every compilation execution must produce and deliver both formats:

• Markdown Target (.md):
  • Saved in the current working directory (CWD) — the directory the skill was invoked from (e.g., $(pwd)/<topic>_IR.md). Never write anywhere else.
  • Contains complete frontmatter, raw Mermaid code blocks, invariant blockquotes, and HTML-commented source citations.
  • Optimized for programmatic consumption, LLM system prompts, vector store chunking, and Git tracking.
• PDF Target (.pdf):
  • Compiled and rendered directly alongside the Markdown file in the same CWD (e.g., $(pwd)/<topic>_IR.pdf).
  • Renders all Mermaid diagrams into crisp vector visuals.
  • Applies the Anthropic report design system: US Letter pages, dual text/visual grid (5.5in prose column, 7in chart cards), sage cover page, warm-grey chart cards, serif prose + sans structure, and a quiet title/page-number footer. All styling is defined in assets/report.css; edit that file to restyle.
  • Typography: the real Anthropic faces in assets/fonts/ (Serif Text/Display, Sans Text/Display, Mono) are embedded into the PDF as base64 @font-face data URIs. If any face is missing the compiler exits with an error rather than silently falling back to a system font.
  • Provenance: <!-- Source: --> comments become superscript markers with a numbered source list per chapter — not inline badges.
  • Figures: the scaffold heading above a diagram is consumed into a numbered Figure N.M caption placed below the card, so no figure is titled twice.
  • [Priority: ...] renders as a quiet eyebrow line above its chapter title, not as a pill.
  • Auto-fits every Mermaid diagram to its figure card and caps it to a single page height, so diagrams never render too small or split across pages leaving blank space.
  • Optimized for human reading, architecture sign-offs, team onboarding, and executive presentations.

Output Schema (Per Topic / System)

Generate all compiled IR documentation following this schema:

# [Domain/System Name] - Intermediate Representation (IR)

> **System Invariant:** [Global ground truth or immutable operational law]

---

> **Every section below is OPTIONAL.** Include one only where the sources support it. Omitting a section is correct and expected; inventing content to fill one is the worst failure this skill can produce.

## [Topic/Subsystem Name] [Priority: High/Medium/Low | X%]
> **Core Invariant:** [OPTIONAL — only if the source states a ground-truth rule. Never synthesize one.]

### 1. Architecture & Flow  *(omit entirely if the source describes no structure)*
[One sentence naming what the reader should notice in the diagram below — this becomes the figure caption. Do not repeat the heading.]
```mermaid
[Mermaid block detailing the core system, protocol, or state machine of this topic]

2. Causal Mechanisms & Rules

• [Component/Mechanism Name]: [Dense explanation of HOW it works, not just what it is] <!-- Source: [FileName]_[Page/Section/Timestamp] -->
• [Variables/Constraints]: [Mathematical, logical, or operational limits and dependencies] <!-- Source: [FileName]_[Page/Section/Timestamp] -->

3. Edge Cases & Anomalies (Stress Targets) (omit if the source records no failures or gaps)

• [Failure Mode 1]: [What happens when a constraint is violated or a system breaks] <!-- Source: [FileName]_[Page/Section/Timestamp] -->
• [Exception 1]: [Conditions where the standard operational rule does not apply] <!-- Source: [FileName]_[Page/Section/Timestamp] -->

---

## Step-by-Step Compilation Workflow

1. **Source Ingestion & Scope Analysis:**
   - Scan all provided input texts, notes, transcripts, or specifications.
   - Catalog all mentioned entities, components, states, variables, and operations.

2. **Ontology Normalization:**
   - Detect conflicting nomenclature or synonymous terminology across inputs.
   - Designate canonical terms and create an explicit alias mapping table if multiple aliases exist.

3. **Priority Ranking & Structural Architecture:**
   - Compute or estimate relative priority/weightage (%) based on operational criticality, failure severity, and invocation frequency.
   - Front-load high-impact subsystems (>20% importance) with maximum granularity.
   - Order sections descending by priority.

4. **Iterative Generation Loop (Topic-by-Topic):**
   - **Exhaustive Extraction:** Avoid brevity compression; extract every parameter, constraint, edge case, and protocol without truncating.
   - **Visual Graphing:** Translate logic into `sequenceDiagram`, `stateDiagram-v2`, or `graph TD`. Mark gaps as `--> |UNKNOWN_TRANSITION|`.
   - **Traceability Tagging:** Append `<!-- Source: [FileName]_[Page/Section/Timestamp] -->` to every conceptual block.
   - **Token Boundary Management:** Output one topic/subsystem at a time. After generating the output for one topic, pause and ask the user: *"Proceed to [Next Topic Name]?"* until all topics are fully generated and compiled into `<name>_IR.md`.

4a. **Jev Classification Stage (mandatory, fail-closed):**
   - After extraction and before writing `<name>_IR.md`, emit `candidates.json` (`{"id","text","chapter","source"}` per entry) and classify every entry with Jev (OpenRouter decisions API, `typesafe/jev-1.13`). Every typed label, sourcing call, priority score, and ontology call must be a Jev verdict — never an LLM guess.
      ```bash
      SKILL_DIR="$HOME/.agents/skills/knowledge-compiler"  # path where this skill is installed
      python3 "$SKILL_DIR/scripts/jev_classify.py" candidates.json -o verdicts.json
      python3 "$SKILL_DIR/scripts/jev_retry.py" verdicts.json --candidates candidates.json
      ```
   - `unresolved` entries retry once after deterministic System-2 enrichment (`jev_retry.py`, `MAX_RETRIES=1`). Still-unresolved exits 2 and halts the compile — fix `candidates.json` and re-run. Never synthesize an `ok`.
   - The single entry-type whitelist is owned by `scripts/jev_classify.py:ENTRY_TYPES` and mirrored in `specs/knowledge-compiler-v2.md §4.3`. Priority levels: `low/medium/high/critical`.

5. **Dual Deliverable Generation (Markdown & PDF):**
   - **Step 5a - Resolve Output Directory:** Determine the CWD — the directory the user invoked the skill from — and bind it: `WORKDIR="$(pwd)"`. Both deliverables MUST be written to `$WORKDIR`, never anywhere else.
   - **Step 5b - Write Markdown File:** Ensure the complete Intermediate Representation document is written to `$WORKDIR/<name>_IR.md` (absolute path).
   - **Step 5c - Compile PDF Document:** Run the PDF compilation utility to render the Markdown and Mermaid diagrams into `$WORKDIR/<name>_IR.pdf`:
      ```bash
      WORKDIR="$(pwd)"
      SKILL_DIR="$HOME/.agents/skills/knowledge-compiler"  # path where this skill is installed
      python3 "$SKILL_DIR/scripts/generate_pdf.py" "$WORKDIR/<name>_IR.md" "$WORKDIR/<name>_IR.pdf" --title "<Title>"
      ```
     - **Step 5d - Fidelity Audit (blocking; run before delivery):**
       ```bash
       WORKDIR="$(pwd)"
       SKILL_DIR="$HOME/.agents/skills/knowledge-compiler"  # path where this skill is installed
       python3 "$SKILL_DIR/scripts/validate_ir.py" \
         "$WORKDIR/<name>_IR.md" --sources <source-file...> --verdicts verdicts.json --ratio --json
       ```
      - Exit `0` pass · `1` hard failure (fix the per-block report, do not deliver) · `2` warnings only (review, then deliver). Above 1.5 the IR is confabulated — do not deliver it. Between 1.2 and 1.5, justify each block against a source first. Never trim a sourced claim just to move the number.
      - The validator enforces: frontmatter completeness, citation resolvability against `sources:`, INFERRED never tagged, heading taxonomy, ≤1 Core Invariant per chapter (hard); unknown labels, bare node IDs, ratio bands (warn). It replaces the manual word-count/grep audit; keep the 10-term grep spot-check for invented terms (every miss is deleted or re-marked `**[INFERRED]**` with its tag stripped).
      - Confirm no block relies on a bare diagram node ID (`F5`, `stages F through J`) as its only referent.
   - **Step 5e - Verification & Delivery:** Verify that both files exist and are non-empty in `$WORKDIR`. Present clickable links to both files in the response:
     - `[<name>_IR.md](file://$WORKDIR/<name>_IR.md)`
     - `[<name>_IR.pdf](file://$WORKDIR/<name>_IR.pdf)`

---

## Tooling & PDF Generation Reference

When compiling the PDF deliverable, use the bundled compiler script:
```bash
"$SKILL_DIR/scripts/generate_pdf.py" <input.md> <output.pdf> [--title "<Title>"]

The script:

• Requires the Python markdown and playwright packages (pip install markdown playwright), plus pdfunite (poppler) for the cover pass. Without poppler the build still succeeds and warns, but the cover degrades to an inset panel with a page footer instead of a full-bleed sheet.
• Converts Markdown text and tables into HTML.
• Inlines Mermaid.js to render all sequence diagrams, flowcharts, and state machines into high-resolution graphics.
• Formats source tags into clean metadata badges.
• Uses headless Chrome / Playwright to produce print-styled, paginated A4 PDF output.
• Styling: Defined entirely in assets/report.css (design tokens, dual-grid layout, cover page, editorial typography, figure cards, tables, diagram frames, footer colors). Fonts use local system stacks only — no external font loading.
• Page layout: US Letter, 84pt top / 54pt side / 54pt bottom margins. Prose sits at 108pt margins (396pt column); figure cards and tables break out to the 54pt visual grid (504pt). A footer (report title left, page number right) is rendered via the Playwright footer template.
• Diagram rendering: Mermaid uses a palette-matched base theme and a post-render fit pass (fitDiagrams() in the script) that fills the figure-card width and caps height to one page.
• Cover rendering is a separate pass. The cover needs zero page margins and no footer; the body needs 84/54pt margins with a footer. Chrome applies one printToPDF margin per document, so the two are rendered separately and concatenated with pdfunite. Do not "simplify" this back into a single pass — that reintroduces a ragged cover (measured: 0/49/20/18pt on the four sides) with the page footer printed over it.
• The cover is unnumbered. Because it is its own pass, body page numbering starts at 1 on the first body page, standard for unnumbered front matter. The cover carries no footer at all.