Ingesting sources into the fact register

You are the extraction stage. Raw documents go in; what comes out is a fact register -- atomic statements, each carrying a verbatim quote that code will locate in the source -- plus an entity table, a coverage report and clarification questions. Everything downstream (capability map, layer models, views, documents) builds on this register, so its quality is the ceiling for the whole pipeline.

Non-negotiables

1. A fact without a quote is not a fact. There is deliberately no assumed field in the register schema. If you believe something the sources do not say, it is a clarification question, not a fact. The validator (FACT004) rejects any quote it cannot locate in the cited file -- fabricated citations are the documented failure mode this stage exists to prevent.

2. Quotes are verbatim. Copy the exact characters, including pipes in table rows. Multi-line is fine (matching is whitespace- and case-insensitive), paraphrase is not (FACT005). Do not tidy grammar, do not reconstruct from memory.

3. One fact, one claim. "The ERP holds order records and runs on-premise" is two facts. Atomic facts are what let the modelling skills cite precisely and let delta ingestion diff meaningfully later.

4. Never edit a source document. Sources are the evidence; the register points into them.

The pipeline

0. If the input is not plain text, convert it first

easkills intake-csv --root <repo> --file inventory.csv      # CSV (and Excel saved as CSV)
easkills intake-doc --root <repo> --file policy.pdf         # PDF, .docx, .pptx, .xlsx

Never convert by hand -- a hand-made copy breaks the verbatim chain at the one point it exists to hold. intake-doc keeps headings, tables and body order, converts every page, slide (speaker notes included) and sheet, and records in its header what it could not read: a PDF page with no text layer (no OCR is done), a formula with no cached value. Read those notes before extracting -- an unread page is a gap in your evidence, and the honest response is a clarification question or a better copy of the document, never a fact reconstructed from what the page probably said.

An application inventory arrives as a spreadsheet more often than as prose, and a spreadsheet is not a document a quote can be located in. The converter writes a markdown table into facts/sources/ whose header records the original file's SHA-256, encoding and delimiter -- so a quote verified against it is still traceable to the bytes it came from. Commit the CSV next to the converted document.

After conversion the row is the quotable unit: cite | CRM Suite | crm-team@x.test | on-premise | verbatim, exactly as you would a sentence from an interview. Do not paraphrase a row into prose and cite that.

The converter interprets nothing -- no column is mapped to a model field, no value is typed. A column headed "Owner" is a claim in a spreadsheet until a fact says so and the model cites the fact. Read the columns yourself and decide what they mean; where a header is ambiguous ("Status", "Type"), that is a clarification question, not a guess.

1. Chunk

easkills chunk --root <repo>                 # all of facts/sources
easkills chunk --root <repo> --file facts/sources/interview.md

Work chunk by chunk, never the whole document in one pass -- small units roughly double entity recall. The chunker is deterministic: line numbers are exact, and re-runs produce identical chunks, so your extraction is reviewable and repeatable.

2. Extract, then glean

For each chunk, extract every atomic factual claim into facts/register/<source>.yaml:

facts:
  - id: fact-erp-role
    statement: The ERP core holds the master order records and performs invoicing.
    provenance:
      - file: facts/sources/interview-operations-2026-07-15.md
        quote: The ERP core holds the master order records and does the invoicing.
    entities: [erp-core, order-record]
    topics: [application]

Then do one gleaning pass per chunk: re-read it asking only "what did I miss -- quantities, dates, owners, constraints, negations?" Negations matter: "the database is not shared with any other system" is a fact people forget to extract.

Use confidence: implied when the quote supports the statement only indirectly (e.g. "when the API is down, customers cannot order" implies the ERP is a single point of failure). Default is stated; reviewers look twice at implied.

Ids are stable slugs (fact-erp-role), one register file per source document. When two sources support the same fact, keep one fact with both quotes in its provenance list -- that is what the duplicate-statement warning (FACT007) pushes you toward.

3. Resolve entities

Maintain facts/entities.yaml: one entry per real-world thing, with every alias the sources use for it.

Every named thing, not just the systems. People and teams, roles, processes, integrations, data, locations and hardware all get entries — a measured run of this skill produced entities for three applications and nothing else, missing the actor, the integration and the server the same source named, because every example here used to be an application. If a source names it and later text refers back to it, it is an entity:

entities:
  - id: erp-core
    name: ERP Core
    kind: application        # informal hint; ArchiMate typing happens later
    aliases: [the ERP, ERP]

  - id: front-desk
    name: Front Desk
    kind: actor              # people and teams count
    aliases: [reception, the front desk staff]

  - id: order-api
    name: Order API
    kind: integration        # so does the thing between two systems

  - id: erp-server
    name: ERP Server
    kind: technology
    aliases: [the server in the back office]

"The portal", "online order portal" and "Order Portal" are one entity or the model downstream will contain three applications. Only record aliases the sources actually use. One term must resolve to exactly one entity -- a collision is an error (ENT002) because it means downstream modelling would silently merge two different things.

4. Validate, repair (three iterations, then stop)

easkills validate-facts --root <repo>

Fix errors and re-run, at most three times. FACT004 means the source does not say what you quoted: find the real sentence or drop the fact -- do not shop around for a different file with similar words. If findings survive three passes, stop and report what is unresolved; do not grind.

5. Measure coverage, ask questions

easkills coverage --root <repo>

The report lists every substantive source statement that no fact cites, with line numbers. For each uncited span, decide honestly:

• extractable fact you missed -> go back to step 2;
• pleasantry, boilerplate, formatting -> say so in your report, do not force a fact;
• something ambiguous or half-said -> a clarification question.

Questions are records, in facts/questions/, not paragraphs in a report that the next report forgets. Start from the skeletons:

easkills propose --root <repo> --from coverage --as-of <date>

That writes one entry per uncited statement into facts/questions/proposed-questions.yaml, each pointing at the file, the lines and the statement's own text, with the question left as PROPOSED --. Work through them: a fact you missed -> extract it and delete the entry; boilerplate -> delete it; genuinely ambiguous -> write the question so a stakeholder can answer it, and set addressedTo to who should. validate-facts holds questions to the fact discipline (QST*): the quote is located in its source, an answered question names the fact the answer produced (answeredBy), a withdrawn one says why.

Questions are a first-class output of intake -- a thin source produces few facts and many questions, and that is a good result, not a failure. When an answer arrives, it is a source like any other: ingest it, extract the fact, and close the question by naming it.

Reporting back

Lead with the numbers: facts extracted, entities resolved, validator verdict, coverage per source. Then the clarification questions, then anything you deliberately did not extract and why. Never call intake finished while validate-facts reports errors, and never pad coverage with facts that restate headings or metadata.