agentleFS
opendatadiscoveryopendatadiscovery/odd-team/.claude/skills/ingest-docs/SKILL.md

ingest-docs

Skill0 starsChanged 2 months agoApache-2.0View on GitHub

Ingest the live documentation manual into the ontology as ground-truth Doc nodes, then enrich them with doc→ontology DESCRIBES links. Runs the mechanical doc extractor (docs-ingest → doc-nodes.jsonl, split by heading/anchor, prose referenced from ../documentation), rebuilds the graph (embeds the doc prose), spawns the doc-analyser subagent per un-enriched page (live-URL verification + DESCRIBES to the concepts/features/code each page documents + doc-claim-vs-code drift), and reports the consistency dashboard. Per adrs/drafts/ground-truth-lineage.md (Phase 1).

1 other project uses a version of this.

Adopted by

Most recently taken up by

The file

We split this skill into the rules it gives an agent

1 rule. About 0 are common, 0 are distinctive, and 1 is very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈0

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 1

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    default → pages in doc-nodes.jsonl with no doc-understanding/{slug}.md (run docs-verify <repo> --json and read …

    line 75

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Ingest docs — the documentation ground-truth layer (ground-truth-lineage Phase 1)

Bring the published documentation manual (../documentation/docs/**, live at https://docs.opendatadiscovery.org/) into the ontology as first-class, searchable, traversable Doc nodes, and wire the reverse DESCRIBES links so a maintainer can start at a doc section and reach the implementing code / concept / feature — and vice-versa.

Source-of-truth contract (do not violate): doc prose is referenced, not copied. ../documentation stays the sole prose SoT. The committed artefacts are addressing only (doc-nodes.jsonl), the per-page agentic sidecars (doc-understanding/*.md), and the drift/completeness manifest (documentation/_manifest.yaml). The graph + vectors are derived/ephemeral. adrs/drafts/ground-truth-lineage.md + lineage/GRAPH-TOPOLOGY.md.

Argument forms

FormBehaviour
/ingest-docs [<repo>]Default. Full pipeline: mechanical ingest → graph-build → enrich every un-enriched page with doc-analyser → graph-build → docs-verify. (default repo odd-platform)
/ingest-docs --mechanical-only [<repo>]Cheap refresh: docs-ingest + graph-build + docs-verify. No agentic pass. Use after a small ../documentation edit to refresh addressing + embeddings + drift.
/ingest-docs --pages "<glob>" [<repo>]Enrich only pages matching the docs-relative glob (e.g. configuration-and-deployment/**).
/ingest-docs --full [<repo>]Re-enrich ALL pages (re-run doc-analyser even on already-enriched pages). Use after a doc-analyser prompt-version bump.
/ingest-docs --show [<repo>]Read-only. Print the docs-verify consistency dashboard. No build, no subagent.

Prerequisites

  • ../documentation is cloned (the prose SoT; the build reads it) and checked out on main, synced to origin/main — the doc layer ingests published truth only; release trains (release/* branches) are invisible to it by design (adrs/drafts/release-train-doc-gating.md). Assert git -C ../documentation branch --show-current → main; refuse to ingest otherwise.
  • The extractor venv is installed (lineage/_extractor, uv sync --extra embeddings for the doc vectors; graph-only still works without).
  • lineage/{repo}/concepts.yaml + feature-flows.yaml exist (the doc-analyser binds DESCRIBES against them).
  • WebFetch available — doc-analyser verifies each page's live URL (the authoritative GitBook slug); stale verification is forbidden.

Protocol

1. Orient (skip if loaded this session)

  • lineage/GRAPH-TOPOLOGY.md — the graph map (labels, edges, the doc layer).
  • adrs/drafts/ground-truth-lineage.md — the decision + consistency contract.
  • .claude/agents/doc-analyser.md — the subagent's system prompt.
  • playbooks/live-site-verification.md (Gate 8) + playbooks/reducer-incremental-mode.md.

2. Mechanical ingest (deterministic, no LLM)

cd lineage/_extractor
uv run lineage-extractor docs-ingest <repo>

Writes doc-nodes.jsonl + documentation/_manifest.yaml. Surfaces completeness (missing/orphan vs SUMMARY.md — the upstream-authoritative denominator). A non-empty missing is a real upstream/SUMMARY inconsistency — surface it, do not paper over it.

3. Build the graph (embeds doc prose from upstream)

uv run lineage-extractor graph-build <repo>

Confirm Doc nodes now carry vectors (vector_count rises). Cold build is minutes; warm is seconds (the (text-hash, model-id) embed cache re-embeds only changed sections).

4. Enrich un-enriched pages (agentic — the DESCRIBES layer)

Compute the page set to enrich:

  • default → pages in doc-nodes.jsonl with no doc-understanding/{slug}.md (run docs-verify <repo> --json and read enrichment.unenriched_examples, or diff the page set against doc-understanding/).
  • --pages <glob> → pages matching the glob.
  • --full → all pages.

Spawn the doc-analyser subagent per page, in parallel batches of ~5 (non-conflicting — each writes its own doc-understanding/{slug}.md). Pass each the input block from .claude/agents/doc-analyser.md (REPO, DOC_PAGE, DOC_PAGE_ABS, DOC_NODES_PATH, LIVE_URL_GUESS, CONCEPTS_YAML_PATH, TARGET_PATH). The agent reads the page, WebFetches the live URL (records the resolved slug + status), graph-searches + confirms the concepts/features/code it documents (DESCRIBES), and records doc-claim-vs-code drift.

Cost note (APPROACH.md §9 / minimal-resources): the agentic pass is one subagent per page (~100 pages). It is the expensive half. For an overnight unattended run, drive this skill via /loop or batch it; the mechanical layer (steps 2-3) already makes docs searchable, so the DESCRIBES enrichment can land incrementally.

5. Rebuild + verify

uv run lineage-extractor graph-build <repo>     # project the new DESCRIBES edges + embed any prose
uv run lineage-extractor docs-verify <repo>      # the consistency dashboard

docs-verify reports the three axes: completeness (SUMMARY denominator), content drift (committed hash vs live upstream prose), and DESCRIBES enrichment coverage + live-URL status. Surface it to the maintainer.

6. Follow-ups

  • doc_claim_vs_code findings the doc-analyser surfaced feed /doc-gap-check (the existing code↔doc drift reducer) — run it to triage DOC-NNN candidates.
  • A non-zero drift count in docs-verify means ../documentation changed since the last docs-ingest — re-run step 2.

Exit

Report: pages ingested (sections), completeness (missing/orphan), vectors, DESCRIBES enrichment coverage (N/total), live-URL statuses, and any drift or doc-claim-vs-code findings worth a DOC-NNN. Never self-mark a doc page "done" — enrichment coverage is the honest metric.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.