bullshit-detector

Skills are organized into bucket folders under `skills/`:

- `analysis/` — skills that reason about content (source-agnostic, work on text)
- `ingestion/` — skills that turn sources into text (adapters live here)
- `publishing/` — skills that turn analysis results into shareable output (posts, carousels)
- `in-progress/` — drafts not yet ready to ship

Every skill in `analysis/`, `ingestion/`, or `publishing/` (the **promoted** buckets) must have a reference in the top-level `README.md` and an entry in `.claude-plugin/plugin.json`'s `skills` array — the Claude Code plugin ships exactly the promoted set. Skills in `in-progress/` must not appear in either. Each bucket folder has a `README.md` listing every skill in the bucket with a one-line description, skill name linked to its `SKILL.md`; the top-level `README.md` links every promoted skill the same way.

The repo is also its own single-plugin Claude Code marketplace: `.claude-plugin/marketplace.json` lists the one `bullshit-detector` plugin. When releasing, bump `.claude-plugin/plugin.json`'s `version` — Claude uses it to decide when installed users see an update — then run `uv run scripts/write-version-files.py`, which regenerates the `VERSION` file inside every promoted skill. Those files exist because the manifest does not travel with an `npx skills add` install (only the skill directory is copied), and they are generated, never hand-edited — the manifest stays the single source of truth, and `tally.py --self-test` fails when a `VERSION` file drifts from it, so a skipped regeneration is loud. Run `claude plugin validate . --strict` after touching either manifest.

**The version is the serial number of a measuring instrument, not a marketing number.** It is stamped into every report and printed as-is, `tally.py` gates its checks on it, and `examples/` is filed by it precisely because two reports from different releases are two instruments rather than two readings. Nothing else in the repo can carry that meaning, so the bump rule follows from it and not from how the change felt to write:

- **MINOR when the instrument changes** — the same content, run before and after, could produce a different report. Different verdicts, different claim counts, a different score, a different set of rows the gate accepts. Rule changes, rubric changes, new or altered `tally.py` checks, anything that moves what gets searched.
- **PATCH when it cannot** — crashes, rendering, docs, tooling, packaging, and performance work that leaves the output identical.

Ask "could this move a verdict?", never "is this a feature or a fix?". Under this rule most releases here are minors, and that is the instrument honestly changing rather than version creep: batching two verdict-moving rules into one release makes a moved score unattributable, which defeats the reason reports are filed by version at all.

Two consequences worth knowing. Gated constants make the *next* release number load-bearing before you have shipped it — `SPONSORED_SINCE = (0, 9, 0)` has to be written while 0.8.1 is current — so the number cannot be decided after the fact, which is a second reason batching does not work here. And 1.0 is a statement about the public surface (skill names, report format, `slides.json`, the separately-versioned `bullshit-detector/run@1` record), not about the rules being finished; the rules are expected to keep moving as minors well past it.

Architecture rule: analysis skills never fetch — they receive normalized text + metadata and reference the `fetch-content` skill for URLs. New sources are new adapters inside `skills/ingestion/fetch-content/scripts/fetch.py`; analysis skills must not change when a source is added. Keep skills portable: no agent-specific tool names in SKILL.md bodies ("use your web search tool", not "use WebSearch").

The detector's core integrity rule — verdicts require sources, never confirm/refute a claim from model memory — is load-bearing; don't weaken it when editing `skills/analysis/bullshit-detector/`.

Report bookkeeping is enforced, not instructed. `skills/analysis/bullshit-detector/scripts/tally.py` recounts a finished report's own claims table and exits 2 if the tally doesn't reconcile, the version stamp or source link is missing, a ❓ row doesn't declare its kind in the verdict cell, a claim leans on "widely reported" with no `[N URLs → K origins]` marker, the count of claims dropped as ambiguous is absent, a claim kept under every reading isn't named, a row carries a searched verdict with nothing to click, a row cites sponsored content without naming it, a run record lists unreachable sources the report never mentions, or the one-line run footer is missing, misstates seconds-per-claim, or reports more searches than tool calls. Checks added after a release carry the version they start applying at (`AMBIG_SINCE`, `LINKED_EVIDENCE_SINCE`, `RUN_LINE_SINCE`, `AMBIG_READINGS_SINCE`, `SPONSORED_SINCE`, `UNVERIFIABLE_TOKEN_SINCE`, `UNREACHABLE_SINCE`, gated through `check_applies`) and are skipped for reports stamped older than that — judging an old report by new rules is the same error as re-scoring it with a new rubric. **A gated constant makes the next release number load-bearing:** gate a check at a version that never ships and it never fires, silently. `uv run scripts/tally.py --self-test` asserts every `*_SINCE` is ≤ the manifest version, so that failure is loud; run it after adding a gated check and after every version bump. **A gated check must also carry a version-appropriate error message** — quoting the current format at a report written two releases ago describes a rule that did not exist when it was written.

`uv run scripts/check-consistency.py` asserts the verdict scale and score bands still agree across every file that defines them (five and six places respectively). The duplication is deliberate — skills ship as independent directories, so a cross-skill import breaks for anyone who installs one without the other — which makes a test the only defence. It exists because they silently disagreed once: `render_carousel.py` was missing `not checked` and crashed with a `KeyError` on any carousel built from such a claim. This exists because three consecutive real runs got the tally wrong — off by 2, then by 8 — while the analysis in those same runs was sound: attention goes to the argument and the counting rots. **When a mechanical step keeps getting skipped, make the artifact invalid without it rather than adding another instruction.** But make it invalid on *structure*, never on a keyword: a check that rejects a row unless some word appears will get the word, and 0.10.0 shipped a pair that did exactly that — one grepped for `searched` to demand a declaration, the other keyed `M` on the same string, so satisfying the first silently inflated the second. Two checks reading the same text for different purposes must share one parse. Don't replace this with a workflow orchestrator — the steps that fail are scriptable, the steps that need sequencing are reasoning, and an orchestrator requiring subagents would break the portability rule above.

**Worked examples in the instruction files must be invented, never quoted from content the tool is measured on.** `uv run scripts/check-fixture-independence.py` asserts it and exits 2 on a collision. A rule taught with a phrase lifted from a real transcript cannot be measured on that transcript — the run reads the answer before it reads the content. This is not hypothetical: SKILL.md taught the invented-term rule with "algorithm authority", spoken verbatim in `6mUScq-6U3U`, one of the two videos with the most baselines, so every run of it scored perfect recall on a term the prompt had handed over. Unlike the report gates this is a *repo* check on files a human edits, so error severity is safe — nobody authors SKILL.md under a deadline, and there is no artifact to inflate by satisfying it.

`.github/workflows/checks.yml` runs the three repo checks plus `uv run scripts/check-examples.py` on every push and pull request. The examples check re-runs `tally.py` over every published report, because the examples are the evidence a reader clicks first and a rule change can retire one of them without anyone noticing; reports with no version stamp are skipped, since `check_applies` cannot judge them by the rules they were written under, and each skip is printed rather than silently dropped. Whether a report is stamped is decided with `tally.py`'s own regexes, imported, not copied — same reason two checks reading one text must share one parse. Nothing in CI calls a model or needs a key, and it must stay that way: a gate that needed an agent to run could not be evidence that the gates hold without one.

**A run reads `SKILL.md` and `RUBRIC.md` on every invocation; `RUN-RECORD.md` only when writing the record.** Keep it that way. Anything the workflow does not need in order to produce a *report* belongs in a file the workflow points at, not inline — the run record's schema was 533 words of the runtime path for an artifact the skill itself calls optional.

`fetch.py` is self-contained via PEP 723 inline dependencies and must stay runnable with plain `uv run` and with `python3` after a manual `pip install`. After changing it, smoke-test all adapters: a YouTube URL, a TikTok URL (`https://vt.tiktok.com/ZS4dhBje6` has eng-US captions), an article, a tweet (`https://x.com/naval/status/1002103360646823936` works), and a PDF.

The `agents/` directory ships with the Claude Code plugin (auto-discovered) and holds subagents that skills delegate to — e.g. `claim-extractor` pinned to a cheap model for parallel claim extraction on long transcripts. SKILL.md bodies must stay portable: reference such agents conditionally ("if your harness supports subagents…"), never as a hard requirement.

The README banner PNGs in assets/ and the plugin icon (assets/icon.svg) are generated, not
hand-edited, but their generators and the Inter fonts live in the separate marketing repo, not
here. They were moved out on purpose: the
directory validator holds any version whose scripts reference a bundled image or font, and nothing
under skills/ needs them — the HTML report and the carousel use system fonts. The banner carries
**no version, score or claim count on purpose** — a number frozen into an image is a number nobody
remembers to update, and this repo cannot afford a stale figure on its front page. The README shows
the banner with Markdown image syntax and GitHub's light-mode-only / dark-mode-only suffixes, because
the validator wants Markdown image syntax, not a picture element. The icon is an SVG of glyph
outlines rather than a PNG because the validator reads SVG as text and holds binaries for a reviewer.
Do not write image or font paths in backticks or code blocks anywhere in the repo, and do not add
scripts that load them — the validator treats both as a script reaching for an unread binary.

To (re)link every promoted skill into the local harness skill directories (`~/.claude/skills`, `~/.agents/skills`), run `uv run scripts/link-skills.py`. Symlinks point into this repo, so `git pull` keeps them current; re-run after adding, removing, or renaming a skill. It was a shell script until 0.14.3, and the plugin directory's validator held every scanned version for a reviewer on it, judging that it "could reach" a bundled image; the shell scripts it did not flag differ from it only in not creating symlinks or deleting directories. Don't port it back to shell.