CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

What this repo is

A skills-first repo: the product is skills/*/SKILL.md — markdown workflows that teach an AI assistant to annotate proteomics SDRF files. Supporting Python lives in tools/. Specification data is not stored here; it is read at runtime from the spec/ git submodule.

Skills are auto-discovered — .claude-plugin/plugin.json carries no skills key, so Claude Code scans the skills/ directory automatically. Each SKILL.md declares its own name and routing description in frontmatter. Currently 4: sdrf-annotate, sdrf-contribute, sdrf-campaign (invoked as /sdrf-skills:sdrf-annotate etc. once installed as a marketplace plugin) plus sdrf-adversarial-review, which sdrf-annotate dispatches into a fresh context at Step 9.5 and which is never typed. sdrf-annotate is the one entry point for everything about one SDRF - create (review always included), review/validate/check/score/fix an existing file, look up cell lines and terms, verify technical metadata from raw files, plan, explain. Its references/ hold what used to be separate skills (setup, knowledge, templates, validate, fix, techrefine, cellline, review, design). Run ls skills/ for the current set — do not trust a hardcoded skill count anywhere in this repo; README.md alone carries three contradictory numbers, and commit d5c4c70 exists only to repair drift.

Commands

# Tests (126, ~2s, no network). CI runs: python -m pytest tests/ -v --tb=short
python -m pytest tests/ -q
python -m pytest tests/test_parser.py -q                                # single file
python -m pytest tests/test_services.py::TestPRIDEClient::test_get_project_mock -q  # single test (tests use classes)
python -m pytest tests/ -q -k unimod                                    # by keyword

# Lint — NOT in CI, and currently RED: 32 errors (28 F401, 4 F541), all auto-fixable.
# tools/review_gate.py is clean; the rest predates it. Run `ruff check --fix` before gating on it.
ruff check tools/ tests/

# Submodule: TWO levels deep (spec/ -> spec/sdrf-proteomics/sdrf-templates).
# `--recursive` is mandatory or templates.yaml is silently MISSING.
git submodule update --init --recursive     # restore pinned state (what you usually want)
git submodule update --remote --recursive   # advance to upstream tip; leaves a dirty gitlink

# tools CLI: `sdrf-tools` console script after `pip install -e .`; `python -m tools` also works from the repo root
sdrf-tools --help   # check, structure, contract, build, score, fix, benchmark, massive-files,
                         # verify, cellline, review-gate, reconcile, audit-existing, curated,
                         # bruker-dia, search-params

python is an alias to python3 here, not a binary — skills invoke bare python, assuming an activated env. Supported: Python 3.10/3.11/3.12 (CI matrix); environment.yml pins only python>=3.10 with no upper bound. conda is not installed on this machine, so the README's "recommended" conda path does not work here; uv is available and requirements.txt already assumes it.

Architecture

Three layers, loosely coupled — the coupling gaps matter more than the layers:

1. skills/ — 4 SKILL.md workflows. sdrf-annotate is a ~3.7k-word core plus seventeen references/ files it reads on demand (gathering, sample and technical values, templates, validation, fix patterns, cellline, techrefine, review checks, reconcile, format rules, explaining, OLS lookup, setup, planning); references/format-rules.md is the single canonical copy of the format rules; the two review-gate skills ship references/review-contract.md and agents/openai.yaml. A SKILL.md is resident every turn once invoked, so its length is a per-turn cost - keep cores short and put detail in references/.
2. tools/ — offline-first Python. contract and build are what sdrf-annotate runs (Steps 3 and 6: the column contract of a template union, and the deterministic SDRF expander from samples.tsv + technical.tsv); massive-files (annotate, review) and cellline lookup (annotate) are the other helpers a skill calls. check, score, fix, benchmark, and verify are called by no skill — reachable only by hand or from CI, which smoke-tests all subcommands. massive-files asks MassIVE's PROXI record for the dataset's FTP root and tries it first (proxi_ftp_url): a bare MSV yielded no root at all, and the ProteomeCentral route yields one with the version directory missing, so both fell through to guessing /v02/ and timed out on the many datasets that live under /v01/. fetch_json honours the response charset for the same ISO-8859-1 reason as the MCP client.
3. spec/ — the runtime data contract (below).

Skill dependency graph. sdrf-annotate is the hub: it calls cellline and techrefine while annotating, ends every annotation with a fresh-context sdrf-adversarial-review (Step 9.5: manifest, track, dispatch, repair, re-review, gate), and in review mode does what sdrf-review and sdrf-annotate-reviewed used to. sdrf-campaign screens a class of studies and loops annotate over the included ones. contribute runs sdrf-tools review-gate gate before publishing. sdrf-tools doctor replaces setup; validate, fix, knowledge, techrefine and cellline are references and tool commands now. Removed 2026-09-23: sdrf-review, sdrf-annotate-reviewed, sdrf-design (folded into annotate's review mode and references/review-checks.md), sdrf-templates (now sdrf-annotate/references/templates.md), sdrf-convert (a README block). None of this routing is covered by CI (tools-tests.yml ignores skills/**).

Bundled paths resolve against the bundle root, not the cwd. The spec references in skills/ are still written bare (spec/sdrf-proteomics/TERMS.tsv), so every skill that reads one opens with a Bundle paths blockquote telling the agent to resolve them against ${SDRF_SKILLS_ROOT:-${CLAUDE_PLUGIN_ROOT}} (else the sdrf-skills checkout) and to run the helpers as sdrf-tools …. .mcp.json, hooks/hooks.json and the plugin venv commands in setup.md stay on CLAUDE_PLUGIN_ROOT: only Claude Code reads them. hooks/check-deps.sh reads SDRF_SKILLS_ROOT, then CLAUDE_PLUGIN_ROOT, then falls back to .. On the Python side, resolve_terms_tsv() finds the bundled TERMS.tsv from __file__ before falling back to a cwd-relative path. Keep both when adding a skill: installed as a plugin, the cwd is the user's project, and a cwd-relative read silently misses (degrading to the hardcoded fallback map).

Frontmatter schema (uniform across the 14 domain skills — match it exactly)

---
name: sdrf-annotate           # matches the directory (skills/sdrf-annotate/); invoked as /sdrf-skills:sdrf-annotate once installed as a marketplace plugin
description: Use when the user wants to ... Triggers on ...   # always starts "Use when the user"
user-invocable: true
argument-hint: "[PXD accession or experiment description]"
---

No skill uses allowed-tools, model, or version. The two review-gate skills are the exception to the schema above: they declare only name + description and are not user-invocable — they are dispatched by other skills into a fresh context, never typed as a slash command.

Adding/renaming a skill: the 6-file lockstep

Nothing in CI or the tests checks this, which is why the copies have already rotted. Update all of: CLAUDE.md, README.md (badge, prose, table, tree), GEMINI.md, .opencode/AGENTS.md, .cursor/rules/sdrf-skills.mdc, .codex/INSTALL.md. .claude-plugin/plugin.json needs no edit — it points at the directory.

Domain policy (value encoding, reserved words, modification syntax, UNIMOD swaps, label types, row identity) exists once, in skills/sdrf-annotate/references/format-rules.md; sdrf-annotate and sdrf-annotate's core and references point to it. Do not paste rules back into a skill.

MCP

.mcp.json wires the bundled mcp/server.py (FastMCP, name sdrf-pride-pmc), launched via ${CLAUDE_PLUGIN_ROOT:-.}/.venv/bin/python. The entry must carry "type": "stdio": without it Claude Code never resolves command and tries to exec a binary literally named stdio, failing with ENOENT: no such file or directory, posix_spawn 'stdio'. It exposes 11 tools: search_projects, search_extensive, get_project_details, get_project_files, get_article_metadata, get_pdf_by_unpaywall, search, searchClasses, getChildren, get_full_text_article, get_full_text_section. fastmcp/httpx are in requirements.txt and environment.yml. The server depends on .venv/ existing next to the plugin (uv venv "$CLAUDE_PLUGIN_ROOT/.venv" && uv pip install --python "$CLAUDE_PLUGIN_ROOT/.venv/bin/python" -r "$CLAUDE_PLUGIN_ROOT/requirements.txt") — a marketplace install ships no venv, and a plugin upgrade replaces the directory, so it has to be re-created. Conda users repoint command.

Skills still call five tools that exist in no bundled server — searchClassesWithEmbeddingModel, listEmbeddingModels, searchWithEmbeddingModel (in sdrf-annotate and its references, formerly sdrf-knowledge), and search_articles / search_preprints (in sdrf-annotate). Those paths need an external OLS/PubMed/ bioRxiv MCP or a rewrite onto searchClasses/getChildren.

search_projects is one page; search_extensive is the whole sweep. The latter takes a LIST of keywords, pages each to a short page, unions and dedupes on all_accessions, and reports what each keyword contributed (per_keyword.new = 0 means that keyword was redundant). PRIDE ANDs the terms in a keyword and then RANKS rather than filters, so recall — not pagination — is the binding limit: a 16-keyword single-cell sweep unions to more datasets than single-cell proteomics returns alone (measured today: nanoPOTS 11 + proteoCHIP 11 = 22 unique, zero overlap). Exhaustion is proven ONLY by a short page; an empty page after a full one is ambiguous, so it triggers a year-partitioned (filter=submissionDate==YYYY) retry and anything still unresolved lands in truncated rather than being reported as complete. PRIDE's historical bare-keyword 100-cap (#28) no longer reproduces — verified 2026-08-19, v2 and v3 both paginate — which is why the partitioning is a detector, not an unconditional workaround.

Article identifiers: one parser — silent-corruption class. Every article tool classifies its input with _parse_article_id: bare forms, PMID: / PMCID: / doi: prefixes, and doi.org / PubMed / PMC / Europe PMC URLs. Bare digits are a PMID everywhere; the full-text tools resolve a PMID or DOI to its PMCID. Unrecognised input still yields an error record, not an exception, which a skill reads as "no evidence" — so teach the parser a new spelling rather than adding a rule to a skill. Pinned by tests/test_mcp_pride.py.

get_project_details / search_projects span PRIDE and MassIVE. MSV… routes to MassIVE's PROXI API; PXD… tries PRIDE then falls back to MassIVE, because MassIVE-hosted ProteomeXchange datasets 404 in PRIDE (PXD003626 is a live example) — a PXD prefix is not evidence of PRIDE residency. Results carry repository and all_accessions (a dataset may hold both an MSV and a PXD; dedupe on it). MassIVE metadata is far thinner — measured over 100 datasets each: instruments PRIDE 100% / MassIVE 3%, experiment_types and quantification_methods never published by MassIVE (PRIDE 100% / 14%). Screen MassIVE candidates from the publication; empty ≠ excluded.

PROXI parsing has three traps, all silent — payload is in value not name; species / publications / contacts nest one list deeper; and a missing value is often the literal string "null". _proxi_values() is the single place all three are handled — do not hand-roll PROXI parsing elsewhere.

MassIVE serves charset=ISO-8859-1. httpx.Response.json() decodes raw bytes as UTF-8, so one accented character (author names, European affiliations) raised UnicodeDecodeError and silently dropped a whole result page — 29% recall loss on a measured query. _cached_get_json now falls back to json.loads(resp.text), which honours the declared charset. It also takes not_found_ok=True to map a 404 to [], because MassIVE 404s when you page past the end and that is exhaustion, not failure.

PRIDE keyword search ANDs its terms. search_projects(keyword=…) narrows hard on multi-word input (metaproteomics → 100+, human gut metaproteomics → 2). Issue several short keywords and union the accessions. PRIDE hits return the structured fields as plain strings, where get_project_details returns CvParam dicts for the same logical fields; _names() handles both. On MassIVE's PROXI, resultType is required, pageNumber is 1-based, and keywords= is silently ignored — it returns the unfiltered listing, so only search= actually filters.

Spec data contract (spec/, read at runtime — never hardcode)

• Columns: spec/sdrf-proteomics/TERMS.tsv
• Manifest: spec/sdrf-proteomics/sdrf-templates/templates.yaml
• Templates: spec/sdrf-proteomics/sdrf-templates/{name}/{version}/{name}.yaml

TERMS.tsv is a glossary, not a column index. 223 rows vs 345 distinct template columns, overlapping only 204. For "is this column valid for template X?", the resolved template YAML is authoritative. 9 tab-separated columns (term/type/ontology_term_accession/usage/values/description/allow_not_available/allow_not_applicable/allow_pooled), CRLF-terminated with one blank row — parse with csv.DictReader(open(p, newline='', encoding='utf-8-sig'), delimiter='\t') and filter empty terms, or allow_pooled == 'false' fails against 'false\r'. Header is type[term] (characteristics[organism]) except type == 'anchor column' (source name, assay name, technology type), which is the bare term — anchor column[source name] is invalid SDRF.

The values field drives ontology routing, but it is prose, not a closed grammar. Common shapes: comma-separated prefixes in preference order (disease → MONDO, EFO, DOID, PATO) → OLS; fixed: a, b, c → closed enum, no OLS call; pattern:/integer/numeric/free text → no OLS; prefix + subtree restriction (MS, PRIDE (children of MS:1000044)). ~26 of 223 rows fit none of these — do not write a parser that assumes total coverage, and never split on , without handling the parenthetical, or you get the bogus ontology PRIDE (children of MS:1000044).

Templates: mutually_exclusive_with appears 0 times in templates.yaml — exclusivity lives in the individual YAMLs and the graph is asymmetric, so compute the symmetric closure or you will propose human + plants. Only ms-proteomics, affinity-proteomics, and ms-metabolomics are usable_alone: true; base and sample-metadata are construction artifacts — never offer them. excludes is inert: resolve_templates.py surfaces it as metadata only, and merge_columns never consults it. The version in extends: sample-metadata@>=1.0.0 is likewise decorative — the resolver splits on @ and always loads latest. Read spec/scripts/resolve_templates.py rather than reimplementing merge semantics.

SDRF invariants (silent-corruption class — wrong output, no exception)

1. NEVER guess ontology accessions — always verify via OLS. This is not self-enforcing: only annotate and cellline state it about accessions specifically, and knowledge — despite being the "background" skill — contains no such language at all. This file is the only global home for it.
2. UNIMOD:1 = Acetyl, UNIMOD:21 = Phospho — the #1 swap. tools/column_ontology_map.py's UNIMOD_KNOWN is what detects these; every row in it was verified against OLS on 2026-09-21 (issue #73 found 8 wrong rows, including UNIMOD:374 → Propionamide, which made check endorse the error it exists to catch). Labels are unique, so UNIMOD_BY_NAME inverts the table: when NT= names a known modification, the accession is treated as the typo. Verify any new row against OLS before adding it — a wrong entry here is invisible by construction.
3. Reserved words: not available / not applicable — never N/A, NA, unknown. Gated per-column by TERMS.tsv's allow_* booleans. Exception: sdrf-campaign's screening phase emits a six-column decision TSV, not an SDRF — label is include/exclude/uncertain, never a reserved word, and uncertain never belongs in an SDRF.
4. Modification format: NT=name;AC=UNIMOD:id;TA=aa;MT=Fixed|Variable. A positional value (Protein N-term) belongs in PP=, not TA=.
5. All ontology terms carry label + accession (e.g. "breast carcinoma" / MONDO:0007254). This file is the correct copy — GEMINI.md:45, .cursor/rules/sdrf-skills.mdc:44, and .opencode/AGENTS.md:50 all say EFO:0000305; fix those toward this file, not the reverse.
6. Duplicate columns are legal and load-bearing: multiple comment[modification parameters] (one per mod) and multiple comment[sdrf template] (one per template) — both cardinality: multiple. tools/sdrf_parser.py disambiguates with __N keys; keying rows by raw name silently reads only the first occurrence.
7. Validate before presenting any SDRF: parse_sdrf validate-sdrf --sdrf_file X --template Y, after refreshing the submodule. Several -t/--template flags validate against the union of those templates (the --help says so and tests/test_build.py proves it on three curated references, 2026-09-23); the older behaviour where only the last --template counted (observed 2026-07-17 on PXD061710) is gone from current sdrf-pipelines. If you are on an old install, run once per declared template. For the authoritative multi-template constraint set (column licensing + reserved-word allow_*), resolve with spec/scripts/resolve_templates.py — parse_sdrf enforces neither. parse_sdrf ships in sdrf-pipelines and is not installed by default (CI installs only requests, pytest, fastmcp, httpx — not sdrf-pipelines[ontology], which is heavy) — run sdrf-tools doctor (install notes: sdrf-annotate/references/setup.md). Keep concurrent parse_sdrf jobs ≤ 2.
8. A producer must never approve its own SDRF. For changed SDRFs, require a passing receipt from sdrf-adversarial-review; any edit invalidates the receipt and requires a fresh reviewer. Enforced by sdrf-tools review-gate (track, pending, status, gate, approve), which discovers changed artifacts from git and binds each receipt to the artifact's SHA-256, so an approval cannot outlive the content it describes. gate exits 1 while review is pending. Enforcement lives in the CLI, not the Stop hook, because four of the five platforms this repo supports cannot run Claude Code hooks at all.
9. Vendor RAW only in comment[data file] — .raw/.d/.wiff/.wiff2, never peak lists (.mgf/.mzML/.mzXML). A peak-list reference validates structurally but breaks reprocessing.
10. Row-uniqueness coordinate: (source name, characteristics[biological replicate], comment[technical replicate], comment[fraction identifier]) must be unique; the spec MUST-unique key is source name+assay name+comment[label]. The sdrf-annotated-datasets review gate rejects collisions, so annotate/fix/contribute must produce it.
11. Value encoding by column type: characteristics[...] = the bare ontology label (never NT=;AC=); comment[...] = NT=<OLS label>;AC=<accession>; structured characteristics (spiked compound CT=/QY=, pooled sample SN=) keep key-value.
12. Acquisition method (comment[proteomics data acquisition method], required for MS) is a descendant of PRIDE:0000659 — DDA PRIDE:0000627, DIA PRIDE:0000450, PRM PRIDE:0000629, SRM PRIDE:0000630; never MS:1000206/NCIT:C161786; comment[dia method] was removed.
13. Contribution hygiene: a datasets/ PR adds exactly one new {ACC}/ folder — git diff --cached --name-status must show 0 deletions/modifications to other datasets; unresolved datasets go to CI-exempt sandbox/ with a BLOCKED: note; no AI/assistant attribution in public commits or PRs. Never invent sample->file/channel maps, demographics, or runs.

Landmines

• CI runs no tests on the product. tools-tests.yml is path-filtered to tools/**, tests/**, requirements.txt. A PR touching only skills/** — the actual product — plus hooks/, mcp/, plugin.json, or docs runs zero tests. check-spec.yml has no path filter so it runs on every PR, but only warns when the spec is stale; it never fails the build.
• Fixture path-filter gap: tests/conftest.py reads examples/PXD_synthetic.sdrf.tsv and tools/cellline_db.py defaults to data/*.tsv, but neither examples/** nor data/** is in the CI filter. Editing them can break main with CI never running — run pytest locally.
• tests/test_services.py mocks fail open: it pre-seeds the private _cache with a key that must byte-for-byte equal f"{base_url}{endpoint}|{params}". Any mismatch issues a live request to Cellosaurus/UniProt/EBI instead of failing. Prefer MagicMock(spec=OLSClient) injection, as tests/test_hallucination.py does.
• Dead code: tools/services.py is imported only by its own test.
• tools/sdrf_fixer.py implements 7 fix patterns, not the 10 its own docstring and the README claim (comments number them 1,3,4,5,6,7,9).