SDRF Annotation Workflow

Bundle paths. spec/, tools/ and data/ ship with this skill, not with your working directory. Resolve every such path below against the bundle root (SDRF_SKILLS_ROOT, else CLAUDE_PLUGIN_ROOT, else your sdrf-skills checkout): ${SDRF_SKILLS_ROOT:-${CLAUDE_PLUGIN_ROOT}}/spec/sdrf-proteomics/TERMS.tsv. The helpers are the sdrf-tools command (install notes: sdrf-annotate/references/setup.md; check with sdrf-tools doctor); no PYTHONPATH or bundle-root variable is needed to run them. Files the user is annotating stay relative to the working directory.

You are performing a complete SDRF annotation. Work through the steps in order, but settle Step 0 first: it decides which of the later steps apply at all. Do not guess a value — verify it, against the record when you have network tools and against the files in front of you when you do not.

Step 0: Settle the operating mode, then stop looking around

Answer the three questions below before any other tool call. Getting them wrong is what turns a ten-turn annotation into a fifty-turn one, because the workflow below assumes lookups that may not exist here and you will spend turns discovering that one failed call at a time.

1. Is the record already on disk? If the working directory holds an evidence/ directory, or the user handed you the PRIDE record, file list and paper as files, then the gathering is already done. You are in offline mode:

• Read those files, and treat them as the whole of the available record.
• Skip Steps 0.5, 1.1, 1.2 and 1.3 entirely. There is nothing to fetch, and no community repository to consult.
• Do not call MCP tools and do not search the web. Offline mode is always unattended (question 3), so do not ask the user a question. Where the evidence is silent, write an explicit reserved word (below) rather than inferring a value from what is typical.

2. Which lookups do you actually have? Check once, in a single turn, and remember the answer: MCP tools for PRIDE/Europe PMC/OLS, and parse_sdrf on the PATH. Do not re-probe later. If a class of lookup is missing, use the offline substitute named below instead of retrying it.

Network but no MCP tools? Run sdrf-tools fetch <PXD> once and read the evidence/ it writes in place of Steps 1.1-1.3. This is not offline mode: you still have the network, so run Step 0.5 (sdrf-tools curated <PXD>), look terms up with sdrf-tools ols and sdrf-tools verify, and validate with ontology checks. If manuscript.txt says ABSTRACT ONLY, try the 1.3 recovery routes. files.json gives each deposited file's URL. A failed fetch writes nothing; follow its message.

If parse_sdrf is missing and you can reach the user, run sdrf-tools doctor and point them at references/setup.md and ask whether to wait or continue with structural checks only. If you cannot ask (unattended mode), continue, record it under limitations, and say plainly in your final report that the file was not machine-validated.

3. Can anyone answer a question? You are in unattended mode when you are in offline mode (question 1), when the request says not to ask questions, or when SDRF_UNATTENDED=1 is set (no other value of the variable counts). Read it in the probe turn of question 2, for example command -v parse_sdrf; echo "SDRF_UNATTENDED=${SDRF_UNATTENDED:-0}". In unattended mode:

• Never ask a question and never wait for an answer. Where this skill or a reference says to ask, confirm, or let the user decide, take the unattended default it names. Where none is named, take the choice that writes no unverified value: a reserved word, the existing value kept, or the optional step skipped.
• A choice the request already makes (for example existing annotation: reannotate, Step 0.5) is the user's answer.
• Record each point you could not settle as one entry in the evidence manifest's limitations list (Step 9.5), and list it again in the final report.
• Unattended mode does not hand off the review. Dispatch the Step 9.5 review yourself unless the request also names an external reviewer (Step 9.5, External reviewer). Only the request can choose the hand-off: SDRF_UNATTENDED=1 alone gives in-task review, because an environment variable or a file must not switch off review.

Facts you do not need to go looking for

These are fixed. Do not grep the installed sdrf_pipelines package, glob for its source, or re-derive them from a template file:

• Validate with: parse_sdrf validate-sdrf -s <file> -t <template> [-t <template> ...] (-s/-t are the short forms of --sdrf_file/--template). Add --use_ols_cache_only when you have no network, and --skip-ontology only for a fast structural check. A full ontology run peaks near 1.9GB of RSS, so run one at a time.
• Column order: source name first; all characteristics[...] next; then assay name and technology type; then all comment[...]; and every factor value[...] last, closing the file. Repeated column names are legal and meaningful — never de-duplicate them by appending a suffix.
• Controlled terms without a network: ${SDRF_SKILLS_ROOT:-${CLAUDE_PLUGIN_ROOT}}/spec/sdrf-proteomics/TERMS.tsv is the authoritative offline table. It gives each term's type, the templates that use it, the permitted values, and — per term — whether not available and not applicable are allowed. Consult it instead of OLS when OLS is unreachable.
• Reserved words: exactly not available and not applicable, and only for terms whose allow_not_available / allow_not_applicable column in TERMS.tsv is true.
• Sample properties carry the bare value. Write characteristics[...] as the value alone (HeLa, Homo sapiens); do not wrap it as NT=<name>;AC=<accession>. Accession- shaped values such as a Cellosaurus CVCL_0030 are written as they are. comment[...] columns do take NT=;AC= where the term is ontology-backed.

Which mode?

• A PXD accession (or a description of an experiment) → annotate: Steps 0a–11 below, ending with the independent review in Step 9.5. Review is not optional; an annotation that has not been independently reviewed is a draft. Only the caller can hand the review to an external reviewer (Step 9.5).
• A path to an existing .sdrf.tsv → review it: skip to Reviewing an existing SDRF at the end of this file. Do not re-annotate a file you were asked to review.
• A question about what to capture, before there is a file → planning: see the last section.
• A question about the format (what a column, value, error or concept means; which template; how to write a modification) → answer it from references/format-rules.md and references/explaining.md; look terms up per references/ols-lookup.md. Do not start an annotation.
• "Validate / check / fix / score this file" → the review mode below does all four; a user who wants only the deterministic part can run parse_sdrf validate-sdrf, sdrf-tools structure, sdrf-tools check, sdrf-tools fix or sdrf-tools score directly.

Tools missing? sdrf-tools doctor says what, and references/setup.md says how to install it.

Step 0a: Isolate this dataset's working files (required)

When annotators run concurrently they collide through a shared scratchpad: generic filenames (files_all.json, build.py, efetch.xml) written by several agents into one directory silently overwrite each other. The failure is silent — the file still parses, it just describes a different dataset — so an agent that trusts the re-read annotates the wrong PXD. This has happened (a cached PRIDE file list overwritten mid-run with another accession's data; an efetch.xml replaced by an unrelated paper).

1. Derive your working directory from the accession — scratchpad/<PXD>/ — and write every temp file there and nowhere else.
2. Never read a scratch file you did not write in this run.
3. Point PDF/full-text fetchers at that directory (get_pdf_by_unpaywall(output_dir="scratchpad/<PXD>/")); mcp/pdf/ is shared by default and two agents fetching different papers will collide.
4. Assert on read anyway (defence in depth — the substitution also comes from outside): after fetching the file list, check every entry's projectAccessions contains your accession; when you pull supplementary files or an efetch result, verify the returned title/accession, not just HTTP 200. (Europe PMC supplementaryFiles has returned another paper's mmc*.xlsx; efetch with id=PMC… silently returns a different article — use the numeric id.)

Output files

An annotation of <PXD> writes these files to the working directory. <PXD> is the dataset accession as deposited (PXD…, MSV…, IPX…); without one, use a short name for the study. A caller may give other names; then use those everywhere below.

samples.tsv and technical.tsv (Steps 3-5) are working files, not outputs. Keep them until the review passes, because every fix edits them and rebuilds. evidence/ holds the project record (pride.json, files.json, manuscript.txt) that the manifest cites. In offline mode it is input: do not change it.

Step 0.5: Is the dataset already annotated? — STOP GATE

Offline mode (Step 0): skip — the community repository is not reachable. After sdrf-tools fetch you are not offline: run this step.

Before annotating a PXD, enumerate its directory in the community repository (it may hold several suffixed SDRFs), audit any existing file against the deposit, and let the user decide whether to re-annotate. Never re-annotate silently or overwrite without an explicit reannotate. Read references/existing-annotation.md for the lookup, the audit, the report format and the unattended default.

Step 1: Gather the project record

Offline mode (Step 0): the record is already in evidence/ (pride.json, files.json, manuscript.txt when a paper exists). Read those; do not fetch anything.

Otherwise gather, in this order: the PRIDE project record; the run list (raw files only — a deposited SDRF is never consulted); the publication and its Methods; sample metadata from the paper; BioSamples where the deposit maps sources; and the plasma guard when the study is plasma. The archive's structured fields (instruments, diseases, organismParts) are submitter dropdowns, not the record — the title, the protocol prose and the run names outrank them. Read references/gather-context.md for each source's endpoint, what to extract, and the traps.

Step 2: Select Templates

Read references/templates.md — layers, exclusivity, the decision tree. Based on the gathered context:

1. Technology: MS → ms-proteomics. Affinity → affinity-proteomics
2. Organism: Human → human. Mouse/rat → vertebrates. Drosophila → invertebrates. Plant → plants. Microbiome → metaproteomics + child
3. Experiment type: DIA → + dia-acquisition. Cell lines → + cell-lines. Single-cell → + single-cell. XL-MS → + crosslinking. Immunopeptidome → + immunopeptidomics
4. Clinical/Oncology: Patient study → + clinical-metadata. Cancer → + oncology-metadata

Present the template selection to the user for confirmation before proceeding. Explain WHY each template was chosen and what columns it adds. In unattended mode, do not wait for the confirmation: put the selection and its reasons in the report and continue.

Step 3: Get the contract, then author two tables — do not write the SDRF by hand

You will not build SDRF structure yourself. Column order, fraction and replicate numbering, channel rows and repeated columns are produced by build from two small tables you write. This is what keeps the file structurally valid on the first try and what keeps this annotation short: no spec reading beyond one command, no editing the SDRF, no re-reading it.

3.1 Print the contract (once)

sdrf-tools contract -t <template1> [-t <template2> ...]

It lists every column of the template union in order, whether it is required, whether it takes multiple values, the value form (bare value, NT=<name>;AC=<accession>, <number> ppm|Da, integer), and which reserved words it permits. Keep it in context; do not read TERMS.tsv, the template YAMLs or the spec README to learn the same thing.

contract and build need sdrf-pipelines; if the python3 on your PATH cannot import it but parse_sdrf is installed, they re-run themselves under parse_sdrf's interpreter. Do not go looking for another Python yourself.

3.2 Write samples.tsv — one row per source and replicate

Tab-separated. Three structural columns, then the sample properties you can support from the evidence:

Replicates are separate rows, not a longer files list. A source measured in three runs is three rows that differ in characteristics[biological replicate] or technical replicate; a comma list means fractions of a single injection. build refuses two rows that share (source, biological replicate, technical replicate, fraction) — that is the row coordinate the community review gate checks — rather than renumber them for you.

Multiplexed runs: one row per (source, channel), each listing the run file(s) in files. A channel the run does not use gets a row with an empty source name; build emits nothing for it. build refuses a run whose channel set is incomplete — it will not fill a channel in, and neither may you: see 6.1 for where the map is found.

3.3 Write technical.tsv — run-level settings shared by every row

Two columns, column and value, one row per comment column that is constant across the file: comment[instrument], comment[cleavage agent details], tolerances, comment[proteomics data acquisition method], comment[sdrf version], and so on. A multiple column such as comment[modification parameters] takes its values separated by | — one column per value is emitted:

column	value
comment[instrument]	NT=Orbitrap Fusion Lumos;AC=MS:1002732
comment[modification parameters]	NT=Carbamidomethyl;AC=UNIMOD:4;TA=C;MT=Fixed|NT=Oxidation;AC=UNIMOD:35;TA=M;MT=Variable

comment[sdrf template] and technology type are filled by build from the templates you pass; do not add them.

Steps 4 and 5 tell you how to find the values for these two tables. Step 6 builds.

Step 4: Find the sample values → samples.tsv

characteristics[...] carry the bare value (Homo sapiens, liver, HeLa), never NT=;AC=; structured exceptions keep their keys (spiked compound CT=/QY=, pooled sample SN=). Demographics are per-sample facts: age, sex, ethnicity need source-level evidence, developmental stage may come from an unambiguous cohort description; when only a cohort summary exists, leave the per-sample field out rather than guess. Verify every term in the ontology the contract names for its column; use not available only where the contract allows it. Cell lines: sdrf-tools cellline lookup <name>; the translation rules are in references/cellline.md. Read references/sample-values.md for the OLS procedure, embeddings/ZOOMA fallbacks, and the specificity and reserved-word rules.

Step 5: Find the technical values → technical.tsv

Instrument, cleavage agent, modifications, labels, acquisition method, tolerances, and the run-level settings, each as NT=<name>;AC=<accession> where the contract says so. Read the modifications out of the deposited search results, never infer them from the paper: sdrf-tools search-params <file> reads a MaxQuant mqpar.xml/summary.txt, a FragPipe fragger.params, a DIA-NN log or a PD .msf/.pdResult and prints technical.tsv rows, plus what it could not map (look those up, do not guess); the acquisition method is a descendant of PRIDE:0000659, PRIDE-first. Read references/technical-values.md for where each value lives in the deposit, and references/techrefine.md to verify it from the raw files with techsdrf.

Step 6: Build the SDRF

sdrf-tools build \
  --samples samples.tsv --technical technical.tsv --files evidence/files.json \
  -t <template1> [-t <template2> ...] -o <PXD>.sdrf.tsv

build assigns one row per (source, file); comment[fraction identifier] from the position in files (1 for a single file); comment[technical replicate] from the column or 1; assay name from the file stem unless you set it; the contract's column order with factor values last. Every file in files.json should be claimed by exactly one source (or, when multiplexed, by one row per channel).

If build refuses, it names the table and row. Fix that row and run it again. It refuses, rather than guesses, an incomplete channel map (6.1). If build cannot be run at all — the command is denied or the tools are missing — stop, keep samples.tsv and technical.tsv, and say so in the report. Do not write <PXD>.sdrf.tsv by hand: a hand-written file is exactly what this step exists to prevent, and on a large dataset it will not even fit in one response.

6.1 Multiplexed: exhaust every source of the channel→sample map before BLOCKING

Inventing a channel identity is the worst failure this skill can produce, and a missing map in PRIDE is not proof that none exists. Read references/channel-map.md and work its ladder (paper and supplementary → deposited PRIDE files → Europe PMC supplementary → repositories cited in Data Availability → cross-validation) before declaring the dataset BLOCKED.

Step 7: Set Factor Values

1. Identify what is being compared (disease vs control? treatment vs untreated?)
2. Add a factor value[<variable>] column to samples.tsv (e.g., factor value[disease]); build places it last
3. Copy values from the corresponding characteristics column
4. If multiple factors → one factor value[...] column per factor

Do not invent a factor value column for sample composition. A quantity that describes what was added to the sample during preparation — a spiked protein/peptide/mixture and how much of it, a carrier amount, a dilution — is sample metadata, not the compared variable. It belongs in characteristics[spiked compound] (CT=/QY=/PS=/AC=/CN=/CV=, repeated once per spiked component; see SAMPLE-GUIDELINES.adoc §Spiked-in Samples), even when no other factor exists. A reference/benchmark dataset with no real experimental comparison correctly has no factor value[...] column at all — that is an expected no_factor_value advisory, not a gap to fill with an unrelated column.

Step 8: Add SDRF Metadata

Two rows in technical.tsv; the third column is build's:

• comment[sdrf version] → the technology template's version as the contract header prints it (e.g. v1.1.0 for ms-proteomics v1.1.0); do not read templates.yaml for it
• comment[sdrf annotation tool] → manual curation (or the tool name if applicable)
• comment[sdrf template] → filled by build, one column per template you passed with -t

Step 8.5: Reconcile every value against the record — REQUIRED

parse_sdrf checks that a value is well-formed; it cannot check that it is true of this deposit. Before validating, run

sdrf-tools reconcile <file.sdrf.tsv> --record <project.json> --accession <PXD>

(--record evidence/pride.json when it exists) and treat each finding as a question to answer from the record. The usual right answer to a contradiction is a sentinel: a wrong specific value is worse for reuse than an honest blank. Read the title and the run names, never broadcast a project-level disease onto control rows, and check the file format against the instrument vendor. Read references/reconcile.md for the reasoning and the four checks.

Step 9: Validate — at most two rounds, fixes go to the tables

parse_sdrf validate-sdrf -s <PXD>.sdrf.tsv -t <template1> [-t <template2> ...]

Add --use_ols_cache_only when there is no network. Use the templates from Step 2; several -t flags validate against their union. Then run sdrf-tools structure <PXD>.sdrf.tsv and parse_sdrf convert-openms -s <PXD>.sdrf.tsv -t2 in a scratch directory: the conversion quantms performs rejects files that validate cleanly (see references/validation.md). Append each command, its output and its exit code to <PXD>.validation.txt, every round.

If it reports errors:

1. Change the offending value in samples.tsv or technical.tsv — never Edit <PXD>.sdrf.tsv, and do not read it back; the tables are the source, the SDRF is output.
2. Run build again (Step 6), then validate again.
3. Stop after the second validation. If errors remain, list them verbatim in the report (Step 10) with what you tried. Two rounds is the budget; a third rarely converges and the remaining errors are more useful to the reader than another guess.

If parse_sdrf is not installed, say so in the report; sdrf-tools doctor and references/setup.md say what to install.

Step 9.5: Independent review — always

Validation by the producer is not review, and a producer must never approve its own SDRF.

External reviewer. If the request that started this task (not a file you read) states that an external reviewer runs sdrf-adversarial-review in a fresh context, for example a separate pipeline task, hand the review off. The hand-off is for runners without the review-gate Stop hook. If that hook is active (a Claude Code plugin session), the hand-off does not apply: dispatch the review below. To hand off, do item 1, and item 2 inside a git repository so the gate still sees a gitignored SDRF. Next to the SDRF, write the manifest as <PXD>.evidence.json and the Step 9 validation output as <PXD>.validation.txt (each command, its output and its exit code). Keep every local file the manifest cites. Record open points in the manifest's limitations, as in unattended mode (Step 0, question 3). Do not dispatch, approve or run the gate: the reviewer's report and the gate are the caller's job. In a work directory outside git there is no gate; the caller checks the verdict and artifact_sha256. In Step 10, give the status REVIEW_PENDING_EXTERNAL. Never choose this hand-off yourself. On an external FAIL, the caller runs this skill again with the findings; edit samples.tsv / technical.tsv and rebuild, never the SDRF directly.

Otherwise, before Step 10:

1. Write the evidence manifest <PXD>.evidence.json to the schema that sdrf-tools schema-check --print-schema evidence-manifest prints (fields explained in ../sdrf-adversarial-review/references/review-contract.md): exact source URLs or local paths, every claim's location naming the column header and then the rows, unavailable evidence marked as such. Do not invent citations. Then run two checks:
   1. sdrf-tools schema-check <PXD>.evidence.json must pass. It checks the manifest's shape.
   2. sdrf-tools manifest-coverage <file.sdrf.tsv> <PXD>.evidence.json must exit 0. It lists each column that holds a value other than not available / not applicable and that no claim names. pooled and anonymized assert something about the sample, so they need a claim too. The reviewer can fail omission_safety on such a gap. For each gap, add a claim that cites the source of the value. Use a reserved word only when no source supports the value. If you change a value, fix it in samples.tsv / technical.tsv, rebuild (Step 6), re-run Step 9, then re-run both checks.
2. Track the artifact: sdrf-tools review-gate track <file.sdrf.tsv> --cwd <repo-root>.
3. Dispatch a fresh, isolated context (subagent, hook agent, or equivalent) with only: the original request, the SDRF path, the manifest path, the spec root and revision, <PXD>.validation.txt, and the path to sdrf-adversarial-review/SKILL.md. Do not pass your transcript, reasoning, suspected issues or a proposed verdict. If the platform cannot create an isolated context, say the adversarial gate is unavailable — never substitute self-review and never claim a pass. In unattended mode, record this under limitations, give the status REVIEW_FAILED and go to Step 10; do not commit or contribute the SDRF. A pipeline without isolated contexts should use the External reviewer hand-off above.
4. Repair and re-review. Evaluate each blocker or important finding against the evidence; push back only with concrete counter-evidence. Fix in samples.tsv / technical.tsv, rebuild, re-validate, then dispatch a new fresh reviewer — never ask the previous one to reuse its verdict. Escalate to the user after two failed rounds, or when a finding needs scientific judgement the evidence does not contain. In unattended mode, stop repairing at that point instead: list each open finding under limitations, give the status REVIEW_FAILED and go to Step 10; do not commit or contribute the SDRF.
5. Enforce the gate: sdrf-tools review-gate gate --cwd <repo-root> must exit 0 with the current hash approved before Step 10, before any commit, and before contribution. Report the reviewer identity, the hash, the validation commands, remaining minor findings, and evidence limitations.

Step 10: Present Results

Write the report to <PXD>.report.md and present the validated SDRF as a TSV code block. The report explains:

• Total rows and columns
• Sample groups and counts per group
• Templates applied (with version)
• File mapping summary
• Validation result (PASS / warnings)
• Any values marked as not available (ask user to fill)
• Any values you're uncertain about (flag for user review)
• The review status: gate passed, REVIEW_PENDING_EXTERNAL or REVIEW_FAILED (a draft)

In unattended mode, do not ask or flag for the user: list the not available values, the uncertain values and every limitations entry as open items.

If no SDRF could be built, the report says why, which sources you exhausted, and what would unblock the dataset.

Step 11: Recommend community contribution

For a ProteomeXchange dataset, tell the user the annotation can be contributed with /sdrf-skills:sdrf-contribute {PXD} (reuse the Step 0.5 result to say whether this is new or an update; an update's PR must state what was wrong before, with evidence). A recommendation only. Recommend it only when the review gate passed, never for REVIEW_PENDING_EXTERNAL or REVIEW_FAILED.

Rules that do not bend

The format rules — value encoding, reserved words, modification syntax, UNIMOD swaps, label types, row identity — live in one place: references/format-rules.md. Read it once per annotation. On top of them:

• NEVER re-annotate an already-annotated dataset silently (Step 0.5)
• NEVER take the archive's structured fields as the record; reconcile (Step 8.5) and write a sentinel on disagreement
• NEVER assert a project-level disease on every row — check for a control arm first
• NEVER match an ontology term on a keyword without checking the matched word's ROLE ("chymotrypsin-like activity" is an assay, not a digest; icat matches inside "quantifi(cat)ion")
• NEVER fabricate accessions, guess file names, or invent sample information
• Always distinguish: extracted from the paper vs inferred vs assumed

Reviewing an existing SDRF

You were given a .sdrf.tsv, not an accession. Review it; do not rebuild it.

1. Load the record. Read the file; read its declared templates from comment[sdrf template] (do not upgrade versions — report a newer one as an optional migration). If a PXD is known, gather the record as in Step 1 (offline: whatever is in evidence/). If this is a pull request, read the diff to see what changed.
2. Validate. parse_sdrf validate-sdrf -s <file> -t <t1> [-t ...] --use_ols_cache_only, then sdrf-tools structure <file> and sdrf-tools check <file> --offline (references/validation.md has the full checklist). Collect every error and warning; do not fix anything yet.
3. Reconcile and cross-reference. sdrf-tools reconcile <file> --record <project.json> --accession <PXD> (Step 8.5), then the checks in references/review-checks.md: sample counts, conditions, instruments, demographics and tissues against the paper; file names, organism and instrument against PRIDE; batch effects, confounders and replication in the design. Write each disagreement as a DISCREPANCY: line with the evidence on both sides.
4. Score. sdrf-tools score <file> (completeness, specificity, consistency, standards, design).
5. Independent review. Step 9.5 applies, including its external-reviewer case: manifest, track, fresh-context sdrf-adversarial-review, gate. If this context edited the file, everything above is advisory — the verdict comes only from the isolated reviewer.
6. Report: verdict (valid / needs minor fixes / needs major fixes / invalid), the errors and discrepancies in priority order, what sdrf-tools fix can repair deterministically (references/fix-patterns.md), what needs a human, and — for a ProteomeXchange dataset with a clean verdict — the offer to contribute (Step 11).

Planning instead of annotating

When the user wants to discuss what to capture before there is a file — templates, columns, design considerations, expected row count — read references/planning.md and follow it instead of the steps above.