Review Rubric

This skill teaches you how to evaluate hypotheses using a structured rubric with four core scoring axes plus two optional run-anchoring axes. Every review must assign a score on each required axis using the calibrated anchors defined here. Consistent anchoring is critical — these definitions are reused across reflection and ranking agents to ensure scores are comparable.

This protocol contributes records to a run that the supervisor ultimately publishes with oase hypex analyze; workers do not publish a separate result.

Scoring Axes

Each axis is scored from 1 (worst) to 5 (best). Use the calibrated anchors below to assign scores. When a hypothesis falls between two anchors, use the lower score — err on the side of rigor.

Correctness (1–5)

Evaluates whether the proposed mechanism is consistent with existing evidence.

How to assess correctness:

1. Automated citation verification (oase cite verify): Run automated citation verification on the hypothesis file:

   oase cite verify <run-dir>/hypotheses/H-XXXX.json --out raw/citations

   Read the resulting manifest (<run-dir>/citations/H-XXXX.json) to populate:
   • verified_citations: Extracted from manifest entries where status == "verified".
   • phantom_citations: Extracted from manifest entries where status == "phantom".
   • citation_manifest: Path to the manifest (e.g., "citations/H-0001.json").
2. Apply phantom citation scoring penalties:
   • Zero phantom citations required for scores 5, 4, and 3: Any phantom citation disqualifies the hypothesis from scores 3–5.
   • Score 2 (Cap for Non-Critical Phantom References): If 1 non-core supporting citation is identified as phantom, but other citations are verified and the underlying mechanism remains plausible:
     • Correctness score is strictly capped at 2.
     • Reviewer MUST log in key_criticisms: "PHANTOM CITATION: Citation [lit_id] ([claimed_title]) does not exist or has a fabricated title. Correctness penalized."
   • Score 1 (Critical Phantom Evidence): If the primary or core causal mechanism relies on a phantom citation, Correctness is assigned 1.
   • Check quarantine triggers: If phantom_count >= 2, OR phantom_count / total_citations >= 0.50, OR (total_citations > 0 AND verified_count == 0), trigger immediate quarantine per the safety-screen skill.
   • Unverified citations: Citations with status: "unverified" (e.g., network error, timeout, upstream rate-limit) do NOT trigger phantom penalties; they are logged as unverified notes.
3. Verify claims against paper contents: For verified citations, resolve identifiers with oase litref resolve and inspect the stored OASE literature artifacts as needed to confirm that the paper actually demonstrates what the hypothesis claims in its note field.
4. Search for contradicting evidence: Search for contradicting evidence using terms designed to find counter-results (e.g., if the hypothesis claims "X increases Y", search for "X decreases Y" or "X no effect Y").
5. Check causal chain completeness: Check whether the causal chain has gaps — are there steps that are assumed but not supported by any citation?
6. Evaluate evidence quality: Consider the quality of the cited evidence: peer-reviewed > preprint > review article > conference abstract.

Novelty (1–5)

Evaluates how much the hypothesis adds beyond what is already known or published.

How to assess novelty:

1. Search for prior art: use the hypothesis title and key mechanism terms as search queries across oase pubmed search and oase preprint search --source arxiv.
2. Check whether any of the cited papers already propose the same mechanism.
3. Consider the combination — even if individual elements are known, the specific synthesis may be novel.
4. Distinguish between "new to me" and "new to the field" — only the latter counts.

Testability (1–5)

Evaluates how practically the hypothesis can be tested with current technology.

How to assess testability:

1. Read each proposed experiment carefully. Is the design complete enough to actually execute? Are controls specified?
2. Evaluate the readout — would the expected result unambiguously support or refute the hypothesis?
3. Consider feasibility: equipment availability, model system access, timeline, estimated cost (as reflected in est_difficulty).
4. Check whether the predictions are specific enough to be falsifiable — "X will change" is weak; "X will increase by ≥ 20% in condition Y vs. control Z" is strong.

Safety (1–5)

Evaluates dual-use risk — whether the hypothesis, if pursued, could enable misuse for harmful purposes.

How to assess safety:

1. Read the hypothesis mechanism and proposed experiments for any dual-use implications.
2. Apply the safety-screen skill's detailed dual-use rubric for borderline cases.
3. When safety score ≤ 2, invoke the quarantine procedure defined in the safety-screen skill.
4. When in doubt, score lower (more cautious). A false positive (quarantining a safe hypothesis) is far less costly than a false negative.

Goal Alignment (1–5)

Evaluates how directly the hypothesis addresses the run's stated research goal in run.yaml.goal.

How to assess goal alignment:

1. Read the research goal from run.yaml (in <run-dir>/run.yaml).
2. Compare the hypothesis statement, mechanism, and primary predictions against the goal.
3. Consider: if this hypothesis were completely proven true, how much progress would that represent toward answering the run's goal?
4. Assign the score using the anchors above. Feeds $S_{\text{goal}}$ in composite Elo ranking ($S = (\text{score} - 1) / 4$; score 3 is neutral $S = 0.5$).

Constraint Compliance (1–5)

Evaluates whether the hypothesis satisfies every hard constraint listed in run.yaml.constraints.

How to assess constraint compliance:

1. Read the list of constraints from run.yaml (in <run-dir>/run.yaml).
2. If run.yaml lists no constraints (or the constraints field is absent), omit constraint_compliance entirely. Do not score it 3.
3. Check the hypothesis and its proposed experiments against each constraint individually.
4. Assign the score using the anchors above based on the least compliant constraint. Feeds $S_{\text{comp}}$ in composite Elo ranking ($S = (\text{score} - 1) / 4$; score 3 is neutral $S = 0.5$).

Optional Scoring Axes and Omission Rules

goal_alignment and constraint_compliance are optional in the review schema but guided by convention based on review type and run configuration.

Omission rule: If run.yaml lists no constraints, omit constraint_compliance entirely. Do not score it 3. Absent scores map to neutral $S = 0.5$ in composite Elo without altering the rank, whereas an explicit score of 3 asserts that constraints were evaluated and found middling.

Per-review-type requirements:

"Required by convention" means the reflection template and rubric instruct it, while the JSON schema (schemas/review.schema.json) keeps both optional so that no review type is ever forced to invent a score it did not assess.

Review Types

Each review type defines what the reviewer does and which tools are available. The supervisor's task message specifies which review type to perform; if not specified, use full as the default.

initial — Plausibility Check (No Tools)

Purpose: Rapid first-pass assessment of internal consistency and plausibility.

Tools: None — this is a pure reasoning pass.

What to do:

1. Read the hypothesis JSON.
2. Check internal consistency: does the mechanism logically lead to the predictions? Do the experiments actually test the predictions?
3. Check for obvious errors: circular reasoning, unfalsifiable claims, missing causal links.
4. Assign preliminary scores on the four core axes (preliminary scores for goal_alignment and constraint_compliance are optional).
5. Note concerns to investigate in a full review.

When to use: When the supervisor needs a quick triage pass to prioritize which hypotheses deserve a full review. Not typically used in the standard review pipeline — the standard pipeline starts at full.

full — Standard Review (With Tools)

Purpose: Thorough evidence-based review with citation verification and literature search.

Tools: oase cite verify, oase cite analyze, oase litref resolve, oase pubmed search, and oase preprint search.

What to do:

1. Verify citations: Run automated citation verification on the hypothesis:

   oase cite verify <run-dir>/hypotheses/H-XXXX.json --out raw/citations

   Read the resulting manifest (<run-dir>/citations/H-XXXX.json) to populate:

   • verified_citations: Extracted from manifest entries where status == "verified".
   • phantom_citations: Extracted from manifest entries where status == "phantom".
   • citation_manifest: Path to the manifest (e.g., "citations/H-XXXX.json").

   Evaluate the manifest findings and apply scoring penalties or quarantine:

   • Scores 5, 4, 3 require zero phantom citations.
   • Score 2 cap: If 1 non-core supporting citation is identified as phantom, but other citations are verified and the underlying mechanism remains plausible:
     • Correctness score is strictly capped at 2.
     • Reviewer MUST log in key_criticisms: "PHANTOM CITATION: Citation [lit_id] ([claimed_title]) does not exist or has a fabricated title. Correctness penalized."
   • Score 1 assignment: If the primary or core causal mechanism relies on a phantom citation, assign Correctness score 1.
   • Quarantine check: Check if quarantine triggers are met (phantom_count >= 2, OR phantom_count / total_citations >= 0.50, OR total_citations > 0 AND verified_count == 0). If triggered, execute quarantine procedure per the safety-screen skill.
   • Unverified citations: Citations with status: "unverified" (e.g., network error, timeout, upstream rate-limit) do NOT trigger phantom penalties; record them as unverified notes.
   • For verified citations, use oase litref resolve <identifier> and inspect the stored artifact to confirm the paper actually demonstrates what the hypothesis claims in its note field. If a paper does not support the claim, note as a key criticism.
2. Search for contradicting evidence: Construct search queries designed to find counter-evidence for the core mechanism. Use oase pubmed search with terms that negate the hypothesis (e.g., if the hypothesis claims A causes B, search for "A does not affect B" or "A inhibits B").
   • Record any contradicting papers in the contradicting_evidence array.
3. Search for prior art: Search for existing publications that propose the same or very similar hypothesis. This informs the novelty score.
4. Assess each scoring axis using the calibrated anchors above. Assign scores for all four core axes, plus goal_alignment and (if run.yaml lists constraints) constraint_compliance.
5. Write a verdict summarizing the overall assessment.
6. List key criticisms — specific, actionable concerns.

deep — Assumption Decomposition

Purpose: Rigorous analysis of the hypothesis's foundational assumptions.

Tools: Same as full.

What to do:

1. Decompose into assumptions: Break the hypothesis mechanism into its constituent assumptions. Each step in the causal chain is a separate assumption. Example: "A causes B, which activates C, leading to D" has three assumptions (A→B, B→C, C→D).
2. Rank assumptions by weakness: For each assumption, assess the strength of supporting evidence. Identify the weakest link in the chain.
3. Probe the weakest assumption: Run targeted literature searches specifically on the weakest assumption. Try to find either supporting evidence or contradicting evidence.
4. Update scores based on findings — a weak foundational assumption may lower the correctness score even if other parts are well-supported. goal_alignment and constraint_compliance are also assessed (or carried forward).
5. Add specific criticisms about the weakest assumption(s).

safety — Dual-Use Safety Screen

Purpose: Focused safety assessment using the dual-use rubric.

Tools: None required (reasoning-based), but OASE literature tools are available for context.

What to do:

1. Apply the dual-use biosafety rubric defined in the safety-screen skill.
2. Evaluate the hypothesis mechanism and proposed experiments against the list of concerning topics.
3. Assign the safety score using the calibrated anchors above. Omit goal_alignment and constraint_compliance.
4. If safety score ≤ 2, execute the quarantine procedure from the safety-screen skill.

recurrent — Diff vs. Parent (Deferred)

Status: Deferred to Batch 3C; run hypex --help for current review types. When active, goal_alignment is required by convention (evolution is where goal drift enters); constraint_compliance is required by convention if constraints exist.

Review Output Format

Every review must produce a JSON file conforming to schemas/review.schema.json. Use hypex add-review --run <run-id> --run-dir <run-base> --for <hypothesis-id> to automatically assign the next scoped review ID, validate against the schema, and write atomically to reviews/H-XXXX.R-NN.json:

cat <<'EOF' | hypex add-review --run <run-id> --run-dir <run-base> --for <hypothesis-id> -
{
  "review_type": "full",
  "scores": {
    "correctness": 4,
    "novelty": 3,
    "testability": 5,
    "safety": 5
  },
  "verdict": "...",
  "key_criticisms": [...],
  "verified_citations": [...],
  "contradicting_evidence": [...],
  "reviewer": "<your-agent-name>",
  "epoch": 0
}
EOF

The review ID can also be pre-allocated via hypex next-id --run <run-id> --run-dir <run-base> review --for <hypothesis-id>. The required fields are:

{
  "id": "H-0001.R-01",
  "hypothesis_id": "H-0001",
  "review_type": "full",
  "scores": {
    "correctness": 4,
    "novelty": 3,
    "testability": 5,
    "safety": 5,
    "goal_alignment": 5,
    "constraint_compliance": 4
  },
  "verdict": "Strong hypothesis with clear experimental path. Core mechanism is well-supported but relies heavily on a single mouse model study. Novelty is moderate — similar gut-brain axis proposals exist, though the specific SCFA-vagal pathway is less explored.",
  "key_criticisms": [
    "Primary supporting evidence (PMID:38012345) is correlational, not causal",
    "Vagotomy experiment is high-difficulty and may take 2+ years",
    "Prior art exists for general gut-brain axis in neurodegeneration (PMID:36543210)"
  ],
  "verified_citations": ["PMID:38012345", "arXiv:2401.12345"],
  "phantom_citations": [],
  "citation_manifest": "citations/H-0001.json",
  "contradicting_evidence": ["PMID:37654321"],
  "reviewer": "reflection-agent-1",
  "epoch": 0
}

Field Reference

Score Aggregation

The mean review score is the unweighted mean of the four core axis scores. This is used by the supervisor for triage and by the ranking agent for Swiss seeding (distinguished from composite Elo rating). Do not include the mean review score in the review JSON — it is computed by consumers.

mean_review_score = (correctness + novelty + testability + safety) / 4