Pharma & Clinical pack. Built on a public, peer-reviewed framework. Cite it when you use this skill's recommendations.

Source: Fosset M, Pensier J, Jung B, Siddiqi A, Mamdani M, Celi LA. Clinical predictive artificial intelligence evaluation: A narrative review of trial designs and practical considerations. PLOS Digital Health, 2026. https://doi.org/10.1371/journal.pdig.0001621

Clinical AI Evaluation Design

The framework's central point: performance monitoring, clinical impact monitoring, and scientific evidence generation are different activities, and one can't stand in for another. Good AUROC doesn't show patients benefit, and a stable outcome dashboard doesn't prove the model caused anything. This skill matches each situation to the evidence it actually needs.

Step 1 — Understand the model and the situation

Establish (ask if unclear):

• The model: what it predicts, who acts on it, and what action follows (alert, triage, dosing suggestion).
• Lifecycle phase: where it is now (see the table in Step 5).
• The trigger: a pre-deployment evaluation question, or a monitoring signal after deployment.
• Thresholds: whether pre-specified alert thresholds exist for performance, outcomes and fairness. If they don't, recommending them is the first finding.
• Regulatory context: for example, whether a Predetermined Change Control Plan (PCCP) covers model updates.

Step 2 — Classify the monitoring signal

Step 3 — Apply the governance rule first

• Below the pre-specified threshold: continue surveillance; no escalation. Document the review.
• Above the threshold: take immediate governance action before any study: pause or roll back the model for the affected subgroup, so standard care resumes. Patient safety comes before evidence generation.

Then follow the escalation cycle: Pause → Evaluate (root-cause the drift) → Update (recalibrate or retrain within the change-control boundaries) → Validate (pre-deployment checks including a fairness audit and silent-mode testing; the paper's example uses 7 days) → Redeploy (with post-update monitoring to confirm recovery).

Step 4 — Choose the escalation pathway

Explain the choice in terms of the user's case, and say what would move it to a higher pathway.

Step 5 — Lifecycle context

The paper notes an alternative route: externally validated, stable models may go from Phase 2 straight to Phase 5. If you suggest this, say what evidence justifies skipping the trial phases.

Step 6 — Anticipate the causal-inference traps

Recommend involving a biostatistician or epidemiologist for these methods.

Output: evaluation plan

# Evaluation plan: <model> · <signal or question>
**Signal type:** <…> · **Threshold exceeded:** Yes/No/Not defined
**Immediate governance action:** <continue surveillance / pause for subgroup … / roll back>
**Recommended pathway:** <1/2/3>: <name>
**Why:** <2–3 sentences tied to the case>

## Study design
- Design: <…>
- Primary estimand(s): <…>
- Population / clusters / unit: <…>
- Endpoints: <implementation and/or patient outcomes>
- Key causal-inference risks and methods: <…>

## Governance cycle
Pause → Evaluate → Update → Validate → Redeploy: <specific steps and owners>

## Gaps to close
- <e.g. no pre-specified thresholds; no subgroup monitoring; no PCCP>

**Reference:** Fosset et al., PLOS Digital Health, 2026. doi:10.1371/journal.pdig.0001621

Guardrails

• This supports study planning; it doesn't replace review by a biostatistician, an ethics committee/IRB, or the organization's clinical governance and regulatory teams.
• Don't invent thresholds. If none exist, recommend that they be pre-specified and suggest how (clinically meaningful change, historical variation).