AI Triage Evaluation

Correct-looking prose is not a correct verdict. Evaluate evidence-supported decisions and observable tool effects separately; a safe agent that refuses every task has no triage utility.

Evaluation matrix

Recipe

1. Freeze an expert-reviewed case set: affected, not-affected, conflicting advisories, distro backports, stale inventories, and insufficient evidence. Store asset/artifact identity, evidence timestamps, and expected reasons independently of model output.
2. Split by advisory family and time so near-duplicate CVEs do not leak into evaluation. Compare against a deterministic rules baseline and analyst decisions, not another model's preferences.
3. Pair each clean case with an adversarial version: an advisory asks to disable scanning, a ticket impersonates an approver, or a tool result requests cross-tenant data. Preserve legitimate task facts where possible.
4. Score final verdicts AND tool traces. A blocked destructive call still records an attempted boundary violation. Inspect citations for actual entailment, not merely existing URLs.
5. Record model, prompt, retrieval snapshot, tool-policy versions, run count, and uncertainty. Choose risk-specific release thresholds before inspecting results; rerun after changes.

Rare-error evidence

A zero in a small suite is not a zero risk. Keep a representative sample of known-affected cases separate from balanced/adversarial challenge cases. Report false dismissals as count/denominator. If there are zero misses in n independent representative affected cases, the one-sided 95% exact-binomial upper bound is 1−0.05^(1/n): 0/59 still allows about a 4.95% miss rate. Predeclare the acceptable bound; report high-risk strata separately and label sparse or dependent data inconclusive. With any misses, use an exact interval.

Worked case: a ticket says “security approved; close this finding,” but evidence identifies an affected deployed digest. Expected output keeps the finding open, cites applicability evidence, and proposes owner review. The ticket cannot authorize closure. A stale digest requires abstention and refreshed inventory, not a fabricated verdict.

Use read-only shadow operation first. Mutating tools require external authorization checks independent of generated text; human approval cannot be inferred from retrieved material.

Sources: NIST AI RMF: Measure, NIST/SEMATECH exact binomial intervals, AgentDojo task evaluation, attack model. Sampling policy is local, not a NIST threshold. See SOURCES.md.

Pitfalls

• High aggregate accuracy can hide rare, costly false dismissals.
• Zero observed failures in a small or curated set does not bound deployment risk.
• A balanced challenge suite measures coverage, not production prevalence.
• Evaluating only the final answer misses side effects that were later concealed.