AI exchange audit

Some code depends on what a model returns. It only works if the model can ask for the shapes it must produce, read what it is about to change, and have its output validated and repaired against the same source of truth the code uses. A model fed a hand-copied schema, whose output is parsed by hope, is a blind interaction: it guesses, the copy drifts, and failures look like success.

1. Inventory every model call

Search for every place a model is invoked: SDK clients (anthropic, openai, generateText, chat.completions, messages.create), agent and tool definitions, prompt files (prompts/, *.prompt.md, template strings with "You are"), structured-output helpers, and any JSON.parse of a completion. State the denominator ("graded 14 of 14 call sites").

2. Classify each call's grounding pattern

The defect in a prompt-stuffed call is the copy itself, not its content. Accurate hand-copied schemas still drift.

3. Grade A–F on these dimensions

The scale:

• A+: tool-mediated, runtime schemas from the source of truth, state read before writing, validated output, repair loop, gated apply.
• B: grounded, but one of those is missing.
• C: prompt-stuffed.
• D: blind, with some parse guard.
• F: blind on a call that writes state, or a prompt copy that has already drifted. Cite the diff.

Back every grade with evidence from both sides: the code-side source (validator, type, file:line) and the model-side entry (prompt file:line, tool list), or proof that one of them is missing.

4. The checks that most often find real defects

• catch { return {} }, ?? [], or || defaultValue after parsing model output. That is success-with-empty: the caller cannot tell the model failed.
• Schema text in a prompt that differs from the validator. Diff them, never eyeball.
• A "repair" loop that retries without passing the errors back.
• Absence scans. Search for the checks that should exist and don't: output used with no validator, a repair loop with no bound, an apply with no approval step, a raw record sent where a filtered read exists. An absence is a finding: cite the empty search, and show the same search hits where the check does exist.

5. Report

Write a tracker (for example docs/AI-EXCHANGE.md):

• one section per call site: pattern, grade, evidence, gaps;
• an ordered path to A+;
• a changelog.

Re-runs update the tracker; they never start it over. For each gap you fix, add a test that pins the prompt schema to the validator, so the drift cannot come back ({{skill:test-necessity}} can show that the test fails when the two diverge).