LLM Judge Alignment Analysis

You are helping analyze how well an LLM-based judge aligns with human labels in an eval dataset. Work through the five phases below in order, confirming with the user at each ambiguous step before proceeding.

The methodology works regardless of eval platform (Braintrust, LangSmith, custom CSV/JSONL, or any other format). Phase 1 covers how to get the data you need from different sources.

Phase 1: Discovery

Goal: Figure out what score fields exist and which are human labels vs. LLM judge scores.

Step 1 — Get the eval results

Ask the user how their eval results are stored:

Option A — Local file (CSV or JSONL) Read the file directly. Look for columns or fields that contain numeric or categorical scores. Common patterns: human_score, llm_score, <judge_name>_score, label, rating.

Option B — Eval platform (Braintrust, LangSmith, etc.) Ask for the experiment/run ID or URL. Use the platform's API or MCP tools to fetch rows. You want: each row's input, the human label field, and the LLM judge score field.

Typical Braintrust query (adapt syntax to other platforms):

select: id, span_id, scores
object_type: experiment
object_ids: ["<experiment-id>"]
where: is_root = true
limit: 500

Option C — The user pastes results directly Accept a JSON array, CSV snippet, or any structured format. Parse it in-context.

Step 2 — Identify human vs. LLM score fields

From the data, extract all score fields. For each:

• What are the distinct values? (binary 0/1, categorical PASS/FAIL, continuous 0–1)
• How many rows have this field populated?
• Does the name suggest human or LLM origin? (e.g., human_*, ground_truth, label are likely human; criterion-named fields are likely LLM)

Present your findings and ask the user to confirm: "Which field is the human label and which is the LLM judge score? I'll skip rows where either is missing — is that OK?"

Technical notes (platform-specific)

• Braintrust: hyphenated field names need bracket notation: scores["metric-scope"] not scores.metric-scope. Root spans are where scores aggregate — filter is_root = true. The id field ≠ span_id; select both for later trace lookups.
• CSV/JSONL: parse scores as floats; watch for string "PASS"/"FAIL" that need mapping to 1.0/0.0.

Phase 2: Confusion Matrix + TPR/TNR

Goal: Compute the full confusion matrix and key classification metrics.

Setup

• Positive (1) = the human label value the user defines as "pass" (default: 1 or "PASS")
• Negative (0) = the other value (default: 0 or "FAIL")
• Threshold: for continuous LLM scores (0.0–1.0), default to ≥ 0.5 = predicted positive. Ask if the score looks continuous.

Compute the confusion matrix

Group rows into four cells:

• TP: human = positive, LLM = positive
• FN: human = positive, LLM = negative (judge misses a real pass)
• FP: human = negative, LLM = positive (judge passes something that should fail)
• TN: human = negative, LLM = negative

Report

Then:

• TPR (sensitivity/recall) = TP / (TP + FN) — how well the LLM catches true positives
• TNR (specificity) = TN / (TN + FP) — how well the LLM rejects true negatives
• Total rows used, rows skipped (missing scores)

Interpret the numbers — don't just show the math. E.g.: "TPR of 85% means the judge misses 15% of cases your human reviewers would pass."

Phase 3: Disagreement Investigation

Goal: Understand why the LLM got each disagreement wrong by reading the scorer's reasoning.

Step 1 — Pull disagreement rows

Filter to rows where human label ≠ LLM score (accounting for threshold if continuous). Show up to 20 disagreements — enough to find patterns without overwhelming.

For each row, gather:

• The input
• The AI output (if available)
• Human verdict vs. LLM verdict
• The LLM's reasoning (from the scorer's trace/metadata if available)

Step 2 — Retrieve scorer reasoning

If your eval platform stores scorer reasoning (a critique or rationale alongside the score):

• Braintrust: fetch the scorer span using root_span_id = '<span_id>' and look for output.metadata.rationale or similar
• LangSmith: check the run's feedback or evaluator output fields
• Custom: look for a critique, rationale, or reason field in the eval output

If no reasoning is stored, re-run the judge on the disagreement rows locally and capture the reasoning output.

Step 3 — Summarize patterns

Group errors into patterns — don't list them one by one. Common patterns:

• Too strict FNs: LLM fails responses for a pedantic reason humans overlook
• Too lenient FPs: LLM passes responses that violate the spirit of the criterion
• Ambiguous criteria: the rubric has a gap the LLM fills differently than humans would

Phase 4: Prompt Fix Suggestions

Goal: Suggest targeted, surgical edits to the scorer prompt — not a full rewrite.

Based on the disagreement patterns from Phase 3:

1. Quote the specific part of the existing scorer prompt that's causing the issue
2. Propose the minimal change — a single sentence addition, a clarification to an example, a new edge case in the rubric
3. Explain how the change addresses the observed failure mode without overcorrecting

Format for each suggestion:

Issue: [what's going wrong and in which cases]

Current prompt text: "[relevant excerpt]"

Suggested change: "[replacement or addition]"

Why this helps: [which disagreements it would fix, and why it won't break the passing cases]

Aim for 2–4 suggestions. Don't suggest rewriting the whole prompt.

Phase 5: Few-Shot Example Suggestions

Goal: Identify the best 2–3 disagreement rows to add as labeled examples to the scorer prompt.

Good few-shot examples have these properties:

• The case is genuinely ambiguous (not obviously right or wrong)
• The human verdict is clearly defensible and can be explained in 1–2 sentences
• The LLM's mistake was predictable (reveals a systematic blind spot, not a one-off error)
• Adding it would generalize to similar future inputs

Selection criteria

• One "too strict" FN: A case the LLM failed that the human passed — include human reasoning as the correct label
• One "too lenient" FP: A case the LLM passed that the human failed
• A second FAIL case covering a different failure mode than the first — shows the nuance you want

Output format

For each suggested example, draft the text to add to the scorer prompt:

### Example N — [PASS/FAIL]
Input: "[input]"
Output excerpt: "[relevant part of the AI output]"
{"critique": "[why this is PASS/FAIL — in the voice of the scorer]", "score": "[PASS/FAIL]"}

Explain why you chose each example and what failure mode it targets.

Reminders & Gotchas

• Threshold matters: A continuous score of 0.49 vs 0.51 may swing a cell. Always clarify the threshold before computing the matrix, and report it alongside the results.
• Null scores: Always filter out rows where either score is missing before computing metrics. Report how many rows were skipped.
• Human review timing: In some platforms, human labels may be updated after the experiment runs. If a row appears in disagreement queries but shows agreement when fetched individually, note this — the label was likely updated.
• Braintrust field names with hyphens: Use scores["field-name"], never scores.field-name.
• Small samples: With fewer than 30 human-labeled rows, TPR and TNR estimates are noisy. Report this caveat explicitly.