agentleFS
coval-aicoval-ai/coval-external-skills/skills/evaluation/coval-calibrate-metric/SKILL.md

coval-calibrate-metric

Skill3 starsChanged 18 days agoMITView on GitHub

Validate one Coval judge against independent human labels, with grouped development/test separation, class-specific errors and uncertainty. Use for trust claims or calibration; without real labels, prepare review rather than inventing ground truth.

The file

We split this skill into the rules it gives an agent

7 rules. About 0 are common, 6 are distinctive, and 1 is very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈6

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 1

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Check success and paginate using the public reviews API when the CLI cannot prove completeness.

    line 25
  2. Very specific:

    If human labels are absent, produce a review-ready sample and criteria using coval-discover-failures.

    line 34
  3. Distinctive:

    Do not create assignments or notify reviewers without the required authorization.

    line 37
  4. Distinctive:

    Keep each group in only one split, including few-shot examples.

    line 43
  5. Distinctive:

    Keep original labels and adjudication provenance.

    line 54
  6. Distinctive:

    If the runtime model cannot be pinned or identified, report that limitation.

    line 62
  7. Distinctive:

    Do not automatically turn a failure-enriched validation sample into a production pass rate.

    line 107

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Calibrate a Coval metric

Measure whether a specific metric version matches a domain expert on its intended population. Successful metric execution and agreement on prompt-tuning examples are not independent validation.

Establish the evidence contract

Confirm organization/workspace, one metric ID, rubric, output type/polarity and decision. Read the definition and output version IDs. Inspect an existing review project and completed annotations, or a human-provided labeled export:

coval --agent metrics get <metric-id>
coval --agent review-projects get <project-id>
coval --agent review-annotations list --filter 'project_id="<project-id>" AND metric_id="<metric-id>" AND completion_status="COMPLETED"' --page-size 100

Check success and paginate using the public reviews API when the CLI cannot prove completeness. Preserve annotation IDs, output/conversation IDs, reviewer, completion status, label values and rubric provenance. Zero is a valid label; null is missing. Record whether each source is a simulated or uploaded conversation: simulation_output_id in an annotation can refer to either. Retrieve its original metrics from the matching simulated-conversations or uploaded-conversations CLI collection/public API, not from a guessed collection. Don't silently treat pending, deferred or AI-generated labels as completed human ground truth.

If human labels are absent, produce a review-ready sample and criteria using coval-discover-failures. Return not yet calibrated. Agent-generated labels, synthetic expected answers and a reviewer-shaped email are not human validation. Do not create assignments or notify reviewers without the required authorization.

Separate development from evaluation

Assign related calls to a stable group_id: same original conversation, near-duplicate scenario, caller/customer thread, or resimulations of a case. Keep each group in only one split, including few-shot examples. Choose a split appropriate to the available sample; do not force tiny datasets into meaningful- sounding percentages. Record the IDs before tuning.

  • train: examples allowed in the judge prompt.
  • dev: inspect disagreements and refine the prompt.
  • test: untouched until the candidate is frozen; measure once.

Have a domain expert resolve conflicting human labels against the rubric; do not overwrite them to match the judge. Multiple reviewers on one conversation are not independent examples. Low human agreement suggests an unclear criterion. Keep original labels and adjudication provenance.

For development, propose a candidate metric separately from production defaults. Fix one observed error pattern at a time. Keep the same frozen metric version across the evaluated split. Re-scoring existing outputs through metrics test is cheaper than resimulating; it still needs a bounded metric-evaluation budget. Inspect per-item queue failures and poll each returned metric_output_ulid, not a prior output. Capture version IDs and available runtime model metadata. If the runtime model cannot be pinned or identified, report that limitation.

Measure the right errors

For binary pass=1/fail=0:

  • Pass recall (TPR) = correctly passed / human passes.
  • Failure detection (TNR) = correctly failed / human failures.
  • False-pass rate = judge passes / human failures. This often matters most for a release gate.

Report the confusion counts, each denominator and uncertainty. Overall agreement can be useful alongside these, but cannot hide an always-pass judge. If a class has no examples, its rate is unknown, not zero or perfect. Pick thresholds with the customer based on consequences; there is no universal 90% certification.

The bundled offline calculator accepts a deliberately normalized, auditable export described in references/calibration-input.md:

python3 scripts/calibration.py --input calibration.json --split dev
python3 scripts/calibration.py --input frozen-test.json --split test

It checks grouping, duplicate conversations, label provenance and metric version, excludes unusable results, and calculates Wilson 95% intervals. It cannot verify that a file's claimed human labels were actually supplied by a human; inspect the source records. Do not relabel AI findings to make the validator accept them.

For categorical metrics, retain categories and report a confusion matrix and per-category error counts. For numerical metrics, predefine tolerance/decision threshold and report absolute error plus boundary disagreements. The bundled calculator is binary-only; don't coerce another metric type into it.

Decide and stop

Inspect development disagreements, never the held-out test during tuning. After examining final test outcomes, that set is no longer untouched for further iterations. Use new held-out groups for a later trust claim. Include failed, skipped and missing score counts so selected successes cannot inflate quality.

Return the metric/version, rubric, label source, split/group manifest, counts, errors, uncertainty and intended population. Distinguish development evidence, held-out measurement, and insufficient evidence. A small test can prove the workflow works without proving the judge reliable.

Do not automatically turn a failure-enriched validation sample into a production pass rate. Prevalence correction requires representative production sampling, stable class-conditional error rates and uncertainty from both samples; those assumptions often do not hold after agent or traffic changes. Report the directly measured errors first. Revalidate after rubric, model, input or population changes.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.