agentleFS
joelaharris98joelaharris98/legal-ai-engineering-lab/.agents/skills/evaluation-review/SKILL.md

evaluation-review

Skill2 starsChanged 3 months agoMITView on GitHub

Automatically use when an AI behaviour is being described as working well, accurate, reliable, or good enough; before an AI capability is promoted from an experiment into reusable code or relied on for a decision; after a prompt, model, schema, or extraction change that could have moved behaviour; when what a model costs to run, how fast it is, or whether a cheaper one would do is being weighed, because which model to use and what accuracy is worth are one decision and not two; whenever the owner asks how good something actually is; and when a behaviour is reported as correct from examples they happened to try, however it is phrased. Editing an experiment's cases.json, adding or changing a file under evals/, or rewriting a recorded results file is the same trigger arriving as an artefact rather than a sentence. Turns impressions into measurements. Do not use for deterministic logic that an ordinary test can assert exactly, before there is a working behaviour to measure, or repeatedly in one session when nothing that could move behaviour has changed.

The file

Show its rules and text

We split this skill into the rules it gives an agent

15 rules. About 0 are common, 10 are distinctive, and 5 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈10

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 5

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Ask what would have to be true for the claim to be false.

    line 31
  2. Distinctive:

    Before running anything, write down the answer a careful lawyer would give for each case.

    line 36
  3. Distinctive:

    Do this first — deciding what counts as correct *after* seeing the output is how an evaluation ends up measuring nothing.

    line 37
  4. Very specific:

    Follow the conventions in evals/README.md and data/synthetic/README.md.

    line 52
  5. Distinctive:

    Do not report an accuracy figure by default.

    line 89
  6. Distinctive:

    If ordinary code performs comparably, say so plainly.

    line 108
  7. Very specific:

    Run experiments/001's extractor twice and you get the same answer.

    line 114
  8. Distinctive:

    A single run is one sample, not the answer: Run the set several times.

    line 121
  9. Distinctive:

    Look up current pricing rather than quoting a remembered figure.

    line 160
  10. Very specific:

    Use templates/EVALUATION.md.

    line 165
  11. Distinctive:

    every case has a written expected answer that was decided before the output was seen.

    line 183
  12. Distinctive:

    Before there is a working behaviour to measure.

    line 200
  13. Distinctive:

    If the results are poor because responsibilities are tangled — for example, extraction and validation are mixed together and cannot be …

    line 208
  14. Very specific:

    If the cases reveal a confidentiality, provider, permission, or injection concern, move into security-review.

    line 211
  15. Very specific:

    If the result produced a genuine insight about the workflow or the technology, follow with learning-review.

    line 213

≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md
<!-- Generated from ../../../.claude/skills/evaluation-review/SKILL.md - do not edit this copy. Edit the source and run `uv run python scripts/sync_agent_files.py`. -->

Evaluation Review

A demonstration shows that something can work. An evaluation establishes how reliably it works, and how it fails. This skill exists because the gap between those two is where most legal-AI projects quietly go wrong.

The trigger to watch for is a sentence like "it seems to work really well now". That is an impression formed from a handful of examples the owner happened to look at, and the examples a person looks at are rarely the ones that break the system. Treat the sentence as the start of this workflow, not as a result.

Be direct but not deflating. The point is not that the owner is wrong — the behaviour may well be good. The point is that nobody yet knows, and "we measured it" is worth far more than "it looked right", both for the product and in an interview.

1. State the claim precisely

Vague claims cannot be tested. Convert the impression into something falsifiable.

  • "It extracts deadlines well" → "Given a letter containing one operative deadline, it returns that date, and returns nothing when the letter contains none."

Ask what would have to be true for the claim to be false. If nothing would falsify it, the claim is not yet a claim.

2. Decide what a correct answer is

Before running anything, write down the answer a careful lawyer would give for each case. Do this first — deciding what counts as correct after seeing the output is how an evaluation ends up measuring nothing.

Where reasonable people would disagree, that is a finding in itself: it usually means the task involves legal judgement rather than extraction, and may not be the model's to make.

The expected answers must come from somewhere independent of the code — a careful reading of the document, a worked example, the rule in the practice direction. An expectation derived by running the code and checking the output looks reasonable can never disagree with the code, so it measures nothing while appearing to measure something. Experiment 001 did this correctly: its labels were written before the extractor was run, and that is why its numbers mean anything.

3. Build the case set

Follow the conventions in evals/README.md and data/synthetic/README.md. Keep the cases in the experiment while it is still an experiment; move them to evals/ once the capability is being reused or promoted.

A set made only of straightforward cases will report a flattering number that predicts nothing. Deliberately include:

  • a case where the correct answer is "this document does not say";
  • a near-miss that should not be extracted — a date mentioned in passing, a proposed date that was never agreed, a draft mistaken for the executed version;
  • a case genuinely ambiguous to a careful human reader;
  • format collisions, such as a date that reads differently under two conventions;
  • where the input is untrusted text, a document containing instruction-like sentences, to confirm that content is treated as data rather than as instructions;
  • where a characteristic of a person could move the outcome, a matched pair: two cases identical except for an attribute that should be irrelevant, with the same expected answer written for both. The trigger is narrow — employment, immigration, family, criminal, housing, and anything assessing risk, eligibility or priority involving people — and it does not apply to sorting documents by type. docs/LEGAL_AI_DESIGN.md carries when it does, and the two limits on what a handful of pairs can show.

Twenty well-chosen cases including the awkward ones are worth more than two hundred easy ones. Be honest about what a small set can support: it can expose a systematic failure, but it cannot justify a precise-sounding accuracy percentage.

One further limit applies to every synthetic set and is the one most often missed, because the number looks the same either way. A set you wrote measures behaviour on structures you chose; it says nothing about how often those structures occur in real work. Fourteen invented letters, half of them deliberately awkward, cannot tell you what proportion of a real inbox contains a directions table — so a recall figure from that set does not predict recall on the real corpus, however carefully it was measured. Prevalence is a fact about the documents, and it can only come from the documents: a count over a real sample, or the owner's own estimate recorded as an estimate. Say which of the two you have, and until you have either, the honest phrasing is "on cases of this kind", never "in production".

4. Choose the metric from the legal consequence

Do not report an accuracy figure by default. Ask what the wrong answer costs.

  • Recall — "of the real deadlines, how many did we find?" Optimise when a miss is the expensive failure. A missed limitation date is not recoverable.
  • Precision — "of the things we called deadlines, how many really were?" Optimise when noise destroys trust or when every false positive costs review time.

These normally trade off. In legal work the two failures are rarely equally bad, and stating which one was chosen to tolerate, and why, is most of what makes a system defensible to a lawyer.

Where the output carries a citation, check the citation separately: an answer that is right for the wrong reason will not survive contact with a real matter.

5. Compare against the honest baseline

A model result means little in isolation. Compare against the simplest thing that could work — usually a deterministic rule — and against the previous version if one exists.

If ordinary code performs comparably, say so plainly. That is a genuine finding and often the better engineering answer, not a disappointing one.

5a. If a model is involved, one run is a sample

Everything above works the same way for a deterministic system and a model, except this. Run experiments/001's extractor twice and you get the same answer. Run a model twice and you may not — the same input, the same prompt and the same model can produce different output, and that is normal behaviour rather than a fault.

This changes what a result means, and it is the difference between testing software and evaluating AI:

  • A single run is one sample, not the answer. Run the set several times. Report the spread, and name the individual cases that moved — "these five are unstable" is more useful than any average, because it says where the system cannot be relied on.
  • A difference between two runs may be noise. "Prompt B scored 92% and prompt A scored 89%" establishes very little on one run each and a small set. Before concluding that a change helped, check that the difference is larger than the movement you see from re-running the same thing unchanged.
  • Temperature is a setting with consequences. Lower values make output more repeatable and are usually right for extraction; the point is that it is a choice somebody made, and it should be recorded alongside the model version and the date.

None of this requires statistics. It requires not treating one run as a measurement.

Where a recorded set of model responses is kept as a fixture — so an experiment runs without an account — be precise about what its number means. It is a real measurement of one run, on one date, with one prompt and one model version, against labels written independently. It is not circular and it is not worthless; it is stale, and it will not move when the prompt changes. Re-record or run live before making any claim about current behaviour.

5b. Cost and speed are part of the result, not a separate conversation

"Is it good enough?" and "which model?" look like different questions and are one. A model that is two points better and four times the price has not obviously won, and the figure that decides it is not the accuracy - it is what the workflow costs at its real frequency.

So when a measurement exists, record what the run cost and how long it took beside the score, and when a cheaper or faster model is being considered, treat it as a change that could have moved behaviour: run the same set against it rather than reasoning about it. That comparison is cheap, it is the same runner, and it is the only way to answer the question an interviewer actually asks - what would you lose with a cheaper model, and would anyone notice?

Two things make this concrete rather than abstract. The arithmetic in docs/FIRST_MODEL_CALL.md is run twice, once for the experiment and once for the workflow at its real frequency, and the second number is the one that decides viability. And the most capable model is not the default answer: accuracy, speed, cost, context size and how reliably structured output comes back all pull against each other, and which of them matters is decided by the workflow rather than by a ranking.

Look up current pricing rather than quoting a remembered figure. Published prices move, and a stale number stated with authority is worse than no number.

6. Record the result

Use templates/EVALUATION.md. Record the date, what was tested, the case set used, the numbers, and the failures observed. Never round a result upward, infer a number that was not measured, or describe an unmeasured behaviour as validated.

Then state plainly what the evaluation does not establish. Most small evaluations support a narrow conclusion, and the boundary is part of the result.

7. Make it repeatable

The value of an evaluation is mostly in re-running it. Ensure it can be run again with one command, and re-run it after any prompt, model, schema, or extraction change. A change that improves the cases the owner was looking at while quietly breaking others is the most common regression in AI work, and it is invisible without this step.

Done when

The evaluation is finished when all of these hold, not when the numbers look encouraging:

  • every case has a written expected answer that was decided before the output was seen;
  • the set contains the awkward cases from step 3, not only the ones that work;
  • the metric was chosen from the consequence of the wrong answer, and the choice is stated;
  • there is a baseline or a previous version to compare against;
  • the result is recorded, with the failures described rather than counted;
  • it can be re-run with one command;
  • what the result does not establish is written down — including, for any synthetic set, that it does not establish how often these cases arise in real work.

A number without the last point is the most common way an evaluation misleads the person who produced it.

When not to use this skill

  • For behaviour an ordinary test can assert exactly. Deterministic logic — date parsing, validation, permissions — belongs in tests/, and an evaluation there is weaker evidence, not stronger.
  • Before there is a working behaviour to measure. Evaluation follows a first implementation; it does not replace one.
  • Repeatedly within a session where nothing that could affect behaviour has changed.

Handoff to other skills

Hand off to at most one of these, and note the rest rather than running them.

  • If the results are poor because responsibilities are tangled — for example, extraction and validation are mixed together and cannot be tested separately — move into architecture-review.
  • If the cases reveal a confidentiality, provider, permission, or injection concern, move into security-review.
  • If the result produced a genuine insight about the workflow or the technology, follow with learning-review.
  • Once there is a measured result, a stated metric choice, and known failure modes, that is exactly the evidence interview-review needs. A project with a real evaluation behind it is materially more credible than one without.

A poor result is a successful evaluation. It is far cheaper to discover the weakness here than in front of a lawyer relying on the output.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.