agentleFS
tjmustardtjmustard/eta-skills/skills/cross-model-judge-eval/SKILL.md

cross-model-judge-eval

Skill1 starsChanged 32 days agoMITView on GitHub

Evaluate, score, and benchmark agent outputs, skills, and code using an ensemble panel of heterogeneous LLMs from different model families. Use when assessing agent reliability across frontier and open-source models, mitigating single-model evaluation bias, or running jury/consensus-based quality gates.

The file

We split this skill into the rules it gives an agent

1 rule. About 0 are common, 1 is distinctive, and 0 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈1

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 0

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Automated Gating: If consensus score is below threshold, feed Phase 4 directives directly back into the generation harness for another …

    line 64

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Cross-Model Judge Evaluation

A structured evaluation protocol that leverages an ensemble of diverse LLMs from competing model families to score, audit, and benchmark target agent artifacts, skills, or tool calling traces.

When to Use

  • Evaluating skills or code generated by one model family using distinct frontier judges.
  • Benchmarking open-source local models against frontier model judging panels.
  • Eliminating self-preference bias, circular blind spots, and family-specific evaluation quirks.
  • Implementing consensus or jury-based gates prior to deploying agent skills into production.

Judge Ensemble Archetypes

To ensure true orthogonal critique, select judges across separate architectural and training lineages:

  • Anthropic Family (Claude): Strong on nuanced instruction following, safety boundaries, edge-case probing, and pedantic logic auditing.
  • Google Family (Gemini): Strong on long-context synthesis, multimodal reasoning, cross-reference checking, and structured schema validation.
  • OpenAI Family (GPT and o-series): Strong on planning heuristics, concise code generation, and standard benchmark alignment.
  • Open-Source Family (e.g., Llama, DeepSeek, Qwen): Strong for local cost-effective scoring, latency baselines, and identifying deployment portability issues.

Evaluation Pipeline

The evaluation follows four sequential phases:

Phase 1: Artifact Framing & Blind Rubric Generation

  1. Extract the target artifact (system prompt, generated skill, tool schema, or execution trajectory).
  2. Strip explicit provenance markers where possible to prevent family-name bias.
  3. Establish a standardized scoring rubric with 4 orthogonal dimensions:
    • Functional Accuracy & Executability (1-5): Does it accomplish the stated objective without fatal logic or syntax flaws?
    • Constraint Adherence (1-5): Does it respect guardrails, schema types, and negative constraints?
    • Edge-Case Resilience (1-5): How does it handle unhandled inputs, malformed parameters, or boundary conditions?
    • Context & Token Efficiency (1-5): Is the instruction set concise and free of unnecessary cognitive bloat?

Phase 2: Independent Parallel Critique

Dispatch the framed artifact and rubric to each designated cross-family judge in isolated prompts. Each judge must output:

  • Numerical Scores: Quantitative ratings across the 4 dimensions.
  • Identified Failure Modes: Concrete line-by-line defects, unhandled states, or hallucinations.
  • Actionable Remediation Points: Specific edits recommended by that judge.

Phase 3: Consensus & Divergence Telemetry

Aggregate the independent critique outputs into an ensemble matrix:

  1. Calculate Concordance: Identify areas where all judges agree on defects or strengths.
  2. Flag Divergent Findings: Highlight points where judges disagree significantly (delta >= 2 points), isolating whether disagreement stems from differing safety guardrails, style conventions, or strictness.
  3. Consensus Score: Compute weighted average across the judging panel.

Phase 4: Synthesis & Patch Directives

  1. Compile the unified feedback into a prioritized remediation list sorted by consensus agreement.
  2. Formulate explicit patch directives for the target agent or skill generator.
  3. Provide pass/fail verdict based on predefined threshold criteria (e.g., minimum score >= 4.0 across all dimensions with zero unanimous critical flags).

Loop Mechanics and Gating

  • Automated Gating: If consensus score is below threshold, feed Phase 4 directives directly back into the generation harness for another iteration.
  • Drift Detection: Monitor if the artifact over-indexes toward the idiosyncratic style of one judge while degrading performance on others.

Output Deliverables

At the conclusion of the evaluation run, produce:

  • Jury Matrix: Tabular comparison of scores from each model family across all rubric dimensions.
  • Consensus Defect Register: Verified flaws confirmed by multiple independent judges.
  • Divergence Notes: Meaningful disputes between model families regarding requirements or safety bounds.
  • Final Verdict: Clear Pass / Conditional / Fail decision accompanied by concrete patch directives.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.