cross-model-judge-eval
Skill1 starsChanged 32 days agoMITView on GitHub
Evaluate, score, and benchmark agent outputs, skills, and code using an ensemble panel of heterogeneous LLMs from different model families. Use when assessing agent reliability across frontier and open-source models, mitigating single-model evaluation bias, or running jury/consensus-based quality gates.
In this repository13 other files it gives its agents.
- AGENTS.mdAGENTS.md
- CLAUDE.mdCLAUDE.md
- adversarial-agent-hardeningSkill
- article-interviewerSkill
- article-outlinerSkill
- build-walkthroughSkill
- narrated-explainerSkill
- narration-handoffSkill
- page-to-videoSkill
- prompt-engineerSkill
- render-page-to-videoSkill
- tts-narration-buildSkill
- {{name}}Skill
The file
We split this skill into the rules it gives an agent
1 rule. About 0 are common, 1 is distinctive, and 0 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈1
Distinctive · estimate
This author's own choices. Worth a look.
0
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 64
Automated Gating: If consensus score is below threshold, feed Phase 4 directives directly back into the generation harness for another …
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Cross-Model Judge Evaluation
A structured evaluation protocol that leverages an ensemble of diverse LLMs from competing model families to score, audit, and benchmark target agent artifacts, skills, or tool calling traces.
When to Use
- Evaluating skills or code generated by one model family using distinct frontier judges.
- Benchmarking open-source local models against frontier model judging panels.
- Eliminating self-preference bias, circular blind spots, and family-specific evaluation quirks.
- Implementing consensus or jury-based gates prior to deploying agent skills into production.
Judge Ensemble Archetypes
To ensure true orthogonal critique, select judges across separate architectural and training lineages:
- Anthropic Family (Claude): Strong on nuanced instruction following, safety boundaries, edge-case probing, and pedantic logic auditing.
- Google Family (Gemini): Strong on long-context synthesis, multimodal reasoning, cross-reference checking, and structured schema validation.
- OpenAI Family (GPT and o-series): Strong on planning heuristics, concise code generation, and standard benchmark alignment.
- Open-Source Family (e.g., Llama, DeepSeek, Qwen): Strong for local cost-effective scoring, latency baselines, and identifying deployment portability issues.
Evaluation Pipeline
The evaluation follows four sequential phases:
Phase 1: Artifact Framing & Blind Rubric Generation
- Extract the target artifact (system prompt, generated skill, tool schema, or execution trajectory).
- Strip explicit provenance markers where possible to prevent family-name bias.
- Establish a standardized scoring rubric with 4 orthogonal dimensions:
- Functional Accuracy & Executability (1-5): Does it accomplish the stated objective without fatal logic or syntax flaws?
- Constraint Adherence (1-5): Does it respect guardrails, schema types, and negative constraints?
- Edge-Case Resilience (1-5): How does it handle unhandled inputs, malformed parameters, or boundary conditions?
- Context & Token Efficiency (1-5): Is the instruction set concise and free of unnecessary cognitive bloat?
Phase 2: Independent Parallel Critique
Dispatch the framed artifact and rubric to each designated cross-family judge in isolated prompts. Each judge must output:
- Numerical Scores: Quantitative ratings across the 4 dimensions.
- Identified Failure Modes: Concrete line-by-line defects, unhandled states, or hallucinations.
- Actionable Remediation Points: Specific edits recommended by that judge.
Phase 3: Consensus & Divergence Telemetry
Aggregate the independent critique outputs into an ensemble matrix:
- Calculate Concordance: Identify areas where all judges agree on defects or strengths.
- Flag Divergent Findings: Highlight points where judges disagree significantly (delta >= 2 points), isolating whether disagreement stems from differing safety guardrails, style conventions, or strictness.
- Consensus Score: Compute weighted average across the judging panel.
Phase 4: Synthesis & Patch Directives
- Compile the unified feedback into a prioritized remediation list sorted by consensus agreement.
- Formulate explicit patch directives for the target agent or skill generator.
- Provide pass/fail verdict based on predefined threshold criteria (e.g., minimum score >= 4.0 across all dimensions with zero unanimous critical flags).
Loop Mechanics and Gating
- Automated Gating: If consensus score is below threshold, feed Phase 4 directives directly back into the generation harness for another iteration.
- Drift Detection: Monitor if the artifact over-indexes toward the idiosyncratic style of one judge while degrading performance on others.
Output Deliverables
At the conclusion of the evaluation run, produce:
- Jury Matrix: Tabular comparison of scores from each model family across all rubric dimensions.
- Consensus Defect Register: Verified flaws confirmed by multiple independent judges.
- Divergence Notes: Meaningful disputes between model families regarding requirements or safety bounds.
- Final Verdict: Clear Pass / Conditional / Fail decision accompanied by concrete patch directives.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

