sc-evaluate
Skill19 starsChanged 8 months agoMITView on GitHub
LLM pipeline evaluation with oracle judge scoring. Runs prompts against gold standard datasets, evaluates output quality via LLM-as-judge, and generates scored reports with improvement recommendations.
1 other project uses a version of this.
Adopted by
Most recently taken up by
- kreighmitchell/SuperClaudeword for word, first seen Nov 1, 2026
Other versions1 project uses a version of this.
Word for word · 1
- kreighmitchell/SuperClaude★ 4100% same
Edited · 0
None found.
Wrote their own version · 0
None found.
How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.
In this repository37 other files it gives its agents.
- AGENTS.mdAGENTS.md
- agent-data-engineerSkill
- agent-fullstack-developerSkill
- agent-kubernetes-specialistSkill
- agent-ml-engineerSkill
- agent-performance-engineerSkill
- agent-react-specialistSkill
- agent-security-engineerSkill
- agent-technical-writerSkill
- ask-multiSkill
- askSkill
- learned-skills-indexSkill
- sc-analyzeSkill
- sc-brainstormSkill
- sc-buildSkill
- sc-cicd-setupSkill
- sc-code-reviewSkill
- sc-designSkill
- sc-documentSkill
- sc-e2eSkill
- sc-estimateSkill
- sc-explainSkill
- sc-gitSkill
- sc-implementSkill
- sc-improveSkill
- sc-log-fixSkill
- sc-mcpSkill
- sc-pr-checkSkill
- sc-pr-fixSkill
- sc-principlesSkill
- sc-pushSkill
- sc-readmeSkill
- sc-researchSkill
- sc-tddSkill
- sc-testSkill
- sc-workflowSkill
- sc-worktreeSkill
The file
We split this skill into the rules it gives an agent
9 rules. About 0 are common, 8 are distinctive, and 1 is very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈8
Distinctive · estimate
This author's own choices. Worth a look.
1
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 8
Run LLM pipeline evaluation against gold standard datasets using oracle LLM-as-judge scoring.
- Distinctive: line 36
Execute: - Run pipeline on gold standard inputs
- Distinctive: line 68
If no standard structure found, ask the user to specify paths.
- Distinctive: line 86
Run the pipeline on gold standard inputs:
- Distinctive: line 122
Read and analyze evaluation report:
- Very specific: line 215
Always pass --verbose for progress visibility
- Distinctive: line 219
Never modify gold standard files
- Distinctive: line 223
Bash: - Run evaluation scripts
- Distinctive: line 226
Grep: - Search for patterns in outputs
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
LLM Evaluation Skill
Run LLM pipeline evaluation against gold standard datasets using oracle LLM-as-judge scoring. Measures output quality across weighted dimensions, identifies weak steps, and suggests prompt improvements.
Quick Start
# Full evaluation (all test cases, all steps)
/sc:evaluate
# Quick spot check
/sc:evaluate --cases=case_1,case_2 --steps=1,2,3
# Re-evaluate existing results without re-running pipeline
/sc:evaluate --skip-pipeline
# Generate outputs only (no evaluation)
/sc:evaluate --skip-eval
# Specify judge model
/sc:evaluate --judge-model=gpt-4o
# Dry run to preview plan
/sc:evaluate --dry-runBehavioral Flow
- Discover - Find evaluation script, gold standards, and prompt files
- Configure - Parse scope (cases, steps, model overrides)
- Execute - Run pipeline on gold standard inputs
- Evaluate - Score outputs against gold standards via LLM-as-judge
- Analyze - Identify weak steps, dimension breakdowns, patterns
- Recommend - Suggest specific prompt improvements for low-scoring steps
- Report - Generate JSON + Markdown evaluation reports
Flags
| Flag | Type | Default | Description |
--cases | string | all | Comma-separated test case IDs to evaluate |
--steps | string | all | Comma-separated step numbers to evaluate |
--model | string | env default | Override pipeline model |
--judge-model | string | env default | Override judge/oracle model |
--skip-pipeline | bool | false | Skip pipeline execution, evaluate existing results |
--skip-eval | bool | false | Run pipeline only, skip evaluation |
--dry-run | bool | false | Preview execution plan without API calls |
--output | string | eval_runs/YYYYMMDD_HHMMSS/ | Output directory |
--concurrency | int | 5 | Parallel judge calls |
--threshold | int | 70 | Score threshold for "needs improvement" |
Phase 1: Discover Project Structure
Locate evaluation components:
| Component | Common Locations | Purpose |
| Evaluation script | scripts/run_eval.py, eval/run.py | Orchestrates pipeline + scoring |
| Gold standards | gold_standards/, test_data/, fixtures/ | Expected outputs |
| Prompts | prompts/, templates/ | Pipeline prompt templates |
| Rubrics | eval/rubrics.py, config/rubrics.yaml | Scoring dimensions and weights |
If no standard structure found, ask the user to specify paths.
Phase 2: Configure Scope
Parse arguments to determine:
- Which test cases to run (default: all discovered)
- Which pipeline steps to evaluate (default: all)
- Model overrides for pipeline and judge
- Output directory (default: timestamped)
Create output directory:
OUTPUT_DIR="${output:-eval_runs/$(date +%Y%m%d_%H%M%S)}"
mkdir -p "$OUTPUT_DIR"Phase 3: Execute Pipeline
Run the pipeline on gold standard inputs:
python <eval_script> \
--output "$OUTPUT_DIR" \
--verbose \
[--cases CASES] \
[--steps STEPS] \
[--model MODEL] \
[--skip-pipeline] \
[--skip-eval]API call estimation:
- Pipeline: steps x cases API calls
- Evaluation: scored_dimensions x cases judge calls
For quick validation, suggest running on 1-2 cases with 2-3 steps first.
Phase 4: Evaluate with LLM-as-Judge
For each step output, compare against gold standard using oracle LLM-as-judge:
Evaluation dimensions (customizable per project):
| Dimension | What It Measures |
| Content Agreement | Do outputs cover the same key points? |
| Structure Match | Is the organization/format similar? |
| Detail Accuracy | Are specific claims and data correct? |
| Completeness | Are all expected elements present? |
Each dimension has a weight (0.0-1.0) summing to 1.0 per step.
Phase 5: Analyze Results
Read and analyze evaluation report:
- Overall similarity score across all cases and steps
- Per-step scores — highlight any below threshold (default: 70/100)
- Per-case scores — identify consistently weak test cases
- Dimension breakdowns for weak steps
Score interpretation:
| Score Range | Assessment | Action |
| 85-100 | Excellent | No changes needed |
| 70-84 | Good | Minor tuning possible |
| 60-69 | Needs improvement | Prompt revision recommended |
| Below 60 | Poor | Prompt likely needs rewrite |
Phase 6: Recommend Improvements
For each step scoring below threshold:
- Read the current prompt template
- Read the gold standard output (expected)
- Read the pipeline output (actual)
- Compare and identify gaps:
- Missing instructions that gold standard captures
- Overly broad instructions causing divergent output
- Format/structure differences
- Specificity gaps
Present actionable suggestions:
### Step N: <step_name> (Score: XX/100)
**Weakest Dimension**: <dimension> (XX/100)
**Gap Analysis**:
- Gold standard includes <X> but prompt doesn't instruct it
- Output format diverges: gold uses <format>, output uses <other>
**Suggested Prompt Changes**:
1. Add instruction: "<specific instruction>"
2. Clarify format: "<format guidance>"
3. Add example: "<example output snippet>"Output Structure
eval_runs/YYYYMMDD_HHMMSS/
results/ # Pipeline outputs
case_1/
step_01_<name>.md
step_02_<name>.md
...
case_2/
...
evaluation/ # Judge scores
evaluation_report.json
evaluation_report.md
per_step_scores.csv
per_case_scores.csvMCP Integration
PAL MCP (Optional)
| Tool | When | Purpose |
mcp__pal__thinkdeep | Low-scoring steps | Deep analysis of why outputs diverge |
mcp__pal__consensus | Prompt revision | Multi-model validation of proposed changes |
mcp__pal__codereview | Eval script | Review evaluation pipeline code |
Rube MCP (Optional)
| Tool | When | Purpose |
mcp__rube__RUBE_REMOTE_WORKBENCH | Large eval runs | Process results in Python sandbox |
mcp__rube__RUBE_MULTI_EXECUTE_TOOL | Notifications | Report results to Slack/email |
Error Handling
| Scenario | Action |
| No eval script found | Ask user for script path |
| No gold standards found | Ask user for gold standard directory |
| API rate limit | Reduce concurrency, add delays |
| Pipeline step fails | Log error, continue with remaining steps |
| Judge returns invalid score | Retry once, then flag for manual review |
| Output directory exists | Append timestamp suffix |
Guardrails
- Always pass
--verbosefor progress visibility - Warn about API call counts before full runs
- Suggest quick validation on subset before full evaluation
- Preserve all intermediate outputs for debugging
- Never modify gold standard files
Tool Coordination
- Bash - Run evaluation scripts
- Read - Inspect prompts, gold standards, outputs, reports
- Write - Generate reports
- Grep - Search for patterns in outputs
- PAL MCP - Deep analysis of score gaps
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

