agentleFS
shinprshinpr/rashomon/skills/recipe-eval-prompt/SKILL.md

recipe-eval-prompt

Skill20 starsChanged 6 months agoMITView on GitHub

Compares original and optimized prompts through repeated blind paired execution in git worktrees. Use when evaluating prompt improvement effects or learning prompt engineering through concrete examples.

The file

We split this skill into the rules it gives an agent

7 rules. About 0 are common, 5 are distinctive, and 2 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈5

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 2

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    No user confirmation required between phases unless explicitly requested.

    line 25
  2. Distinctive:

    Exception: If the request lacks any identifiable target (no file, function, or scope mentioned at all), ask ONE question to establish …

    line 32
  3. Distinctive:

    Extended timeout: If the user mentions needing more time, use up to 1800 seconds (default: 300 seconds)

    line 34
  4. Very specific:

    When a gate is blocked, stop before environment setup and present the gate's missing items as the required input for continuing.

    line 62
  5. Very specific:

    When result.status is original_sufficient, stop before environment setup.

    line 64
  6. Distinctive:

    No prompts, identity mapping, optimization findings, or change summary

    line 111
  7. Distinctive:

    Action: Ask user for feedback on comparison results, then delegate to knowledge-optimizer agent

    line 134

≈ “Common” is an estimate: at least 1 in 5 of the 350,495 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Explicit User Instruction: The user explicitly instructs and authorizes every subagent call named in this recipe. Execute each applicable call when its prerequisites are met.

Prompt Evaluation

Orchestrator Definition

Purpose: Provide accurate feedback on prompt optimization effects, enabling users to learn effective prompting through concrete comparison results.

Core Identity: "I route information between specialized agents. I pass user input to analyzers. I present agent outputs to users."

Pass-through Principle: Pass the user's exact request to prompt-analyzer, execute the original and optimized prompts under identical conditions, and present report-generator's output unchanged.

Execution Protocol:

  1. Delegate specialist work to the named sub-agents; keep workflow routing, worktree setup and cleanup, gate decisions, and user interaction in the orchestrator
  2. Follow the Execution Flow in order, applying its declared early-stop and error transitions

Phase Boundaries

No user confirmation required between phases unless explicitly requested. Each phase must complete all required outputs before proceeding.

Input

The user provides a natural language request. Pass it directly to prompt-analyzer.

Exception: If the request lacks any identifiable target (no file, function, or scope mentioned at all), ask ONE question to establish scope, then pass through.

Extended timeout: If the user mentions needing more time, use up to 1800 seconds (default: 300 seconds)

Execution Flow

Step 1. Run Required Skills

Run worktree-execution skill.

Step 2. Prompt Analysis and Optimization

Invoke: prompt-analyzer agent

Input:

  • User's exact request text

Output:

  • Complete gated JSON from the prompt-optimization skill
  • Analysis results in analysis.pattern_coverage
  • Individual issues in analysis.findings
  • Final prompt in result.final_prompt
  • Applied optimizations in optimization.finding_resolutions

Quality Gate:

  • Input contains user's request text only
  • Agent output parses as JSON
  • analysis_gate, optimization_gate, and balance_gate are pass
  • result.status is optimized or original_sufficient

When a gate is blocked, stop before environment setup and present the gate's missing items as the required input for continuing.

When result.status is original_sufficient, stop before environment setup. Return the analysis evidence and original prompt as the final prompt. Running identical prompts would measure only execution variance, not optimization value.

Step 3. Repeated Paired Execution

Resolve one base SHA, then target three valid trials with at most five total trial attempts. For every trial, create a fresh original/optimized worktree pair at that SHA using worktree-execution. Within a trial, invoke two prompt-executor agents simultaneously:

Subagent 1:
  agent: prompt-executor
  working_directory: {worktree_original_path}
  expected_base_sha: {pinned_base_sha}
  prompt: {original_request}

Subagent 2:
  agent: prompt-executor
  working_directory: {worktree_optimized_path}
  expected_base_sha: {pinned_base_sha}
  prompt: {prompt_analysis.result.final_prompt}

Each subagent executes the prompt as a development task within its isolated worktree. Clean the pair after collecting both results, then create fresh worktrees for the next trial.

CRITICAL: Send both prompt-executor Agent invocations in the same message to achieve true parallel execution.

Pair validity gate:

  • both execution statuses are success;
  • both results used fresh worktrees from the same repository state; and
  • both results report the pinned base SHA; and
  • neither result contains an environment-verification failure.

Keep failed and partial runs as diagnostics only. Continue until three valid pairs are collected or five total trial attempts have run. Compare with reduced confidence when two valid pairs remain. With fewer than two, set status to inconclusive, skip winner/recommendation claims, and report the diagnostics.

Step 4. Environment Cleanup

Execute worktree cleanup per worktree-execution skill "Cleanup" section. Step 4 completes only when the cleanup command exits with code 0; apply the worktree-execution error handling for any other exit code.

Step 5. Blind Report Generation

Invoke report-generator in two phases.

Phase 1 — blind assessment:

  • User task description
  • Anonymized valid pairs as Result A and Result B
  • No prompts, identity mapping, optimization findings, or change summary

The agent must complete and lock its output-quality judgment before Phase 2.

Phase 2 — identity reveal:

  • Identity mapping: A = original, B = optimized
  • Full prompt-analysis JSON, not only optimization.finding_resolutions
  • Execution metadata and diagnostics for every trial

Output:

  • Comparison report (markdown)
  • Improvement classification (structural / context addition / expressive / variance)

Quality Gate:

  • Output presented to user matches agent's output

The report joins analysis.findings and optimization.finding_resolutions by finding_id; pattern, severity, evidence, change, and source must remain traceable. Context delta is derived from resolutions whose source is a named project path or project knowledge entry.

Step 6. Retrospective

Trigger: Report generation completes

Action: Ask user for feedback on comparison results, then delegate to knowledge-optimizer agent

Improvement Classification

Apply the execution quality criteria from the prompt-optimization skill.

ClassificationDefinitionInterpretation
StructuralPrompt structure, clarity, specificity improvementsPrompt writing technique
Context AdditionProject-specific information added from codebase investigationInformation advantage
ExpressiveDifferent phrasing, equivalent substanceNeutral
VarianceWithin LLM probabilistic varianceOriginal prompt sufficient

Key Principle: Distinguish between prompt writing improvements (Structural) and information additions (Context Addition).

Final Output to User

Present report-generator's complete output to user. Optimized prompt must appear in full. This is the core learning value of the report.

The report includes (defined in report-generator):

  • Input Prompts (original and optimized full text)
  • Optimizations Applied
  • Execution Results
  • Comparison Analysis
  • Learning Points

Error Handling

ScenarioBehavior
One side of a trial failsExclude the unpaired trial from quality comparison and retain diagnostics
Fewer than two valid pairsReport inconclusive; no winner or prompt recommendation
All executions failReport full failure with diagnostics
TimeoutTerminate, capture partial results, cleanup
Worktree creation failsReport git error, suggest checking repository state

Prerequisites

  • Git repository with git worktree lock support
  • Claude Code subagent execution permissions
  • Sufficient disk space for worktree copies

Usage Examples

/recipe-eval-prompt
Add error handling to generateResponse in geminiService.ts. Handle 429, timeout, and invalid responses.
/recipe-eval-prompt
Generate code following this skill: .claude/skills/my-skill/SKILL.md

For complex tasks:

/recipe-eval-prompt
Refactor the message pipeline for readability. This may take a while.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.