Evaluator Route Skill

Evaluator Route

Analyze evaluator findings and recommend which model tier to use for the next SDD phase. This is the mechanism that enables the portfolio approach: budget models for routine generation, standard models for review, premium models for critical decisions.

The recommendation is based on:

• Finding severity distribution (critical/high findings → escalate)
• Evidence quality (unsupported claims → need premium reasoning)
• Phase risk profile (implement is higher risk than specify)
• Cost optimization (budget is sufficient when risk is low)

User Input

$ARGUMENTS

Accept:

1. Phase — the next SDD phase to route for (e.g., phase=plan, phase=implement). Required.
2. Result files — evaluator result files to base the recommendation on. If not provided, discover the latest composed result for the current phase.
3. Budget constraint — optional maximum USD budget for the next phase. If provided, the recommendation must stay within budget.

Prerequisites

• Path safety: resolve .specify/extensions/evaluator/results/ — refuse symlinks.
• At least one evaluator result or composed result MUST exist. If none, default to budget tier with a note that no evaluation data is available.

Execution

1. Load Evaluation Results

Read the latest composed result or individual evaluator results for the current phase.

2. Assess Risk Profile

Score the risk of the next phase based on evaluator findings:

Risk score = weighted sum, normalized to 0.0–1.0.

3. Determine Recommended Tier

4. Apply Phase Risk Baseline

Each SDD phase has an inherent risk baseline that shifts the threshold:

5. Apply Budget Constraint (if provided)

If a budget constraint is specified, downgrade the recommendation if the estimated cost exceeds the budget:

1. Calculate estimated tokens for the next phase at the recommended tier
2. Calculate estimated cost at that tier
3. If cost > budget, try the next lower tier
4. If no tier fits the budget, recommend budget with a warning

6. Produce Model Routing Recommendation

Output a model_routing block conforming to the evaluator result schema:

{
  "model_routing": {
    "recommended_tier": "standard",
    "reason": "2 high-severity findings and 1 evidence gap — standard quality recommended for plan phase",
    "escalation_triggers": [
      {
        "condition": "Any new critical finding",
        "escalate_to": "premium"
      },
      {
        "condition": "More than 5 unsupported claims in next evaluation",
        "escalate_to": "premium"
      }
    ],
    "estimated_tokens": 12000,
    "estimated_cost_usd": 0.22,
    "tier_breakdown": {
      "budget": {
        "estimated_tokens": 18000,
        "estimated_cost_usd": 0.01
      },
      "standard": {
        "estimated_tokens": 12000,
        "estimated_cost_usd": 0.22
      },
      "premium": {
        "estimated_tokens": 10000,
        "estimated_cost_usd": 0.90
      }
    }
  }
}

7. Report

Output:

• The recommended tier and reason
• The risk score breakdown
• Cost comparison across all tiers
• Escalation triggers (conditions that would upgrade the recommendation)
• The estimated savings vs always using premium

Model Tier Pricing Reference

Portfolio Approach Rules

1. Default to budget. Start every phase at the budget tier. Only escalate when evaluator findings justify it.
2. Escalate on evidence. Upgrade when findings show critical severity, insufficient_evidence uncertainty, or contradiction between evaluators.
3. Downgrade when clean. If the previous phase had zero high/critical findings, drop back to budget for the next phase.
4. Never use premium for generation. Premium models are for evaluation and decision-making, not for drafting specs or writing boilerplate code.
5. Deterministic evaluators are free. Schema validators, linters, and static analyzers cost near-zero tokens. Run them always, at every phase.
6. Model-backed evaluators use budget tier. Epistemic checks, semantic review, and coverage analysis run on budget models by default. Only escalate the evaluator itself when findings warrant it.

Guardrails

• Never recommend premium for a phase with zero high/critical findings.
• Never recommend budget when there are unresolved block outcomes.
• Always show the cost comparison — let the human see what they're saving.
• The routing recommendation is advisory — the human operator always has final say.

Workflow Integration (Explicit Wiring)

The evaluator's model routing recommendation can be wired into workflows using the existing expression mechanism. This keeps the dataflow visible and intentional — no engine changes needed.

steps:
  - id: evaluate
    type: command
    command: speckit.evaluator.route
    input:
      phase: plan

  - id: implement
    type: command
    command: speckit.implement
    model: "{{ steps.evaluate.output.model_routing.recommended_tier }}"

Each downstream step independently decides whether to use the recommendation. Workflow authors can validate, transform, or ignore it as needed.