Model Router Learn

Analyze usage tracking data and suggest improvements to model recommendations.

Trigger

User runs /model-router:learn

Instructions

1. Check Tracking Status

If tracking is disabled or no tracking file exists:

Usage tracking is not enabled.

To use the learning feature, enable tracking:
  /model-router:configure tracking

Tracking helps Model Router learn which models work best
for your specific use cases.

Exit if no tracking data.

2. Load and Analyze Tracking Data

Read ~/.model-router/tracking.jsonl:

# Count entries by model and task type
# Calculate success rates
# Look for patterns

Parse the data:

{"ts":"2026-02-03T10:15:00Z","task":"code_review","model":"gpt-5.2","outcome":"success","rating":null,"duration_ms":3200}
{"ts":"2026-02-03T10:20:00Z","task":"reasoning","model":"gemini-flash","outcome":"success","rating":"poor","duration_ms":8500}
{"ts":"2026-02-03T11:00:00Z","task":"critical_code","model":"claude-sonnet-4.5","outcome":"failure","rating":null,"duration_ms":12000}

3. Generate Insights

Analyze the data for patterns:

Success Rates:

# For each model + task combination
success_rate = successes / total_attempts
avg_latency = sum(durations) / count
quality_score = good_ratings / total_ratings  # if detailed tracking

Problem Detection:

• Models with low success rates for specific tasks
• Models with poor quality ratings
• Models that are consistently slow

Good Performers:

• Models with high success rates
• Models with consistently good ratings
• Models that exceed community benchmarks

4. Present Analysis

Usage Analysis (Last 30 days)
═══════════════════════════════════════════════════════════

Total tasks: 47
Tracking period: Feb 3 - Jan 4

Task breakdown:
  code_review:      18 (38%)
  reasoning:        12 (26%)
  simple_edits:     10 (21%)
  critical_code:     7 (15%)

Model usage:
  gpt-5.2:              15 (32%)
  deepseek-r1-70b:      12 (26%)
  gemini-flash:         10 (21%)
  claude-sonnet-4.5:     7 (15%)
  claude-opus-4.5:       3 (6%)

═══════════════════════════════════════════════════════════

INSIGHTS:

✓ Strong Performers:
  • gpt-5.2 for code_review: 15/15 success (100%)
    Current: runner_up → Recommendation: Promote to best

  • deepseek-r1-70b for reasoning: 12/12 success (100%)
    Current: best → Keep as-is (performing well)

⚠ Weak Performers:
  • gemini-flash for reasoning: 2/5 poor ratings (40% poor)
    Current: budget option → Recommendation: Demote or remove

  • claude-sonnet-4.5 for critical_code: 2/7 failures (29% fail rate)
    Current: best → Recommendation: Consider Opus instead

💡 Underutilized:
  • llama-3.3-70b: Available but rarely used
    Suggestion: Try for code_review (free alternative to GPT-5.2)

═══════════════════════════════════════════════════════════

5. Generate Specific Recommendations

Based on insights, suggest concrete changes:

Recommended Changes to Your Model Recommendations:

1. code_review: Promote gpt-5.2 to "best"
   Reason: 100% success rate over 15 uses
   Current: best=claude-sonnet-4.5, runner_up=gpt-5.2
   New:     best=gpt-5.2, runner_up=claude-sonnet-4.5

2. reasoning: Demote gemini-flash from budget option
   Reason: 40% poor quality ratings
   Current: best=deepseek-r1-70b, runner_up=claude-opus-4.5, budget=gemini-flash
   New:     best=deepseek-r1-70b, runner_up=claude-opus-4.5, budget=llama-3.3-70b

3. critical_code: Add quality warning
   Reason: Sonnet has 29% failure rate for critical tasks
   Suggestion: Always use Opus for critical_code, not Sonnet

═══════════════════════════════════════════════════════════

Would you like to:
○ Apply all recommended changes
○ Review and select which to apply
○ See detailed statistics
○ Keep current recommendations

6. Review and Select (if chosen)

Select changes to apply:

□ Promote gpt-5.2 to best for code_review
  Impact: GPT-5.2 will be recommended first for code reviews
  Rationale: Your data shows 100% success vs Sonnet's 86%

□ Replace gemini-flash with llama-3.3-70b for reasoning budget
  Impact: Local model becomes budget option (free)
  Rationale: Gemini Flash performed poorly in your tests

□ Update critical_code preferences
  Impact: Warns against using Sonnet for critical tasks
  Rationale: 29% failure rate is too high for production code

7. Apply Changes

If user approves, update ~/.model-router/models.yaml:

# This file has been customized based on your usage patterns
# Last updated: [timestamp]

task_types:
  code_review:
    best: gpt-5.2  # Promoted based on 100% success rate
    runner_up: claude-sonnet-4.5
    budget: gemini-flash

Confirm:

Recommendations updated!

Applied 3 changes:
  ✓ code_review: gpt-5.2 promoted to best
  ✓ reasoning: budget option changed to llama-3.3-70b
  ✓ critical_code: Added quality warning

Your model recommendations are now personalized.

To revert: /model-router:update-models --reset