Model Router Test

Test a model's performance and optionally update recommendations.

Trigger

User runs /model-router:test [model_name] or /model-router:test

Arguments

• model_name (optional): Model to test (e.g., gemini-3-pro, gpt-5.2, deepseek-r1-70b)

If not provided, ask which model to test.

Instructions

1. Select Model to Test

If model_name not provided:

Which model would you like to test?

Available models:
  Claude:
    • claude-opus-4.5
    • claude-sonnet-4.5
    • claude-haiku-4.5

  ChatGPT (requires Codex CLI):
    • gpt-5.2
    • gpt-5.2-codex

  Gemini (requires Gemini CLI):
    • gemini-3-pro
    • gemini-flash

  Ollama (local):
    • deepseek-r1-70b
    • llama-3.3-70b
    • qwen-2.5-32b

Enter model name:

2. Check Model Availability

Verify the model is accessible:

• Claude models: Always available
• GPT models: Check Codex CLI installed and authenticated
• Gemini models: Check Gemini CLI installed and authenticated
• Ollama models: Check Ollama running and model pulled

If not available:

[model_name] is not available.

Reason: [Codex CLI not installed / Ollama not running / etc.]

Would you like to set it up? Run /model-router:configure cli

3. Choose Test Prompts

How would you like to test [model_name]?

○ Standard test suite (5 built-in prompts)
○ Use prompts from your folders (recommended for personalized testing)
○ Custom prompt (enter your own)

If "Use prompts from your folders":

I'll select 10 prompts from your project files.

Which folders should I look in?
(Enter paths, or press enter for current project)

Examples:
  • . (current project)
  • ~/code/my-project
  • Multiple: ./src, ./tests

Folders to use:

After user provides folders, scan for suitable prompts:

# Look for:
# - README files with questions
# - Test files with assertions
# - Code files with TODO comments
# - Documentation with examples

Present found prompts:

Found 15 potential test prompts. Selected 10:

1. [prompt preview...] (from src/auth.ts)
2. [prompt preview...] (from README.md)
3. [prompt preview...] (from tests/api.test.ts)
...

Use these prompts?
○ Yes, run tests
○ No, let me pick different ones
○ Add more folders

4. Run Tests

Testing [model_name]...

Running [N] test prompts:

1. [prompt_type]: [brief description]
   ⏳ Running...
   ✓ Completed (2.3s)
   Quality: [Good/Acceptable/Poor] (based on response coherence)

2. [prompt_type]: [brief description]
   ⏳ Running...
   ✓ Completed (4.1s)
   Quality: [Good/Acceptable/Poor]

... (continue for all prompts)

For each test, use the appropriate method:

• Claude: Use Claude Code directly (Task tool)
• Codex: codex exec -m [model] "[prompt]"
• Gemini: gemini -y -m [model] -p "[prompt]"
• Ollama: HTTP POST to Ollama API

5. Show Results

Test Results for [model_name]
═══════════════════════════════════════════════════════════

Tests run: 10
Passed: 9
Failed: 1

Performance:
  Average latency: 3.2s
  Fastest: 1.8s (simple_edit prompt)
  Slowest: 6.4s (reasoning prompt)

Quality breakdown:
  Good: 7/10 (70%)
  Acceptable: 2/10 (20%)
  Poor: 1/10 (10%)

Task type performance:
  ✓ code_review: Strong (4/4 good)
  ✓ reasoning: Good (2/3 good)
  ⚠ simple_edits: Mixed (1/2 good)
  ✗ critical_code: Weak (0/1 good)

═══════════════════════════════════════════════════════════

6. Offer Recommendation Updates

Based on results, suggest updates:

Based on testing, [model_name] performed well for:
  • code_review (comparable to current best: claude-sonnet-4.5)
  • reasoning (better than current runner_up for your prompts)

Current recommendations:
  code_review:
    best: claude-sonnet-4.5
    runner_up: gpt-5.2

Suggested changes:
  1. Add [model_name] as runner_up for code_review
  2. Promote [model_name] to best for reasoning (replaces current)

Would you like to:
○ Apply all suggested changes
○ Review and choose which to apply
○ Keep recommendations unchanged
○ Run more tests first

If "Review and choose":

Select changes to apply:

□ Add [model_name] as runner_up for code_review
  Current: gpt-5.2 → New: [model_name]

□ Promote [model_name] to best for reasoning
  Current: deepseek-r1-70b → New: [model_name]
  (deepseek-r1-70b moves to runner_up)

7. Apply Updates

If user approves changes, update ~/.model-router/models.yaml:

task_types:
  code_review:
    best: claude-sonnet-4.5
    runner_up: [model_name]  # Updated
    budget: gemini-flash

Confirm:

Recommendations updated!

Changes applied:
  • code_review runner_up: gpt-5.2 → [model_name]

Your personalized recommendations are now saved.
Community defaults are preserved separately.

Run /model-router:recommend to see updated suggestions.

8. Save Test Results (if tracking enabled)

If tracking is enabled, append to ~/.model-router/tracking.jsonl:

{"type":"test","ts":"...","model":"[model]","prompts":10,"passed":9,"avg_latency_ms":3200,"quality":{"good":7,"acceptable":2,"poor":1}}