agentleFS
coyvalyss1coyvalyss1/model-matchmaker/skills/optimize-classifier/SKILL.md

optimize-classifier

Skill168 starsChanged 8 months agoMITView on GitHub

Analyze your Model Matchmaker override patterns and tune the local classifier to match your preferences. Use after collecting 50+ recommendations.

The file

We split this skill into the rules it gives an agent

12 rules. About 0 are common, 11 are distinctive, and 1 is very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈11

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 1

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Analyzes: when you overrode the advisor's recommendations

    line 13
  2. Very specific:

    Read the NDJSON log file at ~/.cursor/hooks/model-matchmaker.ndjson.

    line 26
  3. Distinctive:

    Check if there's enough data:

    line 28
  4. Distinctive:

    If not enough data, tell the user: "You need more usage data.

    line 31
  5. Distinctive:

    If user says "yes" or approves specific keywords:

    line 158
  6. Distinctive:

    Add the new keywords to the list with a comment explaining they were auto-added

    line 162
  7. Distinctive:

    Write the file back using the StrReplace tool

    line 163
  8. Distinctive:

    Only suggest keywords with 3+ override occurrences (confidence threshold)

    line 168
  9. Distinctive:

    Don't add generic words that could cause false positives ("the", "make", "update")

    line 169
  10. Distinctive:

    If the user's overrides are inconsistent (sometimes want Opus, sometimes want Sonnet for the same keyword), note that and ask for …

    line 171
  11. Distinctive:

    After your first 50-100 prompts (to establish baseline patterns)

    line 188
  12. Distinctive:

    After a major project shift (switching from backend to frontend work, for example)

    line 190

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Optimize My Classifier

This skill helps you personalize your Model Matchmaker classifier based on your actual usage patterns. After you've collected 50+ recommendations, this skill analyzes when you disagreed with the advisor and tunes your local classifier to match your preferences.

What This Skill Does

  1. Reads your local Model Matchmaker logs (~/.cursor/hooks/model-matchmaker.ndjson)
  2. Analyzes when you overrode the advisor's recommendations
  3. Finds patterns in the prompts where you disagreed (common words, task types)
  4. Suggests keyword additions to your model-advisor.sh file
  5. Updates your classifier (with your approval) so future recommendations match your preferences

Privacy: Everything happens locally. No data leaves your machine. You review and approve every change.

Instructions for the AI

You are helping the user optimize their Model Matchmaker classifier based on their personal override patterns. Follow these steps:

Step 1: Read and Validate Log File

Read the NDJSON log file at ~/.cursor/hooks/model-matchmaker.ndjson.

Check if there's enough data:

  • Need at least 50 recommendation events total
  • Need at least 5 OVERRIDE events to find patterns
  • If not enough data, tell the user: "You need more usage data. Come back after 50+ prompts with at least a few overrides."

Step 2: Analyze Override Patterns

Filter to action: "OVERRIDE" events and group by model direction:

Group A: User preferred Opus over recommended Haiku/Sonnet

  • These are prompts where the classifier said "use a cheaper model" but you said "no, I need Opus"
  • Extract common words from prompt_snippet fields in this group

Group B: User preferred Sonnet over recommended Opus

  • These are prompts where the classifier said "use Opus" but you said "no, Sonnet is fine"
  • Extract common words from prompt_snippet fields in this group

Group C: User preferred Sonnet over recommended Haiku

  • Extract common words from prompt_snippet fields in this group

Group D: User preferred Haiku over recommended Sonnet/Opus

  • Extract common words from prompt_snippet fields in this group

Step 3: Extract Keyword Candidates

For each group, find words that appear in 3+ override prompts (minimum frequency threshold).

Exclude common stop words:

  • the, a, an, is, are, was, were, be, been, being, have, has, had, do, does, did
  • in, on, at, to, for, of, with, from, by, about, as, into, through, during
  • this, that, these, those, I, you, we, they, it, he, she, me, my, your

Focus on action verbs and technical terms:

  • Examples: debug, investigate, refactor, optimize, analyze, build, create, fix, update, configure

Step 4: Generate Proposed Changes

For each keyword group, map to the correct section of model-advisor.sh:

For Group A keywords (user prefers Opus): → Add to opus_keywords list around line 46

For Group B/C keywords (user prefers Sonnet): → Add to sonnet_patterns list around line 63

For Group D keywords (user prefers Haiku): → Add to haiku_patterns list around line 53

Step 5: Calculate Impact

For each proposed keyword:

  • Count how many past overrides would become correct recommendations if this keyword were added
  • Show confidence: "3 overrides → 'debug' should trigger Opus"

Step 6: Present Recommendations

Output in this format:

# Classifier Optimization Report

## Your Usage Summary

- Total recommendations: N
- Overrides: M (X.X%)
- Ready for optimization: [Yes/No - need 5+ overrides]

## Patterns Found

### You prefer Opus for:

**Keyword: "debug"**
- Frequency: 5 overrides
- Current behavior: Recommends Sonnet
- Proposed: Add "debug" to opus_keywords
- Impact: 5 past overrides would become correct recommendations

**Keyword: "investigate"**
- Frequency: 3 overrides
- Current behavior: Recommends Sonnet
- Proposed: Add "investigate" to opus_keywords
- Impact: 3 past overrides would become correct recommendations

[Repeat for other keywords]

### You prefer Sonnet over Opus for:

[Same structure]

### You prefer Haiku for:

[Same structure]

## Proposed Changes to ~/.cursor/hooks/model-advisor.sh

**Add to opus_keywords (line 46):**
```python
opus_keywords = [
    "architect", "architecture", "evaluate", "tradeoff", "trade-off",
    "strategy", "strategic", "compare approaches", "why does", "deep dive",
    "redesign", "across the codebase", "investor", "multi-system",
    "complex refactor", "analyze", "analysis", "plan mode", "rethink",
    "high-stakes", "critical decision",
    # NEW - Added based on your override patterns:
    "debug", "investigate"  # 8 overrides support this
]

Add to sonnet_patterns (line 63): [Show specific regex additions if any]

Next Steps

Would you like me to apply these changes to your classifier?

  1. Yes - I'll update ~/.cursor/hooks/model-advisor.sh with these keywords
  2. Some of them - Tell me which keywords to add
  3. No - Just show me the analysis, don't change anything

If you approve, I'll:

  1. Read your current model-advisor.sh
  2. Add the new keywords to the appropriate lists
  3. Write the updated file back
  4. Confirm the changes

You can test immediately by restarting Cursor or starting a new composer session.


### Step 7: Apply Changes (If User Approves)

If user says "yes" or approves specific keywords:

1. Read `~/.cursor/hooks/model-advisor.sh`
2. Locate the appropriate keyword list (opus_keywords, sonnet_patterns, or haiku_patterns)
3. Add the new keywords to the list with a comment explaining they were auto-added
4. Write the file back using the StrReplace tool
5. Confirm: "Updated! Your classifier now recommends [model] for prompts containing [keywords]. Changes take effect in your next Cursor session."

### Important Notes

- Only suggest keywords with 3+ override occurrences (confidence threshold)
- Don't add generic words that could cause false positives ("the", "make", "update")
- Show the user exactly what will change before changing it
- If the user's overrides are inconsistent (sometimes want Opus, sometimes want Sonnet for the same keyword), note that and ask for clarification

---

## How to Use This Skill

As a user, invoke this skill by saying something like:

- "Optimize my Model Matchmaker classifier"
- "Tune my classifier based on my overrides"
- "Analyze my Model Matchmaker usage and improve it"
- "Update my classifier to match my preferences"

The AI will analyze your logs, find patterns, and propose keyword additions. You review and approve before any changes are made.

## When to Run This

- After your first 50-100 prompts (to establish baseline patterns)
- Monthly or quarterly as your workflow evolves
- After a major project shift (switching from backend to frontend work, for example)
- Whenever you notice you're overriding the advisor frequently

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.