regression-search
Skill396 starsChanged 6 months agoMITView on GitHub
Search phone-call history for when a feature regressed (find-regression.py) and drill into a single call to see what went wrong (diagnose-call.py). Skips reading 100+ transcripts by hand.
In this repository73 other files it gives its agents.
- AGENTS.mdAGENTS.md
- CLAUDE.mdCLAUDE.md
- agent-activitySkill
- agent-registrySkill
- agent-room-opsSkill
- answer-formatSkill
- audio-transcribeSkill
- bee-actionsSkill
- bot2bot-postSkill
- call-diagnosticsSkill
- claude-codexSkill
- claude-geminiSkill
- claude-piSkill
- claude-routerSkill
- collaboration-intelligenceSkill
- connect-appsSkill
- context-dropSkill
- context-reconstructSkill
- db-browserSkill
- deal-finderSkill
- doc-ingestSkill
- electron-overlay-dimmingSkill
- email-findSkill
- engine-conflict-resolveSkill
- gemini-ttsSkill
- image-generationSkill
- import-claude-contextSkill
- local-workspace-serverSkill
- macos-toolsSkill
- macos-useSkill
- make-viral-videoSkill
- marketplaceSkill
- media-transcriptSkill
- meeting-prepSkill
- meeting-schedulerSkill
- mention-gateSkill
- model-switchSkill
- morning-briefingSkill
- ms365-connectSkill
- observeSkill
- obsidian-vaultSkill
- openai-ttsSkill
- open-sutando-refSkill
- overlay-appsSkill
- phone-conversationSkill
- proactive-loopSkill
- pr-shepherd-contractSkill
- quota-trackerSkill
- relaySkill
- releaseSkill
- report-feedbackSkill
- review-preflightSkill
- room-collabSkill
- schedule-cronsSkill
- screen-companionSkill
- screen-recordSkill
- self-diagnoseSkill
- self-upgradeSkill
- session-recapSkill
- skill-usage-reportSkill
- The first 60 shown.
The file
We split this skill into the rules it gives an agent
2 rules. About 0 are common, 1 is distinctive, and 1 is very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈1
Distinctive · estimate
This author's own choices. Worth a look.
1
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 19
Before shipping a fix — sanity check that the regression is reproducible.
- Very specific: line 49
No semantic understanding.
≈ “Common” is an estimate: at least 1 in 5 of the 350,495 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Regression Search
Two scripts for hunting down bad calls without reading every transcript:
find-regression.py— searchresults/calls/calls.jsonlfor calls touching a feature, classify each as working/broken, print a sorted timeline.diagnose-call.py— drill into a single call by SID, report refusals/errors/silences/repeated requests, optionally show metrics fromdata/call-metrics.jsonl.
Closes #188.
When to use
- "When did the X feature stop working?" — pass the feature keyword.
- "Has feature Y improved?" — see the broken/working trend over time.
- Before shipping a fix — sanity check that the regression is reproducible.
Usage
python3 skills/regression-search/scripts/find-regression.py "record"
python3 skills/regression-search/scripts/find-regression.py "summon" --since 2026-04-01
python3 skills/regression-search/scripts/find-regression.py "play" --jsonFlags:
--since YYYY-MM-DD— only show calls on/after this date--json— machine-readable output--show-snippet— print a one-line transcript snippet for each call
Heuristics
A call is broken for a query if any of:
- Sutando refuses ("I can't", "I'm not able", "I'm unable", "sorry I cannot")
- Sutando reports an error ("error", "failed", "didn't work", "something went wrong")
- The user repeats the same request 2+ times in a row (Sutando didn't respond usefully)
- Sutando says "(Silence)" after the user mentions the feature
Otherwise the call is working if Sutando's response includes the feature keyword and isn't flagged broken.
These are intentionally crude — the goal is "good enough to find the regression window without reading 163 transcripts." Tune as you find false positives.
Limitations
- Keyword matching only. "recording doesn't stop" vs "recording won't start" both match
record. The issue calls this out as future work. - No semantic understanding. A call where Sutando talks about recording but the user wanted something else still matches.
- Doesn't correlate with git commits — manual step for now.
diagnose-call.py
python3 skills/regression-search/scripts/diagnose-call.py de1f04733fc2
python3 skills/regression-search/scripts/diagnose-call.py CA701fc4129779... --metrics
python3 skills/regression-search/scripts/diagnose-call.py de1f04733fc2 --jsonAccepts a full SID or just the last 12 characters. Reports turn counts, refusals, errors, silences, repeated user requests, and the ending style (normal vs abrupt user end vs sutando silence). With --metrics, also pulls per-event tool-call timeline from data/call-metrics.jsonl (requires PR #223). Exit code 1 if any issues are found, 0 if clean — useful for CI.
Typical workflow: run find-regression.py to surface broken candidates, then diagnose-call.py <sid> to drill into the worst one.
Future work
- Auto-correlate regression windows with git log
- Smarter NLP-based query matching (query: "recording doesn't stop" vs "recording won't start")
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.