rigorous-evaluation
Skill8 starsChanged 4 months agoMITView on GitHub
Use when evaluating or reporting model performance and choosing metrics, thresholds, and plots that FIT the problem type. Picks the right metrics per task (binary uses ROC-AUC and PR-AUC; multiclass uses macro F1 and a confusion matrix; detection uses mAP; segmentation uses Dice and IoU; regression uses MAE, RMSE, R2; ranking uses nDCG) and does not force AUC or ROC where they do not apply. Also covers calibration, thresholds chosen on validation, confidence intervals, and honest reporting. Triggers on 'evaluate', 'metrics', 'F1', 'AUC', 'ROC', 'mAP', 'RMSE', 'confusion matrix', 'which threshold', 'is this result good'.
In this repository22 other files it gives its agents.
- accuracy-improvement-loopSkill
- data-rigor-and-leakageSkill
- dataset-profilingSkill
- deployment-explainabilitySkill
- domain-3dSkill
- domain-anomaly-detectionSkill
- domain-audio-speechSkill
- domain-computer-visionSkill
- domain-generativeSkill
- domain-graphSkill
- domain-medical-imagingSkill
- domain-multimodalSkill
- domain-nlp-llmSkill
- domain-recommenderSkill
- domain-reinforcement-learningSkill
- domain-tabularSkill
- domain-time-seriesSkill
- domain-videoSkill
- literature-reviewSkill
- ml-research-methodologySkill
- notebook-deliverySkill
- training-optimizationSkill
The file
We split this skill into the rules it gives an agent
5 rules. About 0 are common, 4 are distinctive, and 1 is very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈4
Distinctive · estimate
This author's own choices. Worth a look.
1
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 28
If a metric does not fit the problem, omit it.
- Distinctive: line 32
When relevant, report two operating points: Youden J (balanced) and a high-sensitivity point (recall >= 0.90) for screening.
- Distinctive: line 40
Check performance across subgroups where it matters (site, device, demographic).
- Very specific: line 44
If a result looks too good, re-run the leakage audit (data-rigor-and-leakage) before believing it.
- Distinctive: line 46
Never report a number you have not actually computed from a real run.
≈ “Common” is an estimate: at least 1 in 5 of the 350,495 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Rigorous Evaluation
Report a number you can defend, not the biggest number. Choose metrics AND plots that fit the problem type. Do not pad a report with metrics that do not apply. For example, ROC-AUC and a ROC curve only make sense for binary classification.
Metrics and plots by problem type
| Problem | Core metrics | Plots to show | Do not use |
| Binary classification | Accuracy, Precision, Recall, F1, ROC-AUC, PR-AUC | confusion matrix, ROC curve, PR curve | none |
| Multiclass (more than 2) | Accuracy, macro and weighted Precision/Recall/F1 | confusion matrix, per-class metric bars | a single ROC curve |
| Multilabel | per-label F1, micro and macro F1, mAP | per-label PR curves | plain accuracy |
| Object detection (faces, vehicles, objects) | mAP@0.5, mAP@0.5:0.95, per-class AP | PR curve per class, sample images with predicted boxes | accuracy, ROC-AUC |
| Segmentation | mean Dice, mean IoU, per-class IoU | mask overlays, per-class IoU bars | ROC-AUC |
| Regression | MAE, RMSE, R2, MAPE | predicted vs actual, residual plot | F1, AUC |
| Ranking or retrieval | nDCG@k, MAP, Recall@k, MRR | recall@k curve | accuracy |
| Face verification | ROC and EER, TAR at fixed FAR | verification ROC | plain accuracy |
| Forecasting | MAE, RMSE, MASE, sMAPE | forecast vs actual, error per horizon | AUC, F1 |
| Clustering | silhouette, ARI, NMI when labels exist | 2D projection | accuracy |
Binary vs multiclass, the common mistake
- ROC-AUC and a ROC curve are for binary problems. With more than two classes do not draw one ROC curve. Use the confusion matrix as the main visual and report macro and weighted F1. Compute one-vs-rest macro AUC only if a ranking metric is genuinely needed.
- Pick ONE headline metric that matches the cost of errors: recall for screening, precision when false positives are expensive, mAP for detection, RMSE for regression.
- If a metric does not fit the problem, omit it. A short relevant report is stronger than a padded one.
Thresholds and selection, on VALIDATION only (classification with a threshold)
- Choosing the decision threshold using test labels is leakage. Select it on validation, then apply it unchanged to test.
- When relevant, report two operating points: Youden J (balanced) and a high-sensitivity point (recall >= 0.90) for screening.
Calibration (for decision support)
Raw softmax is often over-confident. Apply temperature scaling (fit on validation) so a displayed probability is trustworthy. Report reliability, not only discrimination.
Uncertainty and fairness
- Report confidence intervals (bootstrap, or DeLong for AUC) when the test set is small.
- Train each model a few times or use cross-validation; report mean and standard deviation. A single run's validation-best may not be the test-best.
- Check performance across subgroups where it matters (site, device, demographic).
Honest reporting
- Show the confusion matrix and per-class metrics, not only the headline number.
- If a result looks too good, re-run the leakage audit (
data-rigor-and-leakage) before believing it. - State the comparison honestly: same split as the papers, same metric. Note when a result is not comparable.
- Never report a number you have not actually computed from a real run.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.