data-mining
Skill2 starsChanged 12 days agoMITView on GitHub
Data Mining. Unsupervised pattern discovery - clustering and customer segmentation (k-means, k-medoids, hierarchical, DBSCAN, HDBSCAN, Gaussian mixtures), association rules and market basket analysis (Apriori, FP-Growth, support, confidence, lift), anomaly and outlier detection (Isolation Forest, LOF, one-class SVM, robust z-scores, autoencoders), dimensionality reduction for exploration (PCA, t-SNE, UMAP), sequence mining, recommenders and text mining. Use for segmentation, clustering, unsupervised learning, basket analysis, fraud or anomaly screening, topic discovery.
In this repository16 other files it gives its agents.
- ai-prediction-explainerSkill
- data-mining-machine-learningSkill
- data-preparationSkill
- data-profilingSkill
- data-storytellingSkill
- feature-engineeringSkill
- hyperparameter-tuningSkill
- ml-quality-guardSkill
- ml-web-applicationSkill
- model-deploymentSkill
- model-evaluationSkill
- model-explainabilitySkill
- model-trainingSkill
- notebook-auditSkill
- problem-framingSkill
- statistical-analysisSkill
The file
Files
Data Mining
Unsupervised results are only as good as the decision they support. Start with what each segment, rule or alert will trigger.
Clustering
- Choose features that matter for the action; scale them inside a pipeline (
StandardScalerorRobustScaler) before any distance method. - Compare k (or density parameters) with several lenses: elbow (inertia), silhouette, gap statistic, BIC for mixtures, and stability (adjusted Rand index across bootstrap refits). Silhouette favors convex blobs and misleads on non-convex shapes; DBSCAN or HDBSCAN suit those.
- Profile every cluster: size, share, defining features (mean or median versus overall, standardized difference), and a plain-language name grounded in those numbers.
- Validate usefulness: outcome rates by cluster on held-out data, or an A/B test of the cluster-specific action.
Association rules
Report support, confidence, lift, leverage and conviction for each rule. Lift above 1 means items co-occur more than chance; below 1 means they repel. Set a minimum support that yields at least tens of transactions per rule; low-support rules are often spurious. Prune redundant rules and check stability across time windows.
Anomaly detection
State the expected contamination and the review process (who looks at alerts, how many per day). Methods: robust z-score (median and MAD) per feature, Isolation Forest, local outlier factor, one-class SVM, autoencoder reconstruction error. With few labels, evaluate precision at k (k = review capacity) on labelled samples, and monitor alert volume over time.
Dimensionality reduction for exploration
PCA for linear structure with explained variance and loadings. t-SNE and UMAP for visual neighborhoods only: their distances and cluster sizes are not metric-faithful and must not drive quantitative claims or clustering inputs.
Recommenders, sequences, text
Collaborative filtering with implicit feedback (ALS, item-item cosine), popularity baseline, cold-start fallback; evaluate with time-based holdout and precision at k, recall at k, NDCG, coverage. Sequence mining (PrefixSpan) with minimum support. Text: TF-IDF plus linear models first, topic models (NMF, LDA) with coherence, embeddings for semantic similarity.
Methods, parameters and code: references/mining-methods.md.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
Reviews can't be read right now.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.