fine-tuning-ml-models
Skill0 starsChanged 4 months agoMITView on GitHub
Prepares and validates data for fine-tuning ML models (SFT/DPO/LoRA), with a real dataset quality checker tested against synthetic messy data - catches duplicates, malformed rows, and train/eval leakage. Use whenever the user wants to fine-tune, train, or customize an ML model on their own data, or asks about SFT, LoRA, or training configuration.
In this repository19 other files it gives its agents.
- adding-authenticationSkill
- auditing-seoSkill
- automating-git-workflowsSkill
- building-email-sequencesSkill
- building-graphql-apisSkill
- building-wordpress-pluginsSkill
- designing-carouselsSkill
- handling-web-formsSkill
- n8n-automationSkill
- organizing-spreadsheetsSkill
- reviewing-cloudflare-workers-codeSkill
- reviewing-code-for-securitySkill
- scaffolding-landing-pagesSkill
- setting-up-error-monitoringSkill
- setting-up-firebase-authSkill
- using-nextjs-best-practicesSkill
- working-with-postgres-databasesSkill
- writing-marketing-copySkill
- writing-terraform-codeSkill
The file
We split this skill into the rules it gives an agent
2 rules. About 0 are common, 1 is distinctive, and 1 is very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈1
Distinctive · estimate
This author's own choices. Worth a look.
1
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 13
Validate the dataset before training anything: - run scripts/validate_finetuning_dataset.py on the train and eval files.
- Distinctive: line 52
Always budget for at least one discarded training run in time estimates - first-attempt hyperparameters are rarely optimal, and a …
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Fine-tuning ML models
Prepares data for fine-tuning and validates it with a real, tested checker - not a description of what "clean data" should look like.
Workflow
- Clarify the goal before the method: define the target task/metric before picking SFT vs DPO vs LoRA.
- Validate the dataset before training anything - run
scripts/validate_finetuning_dataset.pyon the train and eval files. See below: it's genuinely tested against known-bad data, not just written from a description of best practices. - Pick the method to match the goal: SFT for new format/task, DPO/preference methods for tone/style alignment.
- LoRA over full fine-tuning by default - cheaper, faster, sufficient for most customization.
- Never evaluate on data that overlaps with training - the validator specifically checks for this, since it's the single most common way a fine-tuning result looks better than it actually is.
Tested: the dataset validator catches real, planted issues
Built a synthetic JSONL dataset with 5 known problems planted on purpose, then ran the validator against it - not written from a description of what it "should" catch.
Planted issues: an exact duplicate row, a malformed row (wrong key structure), a near-empty response ("ok"), a genuinely broken JSON line, and an eval-set row that's an exact duplicate of a training row (data leakage).
Confirmed output:
=== Stats ===
eval_rows: 2
eval_leakage_count: 1
total_rows: 6
valid_rows: 5
duplicate_rows: 1
unique_rows: 4
=== 4 ERROR(S) ===
- Line 6: invalid JSON - Expecting ',' delimiter: line 1 column 56 (char 55)
- 1 row(s) missing a valid 'messages' array (need at least 2 turns).
- 1 row(s) have an empty or near-empty final response.
- 1 eval row(s) are exact duplicates of training rows - eval score will be inflated, not a real measure of generalization.
=== 1 WARNING(S) ===
- 1 duplicate row(s) found - inflates apparent dataset size without adding signal.All 5 planted issues were caught, with zero false positives - confirmed separately by running the same validator against a clean 3-row dataset, which returned "No issues found."
Why the eval-leakage check matters most
Of the 5 checks, train/eval leakage is the one most fine-tuning workflows skip entirely, and it's the one that silently invalidates the entire evaluation: if even a few eval examples are duplicates (or near-duplicates) of training examples, the eval score measures memorization, not generalization - a model can score well on the eval set while being genuinely worse at the actual task. This is easy to introduce by accident (e.g. randomly splitting a dataset that already contains near-duplicate rows) and produces no error anywhere in the training pipeline - it just quietly inflates the number that's supposed to tell you whether the fine-tune worked.
Notes
- This environment doesn't have GPU/network access to actually run a fine-tuning job (no access to Hugging Face Hub or similar), so the training step itself wasn't tested here - only the data validation, which is real, tested code.
- Always budget for at least one discarded training run in time estimates - first-attempt hyperparameters are rarely optimal, and a validated-clean dataset doesn't guarantee good hyperparameters on the first try.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

