voice-likeness-eval
Skill231 starsChanged 32 days agoMITView on GitHub
Measure how much a cloned voice sounds like the real character, and whether a change really helped. Covers speaker-embedding similarity (ECAPA) against the centroid of held-out real clips, word error rate with Whisper, comparing per line with paired t-tests, how many runs it takes to beat sampling noise, checking a take-picker with an independent speaker model, and simulating picking strategies on many scored takes. Use when comparing voice engines or settings (Fish Audio Drama 3 vs S2 Pro vs ElevenLabs, sampling, reference clips, best-of-N, fine-tuning), or before telling anyone that one voice setup is better than another.
In this repository25 other files it gives its agents.
- AGENTS.mdAGENTS.md
- CLAUDE.mdCLAUDE.md
- fish-voice-serverSkill
- make-npcs-talkSkill
- npc-actionsSkill
- npc-dialogueSkill
- npc-showcase-clipsSkill
- voice-inputSkill
- voice-likeness-evalSkill
- voice-referencesSkill
- fish-voice-serverSkill
- make-npcs-talkSkill
- npc-actionsSkill
- npc-dialogueSkill
- npc-showcase-clipsSkill
- voice-inputSkill
- voice-likeness-evalSkill
- voice-referencesSkill
- fish-voice-serverSkill
- make-npcs-talkSkill
- npc-actionsSkill
- npc-dialogueSkill
- npc-showcase-clipsSkill
- voice-inputSkill
- voice-referencesSkill
The file
We split this skill into the rules it gives an agent
2 rules. About 0 are common, 2 are distinctive, and 0 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈2
Distinctive · estimate
This author's own choices. Worth a look.
0
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 33
Correct yourself in the same place you claimed it: If a later run undoes a claim, say so where the claim was made.
- Distinctive: line 35
Don't let the picker grade itself: If a setting picks takes by a speaker model, check the picks with a different one too.
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Is the voice actually better?
Ears are good at "that's wrong" and bad at "that's 3% better". This repo's numbers all come from one scorer
(server/finetune/train.py, class Scorer), run on a GPU in Modal:
- Voice match: cosine similarity of the ECAPA speaker embedding (speechbrain
spkrec-ecapa-voxceleb) of the generated line to the centroid of 60 real clips of the character that were never used as references. Real clips of the same character score ~0.8 against it; 0.6-0.8 is the range where engines differ. - Word error: Whisper large-v3 transcribes the line; WER against the text, with
[tags]and punctuation removed. Slang ("busta" → "buster", "Ballas" → "ball is") puts a floor of ~5-8% under every engine, so compare WER between engines, not to zero. - Ten test lines in the character's style with a spread of moods (
TEST_LINESintrain.py).
modal run server/finetune/train.py::score_extra --model drama3best8a,drama3best8b # <model>_<line>.wav files
modal run server/finetune/train.py::pick_study_run --models run1,run2,... # every take, both modelsscore_extra adds <model>_<line>.wav files from data/finetune/<speaker>/ to scores.json and rebuilds
compare.html (every line from every engine, playable, with what Whisper heard).
Rules that kept us honest
- Compare per line. Average each engine over its runs per line, then a paired t-test over the 10 lines (|t| > 2.26 is p < 0.05 with 9 degrees of freedom). Report "better on 8 of 10 lines" next to the mean.
- Run it more than twice. Identical settings, two runs: 0.715 and 0.645. One lucky run made a sampling change look like +0.08 when it was +0.017. With 10 runs of single takes, the baseline settled at 0.660.
- Report every run (
0.704 (runs 0.702 / 0.706)), and say when a difference isn't significant. A tie is a result. - Correct yourself in the same place you claimed it. If a later run undoes a claim, say so where the claim was made.
- Don't let the picker grade itself. If a setting picks takes by a speaker model, check the picks with a different one too. Here: picks by ECAPA raised ECAPA a lot and WavLM a little; picks by WavLM raised WavLM a lot and ECAPA less; both together kept most of each. If only the picker's own metric moves, it's gaming the metric, not cloning better.
Simulate before you deploy
Rendering N takes per line and scoring every take lets you test picking rules offline. 100 takes (10 per line) gave, at 8 takes per line: one take 0.660; WavLM pick 0.706; ECAPA pick 0.723; both 0.721 with the lowest word error (7.4%); the true best of 8 by real-clip ECAPA 0.732 (the ceiling). The deployed rule measured 0.704 directly (2 runs), close to its simulation.
What was measured (Lamar, 10 lines)
| Setup | Voice match | Word error |
| ElevenLabs instant clone | 0.612 | 10.5% |
| Fish Audio Drama 3, defaults (2 runs) | 0.643 | 7.1% |
| Drama 3, temperature 0.7 / top_p 0.8, one take (10 runs) | 0.660 | 7.9% |
| S2 Pro, same reference clip | 0.680 | 6.1% |
| Drama 3, best of 8, checked, ECAPA + WavLM pick (2 runs) | 0.704 | 8.6% |
| S2 Pro, its best single clip | 0.707 | 5.6% |
| S2 Pro fine-tuned (LoRA, step 250; needs training) | 0.785 | 7.5% |
Without training, Drama 3 with best of 8 ties S2 Pro's best zero-shot result; only the fine-tune is clearly
higher. Likeness isn't everything: Drama 3 acts [crying] and [laughing] better, and that's a listening
test, not a number.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

