offline-voice
Skill132 starsChanged 39 days agoMITView on GitHub
How the on-device narration engine (VieNeu-TTS) and voice cloning work in this system - install tiers, the worker protocol, and the verified traps around torchaudio, voice metadata parsing and duration. Read when working on Text to video narration, the Voices page, the /api/voices or /api/tts endpoints, or when a user reports that offline voices or voice cloning fail to install or sound wrong.
- Reads credentials
- Installs packages
1 other project uses a version of this, 1 of them edited.
Adopted by
Most recently taken up by
- notivn/AIEVv2edited, first seen Nov 8, 2026
Other versions1 project uses a version of this.
Word for word · 0
None found.
Edited · 1
- notivn/AIEVv2★ 13681% same
Wrote their own version · 0
None found.
How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.
In this repository20 other files it gives its agents.
- CLAUDE.mdCLAUDE.md
- ai-illustrationsSkill
- auto-cutSkill
- background-musicSkill
- color-gradingSkill
- gsapSkill
- hyperframes-cliSkill
- hyperframes-registrySkill
- hyperframesSkill
- key-layoutSkill
- make-a-videoSkill
- noti-tiktok-full-textSkill
- noti-tiktok-vnSkill
- noti-youtube-editSkill
- remotion-assembleSkill
- short-form-videoSkill
- skill-authoringSkill
- video-pipelineSkill
- website-to-hyperframesSkill
- webui-designSkill
The file
We split this skill into the rules it gives an agent
4 rules. About 0 are common, 2 are distinctive, and 2 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈2
Distinctive · estimate
This author's own choices. Worth a look.
2
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 20
Keep both. Do not "simplify" by deleting one: a weak machine reads at roughly real time, which is unacceptable for a long script, while …
- Distinctive: line 32
If a future engine is evaluated, check the weights licence separately from the code licence.
- Very specific: line 71
Fix: look up each preset's own style (get_preset_voice(name) returns {gender, style, description, ...} as real fields) and pass it into …
- Very specific: line 75
Fix: every duration is measured with ffprobe, exactly as on the Gemini path.
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Offline voice - VieNeu-TTS and voice cloning
The system has two narration engines running side by side. They are deliberately not merged.
gemini | vieneu | |
| Where it runs | Google API | The user's own machine |
| Cost | Per call | Free |
| Needs network | Yes | No |
| Voices | 30 preset | 14 Vietnamese preset + unlimited cloned |
| Voice cloning | No (API only accepts preset names) | Yes |
| Speed | A few seconds | RTF ~1.0 (about real time) |
| Setup | GEMINI_API_KEY | pip install vieneu |
Keep both. Do not "simplify" by deleting one: a weak machine reads at roughly real time, which is unacceptable for a long script, while Gemini costs money on every preview click.
Why VieNeu and not F5-TTS / XTTS / Chatterbox
Checked against the requirement "Vietnamese + English, cloning, and shippable in an open-source repo":
- Chatterbox (MIT, excellent English) has no Vietnamese. Disqualified on language.
- F5-TTS code is MIT but the weights are CC-BY-NC, and every Vietnamese finetune is CC-BY-NC-SA. Disqualified on licence.
- viXTTS / XTTS-v2 inherit Coqui's CPML non-commercial licence. Disqualified on licence.
- VietTTS (dangvansam) is Apache-2.0 code but the installer is Linux-only, and its weights are CC-BY-NC too.
- VieNeu-TTS is Apache 2.0 on the code and on the v3 Turbo weights (
pnnbao-ump/VieNeu-TTS-v3-Turbo, trained from scratch on ~10k hours, not a finetune), plus Apache 2.0 on its codec dependencyOpenMOSS-Team/MOSS-Audio-Tokenizer-Nano. Vietnamese-first with English code-switching, torch-free on CPU via ONNX, clones from 3-8 seconds of reference audio.
If a future engine is evaluated, check the weights licence separately from the code licence. That is where every other candidate failed.
Licence trap inside the package itself. vieneu/assets/voices.json carries "license": "CC BY-NC 4.0" and "Model and voices are for non-commercial use only". That notice is scoped to that file, which is the v2 preset pack: 6 voices named Binh, Tuyen, Vinh, Doan, Ly, Ngoc, default Binh. We run mode="v3turbo", which reads voices_v3_turbo.json (14 Vietnamese voices, default Phạm Tuyên). The two sets have zero overlap - verified by comparing the key sets - so nothing on our path is NC-licensed. If anyone ever switches the engine mode away from v3turbo, or a preset named Binh/Tuyen/Vinh/Doan/Ly/Ngoc shows up in the picker, the commercial position changes and must be rechecked.
Install tiers - keep them separate
Two independent levels. Never collapse them into one check:
- Speech only:
pip install vieneu. Torch-free (ONNX Runtime), ~30 MB of wheels. This is enough for all 14 preset voices. - Cloning: additionally
pip install torch torchaudio, 2-3 GB.
start/doctor.mjs reports these as vieneu and vieneu-clone, and only asks about torch once vieneu is present. Forcing a 2-3 GB torch download on someone who only wants offline narration is the mistake to avoid.
The first synthesis after a server start downloads ~1 GB of model from HuggingFace and takes 30s; subsequent process starts take ~15s to load. Always surface this in the UI, otherwise the first click looks like a hang.
Known issues
Symptom: Cloning fails with ImportError: TorchCodec is required for load_with_torchcodec, or RuntimeError: Could not load libtorchcodec.
Cause: add_voice() reads the reference through torchaudio.load(). From torchaudio 2.9 every backend routes through torchcodec, which needs FFmpeg shared libraries - the usual Windows FFmpeg builds are static, so it can never load. Installing torchcodec does not fix it. Passing backend="soundfile" does not fix it either (verified: all three backends raise the same error).
Fix: monkeypatch torchaudio.load to use soundfile before importing vieneu. soundfile is already a vieneu dependency, so this adds nothing to install. Guard it in try/except so that a machine without torch still gets preset voices.
import torch, torchaudio, soundfile as sf
def _load(uri, *a, **kw):
data, sr = sf.read(str(uri), dtype="float32", always_2d=True)
return torch.from_numpy(data.T).contiguous(), sr
torchaudio.load = _loadSymptom: Voices are filed under the wrong region, or every male voice looks southern.
Cause: list_preset_voices() returns labels shaped Name — <Nam|Nữ> · <Bắc|Trung|Nam> · Phong cách <...>. Field 1 is gender, field 2 is region, and the word "Nam" means both "male" and "southern".
Fix: parse strictly by position, never by searching for the substring "Nam". Regression cases: Thái Sơn = gender nam / region nam; Mai Anh = gender nu / region bac.
Symptom: Cloned voices vanish after a restart, or a stray voice appears in the preset list and leaks into vieneu's own error messages.
Cause: tts.save_voices() (and add_voice(..., save=True)) rewrites vieneu/assets/voices_v3_turbo.json inside site-packages, permanently adding your test voice as a 15th "preset". This happened for real during development.
Fix: never call either. The store lives at assets/voices/ (library.json plus <id>/ref.wav) and voices are re-registered into the worker on demand with add_voice(..., save=False). Registration costs ~3s and is lost whenever the worker restarts, so track what the live process actually has. As defence in depth the worker drops any preset whose description does not parse into gender · region · style, and listLocalVoices() drops presets colliding with a store id - a contaminated install still yields a clean 14. To repair one by hand, delete the offending key from voices_v3_turbo.json or pip install --force-reinstall vieneu.
Symptom: A storytelling voice reads like a newsreader.
Cause: infer() has its own style= argument defaulting to "tu_nhien", which is independent of the voice you picked.
Fix: look up each preset's own style (get_preset_voice(name) returns {gender, style, description, ...} as real fields) and pass it into infer(). Only region has to be parsed out of the description string.
Symptom: Timeline drifts against the narration.
Cause: estimating duration from character count.
Fix: every duration is measured with ffprobe, exactly as on the Gemini path. This is not engine-specific advice - see the video-pipeline skill.
Architecture notes
apps/server/python/vieneu_worker.pyis a long-lived worker speaking JSON-lines on stdin/stdout. stdout is protocol only; all logging goes to stderr. It mustsys.stdout.reconfigure(encoding="utf-8")first thing or Vietnamese corrupts on the Windows cp1252 console.pingmust answer without loading the model - it backs the availability check.- One worker, requests serialised (the model is not re-entrant), 10-minute idle shutdown because it holds ~1 GB of RAM.
- Output is already 48 kHz mono, which equals the pipeline's
OUT_SAMPLE_RATE, so no resampling. It still goes through the sameTRIM_FILTERand gap-insertion path as Gemini insynthScript()- two engines with two joining strategies would drift audibly at the seams. - The Perth watermark is left at its default (on). It is inaudible and is the responsible default for cloned speech.
Reference audio guidance for users
3-8 seconds, clean speech, no background music, no noise. Longer is not better - the model only uses the head, so anything past ~8s is discarded. Below 3s the embedding is too weak and the clone stops resembling the speaker.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

