agentleFS
notivnnotivn/AIEV/.claude/skills/offline-voice/SKILL.md

offline-voice

Skill132 starsChanged 39 days agoMITView on GitHub

How the on-device narration engine (VieNeu-TTS) and voice cloning work in this system - install tiers, the worker protocol, and the verified traps around torchaudio, voice metadata parsing and duration. Read when working on Text to video narration, the Voices page, the /api/voices or /api/tts endpoints, or when a user reports that offline voices or voice cloning fail to install or sound wrong.

  • Reads credentials
  • Installs packages

1 other project uses a version of this, 1 of them edited.

Adopted by

Most recently taken up by

The file

We split this skill into the rules it gives an agent

4 rules. About 0 are common, 2 are distinctive, and 2 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈2

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 2

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Keep both. Do not "simplify" by deleting one: a weak machine reads at roughly real time, which is unacceptable for a long script, while …

    line 20
  2. Distinctive:

    If a future engine is evaluated, check the weights licence separately from the code licence.

    line 32
  3. Very specific:

    Fix: look up each preset's own style (get_preset_voice(name) returns {gender, style, description, ...} as real fields) and pass it into …

    line 71
  4. Very specific:

    Fix: every duration is measured with ffprobe, exactly as on the Gemini path.

    line 75

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Offline voice - VieNeu-TTS and voice cloning

The system has two narration engines running side by side. They are deliberately not merged.

geminivieneu
Where it runsGoogle APIThe user's own machine
CostPer callFree
Needs networkYesNo
Voices30 preset14 Vietnamese preset + unlimited cloned
Voice cloningNo (API only accepts preset names)Yes
SpeedA few secondsRTF ~1.0 (about real time)
SetupGEMINI_API_KEYpip install vieneu

Keep both. Do not "simplify" by deleting one: a weak machine reads at roughly real time, which is unacceptable for a long script, while Gemini costs money on every preview click.

Why VieNeu and not F5-TTS / XTTS / Chatterbox

Checked against the requirement "Vietnamese + English, cloning, and shippable in an open-source repo":

  • Chatterbox (MIT, excellent English) has no Vietnamese. Disqualified on language.
  • F5-TTS code is MIT but the weights are CC-BY-NC, and every Vietnamese finetune is CC-BY-NC-SA. Disqualified on licence.
  • viXTTS / XTTS-v2 inherit Coqui's CPML non-commercial licence. Disqualified on licence.
  • VietTTS (dangvansam) is Apache-2.0 code but the installer is Linux-only, and its weights are CC-BY-NC too.
  • VieNeu-TTS is Apache 2.0 on the code and on the v3 Turbo weights (pnnbao-ump/VieNeu-TTS-v3-Turbo, trained from scratch on ~10k hours, not a finetune), plus Apache 2.0 on its codec dependency OpenMOSS-Team/MOSS-Audio-Tokenizer-Nano. Vietnamese-first with English code-switching, torch-free on CPU via ONNX, clones from 3-8 seconds of reference audio.

If a future engine is evaluated, check the weights licence separately from the code licence. That is where every other candidate failed.

Licence trap inside the package itself. vieneu/assets/voices.json carries "license": "CC BY-NC 4.0" and "Model and voices are for non-commercial use only". That notice is scoped to that file, which is the v2 preset pack: 6 voices named Binh, Tuyen, Vinh, Doan, Ly, Ngoc, default Binh. We run mode="v3turbo", which reads voices_v3_turbo.json (14 Vietnamese voices, default Phạm Tuyên). The two sets have zero overlap - verified by comparing the key sets - so nothing on our path is NC-licensed. If anyone ever switches the engine mode away from v3turbo, or a preset named Binh/Tuyen/Vinh/Doan/Ly/Ngoc shows up in the picker, the commercial position changes and must be rechecked.

Install tiers - keep them separate

Two independent levels. Never collapse them into one check:

  1. Speech only: pip install vieneu. Torch-free (ONNX Runtime), ~30 MB of wheels. This is enough for all 14 preset voices.
  2. Cloning: additionally pip install torch torchaudio, 2-3 GB.

start/doctor.mjs reports these as vieneu and vieneu-clone, and only asks about torch once vieneu is present. Forcing a 2-3 GB torch download on someone who only wants offline narration is the mistake to avoid.

The first synthesis after a server start downloads ~1 GB of model from HuggingFace and takes 30s; subsequent process starts take ~15s to load. Always surface this in the UI, otherwise the first click looks like a hang.

Known issues

Symptom: Cloning fails with ImportError: TorchCodec is required for load_with_torchcodec, or RuntimeError: Could not load libtorchcodec. Cause: add_voice() reads the reference through torchaudio.load(). From torchaudio 2.9 every backend routes through torchcodec, which needs FFmpeg shared libraries - the usual Windows FFmpeg builds are static, so it can never load. Installing torchcodec does not fix it. Passing backend="soundfile" does not fix it either (verified: all three backends raise the same error). Fix: monkeypatch torchaudio.load to use soundfile before importing vieneu. soundfile is already a vieneu dependency, so this adds nothing to install. Guard it in try/except so that a machine without torch still gets preset voices.

import torch, torchaudio, soundfile as sf
def _load(uri, *a, **kw):
    data, sr = sf.read(str(uri), dtype="float32", always_2d=True)
    return torch.from_numpy(data.T).contiguous(), sr
torchaudio.load = _load

Symptom: Voices are filed under the wrong region, or every male voice looks southern. Cause: list_preset_voices() returns labels shaped Name — <Nam|Nữ> · <Bắc|Trung|Nam> · Phong cách <...>. Field 1 is gender, field 2 is region, and the word "Nam" means both "male" and "southern". Fix: parse strictly by position, never by searching for the substring "Nam". Regression cases: Thái Sơn = gender nam / region nam; Mai Anh = gender nu / region bac.

Symptom: Cloned voices vanish after a restart, or a stray voice appears in the preset list and leaks into vieneu's own error messages. Cause: tts.save_voices() (and add_voice(..., save=True)) rewrites vieneu/assets/voices_v3_turbo.json inside site-packages, permanently adding your test voice as a 15th "preset". This happened for real during development. Fix: never call either. The store lives at assets/voices/ (library.json plus <id>/ref.wav) and voices are re-registered into the worker on demand with add_voice(..., save=False). Registration costs ~3s and is lost whenever the worker restarts, so track what the live process actually has. As defence in depth the worker drops any preset whose description does not parse into gender · region · style, and listLocalVoices() drops presets colliding with a store id - a contaminated install still yields a clean 14. To repair one by hand, delete the offending key from voices_v3_turbo.json or pip install --force-reinstall vieneu.

Symptom: A storytelling voice reads like a newsreader. Cause: infer() has its own style= argument defaulting to "tu_nhien", which is independent of the voice you picked. Fix: look up each preset's own style (get_preset_voice(name) returns {gender, style, description, ...} as real fields) and pass it into infer(). Only region has to be parsed out of the description string.

Symptom: Timeline drifts against the narration. Cause: estimating duration from character count. Fix: every duration is measured with ffprobe, exactly as on the Gemini path. This is not engine-specific advice - see the video-pipeline skill.

Architecture notes

  • apps/server/python/vieneu_worker.py is a long-lived worker speaking JSON-lines on stdin/stdout. stdout is protocol only; all logging goes to stderr. It must sys.stdout.reconfigure(encoding="utf-8") first thing or Vietnamese corrupts on the Windows cp1252 console.
  • ping must answer without loading the model - it backs the availability check.
  • One worker, requests serialised (the model is not re-entrant), 10-minute idle shutdown because it holds ~1 GB of RAM.
  • Output is already 48 kHz mono, which equals the pipeline's OUT_SAMPLE_RATE, so no resampling. It still goes through the same TRIM_FILTER and gap-insertion path as Gemini in synthScript() - two engines with two joining strategies would drift audibly at the seams.
  • The Perth watermark is left at its default (on). It is inaudible and is the responsible default for cloned speech.

Reference audio guidance for users

3-8 seconds, clean speech, no background music, no noise. Longer is not better - the model only uses the head, so anything past ~8s is discarded. Below 3s the embedding is too weak and the clone stops resembling the speaker.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.