fish-voice-server
Skill231 starsChanged 32 days agoMITView on GitHub
Host Fish Audio's open-weights voice models (Drama 3 and S2 Pro) on a serverless GPU (Modal, or fal) behind one small HTTP API a game can call. POST /speak takes text and a voice and streams 16-bit PCM back with first audio in ~0.4 s, cloned zero-shot from a reference clip, with [tags] like [angry] or [laughing] for delivery. Covers deploying with SGLang-Omni, the API, preparing voices at startup, keep-warm and cold starts, scaling to zero and cost, sampling settings that measurably help likeness, picking the best of several takes for pre-rendered lines, an optional S2 Pro fine-tune, and the setup gotchas. Use when a game, mod or app needs realtime cloned character voices, the user mentions Fish Audio, Fish Speech, S2 Pro, Drama 3 or a self-hosted TTS server, or a voice server is slow, cold or failing.
In this repository25 other files it gives its agents.
- AGENTS.mdAGENTS.md
- CLAUDE.mdCLAUDE.md
- fish-voice-serverSkill
- make-npcs-talkSkill
- npc-actionsSkill
- npc-dialogueSkill
- npc-showcase-clipsSkill
- voice-inputSkill
- voice-likeness-evalSkill
- voice-referencesSkill
- make-npcs-talkSkill
- npc-actionsSkill
- npc-dialogueSkill
- npc-showcase-clipsSkill
- voice-inputSkill
- voice-likeness-evalSkill
- voice-referencesSkill
- fish-voice-serverSkill
- make-npcs-talkSkill
- npc-actionsSkill
- npc-dialogueSkill
- npc-showcase-clipsSkill
- voice-inputSkill
- voice-likeness-evalSkill
- voice-referencesSkill
The file
We split this skill into the rules it gives an agent
1 rule. About 0 are common, 0 are distinctive, and 1 is very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈0
Distinctive · estimate
This author's own choices. Worth a look.
1
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 33
Keep the URL private: /health needs no token and wakes the GPU.
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
A Fish Audio voice server for games
server/drama3.py (default) and server/s2pro.py are Modal apps with the same API, so a game switches models by
changing one URL. Both run Fish Audio's open weights through SGLang-Omni on one H100 and stream audio as it's
generated.
Pick the model
Drama 3 (fishaudio/drama-3) | S2 Pro (fishaudio/s2-pro) | |
| Size | 35B mixture of experts, 3B active, FP8 (~37 GB) | Smaller; leaves room for fine-tuned voices on the same GPU |
| First audio from a PC | ~0.33-0.5 s warm | ~0.5 s warm |
Acting [tags] | Better (crying, laughing, whispering) | Good |
| Voice likeness, zero-shot (Lamar, below) | 0.643 at defaults, 0.660 tuned, 0.704 best of 8 | 0.680 with the same clip, 0.707 with its best clip |
| Fine-tuning | Not done here | LoRA in server/finetune/ (0.785, the best result, but it needs training) |
Deploy (Modal)
uv tool install modal && modal setup
modal secret create npcvoice-auth NPCVOICE_TOKEN=$(python -c "import secrets; print(secrets.token_urlsafe(32))")
modal run server/drama3.py::download_weights # once: ~40 GB into the npcvoice-models volume
npctalk voice make --name Narrator --audio narrator.wav && npctalk voice upload voices/Narrator
modal deploy server/drama3.py # prints https://<workspace>--npcvoice-drama.modal.run
export NPCVOICE_URL=https://<workspace>--npcvoice-drama.modal.run NPCVOICE_TOKEN=<the token>
npctalk health # waits through a cold start (1-3 min)
npctalk say "[laughing] Nah, you trippin'." --voice Narrator/clone --out test.wavEverything is named npcvoice-* (apps, volumes, secrets) so it stays apart from the rest of a Modal workspace.
Keep the URL private: /health needs no token and wakes the GPU.
The API
POST /speak Authorization: Bearer <NPCVOICE_TOKEN>
{"text": "[angry] Get out of my face!", "voice": "Trevor/clone"} a voice prepared on the server
{"text": "...", "reference_audio_url": "https://.../me.wav", "reference_text": "what me.wav says"}
optional: "temperature", "top_p", "multi" (all clips in the voice's folder), "best_of" (1-8, drama3.py)
→ 200, X-Sample-Rate: 44100, body: 16-bit mono PCM, streamed as it's generated
(with best_of > 1: the whole line at once, and X-Takes: per-take likeness / word error / length)
GET /health no auth → {"ok": true, "model": "drama-3", "voices": 21}- Voices live in the volume at
/voices/<Name>/clone.wav+clone.lab(transcript), plusref_0..2and mood folders (/voices/<Name>/angry/ref_0.wav). The server encodes every clip when it starts, so a named voice costs nothing per request; a clip sent by URL costs ~9 s the first time. Restart the server after uploading voices (modal app stop npcvoice-drama). - Tags go where they happen in the text:
"[sigh] Man... [laughing] nah.". Drama 3 and S2 Pro both voice them; other engines treat them as hints. - Client:
examples/gta-v/mod/Tts.cs(ModalTts, C#) andnpctalk/server.py(Python) both read the stream in 4 KB chunks and start playing at the first chunk.
Settings that were measured (Lamar, 10 lines, voice match = ECAPA similarity to 60 real clips)
| Change | Result |
Sampling temperature 0.7, top_p 0.8 instead of Drama 3's defaults (1.0 / 0.9) | 0.660 (10 runs) vs 0.643, better on 8 of 10 lines (t = 2.5) |
| Temperature 0.5 or 0.3 | Worse (0.609, 0.628) and more rambling takes |
The ~30 s clone.wav vs one 6-10 s clip | 0.680 vs 0.610-0.630: clone the long clip |
| All four clips as references at once | No better (0.641 vs 0.637) |
best_of: 8 with checks (below) | 0.704 vs 0.660 for one take, better on 9 of 10 lines (t = 3.9) |
So SAMPLING = {"temperature": 0.7, "top_p": 0.8} is the server default, and pre-rendered lines use best of 8.
best_of: for lines made ahead
About 1 take in 10 rambles to the token limit (47.6 s for a 15-word line) or trails off into noises. For lines
that don't need streaming (cutscenes, a director's clips) best_of: N renders N takes in one batch and:
- stops each take just past the longest the words could take (
1.5 + 0.45 s per word, + 1 s); - drops takes that are too long or that a small ASR (Whisper base, on the GPU) can't hear the line in (word error above the clearest take's + 0.1, or above 0.5);
- keeps the one most like the reference by ECAPA plus WavLM similarity, each scaled across the takes.
Picking by one model alone over-fits it: WavLM-only picks scored 0.692 at 4 takes; ECAPA-only raised ECAPA a lot but WavLM less; both together kept most of each. An unchecked pick scored 0.708 but let a 14-second whimper through. Best of 8 takes ~4-6 s per line with the scoring models on the GPU.
Keep-warm, cold starts, cost
MIN_CONTAINERS = 0,SCALEDOWN = 300: the H100 (about $4/hour) stops 5 minutes after the last request; the next request waits 1-3 minutes (weights load, kernels warm up, every voice is encoded).- A game should ping while it runs (the GTA mod sends a one-word line every 90 s of silence) and fall back to a no-GPU engine (MiniMax on fal) while the GPU wakes.
- For a demo without the game:
modal deploy server/keepwarm.py, thenmodal run server/keepwarm.py --url $NPCVOICE_URL --hours 12(it says what that costs: ~$45-50).
Fine-tuning S2 Pro (optional, needs training)
server/finetune/: prepare_dataset.py (Demucs → Whisper word timings → 2-15 s clips → speaker filter →
loudness), train.py (LoRA r8, α16, lr 1e-4; step 250 was best), fish_finetune.py (two fixes fish-speech
needed for S2 Pro). Read references/finetune.md before starting.
Gotchas for all of this: references/gotchas.md.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

