agentleFS
christos-antonochristos-antono/NPCTalk/.agents/skills/fish-voice-server/SKILL.md

fish-voice-server

Skill231 starsChanged 32 days agoMITView on GitHub

Host Fish Audio's open-weights voice models (Drama 3 and S2 Pro) on a serverless GPU (Modal, or fal) behind one small HTTP API a game can call. POST /speak takes text and a voice and streams 16-bit PCM back with first audio in ~0.4 s, cloned zero-shot from a reference clip, with [tags] like [angry] or [laughing] for delivery. Covers deploying with SGLang-Omni, the API, preparing voices at startup, keep-warm and cold starts, scaling to zero and cost, sampling settings that measurably help likeness, picking the best of several takes for pre-rendered lines, an optional S2 Pro fine-tune, and the setup gotchas. Use when a game, mod or app needs realtime cloned character voices, the user mentions Fish Audio, Fish Speech, S2 Pro, Drama 3 or a self-hosted TTS server, or a voice server is slow, cold or failing.

The file

We split this skill into the rules it gives an agent

1 rule. About 0 are common, 0 are distinctive, and 1 is very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈0

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 1

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    Keep the URL private: /health needs no token and wakes the GPU.

    line 33

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

A Fish Audio voice server for games

server/drama3.py (default) and server/s2pro.py are Modal apps with the same API, so a game switches models by changing one URL. Both run Fish Audio's open weights through SGLang-Omni on one H100 and stream audio as it's generated.

Pick the model

Drama 3 (fishaudio/drama-3)S2 Pro (fishaudio/s2-pro)
Size35B mixture of experts, 3B active, FP8 (~37 GB)Smaller; leaves room for fine-tuned voices on the same GPU
First audio from a PC~0.33-0.5 s warm~0.5 s warm
Acting [tags]Better (crying, laughing, whispering)Good
Voice likeness, zero-shot (Lamar, below)0.643 at defaults, 0.660 tuned, 0.704 best of 80.680 with the same clip, 0.707 with its best clip
Fine-tuningNot done hereLoRA in server/finetune/ (0.785, the best result, but it needs training)

Deploy (Modal)

uv tool install modal && modal setup
modal secret create npcvoice-auth NPCVOICE_TOKEN=$(python -c "import secrets; print(secrets.token_urlsafe(32))")
modal run server/drama3.py::download_weights        # once: ~40 GB into the npcvoice-models volume
npctalk voice make --name Narrator --audio narrator.wav && npctalk voice upload voices/Narrator
modal deploy server/drama3.py                       # prints https://<workspace>--npcvoice-drama.modal.run
export NPCVOICE_URL=https://<workspace>--npcvoice-drama.modal.run NPCVOICE_TOKEN=<the token>
npctalk health                                      # waits through a cold start (1-3 min)
npctalk say "[laughing] Nah, you trippin'." --voice Narrator/clone --out test.wav

Everything is named npcvoice-* (apps, volumes, secrets) so it stays apart from the rest of a Modal workspace. Keep the URL private: /health needs no token and wakes the GPU.

The API

POST /speak    Authorization: Bearer <NPCVOICE_TOKEN>
  {"text": "[angry] Get out of my face!", "voice": "Trevor/clone"}            a voice prepared on the server
  {"text": "...", "reference_audio_url": "https://.../me.wav", "reference_text": "what me.wav says"}
  optional: "temperature", "top_p", "multi" (all clips in the voice's folder), "best_of" (1-8, drama3.py)
→ 200, X-Sample-Rate: 44100, body: 16-bit mono PCM, streamed as it's generated
  (with best_of > 1: the whole line at once, and X-Takes: per-take likeness / word error / length)
GET /health    no auth → {"ok": true, "model": "drama-3", "voices": 21}
  • Voices live in the volume at /voices/<Name>/clone.wav + clone.lab (transcript), plus ref_0..2 and mood folders (/voices/<Name>/angry/ref_0.wav). The server encodes every clip when it starts, so a named voice costs nothing per request; a clip sent by URL costs ~9 s the first time. Restart the server after uploading voices (modal app stop npcvoice-drama).
  • Tags go where they happen in the text: "[sigh] Man... [laughing] nah.". Drama 3 and S2 Pro both voice them; other engines treat them as hints.
  • Client: examples/gta-v/mod/Tts.cs (ModalTts, C#) and npctalk/server.py (Python) both read the stream in 4 KB chunks and start playing at the first chunk.

Settings that were measured (Lamar, 10 lines, voice match = ECAPA similarity to 60 real clips)

ChangeResult
Sampling temperature 0.7, top_p 0.8 instead of Drama 3's defaults (1.0 / 0.9)0.660 (10 runs) vs 0.643, better on 8 of 10 lines (t = 2.5)
Temperature 0.5 or 0.3Worse (0.609, 0.628) and more rambling takes
The ~30 s clone.wav vs one 6-10 s clip0.680 vs 0.610-0.630: clone the long clip
All four clips as references at onceNo better (0.641 vs 0.637)
best_of: 8 with checks (below)0.704 vs 0.660 for one take, better on 9 of 10 lines (t = 3.9)

So SAMPLING = {"temperature": 0.7, "top_p": 0.8} is the server default, and pre-rendered lines use best of 8.

best_of: for lines made ahead

About 1 take in 10 rambles to the token limit (47.6 s for a 15-word line) or trails off into noises. For lines that don't need streaming (cutscenes, a director's clips) best_of: N renders N takes in one batch and:

  1. stops each take just past the longest the words could take (1.5 + 0.45 s per word, + 1 s);
  2. drops takes that are too long or that a small ASR (Whisper base, on the GPU) can't hear the line in (word error above the clearest take's + 0.1, or above 0.5);
  3. keeps the one most like the reference by ECAPA plus WavLM similarity, each scaled across the takes.

Picking by one model alone over-fits it: WavLM-only picks scored 0.692 at 4 takes; ECAPA-only raised ECAPA a lot but WavLM less; both together kept most of each. An unchecked pick scored 0.708 but let a 14-second whimper through. Best of 8 takes ~4-6 s per line with the scoring models on the GPU.

Keep-warm, cold starts, cost

  • MIN_CONTAINERS = 0, SCALEDOWN = 300: the H100 (about $4/hour) stops 5 minutes after the last request; the next request waits 1-3 minutes (weights load, kernels warm up, every voice is encoded).
  • A game should ping while it runs (the GTA mod sends a one-word line every 90 s of silence) and fall back to a no-GPU engine (MiniMax on fal) while the GPU wakes.
  • For a demo without the game: modal deploy server/keepwarm.py, then modal run server/keepwarm.py --url $NPCVOICE_URL --hours 12 (it says what that costs: ~$45-50).

Fine-tuning S2 Pro (optional, needs training)

server/finetune/: prepare_dataset.py (Demucs → Whisper word timings → 2-15 s clips → speaker filter → loudness), train.py (LoRA r8, α16, lr 1e-4; step 250 was best), fish_finetune.py (two fixes fish-speech needed for S2 Pro). Read references/finetune.md before starting.

Gotchas for all of this: references/gotchas.md.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.