Narrated video from a document

Long documents don't get read. The fix is not a shorter document — it's the same content as something a person will sit through. This pipeline turns written material into a narrated MP4 with subtitles and visuals, entirely on the local machine.

The thing that makes it work: the visuals are computed from the document's own numbers. A generative video model cannot draw your data — it will produce handsome, unrelated footage, which is worse than nothing when the document's value is that it's specific and checkable. Everything here is SVG drawn from JSON, so if a bar moves on screen, a number moved in the source.

The shape

document + its underlying data
        │
        ├─► narration script  (you write this — the only judgment call)
        │        │
        │        └─► tts_narrate.py ──► per-sentence WAV + manifest + subtitles.srt
        │                                        │  measured durations
        │                                        ▼
        └─► visual functions ──────────► render loop ──► f000000.png … (1080p)
                                                 │
                                                 ▼
                                          assemble.sh ──► narrated.mp4

One model call writes the narration. Everything after it is deterministic code. Cost is roughly $0.15–0.40 per video at API rates plus ~18 minutes of CPU for a 10-minute runtime.

Before you start

Run references/setup.md once per machine — Python 3.12 venv, kokoro-onnx, cairosvg, espeak-ng, and the Kokoro model weights. It has the two failure modes that will otherwise cost you twenty minutes (espeak-ng phoneme data path, and system Python 3.9 being too old).

Step 1 — Write the narration script

Create a module with a SEGMENTS list. Each segment is one visual and the sentences spoken over it:

SEGMENTS = [
    {
        "id": "open", "visual": "gravities",
        "sentences": [
            "Almost everything in your chart drains toward a single point.",
            "It is the strongest thing you carry.",
        ],
    },
]

Store sentences individually, not as a paragraph. The TTS step synthesises one sentence at a time and records each one's measured start and end. That is what lets subtitles sit exactly on the audio — an estimate drifts badly over ten minutes, and the drift is the first thing a viewer notices.

Three things decide whether the narration lands:

Strip the jargon, or the audience stops listening. If the source document uses a specialist vocabulary, translate it. Nobody outside the field knows what an antardasha or a p-value adjustment for family-wise error is, and a narrator saying it aloud sounds like a robot reading a foreign menu. Keep a banned-term list in the module docstring and assert against it — a mechanical check catches what your ear won't:

BANNED = ["dasha", "lagna", "ashtakavarga", "yogakaraka"]
text = " ".join(s for seg in SEGMENTS for s in seg["sentences"]).lower()
assert not [b for b in BANNED if b in text]

Keep the document's best lines verbatim. If the source says "you do the sovereign's work and then introduce yourself as the assistant", that sentence is why the document works. Paraphrasing it into "you tend to under-credit yourself" throws away the only part a viewer will repeat afterwards.

Length follows from the material, not a target. Roughly 150 words per minute at a measured pace. A five-minute video is ~750 words; ten minutes is ~1,500. Write what the material earns and check the estimate before synthesising.

Step 2 — Synthesise the narration

./ttsenv/bin/python scripts/tts_narrate.py \
    --script my_video_script.py --out audio --voice bm_george --speed 0.88

Emits per-sentence WAVs, a joined narration.wav, manifest.json (measured start/end for every sentence and segment), and a standards-compliant subtitles.srt.

Voices worth knowing: bm_george and bm_lewis (British male, measured — good for anything serious), am_michael (US male), af_bella and af_nicole (US female). --speed around 0.85–0.90 reads as considered; 1.0 is noticeably brisk for reflective material. Kokoro runs faster than you expect — synthesise first and read the total, rather than tuning speed blind.

Step 3 — Write the visuals

Import the primitives from scripts/video_kit.py and write one function per visual, each taking (p, data) where p is 0→1 progress through that segment:

from video_kit import frame, txt, bars, ring, fade, ease, W, H, BLUE

def v_scores(p, d):
    o = fade(p)                       # opacity envelope: fade in, hold, fade out
    return frame(
        txt(W/2, 180, "how it scores", size=44, op=o) +
        bars([(k, v) for k, v in d["scores"].items()], reveal=ease(p * 1.6))
    )

VISUALS = {"scores": v_scores}

The kit provides frame, txt, bars, ring, timeline, orb, wave, fade, ease, stars, and a palette. Read scripts/video_kit.py — it is short and the function signatures are the documentation.

Two habits separate visuals that add value from ones that fill space:

Pass real values in. bars(d["scores"]), not bars([("a", 80), ("b", 60)]). The moment a number is typed by hand it can drift from the document, and nobody will catch it.

Show the comparison the sentence is making. If the narration says a value is unusual, put the other values next to it. A lone number means nothing; the same number against six smaller ones is an argument. The strongest frame in the reference build was a plain bar chart — one planet at 15, everything else at 5 or below — because it made a claim visible rather than decorating one.

Animate with ease() rather than linear interpolation, and wrap each segment in fade() so cuts land soft. There is no crossfade filter in the assembly step because the opacity envelope already does it.

Step 4 — Render frames

Use render_frames() from the kit — it walks the manifest, picks the active segment and the active subtitle for each frame, and rasterises:

from video_kit import render_frames
render_frames(VISUALS, data, "audio/manifest.json", "frames", fps=12)

12 fps is the right default. These are typographic visuals, not motion footage; 12 fps looks identical to 24 and halves the render. cairosvg costs ~50 ms/frame for simple compositions and ~100 ms for busy ones at 1080p, so ten minutes of video is ~7,900 frames and roughly 7–13 minutes of single-threaded CPU. That is the pipeline's real bottleneck — if you ever need volume, parallelise this step, not anything else.

Preview every visual before committing to the full render. One frame per visual at p=0.6 takes under a second and catches the composition problems that would otherwise cost you a full render:

for name, fn in VISUALS.items():
    svg = fn(0.6, data).replace("</svg>", subtitle("a sample line") + "</svg>")
    cairosvg.svg2png(bytestring=svg.encode(), write_to=f"preview/{name}.png",
                     output_width=W, output_height=H)

Then actually look at them. In the reference build this caught a flow diagram whose inflow arrows didn't reach the reservoir they were supposed to fill, and leak lines too thin to see at 1080p — both invisible in code, both obvious in the image.

Step 5 — Assemble

scripts/assemble.sh . out.mp4 12 frames audio

Takes the working directory, output path, fps, frame directory, and audio directory — the last two are parameters rather than defaults on purpose. When a project has more than one video in flight, hardcoded frames/ will silently mux the wrong project's frames onto the right project's narration, and the result looks plausible enough to ship.

Step 6 — QA the encoded file, not the source

Pull frames back out of the MP4 and look at them:

for t in 12 90 240 400; do
  ffmpeg -v error -ss $t -i out.mp4 -frames:v 1 -y qa/t$t.png
done

Check: subtitles readable and on the audio; every visual reaches the state the narration describes; no text overlapping a graphic; the last segment isn't cut off. ffprobe the duration and confirm it matches the manifest's total.

If a frame looks sparse, check p before assuming it's broken — a frame sampled at the start of a segment is supposed to be mid-fade-in.

Where this goes wrong

When something else is the better answer

If the user wants generative footage — a dreamlike sequence, b-roll, an imagined scene — this is the wrong tool; that needs a video model (Wan, LTX, Hunyuan, or a hosted service), and on a Mac it needs rented GPU.

If the deliverable is a talking head, this pipeline still produces the narration and subtitles; only the face needs an external service.

And if the document has no data behind it — pure prose with nothing to plot — say so before building. You can still make a handsome typographic video, but be honest that the visuals are illustration rather than evidence, because the whole argument for this approach is that they usually aren't.