video-captions

Overview

Captions support the audio and decide whether a short looks edited or amateur. Audio leads: most Reels views are sound-on, so captions serve the sound-off minority, noisy settings, accessibility, and search. This skill is the one caption pipeline for the video domain: real word timings, a vetted renderer with platform-tuned presets, and a libass burn. It replaces hand-written frame-by-frame text drawing (PIL, moviepy, OpenCV) and plain force_style SRT burns, which produce the flat, mistimed captions viewers read as low effort.

Mental model

Three inputs make a caption good, and the pipeline fixes each one at the source:

• Timing comes from per-word timestamps of the final audio, never from even spacing or sentence-level SRT.
• Look comes from a preset designed for the format: page size, case, outline, active-word highlight, position.
• Rendering goes through libass (the ASS subtitle renderer inside ffmpeg), which shapes text properly and animates per word at full frame rate.

The renderer is the video-captions command, installed on PATH with the captions capability (the same file is the bundled scripts/captions.py, listed under Skill Resources). It turns any word-timing JSON into a styled .ass file and can burn it. It needs only python3 and finds an ffmpeg with libass on its own, including Homebrew's keg-only ffmpeg-full. video-captions --help lists every option.

Rules

Pipeline

1. Caption last. Lock the cut and the audio mix first; caption timings must match the final file.
2. Extract the speech: ffmpeg -i cut.mp4 -vn -ar 16000 -ac 1 captions/vo.wav.
3. Get word timings (table below). Save the JSON under ./video-out/<slug>/captions/.
4. Proofread the words against the script or brief. Fix single-word misspellings with --replace Wrong=Right (repeatable); fix multi-word errors by editing the word/text fields in the JSON and keeping the timings.
5. Check the font (Fonts below): the default family ships with the captions capability.
6. Render and burn:

video-captions captions/vo.json --video cut.mp4 -o captions/captions.ass \
  --preset pop --srt captions/captions.srt --burn cuts/master-9x16.mp4

7. Extract three frames at word-dense moments (ffmpeg -ss 2.4 -i out.mp4 -frames:v 1 check-2.4.png) and look at them with read_media_file: face clear, text inside the safe zone, highlight on the spoken word. Adjust --y, --max-words, or the preset and re-render. If an image can't be read, rely on the script's font and safe-zone warnings, and tell the user which frame files to check and that placement is unverified.

Without --burn the script prints the exact burn command with the right ffmpeg binary; use that when the burn joins a larger filter chain (-vf "scale=...,ass=captions.ass:fontsdir=$HOME/.local/share/octomind/fonts").

Word-timing sources

The AssemblyAI MCP tools return transcript text only, so they can't time captions. An .srt input works but the script interpolates word timing inside each cue and warns; use it only when no audio is available.

Presets

Override per brand with --highlight "#RRGGBB", --color, --outline-color, --case, --max-words, --size. Keep one accent color per video and white text on a dark outline for contrast. Keep pages short: TikTok's creative guidance caps on-screen text at 5–10 words per second, and the default pages follow speech word by word.

Placement

Keep the block out of the 9:16 UI bands: Meta Reels reserves the top 14% and bottom 35%, TikTok the right-side action rail and a caption area that grows from the bottom. The script warns when a page reaches into those bands.

Captions and audio

• Audio leads, captions support. Over 75% of Reels views are sound-on (Meta, 2024-11, official), so the script must work when heard; captions mirror it for the sound-off minority.
• Don't write a hook that only works silently, and don't rely on the caption to deliver the hook the voice misses.
• For TikTok Search Ads, show keywords as on-screen text or overlays and put keywords in the title and description (TikTok Search Ads creative tips, 2026-01, official, ads-scoped). When the claim allows it, put the searchable topic keyword in the --hook line.

Fonts

The caption font must exist as a file; otherwise libass silently substitutes a plain fallback, the most common cause of ugly burns. The captions capability installs five heavy display families into ~/.local/share/octomind/fonts, and video-captions uses that directory by default, so --font works by family name:

All five are SIL Open Font License. For a brand font, put its .ttf/.otf in ./video-out/<slug>/fonts/ and pass --fontsdir plus --font with the font's internal family name; the script warns if it can't find the family. Use --bold only for fonts without a heavy cut.

Sidecars and other renderers

• --srt file.srt writes sentence-level cues for platform upload (YouTube, LinkedIn) alongside the burned captions.
• --remotion file.json writes Remotion Caption[] JSON. For Remotion compositions, feed it to createTikTokStyleCaptions() from @remotion/captions; in the create-video --tiktok template, save it as public/<video>.json next to public/<video>.mp4 instead of running the template's whisper.cpp step. Use the Remotion path when captions need emoji or motion libass can't draw.
• Burned ASS can't render color emoji; keep emoji out of burned captions.
• Non-Latin scripts are untested with the bold presets: render one frame first, use clean, and a font that covers the script (Noto Sans CJK, Noto Sans Arabic).

Examples

Example 1: SRT force_style burn

❌ ffmpeg -i ad.mp4 -vf "subtitles=captions.srt:force_style='Fontsize=44,Outline=2'" out.mp4 — one style for the whole file, sentence-length cues, no active-word highlight, and stock Homebrew ffmpeg fails outright (no libass).

✅ video-captions captions/vo.json --video ad.mp4 -o captions/captions.ass --preset pop --burn cuts/ad-captioned.mp4

Example 2: hand-rolled frame rendering

❌ A Python loop that draws text onto every frame with PIL, splitting the script evenly across the clip's duration. Timing drifts from speech, text has no shaping, and the render is slow.

✅ Transcribe the final audio with word timestamps, proofread, render with a preset. No custom drawing code.

Example 3: captions over the face

❌ --y 0.40 on a talking-head close-up: the caption block covers the mouth.

✅ Default --y 0.58, checked on an extracted frame; --y 0.62 if the speaker sits low in frame.

Checklist

• Captions rendered from word timings of the final cut's audio, not a draft and not interpolated SRT.
• Words proofread against the script; brand names spelled right.
• No font warning from the script (installed family, or a brand font passed with --fontsdir).
• Preset matches the format; one accent color.
• Three extracted frames checked (face clear, block inside the safe zone, highlight on the spoken word), or reported to the user as unchecked.
• captions.ass, captions.srt, and vo.json saved under ./video-out/<slug>/captions/.
• Script works heard; captions mirror the voice rather than carry the hook alone.
• Searchable topic keyword in the on-screen hook text when the claim allows it.

Composition / References

• video-spec-sheet — platform safe zones, export presets, disclosure rules.
• video-editing — where captions sit in the edit pipeline (after the cut and the audio mix, before the final encode).
• video-hooks — the hook line passed to --hook.
• ASS override tags (Aegisub)
• ffmpeg ass and subtitles filters
• TikTok creative best practices
• Remotion captions
• Meta for Developers: Reel ads — over 75% of Reels views sound-on (official, 2024-11).
• TikTok Search Ads creative tips — keywords as on-screen text or overlays; keywords in title and description (official, ads-scoped, 2026-01).