agentleFS
Sign inStart free
zuckenbertzuckenbert/video-from-code/skills/captions/SKILL.md

captions

Skill0 starsChanged 13 days agoMITView on GitHub

Burns captions into a spoken video and adds an animated lower third (name, role, optional logo). Transcribes locally with whisper.cpp (English, Portuguese or any whisper language), groups words into cues that never split a locked expression, paints highlight terms in an accent color, and composes everything at 1080p with ffmpeg. Outputs the MP4 plus a loose .srt. Use when the user asks to "caption this video", "add subtitles", "burn in captions", "add a lower third", "name card for the speaker", "legendar esse vídeo", "põe legenda", or hands over a talking-head clip to publish on LinkedIn, Reels or YouTube.

  • Installs packages

9 other projects use a version of this.

Adopted by

Most recently taken up by

The file

We split this skill into the rules it gives an agent

1 rule. 0 are common, 0 are distinctive, and 1 is very specific to this project.

  • 0

    Common

    Most of the 6 edited versions we read keep these.

  • 0

    Distinctive

    This author's own choices. Worth a look.

  • 1

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    Run stage 2 alone (with --skip-asr, it is the first thing printed) and read the cue list in the terminal before spending the render minutes.

    line 92

“Common” is measured: kept by at least 6 in 10 of the 6 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

captions

Local pipeline, no external service. Input: a spoken video. Output: a 1920x1080 MP4 with burned-in captions and an animated lower third, plus a loose .srt to upload alongside on LinkedIn or YouTube.

Usage

./scripts/run.sh -i talk.mp4 -o talk_captioned.mp4 \
                 -n "Ada Example" -r "Head of Product" -l en

Portuguese (or any whisper language code; auto lets whisper detect it):

./scripts/run.sh -i fala.mp4 -o fala_legendada.mp4 -n "Ana Exemplo" -r "Coordenadora" -l pt

With your own config (deep merged over defaults.json, see examples/):

./scripts/run.sh -i talk.mp4 -o out.mp4 -c examples/config.en.json

Reusing the transcription from a previous run (useful to tune style, highlights, locks and fixes without paying for whisper again):

./scripts/run.sh -i talk.mp4 -o out.mp4 -c my_config.json --skip-asr

Flags: -i input, -o output (default <input>_captioned.mp4), -n name, -r role, -l language, -c config, -w work dir (default .work/<video-name>/ inside the skill), --skip-asr.

Without a name (no -n and no card.name) the lower third is skipped and the captions stay at the low margin for the whole video.

Prerequisites

  • ffmpeg and ffprobe (brew install ffmpeg)

  • whisper-cli (brew install whisper-cpp) and a ggml-*.bin model. The default path is ~/.cache/whisper/ggml-large-v3-turbo.bin. Download it once (about 1.6 GB):

    mkdir -p ~/.cache/whisper && curl -L -o ~/.cache/whisper/ggml-large-v3-turbo.bin \
  https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-large-v3-turbo.bin

    Any other ggml model works: point whisper_model in your config at it.

  • Python 3 with Pillow (pip3 install pillow)

Heads up: Homebrew's ffmpeg usually ships without libass and without drawtext. That is why this pipeline does not use any ffmpeg caption filter: captions and card are drawn as RGBA PNGs with Pillow and composed with a single overlay. Side benefit: full control over kerning, outline, shadow and easing.

What each stage does

StageFileInput -> output
1run.sh (inline)video -> audio.wav -> words.json (whisper -ml 1, one timestamp per token)
2scripts/02_cues.pywords.json -> cues.json + captions.srt
3scripts/03_card.pyconfig -> card.png (RGBA with shadow)
4scripts/04_overlay.pycues.json + card.png -> frames/%05d.png
5run.sh (inline)video + frames -> final MP4

Intermediate files live in the work dir.

The two decisions that make the captions good

1. Grouping by dynamic programming, not greedy. A greedy version (fill the cue up to the limit) splits a two-word term across two cues and leaves its last word alone on screen. 02_cues.py solves a DP that minimizes a cost made of: distance from the ideal length, a penalty for breaking away from punctuation, and a penalty for cues that are too short.

2. Locked expressions. Everything in highlights or locked becomes a forbidden boundary: the DP cannot cut there, and the line break inside a cue also avoids that point. That is what keeps "August 10th", "6 in the evening", "public hearing" and proper names whole. Measured on the test clip with a tight max_cue of 30: without locks the DP produced "No dia 10 / de agosto" and "A rede de / ciclovias liga"; with locks it produced "No dia / 10 de agosto" and "A rede de ciclovias / liga".

Run stage 2 alone (with --skip-asr, it is the first thing printed) and read the cue list in the terminal before spending the render minutes. It is cheap and it is where problems show up. Highlighted runs print in [brackets].

Lock the form whisper actually writes, not the one you say: whisper writes numbers as digits ("10 de agosto", "6 in the evening"), so a lock on "six in the evening" never matches.

Config (defaults.json)

KeyWhat it does
langwhisper language (en, pt, ... or auto). -l overrides it
whisper_modelpath to the ggml model
highlightsterms painted in theme.accent, whole words only, case insensitive. Locked automatically
lockedexpressions that never split, even without highlight
fixtranscription corrections, lowercase word -> right form, applied word by word (trailing punctuation kept)
theme.accent / bar / background / text / mutedhighlights (Beacon #EF681E), card edge bar (Cobalt #1D4AA5; falls back to accent when empty), card background (Ink #12141A), caption and name text (Field #F3EFE5), role text (Slate #8195B0)
card.name / rolelower third text. -n and -r override them. Empty name = no card
card.logooptional PNG with alpha (white logo on transparent works best), drawn after a thin divider. Relative paths resolve from the skill root
card.in_at / out_atwhen the lower third enters and leaves (seconds)
caption.margin_low / margin_highcaption baseline outside and during the card window
caption.max_line / max_cue34 and 62 characters, calibrated for 58px at 1080p
caption.font, card.font_name, card.font_roleTTF paths

Look

Font: Manrope SemiBold (captions and name) and IBM Plex Mono Medium (role, a label), shipped in assets/fonts/ under the SIL Open Font License (assets/fonts/Manrope-OFL.txt, assets/fonts/IBMPlexMono-OFL.txt).

Colors: the default theme is the repo palette. The accent is reserved for the highlighted terms (the one thing to notice); the card edge is structure, so it uses theme.bar and never competes with a highlight.

Card: edge bar on the left, name, role, and an optional divider plus logo. It slides in from the left with a cubic ease-out over 0.55s, holds, and leaves with a fade plus a push to the left. While it is on screen the captions rise to margin_high so they never collide.

Known limits

  • Calibrated and tested on 16:9 sources rendered at 1920x1080. The input is scaled to out_w x out_h without letterboxing, so a source with a different aspect ratio gets stretched. out_w/out_h accept 9:16, but then max_line has to drop (around 22 characters) and the card has to shrink. Not validated in that format.
  • The PNG sequence takes about 100 MB per minute of video. The work dir can be deleted after the render.
  • Audio is normalized to -16 LUFS / -1.5 dBTP, the usual target for social networks. For another target, change the loudnorm in run.sh.
  • fix matches single words only.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

Check your setup, free