sync-narration
Skill3 starsChanged 4 months agoMITView on GitHub
Syncs a separately-recorded narration track onto a silent screen recording — pads audio with calculated pauses to match on-screen pacing (no voice during cursor movement), then burns styled captions. Use when the user provides a silent screen-recorded video plus a narration .mp3 and asks to sync/combine/caption them.
- Installs packages
In this repository4 other files it gives its agents.
- demo-setupSkill
- film-demoSkill
- narrateSkill
- produce-videoSkill
The file
We split this skill into the rules it gives an agent
12 rules. About 0 are common, 4 are distinctive, and 8 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈4
Distinctive · estimate
This author's own choices. Worth a look.
8
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 84
Default model is base (~150 MB, one-time download) — its word boundaries are noticeably tighter than tiny.
- Very specific: line 102
Do not: use silencedetect alone for caption timing — comma pauses inside a sentence are not sentence boundaries.
- Very specific: line 138
Read the grid (cell i ≈ i×2 s).
- Distinctive: line 150
If a beat requires both travel *and* a click, speak after the click completes and the UI has updated — never over the travel or click …
- Distinctive: line 153
When in doubt, pause 0.5–1 s longer before the line rather than talking over cursor motion.
- Distinctive: line 202
Only cut when you have verified the removed section adds nothing:
- Very specific: line 220
When in doubt, ship the full recording.
- Very specific: line 232
Read the printed timeline.
- Distinctive: line 238
Verify sync and payoff — mandatory, on the FINAL render:
- Very specific: line 258
Default look (override any field via plan.style): full-width black bar at bottom (barColor: 'black@0.82'), 1px white top border, centered …
- Very specific: line 269
Only if Whisper cannot run.
- Very specific: line 282
Use textfile= for captions, never text= (apostrophes/colons break the graph).
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Sync narration onto a silent screen recording
Input: silent screen recording (from film-demo) + a narration .mp3 (from
narrate, or any TTS/recorded voice). Output: final .mp4 with paced
narration + burned-in captions.
Never ship with (a) captions out of sync with audio, (b) long video
stretches with no narration at the start or end — that reads as broken/empty,
(c) clipped TTS onsets or release tails (last syllable cut off — use
--seg-postroll, see below), or (d) narration playing while the cursor is
still moving — voice should land only after the pointer has settled on the
target element.
Quick start (copy-paste per video)
# 0 — once
pip install -r scripts/requirements.txt
# 1 — align audio to script (local Whisper, no API key)
python scripts/align-narration.py \
demos/<slug>/narration.mp3 \
--script-file script.txt \
--out /tmp/aligned.json
# 2 — contact sheet → pick segmentTargets (when each sentence should start on screen)
ffmpeg -y -i demos/<slug>/no-sound.mp4 -vf "fps=1/2,scale=384:-1,tile=5x6" -update 1 -frames:v 1 /tmp/contact.png
# 3 — write cues.txt (one caption line per readable cue, exact script wording, ≤ 60 chars)
# 4 — build plan (defaults: seg-preroll 0.12, seg-postroll 0.20, min-silence 0.22, max-cue-chars 80)
node scripts/build-plan-from-align.js \
--align /tmp/aligned.json --out /tmp/plan.json \
--video /tmp/work.mp4 \
--audio demos/<slug>/narration.mp3 \
--output demos/<slug>/final.mp4 \
--targets 0,2,8,… \
--cues-file cues.txt \
--tail-pad 1.0
# 5 — work video (default: full no-sound.mp4, no trim)
ffmpeg -y -i demos/<slug>/no-sound.mp4 -c:v copy -an /tmp/work.mp4
# Only trim if you verified dead footage — see "Video trim (conservative)" below
# 6 — validate + render
node scripts/sync-narration.mjs /tmp/plan.json --validate-only
node scripts/sync-narration.mjs /tmp/plan.json
# 7 — spot-check narration ↔ picture on the FINAL render (mandatory)
node scripts/spot-check.mjs /tmp/plan.json
# → read EVERY PNG in spot-check/: at each sentence start the screen must already
# show what the sentence talks about; during long gaps something must be happening.How the three layers fit together
| Layer | Tool | What it controls |
| When each word is spoken | align-narration.py (faster-whisper) | Word + sentence timestamps in the raw .mp3 |
| When each sentence lands on screen | segmentTargets in build-plan-from-align.js | Inserted gaps between sentences |
| When each caption appears | Caption start/end from word alignment | Burned subs locked to audio |
| Audio attack (no clipped onset) | --seg-preroll (default 0.12 s, silence-aware) | atrim starts slightly before Whisper word onset |
| Audio release (no clipped ending) | --seg-postroll (default 0.20 s, silence-aware) | atrim extends after last word; final segment runs to mp3 end |
| No cut inside a word | orphan-word absorption + --min-silence (default 0.22 s) | boundary words are absorbed into the earlier sentence; sentences without usable silence between them are merged into one segment (one target for the block) |
Gaps stretch narration to match UI pacing. Video length defaults to the full silent recording — the padded audio track is often shorter than the screen capture, and the tail of the video carries visual payoffs after the last spoken word.
1. faster-whisper alignment (no API key)
Install: pip install -r scripts/requirements.txt
Default model is base (~150 MB, one-time download) — its word boundaries are noticeably
tighter than tiny; clipped onsets (first letter of a word missing) usually trace back to
running tiny. Only drop to --model tiny when iterating fast on something non-final.
python scripts/align-narration.py narration.mp3 \
--script-file script.txt \
--out aligned.jsonOutput:
words[]—{ word, start, end }for every tokensentences[]— sentence blocks matched to the script
Use this for:
- Audio
segmentsboundaries (one per sentence — never split at comma pauses) - Caption cue
start/end(viabuild-plan-from-align.js+cues.txt)
Do not use silencedetect alone for caption timing — comma pauses inside a sentence are
not sentence boundaries.
Word-safe segment boundaries (fixes clipped attack/release AND mid-word cuts)
Whisper word timestamps lag the audible envelope on both sides (~50–350 ms), and its sentence
bounds sometimes leave a split word's tail token stranded between two sentences — an inserted gap
there cuts the word in half in the output. build-plan-from-align.js handles all of this
deterministically:
- Orphan-word absorption — any word that ends before the next sentence starts is absorbed into the earlier sentence. A sentence's audio runs until the next one begins; nothing is left in no-man's land.
- Tight-boundary merge (
--min-silence, default 0.22 s) — when the silence between two sentences is smaller than this, cutting there would clip the TTS release/attack, so the two sentences are merged into one audio segment. The script prints which sentences merged; pass one--targetsvalue for the merged block. - Silence-aware padding —
--seg-preroll(default 0.12) and--seg-postroll(default 0.20) extend each cut into the surrounding silence, but never more than half of it (the neighbouring sentence keeps its share) and the final segment always runs to the full.mp3duration.
The padded bounds are baked into plan.segments (segmentPreRoll/segmentPostRoll are 0
in the emitted plan) — sync-narration.mjs uses them as-is.
- Listen-check before ship: scrub the last 1 s of audio in the rendered file — the final word should decay naturally, not stop mid-syllable.
2. Contact sheet → segmentTargets
ffmpeg -y -i no-sound.mp4 -vf "fps=1/2,scale=384:-1,tile=5x6" -update 1 -frames:v 1 contact.pngRead the grid (cell i ≈ i×2 s). For each sentence in aligned.json, pick the output time
when that topic is visible — pass as comma list to --targets.
No voice during cursor movement
Do not start a sentence while the pointer is traveling. Narration competes with motion and reads as rushed or out of sync. For each sentence:
- Find the frame where the cursor lands on the element you are describing — not the frame where movement begins.
- Set
--targets[i]to that settled moment (add gap before the sentence if the raw.mp3would speak too early). - If a beat requires both travel and a click, speak after the click completes and the UI has updated — never over the travel or click animation.
When in doubt, pause 0.5–1 s longer before the line rather than talking over cursor motion.
3. Captions (cues.txt)
One line per readable cue — ≤ 60 characters preferred, 80 hard max (the plan builder fails
above --max-cue-chars, default 80; a longer line physically overflows the caption bar at the
default fontsize). Split long sentences at natural clause breaks; lines starting with only /
same get an em dash prefix in the plan builder.
Compound sentence = one audio segment, multiple caption lines. Whisper word times set each
cue's [start,end] inside that segment.
4. Build plan + trim video
node scripts/build-plan-from-align.js \
--align aligned.json --out plan.json \
--video work.mp4 --audio narration.mp3 --output final.mp4 \
--targets 0,2,8,16,19.64,24.42,28 \
--cues-file cues.txt \
--tail-pad 1.0Prints: speech ends at Xs → trim video to Ys (tail-pad 1s) — plus any sentence merges
(then --targets needs one value per merged segment, not per script sentence).
Important: that expectedVideoDuration is the padded audio length, not a mandate to cut
the screen recording. If the silent video is longer, keep it — the extra tail is usually the
visual payoff after the last word (see validate warning: video is longer than built timeline → OK).
Plan fields:
segments—[start,end]in original mp3 time, word-safe padding already baked insegmentPreRoll/segmentPostRoll— 0 in emitted plans (padding baked intosegments)audioDuration— full.mp3length from alignment (used for final-segment tail)gaps— silence inserted between segments so each starts at--targets[i]trailGap—--tail-pad(default 1.0 s after last word)expectedVideoDuration— padded audio length (speech end + tail-pad); informational onlysegmentTargets/segmentTexts— echoed forspot-check.mjs
Video trim (conservative — default: no trim)
Start from the full silent video. The sync script accepts a video longer than the padded audio; output duration follows the video. After the last caption, the viewer should still see key beats from the beat script.
Only cut when you have verified the removed section adds nothing:
- Before any cut — re-read the beat script and every planned narration clause.
- Contact sheet — generate one for the full recording; map each beat to a cell/time.
- Decide — if a proposed cut removes any mapped beat, do not cut.
- Spot-check — grab frames at the proposed cut point and 2 s after the planned end:
ffmpeg -y -ss <t> -i no-sound.mp4 -update 1 -frames:v 1 check.png
| Situation | Action |
| Default | -c:v copy the full silent video → work.mp4; ignore expectedVideoDuration |
| Video longer than padded audio (validate warning) | Expected — keep the video; tail holds visual payoff |
| Dead tail after the last beat (cursor idle, blank screen, stop-recording artefact) | Trim only that dead tail (typically ≤ 1–2 s) |
| Dead head before the first beat (accidental pre-roll) | -ss head trim only if the first 2+ s contain no beat content |
trailGap > 1.5 s in validate output | Wrong gap/targets — fix --targets, do not inflate trailGap to fill dead video |
Tempted to trim to expectedVideoDuration to "tighten" runtime | Stop — check whether that removes a payoff first |
Never combine head trim + tail trim to match audio length without cross-checking every beat.
When in doubt, ship the full recording. Keep the silent master and .mp3 untouched — only the
final .mp4 ships.
5. Validate + render
node scripts/sync-narration.mjs plan.json --validate-only # mandatory
node scripts/sync-narration.mjs plan.jsonRead the printed timeline. Every caption window must sit inside a speech block. trail silence
should be ≤ --tail-pad (≈ 1 s).
A validate warning like video is longer than built timeline is normal when keeping the full
recording for post-narration visual payoffs — confirm the extra tail matches a beat, then render.
Verify sync and payoff — mandatory, on the FINAL render:
node scripts/spot-check.mjs plan.jsonIt extracts a frame at the start and end of every audio segment plus the midpoint of every
long gap, and writes spot-check/spot-check.json. Read every PNG:
- At a sentence's start frame, the screen must already show what the sentence talks about — never narrate a result ("it's saved") while a spinner is still visible.
- The pointer should be idle on the described element, not mid-travel.
- During a long gap frame, something must visibly be happening (otherwise tighten the gap).
Re-run build-plan-from-align.js with later --targets if any line starts too early.
Caption style & fonts
Default look (override any field via plan.style): full-width black bar at bottom
(barColor: 'black@0.82'), 1px white top border, centered monospace, no drop shadow.
sync-narration.mjs auto-detects a system monospace font (Windows: Consolas; macOS: Menlo;
Linux: DejaVu Sans Mono / Liberation Mono). If none is found on your system, set
plan.style.fontfile to an absolute path to a .ttf/.ttc file.
Manual plan.json (fallback)
Only if Whisper cannot run. Same rules: sentence segments only, gaps between sentences,
captions via { text, segment, start, end }, trailGap ≤ 1.5 s, segmentPreRoll 0.12 s,
segmentPostRoll 0.20 s (unless ending still clips), never cut where silence < 0.22 s,
trim video to match.
ffmpeg gotchas (cross-platform)
- Pass
-filter_complexviaspawnSyncargs — not-filter_complex_scriptwith raw drive paths on Windows. - Windows only: escape drive colons in filter values (
C\\:/Windows/...) —sync-narration.mjsdoes this automatically viaffPath(). - Use
textfile=for captions, nevertext=(apostrophes/colons break the graph).
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

