Generating video clips (still to clip, then lip-sync)

Overview

A finished still becomes a 4-9 s clip with an image-to-video model. Then, if a character talks on screen, a lip-sync model re-times their mouth to your own dialogue. You cannot watch the result: judge every clip from extracted frames (scripts/take_sheet.py, and the viewing-videos skill), and let the human have the final say on motion.

Requirements: Python 3 (stdlib only) and ffmpeg/ffprobe. Keys go in env vars: GEMINI_API_KEY (a Google AI Studio key; an image-scoped one works, and GEMINI_IMAGE_API_KEY is read as a fallback) for Omni, and FAL_KEY (the full key exactly as fal shows it) for lip-sync. Never print them.

Choose the model

For lip-sync, Sync 3 on fal (fal-ai/sync-lipsync/v3) is the default. It works video-to-video, so it keeps the clip's motion and re-times only the mouth. Comparison and prices: references/providers.md.

Animate a still

python3 <this-skill-dir>/scripts/omni_generate.py jobs.json clips/ [--takes 3] [--only S05]

{"base": "Bring this still to life as it is: same art style, palette, light and layout, and every character keeps the face, hair, build and clothes they have in it. Natural, fluid movement. Nothing written on screen, no speech bubbles, nobody new, one continuous shot. ",
 "jobs": {"S05": {"image": "panels/p05.png",
                  "prompt": "[0-3s] The woman in the red jacket lifts the phone and talks, excited. [3-6s] The friend in the green hoodie leans back on the sofa and laughs. Slow camera push-in."}}}

How to write the prompt (full guide with examples in references/gemini-omni.md):

• A per-second timeline, one beat per bracket. The clip runs about as long as the timeline.
• Name people by what they wear, never by name: the model doesn't know who "Dana" is.
• Say "talks" for anyone who will be lip-synced, so the mouth already moves.
• Pin what must not change: "the camera stays still", "one single window on the left wall", "exactly ONE hooded figure", "her hands stay flat on the table". Each pin in references/failure-modes.md came from a real failure.

Review every take before using it

python3 <this-skill-dir>/scripts/take_sheet.py sheet review/S09.jpg clips/S09_t1.mp4 clips/S09_t2.mp4 clips/S09_t3.mp4 --times 0.3,1,2,3,4,5,6

Read the sheet: rows are takes, columns are times. Look for props that double, doors or people that appear, hands crossing bodies, faces that drift, and a gesture done twice. Name the exact second of each problem, because the fix depends on where it is.

A hard cut is easy to miss between sampled frames. List them directly; no output means one continuous shot:

ffmpeg -i clips/S09_t1.mp4 -vf "select='gt(scene,0.3)',showinfo" -an -f null - 2>&1 | grep pts_time

Fit a clip to its shot

An edit needs a clip that fills its shot exactly, at the edit's frame rate:

lipsync_parts.py normalize clip.mp4 shot.mp4 --frames N --off S --speed K does the offset, speed, fps, scale (cover and center-crop, never stretch) and last-frame hold in one step, and always returns exactly N frames.

Lip-sync dialogue

1. Normalize the clip to the shot: python3 <this-skill-dir>/scripts/lipsync_parts.py normalize clip.mp4 shot.mp4 --frames 132 --off 1.5 --speed 1.22.
2. Split at line boundaries so each part has one speaker.
   • Write shot.json: each line with its start/end in the shot (seconds) and its speaker; frames equals the --frames you normalized to. Add "skip": true for a speaker seen from behind, off screen, or a crowd shout:

     {"id": "S05", "video": "shot.mp4", "fps": 24, "frames": 132,
      "lines": [{"id": "05-1", "audio": "lines/05-1.mp3", "start": 0.72, "end": 2.31, "speaker": "maya"},
                {"id": "05-2", "audio": "lines/05-2.mp3", "start": 2.43, "end": 4.10, "speaker": "ido"}]}

   • Run lipsync_parts.py split shot.json parts/. Besides the parts it writes parts/S05_lipsync_jobs.json: one entry per non-skip part, with absolute paths, out already named the way join expects (synced/S05_<k>.mp4), and face: null.
3. Point at the speaker in each part, and write it into that jobs file as "face": "x,y@<mid>" (mid is in each entry).
   • Pick a pixel on their face on the part's middle frame.
   • Confirm it: take_sheet.py point parts/S05_1.mp4 check.jpg --frame 30 --xy 410,560, then read the image.
   • Face detectors are unreliable on drawn faces, so pick and confirm by eye. Leave null only when one face is visible.
4. Sync: lipsync_fal.py --jobs parts/S05_lipsync_jobs.json. Expect ~$0.13 per second and ~70-150 s per part; 6 run in parallel. Existing outputs are skipped. To redo one part after correcting its face, run that part alone: lipsync_fal.py parts/S05_1.mp4 parts/S05_1.wav synced/S05_1.mp4 --face 410,560@30 --force (--force with --jobs would redo every part).
5. Join: lipsync_parts.py join parts/S05_plan.json synced/ final/S05.mp4. Each part is forced to its exact frame count, skipped parts keep their original frames, a part with no synced file is reported (--strict makes that an error), and the total frame count is checked.

Details, model choice and failure cases: references/lipsync.md.

Costs and times (measured 2026-10)

Common mistakes

Related skills

• image-generation: the stills (character cards, panels).
• elevenlabs-tts: the dialogue the mouths sync to.
• viewing-videos: frame extraction for reviewing clips.
• producing-animated-episodes: the whole episode workflow that uses these clips.