Video Localization

Overview

Localize one video into another language with reproducible, auditable, real deliverables. The workflow favors verifiable artifacts over black-box automation or studio-grade promises.

Core principle: every step must leave evidence. Save commands, failures, fixes, transcripts, scripts, generated audio, subtitles, timing reports, and final outputs in a work directory next to the source video. Do not claim completion from prompt scaffolds, placeholder audio, total-duration-only alignment, or theoretical steps.

Default dubbing mode:

• Use reference-audio voice cloning unless the user explicitly requests standard TTS or no authorized reference audio is available. Prefer a user-supplied reference clip; otherwise extract a clean 3-15 second single-speaker clip from the source video when rights and audio quality are acceptable. If voice cloning cannot be used, fall back to standard target-language TTS and record the reason in notes/issues.md or report.md.

Supported modes:

• Reference-audio voice cloning: default mode; use authorized reference audio. Default to a clean 3-15 second single-speaker clip extracted from the source video only when appropriate; switch to user-provided reference audio if supplied.
• Standard target-language dubbing: fallback or explicit mode; synthesize narration with a TTS voice that explicitly supports the target language.

Out of scope: batch pipelines, automatic voiceprint purification, voice authentication, lip sync, rights bypassing, or guaranteeing 100% speaker replication.

When to Use

Use this when:

• User provides a YouTube URL, web video, or local video and asks for translation, dubbing, subtitles, or localized video output.
• The task needs source-language transcription, target-language narration, sentence-level sync, bilingual subtitles, or title-named final files.
• User asks for voice cloning with a clear reference-audio source and acceptable usage rights.
• Manual script review is needed before TTS to avoid literal or awkward translation.

Do not use this when:

• Processing many videos as a batch system.
• The source, subtitles, translated script, or reference audio lack clear processing permission.
• User requests unaudited processing with no intermediate artifacts.
• The goal is lip-sync, identity verification, or impersonation without authorization.

Language and Naming Conventions

Confirm the language direction before producing final artifacts:

Use ISO 639-1, BCP-47, or clear language names consistently. For edge-tts, use a concrete voice locale such as zh-CN-XiaoxiaoNeural, en-US-AriaNeural, fr-FR-DeniseNeural, or ja-JP-NanamiNeural; run edge-tts --list-voices to choose a voice matching {tgt_lang}.

Work Directory Convention

Create an isolated work directory next to the source video. The directory name is the source filename stem:

/path/to/Source-Video.mp4
/path/to/Source-Video/
/path/to/Source-Video/output/

Internal layout:

/path/to/Source-Video/
  source/       # normalized inputs and reference audio/text
  artifacts/    # transcripts, scripts, TTS, subtitles, intermediate video
  notes/        # commands, issues, manual checks, constraints
  output/       # user-facing final files
  scripts/      # copied skill scripts
  report.md     # final summary

Keep shared .venv/ and models/ in the source video's parent directory, not inside the per-video work directory, so multiple localization tasks can reuse dependencies and model weights. Ensure that parent-level .gitignore ignores .venv/ and models/.

Minimum artifacts:

Recommended Workflow

1. Acquire and verify source media

• For YouTube or web links, try yt-dlp first, but do not treat 429, bot verification, login checks, or CAPTCHA as ordinary parameter errors.
• If browser state, login, network requests, or CAPTCHA need inspection, use the repository's browser tooling rules.
• If browser access is also blocked, stop bypass attempts and ask the user for a local media or subtitle file.
• Record acquisition method, failures, final source path, and source permissions.

2. Normalize media inputs

• Use ffprobe to save metadata and confirm duration, video stream, and audio stream.
• Copy or transcode to stable paths: source/video.mp4 and source/audio.mp3.
• Quote paths with spaces. Prefer canonical internal paths for scripts.

3. Transcribe source audio

• Prefer faster-whisper for text and SRT.
• Specify --language/--source-lang when the source language is known; otherwise allow auto-detection.
• Sample-check transcript quality: names, numbers, technical terms, sentence breaks, and SRT block count.
• If model download fails, check proxy/cache first; do not fabricate subtitles.

4. Produce target-language script

• Write natural, concise, spoken target-language narration; preserve meaning over literal translation.
• For continuous narration, create artifacts/script.{tgt_lang}.txt.
• For sentence-level sync, create artifacts/script.{tgt_lang}.segments.txt with exactly one line per SRT block.
• Treat scaffold_rewrite_prompt.py output as a manual prompt scaffold, not the final script. Overwrite it with reviewed final target-language text before TTS.

5. Prepare reference audio for voice cloning

Default source-extracted reference:

ffmpeg -y -ss 00:00:02.640 -to 00:00:15.000 \
  -i source/video.mp4 \
  -vn -ac 1 -ar 24000 -c:a pcm_s16le \
  source/reference.wav

printf '%s\n' 'Exact spoken text for this reference clip.' > source/reference.txt

Reference audio should be single-speaker, low-noise, 3-15 seconds, no music/crosstalk/reverb, and paired with exact reference text unless using an explicit x-vector-only mode. Record time range or file path, source, authorization, quality constraints, and text source. If no authorized reference can be prepared, switch to standard TTS rather than fabricating clone quality.

6. Generate real target-language TTS

• Default mode: use Qwen3-TTS or another authorized zero-shot voice-cloning backend with source/reference.wav and source/reference.txt.
• Fallback or explicit standard mode: use edge-tts or another backend that supports {tgt_lang}.
• Do not use a backend or voice trained for a different target language. Wrong pronunciation, romanization/pinyin output, garbled speech, or silence means the backend choice failed.
• Silent, blank, corrupt, or placeholder files are not completion.
• Verify generated audio is playable, non-silent, reasonable duration, and usable by ffmpeg.

7. Align to source timeline

• For sentence-level sync, do not stretch one full narration to the total video duration.
• Use each SRT block start time as an anchor and place generated target-language segments on the original timeline.
• If a target segment is too long, first apply bounded speed-up, then trim only if necessary. Save and report speed-up/truncation data.

8. Generate subtitles

• Produce standalone subtitles before deciding whether to burn them into video.
• Bilingual subtitles must reuse source SRT timing and pair each source block with its matching target-language segment.
• Distinguish external subtitles, hard subtitles, and soft subtitles in the report.
• Name subtitle files with {lang_pair} (e.g. subtitles.en-zh.ass) so multiple language pairs can coexist without filename collision.

9. Compose final video

• Replace or overlay the source audio track with the dubbed audio.
• Unless the user specifies another codec, encode deliverable MP4 files with H.264 video and AAC audio: -c:v libx264 -c:a aac.
• Use ffprobe to confirm final MP4 contains both video and audio streams.
• Export final named files into output/; do not leave only internal artifacts/final*.mp4 files.

10. Package and report

• Copy the clean video, subtitle file, and burned subtitle version to output/ using user-requested names.
• Use {lang_pair} in subtitle and bilingual video names; use {tgt_lang} in the clean dubbed video name.
• report.md must describe only commands and artifacts that actually ran in the current environment.
• Include limitations: transcription quality, TTS backend, voice-clone reference quality, speed-up/truncation, network/model constraints.

Bundled Scripts

Copy scripts into the work directory's scripts/ folder before running so the task archive is self-contained.

Common setup and fallback standard-TTS commands:

python scripts/transcribe_with_faster_whisper.py \
  --audio source/audio.mp3 \
  --txt-out artifacts/transcript.{src_lang}.txt \
  --srt-out artifacts/transcript.{src_lang}.srt \
  --language {src_lang}

python scripts/scaffold_rewrite_prompt.py \
  --transcript artifacts/transcript.{src_lang}.txt \
  --output artifacts/script.{tgt_lang}.txt \
  --source-lang <SourceLanguageName> \
  --target-lang <TargetLanguageName> \
  --mode continuous

python scripts/generate_edge_tts.py \
  --script artifacts/script.{tgt_lang}.txt \
  --voice <voice-id-for-{tgt_lang}> \
  --mp3-out artifacts/narration.{tgt_lang}.mp3 \
  --wav-out artifacts/narration.{tgt_lang}.wav

python scripts/build_aligned_dub.py \
  --srt artifacts/transcript.{src_lang}.srt \
  --tgt-segments artifacts/script.{tgt_lang}.segments.txt \
  --video source/video.mp4 \
  --backend edge-tts \
  --voice <voice-id-for-{tgt_lang}> \
  --wav-out artifacts/narration.{tgt_lang}.aligned.wav \
  --report-out artifacts/narration.{tgt_lang}.aligned.json \
  --segment-dir artifacts/aligned-segments

python scripts/build_bilingual_ass.py \
  --srt artifacts/transcript.{src_lang}.srt \
  --tgt-segments artifacts/script.{tgt_lang}.segments.txt \
  --lang-pair {src_lang}-{tgt_lang} \
  --ass-out artifacts/subtitles.{src_lang}-{tgt_lang}.ass

Default voice-cloned aligned dubbing:

python scripts/generate_qwen3_voice_clone.py \
  --script artifacts/script.{tgt_lang}.txt \
  --reference-audio source/reference.wav \
  --reference-text source/reference.txt \
  --language <TargetLanguageName> \
  --mp3-out artifacts/narration.{tgt_lang}.clone.mp3 \
  --wav-out artifacts/narration.{tgt_lang}.clone.wav

python scripts/build_aligned_dub.py \
  --srt artifacts/transcript.{src_lang}.srt \
  --tgt-segments artifacts/script.{tgt_lang}.segments.txt \
  --video source/video.mp4 \
  --backend qwen3-tts \
  --reference-audio source/reference.wav \
  --reference-text source/reference.txt \
  --language <TargetLanguageName> \
  --model-id ../models/Qwen3-TTS-12Hz-0.6B-Base \
  --segment-dir artifacts/aligned-segments-clone \
  --wav-out artifacts/narration.{tgt_lang}.clone.aligned.wav \
  --report-out artifacts/narration.{tgt_lang}.clone.aligned.json

# Compose clean localized video with default delivery codecs
ffmpeg -y \
  -i source/video.mp4 \
  -i artifacts/narration.{tgt_lang}.clone.aligned.wav \
  -map 0:v:0 -map 1:a:0 \
  -c:v libx264 -c:a aac -shortest \
  output/{title}.{tgt_lang}.mp4

# Burn bilingual ASS subtitles with the same default delivery codecs
ffmpeg -y \
  -i output/{title}.{tgt_lang}.mp4 \
  -vf "ass=artifacts/subtitles.{src_lang}-{tgt_lang}.ass" \
  -c:v libx264 -c:a aac \
  output/{title}.{src_lang}-{tgt_lang}-bilingual.mp4

Scripts accept compatibility aliases: --translated-segments, --target-segments, and --tgt-segments; --language, --target-lang, and --target-language; --source-lang and --source-language.

Note: --source-lang / --target-lang / --language for scaffold_rewrite_prompt.py and generate_qwen3_voice_clone.py accept language names (e.g. English, Chinese, French). --language for transcribe_with_faster_whisper.py and build_aligned_dub.py accepts ISO 639-1 codes (e.g. en, zh, fr).

Minimal Toolchain

• ffmpeg / ffprobe: media extraction, conversion, stream verification, final composition.
• Python virtual environment in the parent directory.
• faster-whisper: transcription and SRT generation.
• edge-tts: standard TTS for supported locales.
• qwen-tts, PyTorch, soundfile: reference-audio voice cloning.
• numpy: segment mixing and WAV timeline construction.
• modelscope: recommended fallback for Qwen model weights when Hugging Face downloads fail.

Tesla P4 note: P4 compute capability is 6.1 and may require Python 3.12 plus torch==2.4.1+cu121. Verify CUDA with a minimal tensor script before Qwen inference. P4 does not support flash-attn; expect slow per-segment synthesis. Segment caching allows resume after interruption.

Troubleshooting

Red Flags

Do not claim completion if any are true:

• Source/target language direction is unclear.
• Only a prompt scaffold exists; final target-language script has not been reviewed.
• Default voice cloning was skipped without a recorded reason.
• TTS backend does not support the target language, or output is silent, blank, corrupt, garbled, romanized/pinyin, or placeholder.
• Voice cloning is claimed without authorized reference audio and quality notes.
• Only total-duration alignment was done but sentence-level sync is claimed.
• Final MP4 was not verified with ffprobe for video and audio streams.
• Final MP4 was delivered without user-requested codecs, or without default H.264/AAC when codecs were unspecified.
• notes/commands.md and notes/issues.md do not record real commands, failures, fixes, and constraints.
• Final user-requested files are missing from output/.

Pre-Delivery Checklist

• {src_lang}, {tgt_lang}, {lang_pair}, output filenames, and subtitle requirements are clear.
• Work directory is next to the source video and contains source/, artifacts/, notes/, output/, scripts/, and report.md.
• Source video/audio, transcript, reviewed target script, TTS audio, subtitles, timing report, and final video have documented paths.
• Key commands are written to notes/commands.md.
• Blockers, fallbacks, quality limits, and authorization notes are written to notes/issues.md or report.md.
• If sentence-level sync was requested, timing report records speed-up/truncation.
• If voice cloning was used, reference audio source, quality, reference text, and authorization are recorded in notes/issues.md or report.md.
• If voice cloning was not used, the reason and standard-TTS fallback are recorded.
• Final MP4 passed ffprobe video/audio stream verification.
• Final MP4 uses user-requested codecs, or default H.264 video and AAC audio when unspecified.
• All requested final filenames and subtitle formats exist in output/.