Video Thumbnail Extraction

Overview

Automatically scan a local MP4 video (or YouTube URL via yt-dlp) and extract the 4 most visually interesting frames — prioritizing expressive faces (laughing, shocked, smiling) and engaging presentation slides. Outputs full frames, face crops, and background-removed transparent PNGs ready for compositing.

When to Use

• Before creating YouTube thumbnails (feeds into youtube-thumbnails skill)
• When you need the best screenshot from a long video recording
• When compositing a thumbnail and need transparent guest cutouts
• Processing Zoom gallery recordings, interviews, or presentations

Dependencies

Python packages (install once)

# In sandbox (Cowork VM):
pip install opencv-python scenedetect deepface pillow numpy --break-system-packages

# On host Mac (for background removal — sandbox can't download the model):
pip3 install 'rembg[cpu]' pillow --break-system-packages

System tools

• ffmpeg (usually pre-installed)
• python3 (3.10+)
• yt-dlp (optional, for YouTube URLs): pip install yt-dlp --break-system-packages

Model downloads (first run)

• DeepFace expression model (~1MB): Downloads automatically on first use. If blocked by proxy, expression detection falls back to OpenCV smile cascade (still effective).
• rembg u2net model (~176MB): Downloads on first use. Must run on host Mac if sandbox blocks GitHub releases.

Pipeline Architecture

Two-Pass Design (memory-efficient)

Pass 1 — Quick Scan (OpenCV only, no deep learning)

• Sample a frame every 10 seconds across the video
• Skip first/last 60 seconds (intro/outro)
• For each frame:
  • Detect faces via Haar cascade (fast, no GPU needed)
  • Detect smiles within face regions
  • Compute visual variance (proxy for "interesting" content)
  • Detect presentation slides (high edge density + low color saturation)
• Score each frame based on: face count, smile count, smile size, visual variance
• Select top 12 diverse candidates using quadrant system: divide video into N time segments, pick best from each → ensures temporal spread

Pass 2 — Deep Analysis (DeepFace, only on top 12 candidates)

• Re-read only the selected frames from video
• Run DeepFace emotion detection (happy, surprise, fear, sad, angry, disgust, neutral)
• Weight emotions by thumbnail value: happy > surprise > fear > angry > sad > neutral
• Combine Pass 1 score with expression score
• Final selection: divide candidates into N time segments, pick best from each → guarantees spread across the full video

Pass 3 — Output (rembg, only on final 4 frames)

• Save full frame as JPG (95% quality)
• Crop largest detected face with generous padding (0.5x)
• Run background removal on face crop → transparent PNG
• Generate manifest JSON with metadata

Scoring Heuristics

Temporal Diversity Algorithm

The pipeline enforces temporal spread to avoid clustering picks in one segment:

1. Quadrant selection (Pass 1 → Pass 2): Divide video duration into N segments, pick the highest-scoring frame from each segment
2. Segment-forced selection (Pass 2 → Final): Divide top candidates into top_n equal time segments, pick best from each
3. Fallback: If any segment is empty, fill from overall top scores

This ensures a 76-minute video yields picks from different parts (e.g., 1:00, 2:10, 21:50, 48:50) rather than clustering in the most face-heavy section.

Usage

Command Line

python3 thumbnail_extractor.py <video_path> [output_dir] [top_n]

Arguments:

• video_path — Path to MP4 file (required)
• output_dir — Where to save outputs (default: ~/Downloads/thumb_candidates)
• top_n — Number of candidates to extract (default: 4)

Examples:

# Basic — extract 4 best frames
python3 thumbnail_extractor.py "GMT20260130-210038_Recording_gallery_2380x1544.mp4"

# Custom output dir and count
python3 thumbnail_extractor.py recording.mp4 ./thumbs 6

# YouTube video (download first)
yt-dlp -o "video.mp4" "https://youtube.com/watch?v=..."
python3 thumbnail_extractor.py video.mp4

Output Files

For each candidate, the pipeline generates:

Naming example: GMT20260130-210038_3_happy_2-10_full.jpg

• GMT20260130-210038 — video name (truncated for Zoom recordings)
• 3 — candidate number (ranked by score)
• happy — detected dominant emotion
• 2-10 — timestamp (2 minutes 10 seconds)
• full / face / transparent — file type

Manifest JSON Structure

{
  "video": "GMT20260130-210038",
  "candidates": [
    {
      "index": 1,
      "timestamp": "2:10",
      "timestamp_sec": 130.0,
      "emotion": "happy",
      "emotion_score": 0.85,
      "combined_score": 12.4,
      "num_faces": 3,
      "is_presentation": false,
      "files": {
        "full": "..._full.jpg",
        "face_crop": "..._face.jpg",
        "transparent": "..._transparent.png"
      }
    }
  ]
}

Background Removal (Separate Step)

Since the Cowork sandbox may block model downloads, run rembg on the host Mac:

# On host Mac (via osascript or Terminal)
cd ~/Downloads/thumb_candidates
python3 -c "
from rembg import remove
from PIL import Image
import glob, os

for f in sorted(glob.glob('*_face.jpg')):
    out = f.replace('_face.jpg', '_transparent.png')
    print(f'Processing {f}...')
    img = Image.open(f)
    result = remove(img)
    result.save(out)
    print(f'  -> {out} ({os.path.getsize(out)//1024}KB)')
"

This takes ~10-15 seconds per image on Apple Silicon. The u2net model downloads automatically on first run (~176MB).

Integration with Other Skills

Feeding into youtube-thumbnails

After extraction, use the transparent PNGs as compositing elements:

1. Pick the best face cutout from the candidates
2. Use it in the Gemini thumbnail prompt as a reference, or
3. Composite it manually onto the generated Gemini background using ImageMagick:

# Composite transparent face onto Gemini-generated background
convert gemini_background.jpg transparent_face.png \
  -gravity southeast -geometry +50+50 \
  -composite final_thumbnail.jpg

Pipeline flow

[Video MP4] → thumbnail-extraction → [face crops + transparent PNGs]
                                          ↓
                                   youtube-thumbnails → [Gemini background]
                                          ↓
                                   [Composite final thumbnail]

Tuning Parameters

Edit these at the top of thumbnail_extractor.py:

For short videos (<10 min), consider SAMPLE_INTERVAL_SEC=5 for finer coverage.

Troubleshooting

• OOM / killed process: The v2 pipeline never holds more than 1 frame in memory during Pass 1. If still OOM, increase SAMPLE_INTERVAL_SEC to 15-20.
• All emotions "neutral": DeepFace model couldn't download (proxy block). Pass 1 smile detection still works — look at the num_smiles field in the manifest.
• Face crop is wrong person: The pipeline picks the largest detected face. In screenshare mode, this may be a profile picture rather than a webcam face. Check the full frame to verify.
• No faces detected: Zoom gallery recordings with "shared screen with gallery view" work best. Solo speaker view may have the face too close/large for the cascade detector — try lowering ANALYSIS_SCALE to 0.3.
• Background removal artifacts: rembg's u2net can produce halos around hair. For cleaner results, try the u2net_human_seg model: remove(img, model_name='u2net_human_seg').
• Slow processing: A 76-minute video takes ~2 minutes for Pass 1, ~15 seconds for Pass 2 (12 candidates), and ~60 seconds for bg removal (4 faces). Most time is in Pass 1 scanning.