Video Analysis Workbench

Understand a video with a fixed loop: probe the environment → probe the input → extract evidence → read the evidence → verify before answering. The scripts don't understand anything — they turn a video into frames and a transcript that you (Claude) read. Every claim must be grounded in a frame timestamp or a transcript segment. Never assume; always look.

Local and free is a hard constraint. Transcription runs on a local Whisper install; frame understanding is your own vision. Do not call any cloud transcription/vision API, and do not fall back to guessing from the filename or a thumbnail. If a required piece is missing, relay the install hint and stop.

The core loop

1. Probe the environment (once per session)

Before the first analysis in a session, run:

node "${CLAUDE_PLUGIN_ROOT}/skills/video-analysis/scripts/probe_env.mjs"

(${CLAUDE_PLUGIN_ROOT} is exported by Claude Code for plugin skills. If it's unset — e.g. running this skill outside the plugin runtime — substitute this skill's own directory.)

It prints JSON: ffmpeg/ffprobe versions, which local Whisper backends are installed, the preferred_transcriber (whisper.cpp first), any downloaded whisper.cpp models, and install_hint when something is missing. Rules:

• No ffmpeg → relay install_hint.ffmpeg and stop.
• transcription_available: false → you can still do visual-only analysis (frames), but say transcription is unavailable and relay install_hint.whisper so the user can enable speech. Do not invent narration.
• Never invoke a backend the probe didn't confirm.

2. Probe the input

ffprobe -v error -print_format json -show_format -show_streams "input.mp4"

Read duration (drives sampling density), resolution (drives frame --width — small frames make on-screen text unreadable), whether there's an audio stream at all (no audio → skip transcription, analyse visually), and fps. A 20-second clip and a 90-minute lecture demand very different plans.

3. Extract the evidence

Two independent artifacts. Run whichever the task needs (usually both).

Frames — timestamped stills you will look at:

node "${CLAUDE_PLUGIN_ROOT}/skills/video-analysis/scripts/extract_frames.mjs" "input.mp4" \
  --outdir ./analysis/frames --mode scene --scene-threshold 0.4 --max-frames 60 --width 640 --montage

Modes: scene (default — one frame per visual change; best for most content), interval (every N seconds — best for talking-head/screencast with few cuts), keyframe (I-frames). It writes a frames.json manifest pairing every file with its exact timestamp, and (with --montage) a contact sheet. See references/frame-sampling.md for density heuristics and reading on-screen text.

Transcript — timestamped speech:

node "${CLAUDE_PLUGIN_ROOT}/skills/video-analysis/scripts/transcribe.mjs" "input.mp4" \
  --outdir ./analysis/transcript --backend auto --model base.en --language auto

--backend auto picks the preferred installed backend. It normalises every backend to transcript.json (segments with start/end/text, plus speaker when diarized), transcript.srt, and transcript.txt. A silent source short-circuits cleanly (has_audio: false). See references/transcription.md for models, languages, diarization, and accuracy.

4. Read the evidence (this is where understanding happens)

Read the montage first for a whole-video overview, then Read individual frames at the timestamps that matter — use the Read tool on the JPEGs; you see them as images. Read transcript.txt/transcript.json and correlate segments with frames by timestamp. Then answer.

• Ground every claim. Cite a frame time (at 02:14 the screen shows…) or a transcript segment. If you can't point to evidence, don't assert it.
• Ambiguous? Extract more — denser sampling or higher --width at the relevant window — rather than guessing. It's cheap.
• Respect the context budget. Don't Read 200 frames. Overview → zoom into the relevant span. See references/reading-frames.md.

5. Verify before you answer

node "${CLAUDE_PLUGIN_ROOT}/skills/video-analysis/scripts/verify_analysis.mjs" \
  --frames-manifest ./analysis/frames/frames.json --expect-frames-min 3 \
  --transcript ./analysis/transcript/transcript.json --expect-transcript

Confirms frames exist and decode, timestamps are monotonic, and the transcript is non-empty (when the source had audio). Exit 0 = trustworthy evidence. Only then present the summary/answer, and report what it's based on (frame count, transcript length, backend used) — not just the conclusion.

Non-negotiable rules

• Stay local and free. Local Whisper for speech; your own vision for frames. No cloud APIs, no API keys, no uploading the video anywhere. This is a privacy feature — safe for confidential/internal recordings — so preserve it.
• Evidence or silence. Every statement about content maps to a frame timestamp or a transcript segment. No filename-based guessing, no inventing what "probably" happens.
• Read on-screen text at adequate resolution. UI, code, slides, terminals: extract higher---width frames at those timestamps; a 640px frame won't render small text legibly.
• No audio ≠ failure. Report it, analyse visually, don't fabricate narration.
• Sample smart on long videos. Probe first; do a sparse overview pass before dense extraction. Cap frames (--max-frames) to stay within context.
• Quote every path. Assume spaces and unicode. Write artifacts to a work dir (e.g. ./analysis/), never into the user's media folder unless asked.
• Don't transform — that's the other skill. Compress, convert, clip, make a GIF, burn/attach subtitles → ffmpeg-workbench. Frame extraction here is a means to understanding, not a deliverable.

Recipe references — read the one that matches the task:

Batch jobs

For "analyse every video in this directory": probe one representative file, validate the whole pipeline on it end-to-end (extract → read → verify), then loop over the rest, giving each its own --outdir, running verify_analysis.mjs on each, and reporting a per-file pass/fail summary. Continue past individual failures; list them at the end.

Scope boundary

This skill understands media; it does not change it. Extracting frames and transcripts is how it reads a video — the outputs are evidence, not products. If the user wants a smaller file, a different format, a clip, a GIF, or subtitles burned in, that's ffmpeg-workbench. If the user wants to know what's in the video, you're in the right place.