Vision Analysis

Interval-based and shot-level visual analysis: frame extraction, agent-driven image analysis, and timecoded text reports.

When to use

• This skill — extract frame sequences, inventory on-screen text/graphics with timecodes, agent-guided visual QC.
• forced-narrative-exact — exhaustive, frame-accurate burned-in dialogue inventory with exclusive-end SMPTE boundaries and completeness audit.
• shots — cut detection and midpoint stills when the unit of analysis is a shot.
• image — image.ocr for a single still without a sequence pipeline.
• captions-compliance — validate separate caption sidecar files (SRT/VTT), not burned-in text.

Output paths

All script outputs write to the workspace root .mediaskills/generated/ (not inside this skill folder). Frame JPGs, manifests, and reports share that directory across skills. Override with $MEDIASKILLS_DATA_DIR/generated/ when set.

Agent vs scripts

This skill does not call Ollama, OpenAI, or any local vision server. If you cannot view images, use the legacy OCR path or tell the user that vision analysis requires an agent with image understanding.

Text types (text_type)

Reports condense repeated identical text into single rows. Scene-description hallucinations in on_screen_text are filtered at report time. Use --force to regenerate reports after re-analysis.

Running scripts

This skill's scripts live in scripts/ alongside this file. Before running any script, resolve its absolute path from this skill's own directory — do not assume the shell cwd is the skill folder, the repo root, or the workspace. Stay in the workspace (so media paths resolve) and pass the absolute script path to uv run:

uv run <skill_directory>/scripts/extract_interval_frames.py --input "<path-to-media>"

<skill_directory> is a placeholder. Never pass those angle brackets to the shell — the command will fail. Replace it with a real absolute directory before invoking uv run.

How to get that directory (do this yourself; do not ask the user first; do not skip the skill):

1. If your tools showed a filesystem path when loading this SKILL.md, strip SKILL.md and use that folder.
2. If you already ran another mediaskills script (for example .../audio/scripts/extract.py), this skill is a sibling of that folder: same parent, directory name vision-analysis.
3. Otherwise search standard install locations and confirm the script exists:

NAME=vision-analysis
for d in \
  "$HOME/.agents/skills/$NAME" \
  "$HOME/.cursor/skills/$NAME" \
  "$HOME/.codex/skills/$NAME" \
  "$HOME/.claude/skills/$NAME" \
  "$PWD/.agents/skills/$NAME" \
  "$PWD/.cursor/skills/$NAME" \
  "$PWD/skills/$NAME"
do
  if [ -f "$d/SKILL.md" ] && [ -e "$d/scripts/extract_interval_frames.py" ]; then
    SKILL_DIR=$(cd "$d" && pwd)
    break
  fi
done
[ -n "$SKILL_DIR" ] || { echo "mediaskills: $NAME not found" >&2; exit 1; }
uv run "$SKILL_DIR/scripts/extract_interval_frames.py" --input "<path-to-media>"

Prefer absolute paths for --input / --output when cwd is uncertain.

Missing binaries vs permissions

Before running any script, treat these two failures differently:

• If a required binary (uv, ffmpeg, ffprobe, tesseract) reports No such file or directory or command not found: this is a missing or unreachable binary, not a permissions issue. Escalating sandbox permissions will not fix it. Run command -v uv, command -v ffmpeg, command -v ffprobe, and command -v tesseract to see whether they resolve at all. If not, install or re-link them (install-media-tools doctor/install) rather than retrying with wider permissions. tesseract is required for the legacy OCR path (compile_report.py); agent-vision reports still need ffmpeg/ffprobe/uv.
• Only escalate sandbox permissions if the error is explicitly a permission denial (e.g. Operation not permitted, Permission denied).

Recipe: interval frames for text / forced narrative

# 1. Extract 1 frame per second
uv run <skill_directory>/scripts/extract_interval_frames.py --input program.mp4

# 2. For each batch: get paths, analyze images yourself, write batch JSON
uv run <skill_directory>/scripts/get_frame_batch.py --manifest-path .mediaskills/generated/program_interval_frames.json --batch-index 0

# 3. After analyzing images in batch 0, merge your JSON
uv run <skill_directory>/scripts/merge_analysis.py \
  --manifest-path .mediaskills/generated/program_interval_frames.json \
  --frames-json batch0_analysis.json

# 4. Repeat step 2–3 for batch-index 1, 2, ... until all frames analyzed

# 5. Required validation (do not skip before reports)
uv run <skill_directory>/scripts/validate_analysis.py --analysis-path .mediaskills/generated/program_frame_analysis_....json

# 6. Reports
uv run <skill_directory>/scripts/text_on_screen_report.py --analysis-path .mediaskills/generated/program_frame_analysis_....json
uv run <skill_directory>/scripts/forced_narrative_report.py --analysis-path .mediaskills/generated/program_frame_analysis_....json

Read references/AGENT_FRAME_ANALYSIS.md for the exact per-frame JSON schema and classification rules.

Forced narrative deliverable format

After forced_narrative_report.py (agent vision path) or compile_forced_narrative_report.py (OCR path), present burned-in dialogue in this table — not the raw on-screen text dump:

Rules for the agent:

1. Embedded TC — use embedded_tc from the report JSON when the source has a tmcd track; otherwise use file-relative start_timecode.
2. Speaker — parse from burned-in caption labels (SQUATTER:, OFFICER:, etc.); use — when no label is visible.
3. Text — literal burned-in dialogue only; exclude slates, legal cards, credits, show logos, and OCR garbage.
4. Full coverage — include every dialogue line in the report; do not sample or truncate.
5. JSON — cite the paired *_forced_narrative_*.json for programmatic use; CSV has the same columns.

Recipe: shot-based analysis (graphics, title cards)

# 1–2. From shots skill: detect cuts, extract midpoint frames
# uv run <shots_skill_directory>/scripts/detect.py --input program.mp4
# uv run <shots_skill_directory>/scripts/extract-frames.py --shots-path .mediaskills/generated/program_shots_....json

# 3. Batch + agent-analyze + merge (same loop as above)
uv run <skill_directory>/scripts/get_frame_batch.py --manifest-path .mediaskills/generated/program_shots_....json --batch-index 0
uv run <skill_directory>/scripts/merge_analysis.py --manifest-path ... --frames-json batch0_analysis.json

# 4. Graphics reports
uv run <skill_directory>/scripts/graphics_on_screen_report.py --analysis-path .mediaskills/generated/program_frame_analysis_....json
uv run <skill_directory>/scripts/extract_title_text.py --analysis-path .mediaskills/generated/program_frame_analysis_....json

Legacy OCR path (no agent vision)

tesseract only — no text-type classification:

uv run <skill_directory>/scripts/compile_report.py --input program.mp4
uv run <skill_directory>/scripts/compile_forced_narrative_report.py \
  --onscreen-json .mediaskills/generated/program_onscreen_text_....json \
  --input program.mp4

The second step filters OCR rows to burned-in dialogue and writes the forced-narrative table format above.

Gotchas

• Agent vision required for classified on-screen text — scripts extract frames and merge JSON; they do not interpret images.
• Long programs — 1s interval on a 1-hour show = 3600 frames; use --interval 2 or --max-frames on extract, or analyze a subset.
• Incremental merge — merge_analysis.py appends batches; re-run with --force to replace all frame entries.
• Report reuse — report scripts skip regeneration unless --force when a matching report exists.

Available scripts

Acceptance checks (agent must pass before delivery)

1. Contract: exit 0, ok: true, every output_paths entry exists and is non-empty.
2. Skill gate: run validate_analysis.py successfully before any delivery report (text_on_screen_report, forced_narrative_report, etc.).
3. Coverage: analyzed_count equals frame_count; do not sample-deliver partial analysis.
4. Spot-check: verify first/middle/last report rows against source frames; exclude hallucinated scene text from dialogue tables.
5. On failure: re-merge or re-analyze; do not ship unvalidated reports. For exhaustive frame-accurate dialogue, prefer forced-narrative-exact.

Do not use for

• Built-in vision/LLM calls (agent analyzes frames; scripts merge JSON only)
• Replacing speech-captions for dialogue transcription
• Real-time stream analysis

Related skills

• shots — cut detection and midpoint stills
• image — single-still OCR
• video-transformation — frame extraction at arbitrary timestamps