agentleFS
badmoonrbadmoonr/loopilot/skills/video-content-understanding/SKILL.md

video-content-understanding

Skill1 starsChanged 27 days agoApache-2.0View on GitHub

Extract and understand visual, spoken, OCR, and timeline evidence from local video files with native Ark video models and FFmpeg signal inspection. Use for reference-video analysis, generated-video review, shot and action breakdown, model-timestamp keyframe extraction, or preparing structured video evidence for another skill. This skill is read-only and does not edit, publish, or regenerate videos.

  • Reads credentials

The file

We split this skill into the rules it gives an agent

7 rules. About 0 are common, 5 are distinctive, and 2 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈5

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 2

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Use this skill as a read-only evidence and understanding layer.

    line 17
  2. Distinctive:

    Run complete native video understanding:

    line 33
  3. Very specific:

    Read video_understanding_validation.json.

    line 59
  4. Very specific:

    If a legacy local-only semantic task is explicitly requested, run prepare, load every selected frame, write the result, and validate it.

    line 91
  5. Distinctive:

    If a legacy or local-review result remains ambiguous, extract only the affected interval:

    line 92
  6. Distinctive:

    Never overwrite the source video.

    line 118
  7. Distinctive:

    Never generate an edit plan unless the consumer explicitly requests one.

    line 119

≈ “Common” is an estimate: at least 1 in 5 of the 350,495 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

任务方案与交付

当前运行 Agent 在本阶段前读取相邻 ecommerce-operations/references/task-execution.md,沿用本任务最新有效方案和适用偏好,按真实结果交付并记录成功操作。已在同一任务读取时直接复用;不反向重启总控或重复业务动作。Agent 主导顺序、分支与恢复,本 Skill 返回实际结果和正式恢复线索。该参考暂不可读时说明缺口并继续当前可执行工作,不新增文档或格式硬门槛。

Video Content Understanding

理解中的置信度、每镜证据张数、时间轴间隙/重叠、文字行数和动画观察长度是事实,不能单凭默认阈值终止任务。原生入口保留实际值并输出 observations,由当前模型决定补看、补帧、修订或继续;实际时间越界、缺文件或不可执行格式仍经原入口恢复。参考观察不等于目标 renderer 支持的效果。静音本地取证如实标注 semantic_source=local_visual_evidence,其 signal facts 可供混剪消费,不要求无关 transcript;本地解码不代表质量审核通过。

下游应直接消费上述事实与 observations:可恢复状态、证据引用或投影差异返回 needs_model_action 并重入原理解入口;权利、身份、账号或客观文件冲突返回 requires_external_resolution。下游不得重新引入固定最低置信度或证据数量门槛。

Use this skill as a read-only evidence and understanding layer. Keep editing, publishing, script generation, and business decisions in consumer skills.

已有历史媒体缺当前 receipt family 时,先回读真实文件而不是因 V1/V2 凭证差异停止:

python3 scripts/video_content_understanding.py admit-existing-media \
  --video /abs/path/existing.mp4 \
  --output-dir /abs/path/existing-media-fact \
  --historical-sha256 <optional-old-sha> \
  --historical-contract-version <optional-old-version>

该入口用本地 ffprobe + 完整解码指纹生成 existing-media-business-fact-v1,Provider 调用固定为 0。旧 SHA/版本不一致记录为 stale / business_veto_allowed=false;媒体缺失或不可解码仍按客观事实失败,权利/账号适配性由执行模型结合当前方案上下文另行校验。

Workflow

  1. Run complete native video understanding:
python3 scripts/video_content_understanding.py analyze \
  --video /abs/path/input.mp4 \
  --output-dir /abs/path/video-understanding

视频复刻新 revision 使用统一持久化证据:

python3 scripts/video_content_understanding.py analyze \
  --video /abs/path/input.mp4 \
  --output-dir /abs/path/video-understanding \
  --account-key account-key \
  --rights-receipt /abs/path/rights-receipt.json \
  --state-root /abs/path/runtime-state \
  --max-provider-attempts 1 \
  --seed-usage-ledger /abs/path/video-replication/seed_usage.json \
  --seed-request-id <creative-runner-reservation-id>

账号级视频复刻必须由 creative-runner 先在 video_generation_task_record.json 登记 task-intake 的 reference_understanding 授权 scope,并为当前视频 SHA 创建、reserve 唯一 operation;analyze 只消费与 run、账号、stage、视频 SHA256、output dir 完全一致的 seed_usage.json 投影,并在 Ark 边界前把主档 operation 标记为 dispatching。主档是预算与防重权威,seed_usage.json 不再独立裁决额度。缺 operation、重复使用、状态或绑定漂移时 Provider 调用数为 0;合法缓存命中回写 billing_certainty=not_generated。无账号的 legacy/local analyze 保持兼容。

默认 Provider 参数固定为 thinking=enabled + reasoning_effort=minimal + max_output_tokens=24000,完整 Schema 不缩减。命令构造前必须校验 ArkCLI 支持的 thinking/effort;thinking=disabled 与非空 reasoning_effort 同时存在时返回 provider_reasoning_config_invalid / provider_call_count=0,账号级 reservation 保持未 claim。长视频只有显式 Profile 才能按需提高输出上限。

  1. Read video_understanding_validation.json. Continue only when status=success. 原始 JSON 可解析但表达/结构无法标准化时,原生入口输出 handoff_diagnostic.json 和 needs_model_action,不创建中央事务或付费重试。当前模型读原始响应、诊断及 ARK_SYSTEM_PROMPT,在诊断指定的 model_adapted_analysis.json 写入 raw_input_sha256 + target_payload,再进入 recover-provider-response;候选经过原 normalizer/validator,并须交回创作消费者确认实际恢复。不得修改原事实或补造缺失内容;unknown 回原查询对账。 若适配事务已完成但共享 cache 未落盘,只能在 result + receipt + transaction 全血缘校验通过后复用其 payload,并重新运行当前 normalize_model_analysis + validate_timeline_and_evidence;该重入仍固定 Provider 0。 升级导致 validator 源文件 SHA 变化时,冻结 target contract 不改写;旧/新 snapshot_sha256 必须各自校验通过,且排除 validator.source_sha256 及其联动 snapshot SHA 后合同完全相同、事务终态和零 Provider 账本有效,才允许用当前运行时 validator 重新校验。字段、约束、validator ID/version 或其它合同内容变化仍返回 contract_adaptation_input_drift。
  2. 普通任务消费 video_understanding.json;带账号的新视频复刻消费 video_understanding_v2.json + video_evidence_ref.json,消费者不得重新抽取等价帧。storyboard_table.md 只供人工查看。 新分析会默认对前 3 秒 Hook 以 6fps 生成 supplemental-frames/manifest.json。快速动作必须登记 start/transition/result 阶段;缺密集证据时返回 reference_action_timeline_invalid。
  3. Consume video_understanding.json; use storyboard_table.md only for human review. Reference semantics, including spoken_text, come only from the native Ark audio-video result. FFmpeg signal facts may report stream presence, duration, decoded bytes and silence, but never create or correct semantic text.

product_asset_remix_v1 可在已完成且 SHA 绑定的 Provider 响应唯一失败为 speech_coverage_incomplete 时执行零调用视觉恢复:

python3 scripts/video_content_understanding.py recover-provider-response \
  --video /abs/path/source.mp4 \
  --output-dir /abs/path/video-understanding \
  --account-key account-key \
  --semantic-scope visual_only \
  --source-audio-policy mute

该入口不提交 Provider,只生成 semantic_scope=visual_only 的受限 V2;音频语义明确排除,FFmpeg 仅保留信号事实。视觉时间轴、关键帧、置信度或其它校验失败仍按客观事实处理。参考视频、视频复刻及默认恢复继续使用 full_av + preserve,语义只认 Ark 结果。 4. 已知 Storyboard / timeline 的生成 Segment 或 final 审核使用显式本地证据模式,绝不调用 Seed:

python3 scripts/video_content_understanding.py prepare-local-review \
  --video /abs/path/generated.mp4 \
  --output-dir /abs/path/local-review \
  --sampling-json /abs/path/local_review_inputs.json \
  --shot-specs-json /abs/path/local_review_shots.json \
  --account-key example_account

该命令输出 local_review_evidence.json,包含每帧 evidence ID、时间码、SHA256 和镜头归属。提供 --account-key 时还以当前媒体指纹和既有分镜时间轴生成账号域 video-evidence-ref-v1,provider_call_count=0;候选池可高频抽取,但消费者只提交预算内精选图片。 5. If a legacy local-only semantic task is explicitly requested, run prepare, load every selected frame, write the result, and validate it. Do not silently use this fallback when Ark fails. 6. If a legacy or local-review result remains ambiguous, extract only the affected interval:

python3 scripts/video_content_understanding.py supplement \
  --output-dir /abs/path/video-understanding \
  --start 4.2 --end 6.0 --fps 6

Load the new supplemental_frame_paths, revise the result, then validate:

python3 scripts/video_content_understanding.py validate-result \
  --output-dir /abs/path/video-understanding

supplement works after either native analyze or legacy prepare. It writes supplemental-frames/manifest.json with frame paths and approximate timeline timestamps. 同一区间不同 FPS 使用不同不可变目录;仅显式 --force-reanalyze 可新增理解 revision,旧 revision 不覆盖。

Boundaries

  • arkcli +understand with the model frozen in Provider Profile v3 is the default semantic provider. Require --ark-provider-profile; resolve production credentials only from Keychain and inject ARK_API_KEY at the isolated subprocess boundary. Never pass --profile/--api-key, persist a Key, inspect SSO, or query control-plane resources during a business run.
  • analyze 用于未知参考视频的完整音画理解;prepare-local-review 用于已知事实的生成视频审核,两者不得静默互相回退。后者不得把 Storyboard 事实扩写成未观察到的音频或商品结论。
  • FFmpeg / ffprobe are the media base for probing, decoding, per-shot keyframe extraction, audio extraction, and basic signal detection.
  • New runs do not invoke Whisper or any independent ASR. Historical transcript fields may be read for compatibility but never participate in canonical facts, correction, composite decisions, or blocking.
  • whisper_call_count=0 and semantic_source=ark_native_av are explicit output facts for new reference-understanding results.
  • Visual, OCR, speech, and audio-event evidence must share one time axis in video_understanding.json.
  • Product, flavor, OCR, material, object-state, action, scene, or shot-size changes require a new shot even without a hard cut.
  • Never overwrite the source video.
  • Never generate an edit plan unless the consumer explicitly requests one.
  • 低置信度保留为观察,当前模型结合实际任务决定补看、补帧、原响应恢复或继续,不单凭默认阈值停止 consumer。video_understanding_result_invalid 等真实输入/证据问题先回读诊断,通过原入口恢复后重入 consumer;不能把无效结果或未观察事实标为合格。身份/账号/权利、付费与 submitted/unknown 边界仍有效,恢复完成不替代业务终审。
  • 新交接由原 normalizer 直接消费;模型候选只转换已有事实,保留原响应 hash 与任务内来源。consume_adapted_analysis() 的通过不等于下游业务成功;下游消费后补介入结果。已有 V1/V2 事务继续明确的历史恢复路径,新任务不使用其版本窗口、草稿次数或中央回执要求。
  • status=incomplete / incomplete_reason=length 表示本次 Provider 调用已消耗但结果不可注册:保留脱敏 response、usage、attempt receipt 和 request hash,输出 recoverable=true / retry_policy=new_authorized_request_required / recovery_profile=concise_minimal,且不得自动重试、解析部分 JSON 或写入缓存。恢复必须使用新的 request ID、output dir、request hash、seed reservation 与人工授权;recover-provider-response 不接受 length 部分响应。

叙事屏显文字理解

text_layers[].role 使用 narrative_caption 表达不对应口播、主要承担叙事推进或情绪强化的屏显文字。模型必须同时返回其功能、时间证据、bbox 和结构样式观察(对齐、行数/断行、字体类别/字重、颜色、描边/阴影、相对字号、行距、动画观察)。style_observation.animation 使用去除首尾空格后的非空自由描述,不设固定长度上限,不套用固定词表;理解层只保存参考事实证据,不自行决定目标视频是否加字或使用哪种可执行动画。

Resources

  • Contract: references/input-output-contract.md
  • Runtime profile: templates/video-understanding-profile.json
  • CLI: scripts/video_content_understanding.py

原生交接与历史恢复

合法新输入不导入中央适配器、不写 pass-through 回执。差异由当前模型按需救援,任务日志使用正常创作运行入口的 handoff_report.py;原响应、时间线、账号/权利和 validator 不可改写。只有目录内已有 V2 请求,或明确指定历史 contract_adapter.mode=legacy_v2 时,才加载可选旧引擎;普通新任务不受其是否安装影响。历史 receipt 由轻量只读兼容器校验,不签发新 receipt 冒充旧成功。

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.