agentleFS
notivnnotivn/AIEV/.claude/skills/video-pipeline/SKILL.md

video-pipeline

Skill132 starsChanged 39 days agoMITView on GitHub

End-to-end video production workflow for the AI Edit Video system - from the user request to the MP4 in outputs/, coordinating HyperFrames (scenes) and Remotion (assembly). Read this when starting any video or when building the backend render queue.

  • Sends data out

6 other projects use a version of this, 1 of them edited.

Adopted by

Most recently taken up by

The file

We split this skill into the rules it gives an agent

7 rules. 0 are common, 0 are distinctive, and 7 are very specific to this project.

  • 0

    Common

    Most of the 3 edited versions we read keep these.

  • 0

    Distinctive

    This author's own choices. Worth a look.

  • 7

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    Create video-projects/<kebab-case-name>/ with index.html, compositions/, assets/, renders/, hyperframes.json, meta.json.

    line 20
  2. Very specific:

    Update meta.json: status: "done", output: "outputs/...".

    line 83
  3. Very specific:

    When using one in a video: copy it into video-projects/<name>/assets/sound-effects/ then declare it in meta.json - a project must contain …

    line 103
  4. Very specific:

    Small edit -> re-render only what changed: if you edited one scene, re-render that scene (render -c <scene>), do not re-render the whole …

    line 112
  5. Very specific:

    Transcription runs only ONCE: - if transcript.json already exists, reuse it, NEVER transcribe again.

    line 117
  6. Very specific:

    Rendering a single scene: use the -c flag, not a positional argument: npx hyperframes render -c compositions/s01.html --quality draft …

    line 130
  7. Very specific:

    The sub_timeline_readiness_timeout warning: when rendering a template file with -c: the render still comes out correct (best-effort) but …

    line 132

“Common” is measured: kept by at least 6 in 10 of the 3 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Video Pipeline - from request to MP4

Engine roles (never swap them)

TaskEngineWhy
Scene motion graphics: kinetic typography, karaoke captions, count-ups, data tables, callouts, shadersHyperFramesHTML + GSAP is its strength, and the Vietnamese skills already ship the fixes
Stitching scenes + source footage, transitions between scenes, mixing audio + sound effects, full-video caption overlay, final exportRemotionProgrammatic assembly, <Sequence>/<Audio>/<OffthreadVideo>

How the two engines talk to each other: intermediate MP4 files in video-projects/<name>/renders/. HyperFrames does not know Remotion exists and vice versa - the only shared thing is the manifest.

Project lifecycle

1. Initialize

  • Create video-projects/<kebab-case-name>/ with index.html, compositions/, assets/, renders/, hyperframes.json, meta.json.
  • meta.json is the central manifest:
{
  "id": "tiktok-paper-gpt5",
  "name": "Mổ xẻ paper GPT-5",
  "width": 1080, "height": 1920, "fps": 30,
  "status": "draft",
  "scenes": [
    { "id": "s01-hook",   "src": "compositions/s01-hook.html",   "durationInFrames": 90,  "render": "renders/s01-hook.mp4" },
    { "id": "s02-talking","srcVideo": "assets/talking-head.mp4", "from": 2.5, "to": 14.0 }
  ],
  "audio": {
    "voice": "assets/voice.mp3",
    "sfx": [ { "file": "assets/sound-effects/whoosh-01.mp3", "atFrame": 88 } ]
  },
  "output": null
}
  • scenes[] is the contract between the two engines: a scene with src is rendered by HyperFrames; a scene with srcVideo is footage used as-is. Remotion reads this file to assemble - never hardcode the scene list in the Remotion code.

2. Build the scenes (HyperFrames)

  • Copy assets/brand/brand-tokens.css into the project, write compositions following the window.__timelines convention.
  • Vietnamese videos: apply the verified fixes (gradient text losing diacritics -> render text as real elements + use background-clip correctly; check that every diacritic is present on the first/last frame of each reveal).
  • Lint clean before rendering: npx hyperframes lint.
  • Render a draft of each scene: npx hyperframes render --quality draft --output renders/<scene>.draft.mp4.

3. Frame verification (MANDATORY, before assembly)

ffmpeg -ss <hero-moment-seconds> -i renders/<scene>.draft.mp4 -frames:v 1 verify/<scene>.png

Inspect every image: does the Vietnamese text keep all its diacritics? Is any text overflowing the edges? Any unexpected white/black frames? Is a face cropped? If anything is wrong, fix the scene - do not move on.

4. Assembly (Remotion) - see the remotion-assemble skill for details

  • The Remotion composition reads meta.json, builds <Sequence> entries from scenes[], and inserts sfx at atFrame.
  • Render a full draft -> review it in the web UI -> approve.

5. Automated QC (MANDATORY, blocks final)

curl -s -X POST http://localhost:6869/api/projects/<id>/qc -H "content-type: application/json" -d "{}"

The server measures the latest draft with ffmpeg and returns { report: { status, checks[] } }:

checkmeaningwhat to do on fail
resolutiondimensions/fps do not match meta.jsonre-render with the correct settings
loudnessfar off -14 LUFSadjust the mix volume, do not fix it by pulling the peak
truepeakclipping (> -0.5 dBTP)lower the source gain, see "SFX loudness and lead silence" in the noti-tiktok-vn skill (voice headroom before layering SFX)
blackframesblack frames in the MIDDLE of the video (leading/trailing fades are ignored)a gap at a scene change - fix transitionOverlap/durationInFrames
freezefrozen picture >= 2sthe scene is missing animation, or the footage is broken
tail-silencesilent tail > 1.2strim the tail
av-durationpicture and sound differ by > 0.5swrong total durationInFrames
safe-areaALWAYS passes, returns frames = images with the obstructed bands outlined in redyou MUST Read every image and judge for yourself; if any text sits in the red zone, pull it inward, see the key-layout skill
  • status: "fail" -> the server rejects the assemble-final job with 409 QC_REQUIRED / QC_FAILED. Fix the root cause, re-render the draft, run QC again. Do not use force: true to dodge it unless the user explicitly asks.
  • status: "warn" -> consider fixing, not blocking.
  • The final report must state the QC results and how each fail/warn check was handled.

6. Final

  • Re-render the HyperFrames scenes at --quality standard.
  • Remotion final render -> outputs/<project>-v<N>.mp4.
  • Update meta.json: status: "done", output: "outputs/...". The web UI reads status from here.

7. Thumbnail + publish package

curl -s -X POST http://localhost:6869/api/projects/<id>/thumbnail -H "content-type: application/json" -d "{\"title\":\"...\",\"frameAt\":12}"
curl -s -X POST http://localhost:6869/api/projects/<id>/publish   -H "content-type: application/json" -d "{}"

publish generates .srt/.vtt from the transcript and has the AI write the title/description/hashtags for TikTok, YouTube and Facebook following the Style Design. Precondition: the transcript must be the FINAL one. If it was cut/remapped, write the final version to assets/transcript.final.json (the transcript loader prefers this file) - otherwise the subtitles will drift out of sync with the cut video.

Render queue rules (backend)

  1. Every render goes through the queue - including the ones Claude runs by hand. Jobs are written to SQLite: id, projectId, type (scene-draft|scene-final|assemble-draft|assemble-final|image-gen), status, progress, log, startedAt, finishedAt.
  2. Jobs run in parallel up to QUEUE_CONCURRENCY (default 2, configurable via env; set 1 on weak machines). Safety constraint: two jobs from the same project never run at the same time (to avoid trampling renders/meta) - parallelism only happens across different projects.
  3. Progress: parse the CLI stdout (both engines print frame progress) -> update the DB -> push SSE to the web UI.
  4. Failed jobs: keep the complete log in the DB, show it in the UI, and do not auto-retry more than once.
  5. Draft always comes before final. The backend rejects a final job if the project has no successful draft for the current version of the scenes.

Sound effects

  • Shared library: assets/sound-effects/, every file has an entry in assets/sound-effects/library.json (file, tags, durationMs, and a description written in Vietnamese).
  • When using one in a video: copy it into video-projects/<name>/assets/sound-effects/ then declare it in meta.json - a project must contain all of its own assets (re-rendering must not depend on a library that may change).
  • The Sound Effects page in the web UI reads library.json, plays previews inline, and allows uploading new files (the backend updates the json).

Speeding up renders (this machine has a GTX 1660 - verified 2026-07)

The number one reason the pipeline burns an entire hour: re-rendering a draft of the WHOLE video after every small edit. The rules:

  1. Verify layout/text with npx hyperframes snapshot or inspect (a few seconds) instead of rendering a full draft (tens of minutes). Render a full draft only ONCE, when every snapshot is good, and final ONCE.
  2. Small edit -> re-render only what changed: if you edited one scene, re-render that scene (render -c <scene>), do not re-render the whole composition. The split-scene + Remotion-assemble pipeline is built exactly for this.
  3. HyperFrames speed flags (the backend queue adds them automatically; when running by hand you MUST remember them): -w 8 --browser-gpu for every render; add --gpu (NVENC) for drafts. Final keeps CPU encoding (libx264) for maximum quality. Measured in practice: ~30% faster on short scenes, more on long ones.
  4. Transcription runs only ONCE - if transcript.json already exists, reuse it, NEVER transcribe again.
  5. Remotion speed flags (when running npx remotion render/still by hand you MUST add): --concurrency 8 --gl angle. WITHOUT --gl angle, Remotion's Chrome renders with the software renderer (SwANGLE) - the CPU carries 100% while the GPU idles (symptom: Task Manager CPU ~95%, GPU ~5%). On Linux use --gl angle-egl instead of angle.

Known issues (verified in production 2026-07)

  • meta.json must respect the data-type contract (the web UI reads it directly - a wrong type crashes the page): output is a path STRING (e.g. "outputs/<id>-v1.mp4"), NOT an object. Extra metadata (duration, quality, renderedAt, ...) goes into a separate outputInfo field if needed. scenes[].durationInFrames is a number; sfx are placed with atFrame (a frame number) as in the schema at the top of this file.
  • Rendering a single scene: use the -c flag, not a positional argument: npx hyperframes render -c compositions/s01.html --quality draft --output renders/s01.draft.mp4. A <template> sub-composition must be referenced by index.html through data-composition-src for -c to be able to render it.
  • Words running together on per-word reveals: the HyperFrames pipeline swallows the whitespace between inline-block <span>s - separate words with margin: 0 0.14em on .word, do not rely on whitespace in the HTML.
  • The sub_timeline_readiness_timeout warning when rendering a template file with -c: the render still comes out correct (best-effort) but wastes another 45s waiting - acceptable for drafts; if you want it gone entirely, render through index.html.
  • A TS comment containing the glob path */ (e.g. video-projects/*/meta.json) closes the block comment early -> a baffling compile error. Write video-projects/<id>/meta.json instead.
  • Vietnamese JSON sent with curl -d from Git Bash on Windows loses its diacritics (verified 2026-08: the thumbnail title arrived at the server as "B?n th? kh?ch..." and rendered broken). The console codepage mangles UTF-8 before curl sends it. For any API body containing Vietnamese, POST from Node instead: node -e "fetch(url, { method: 'POST', headers: {'content-type': 'application/json'}, body: JSON.stringify({...}) })" (or write the JSON to a file and use curl --data-binary @file).

Checklist before reporting completion

  • If the brief enables autoCut: the cut was done per the auto-cut skill, a second silencedetect pass verified the cut version, and the report states the seconds/segments removed
  • Every scene passed frame verification, Vietnamese text keeps all its diacritics
  • Audio is in sync at the start/middle/end (check 3 random points)
  • Sound effects land on the right frames, and their volume does not bury the voice (sfx ~10dB below voice)
  • The output matches the dimensions/fps in meta.json
  • Automated QC ran on the draft with no remaining fail checks (the report spells out any leftover warn checks)
  • meta.json has updated status + output
  • The thumbnail was generated and Read to verify the diacritics are complete
  • The publish package was generated (.srt/.vtt + metadata), using the FINAL post-cut transcript
  • Any new lesson learned was written into the relevant skill

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.