agentleFS
PicsArtPicsArt/media-studio/skills/motion-tracking-from-video/SKILL.md

motion-tracking-from-video

Skill1 starsChanged 10 days agoMITView on GitHub

Measure what objects visibly do in a video - where each one moves, turns and grows, frame by frame - with picsart_media_track_motion, work out each move's easing (curve, overshoot, bounce) and turn it into scene keyframes; then measure your render the same way and compare them with picsart_media_track_motion_diff. Covers targets, estimators, tolerances, reading results, and placing tracked motion on a layer. Use when a scene must reproduce a reference video's motion, or a render's motion must be checked against one.

1 other project uses a version of this.

Adopted by

Most recently taken up by

The file

We split this skill into the rules it gives an agent

8 rules. About 0 are common, 3 are distinctive, and 5 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈3

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 5

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    Leave fps out, so it samples at the video's own rate, and run the window over the whole move: from the frame before it starts to the frame …

    line 39
  2. Very specific:

    End the window before the object is covered: When it slides behind something, its confidence falls and its pose freezes.

    line 97
  3. Distinctive:

    A track that is covered, then reacquired: After the confidence collapses, the later samples can be a valid track of the object that …

    line 106
  4. Very specific:

    An expression: instead of keyframes means the channel fit one within tolerance: orbit (the path is a circle) or sine (a rotation that sways …

    line 139
  5. Very specific:

    Use the tracker's own fitted keyframes and easings where the scene takes them (a preset or a cubic_bezier).

    line 177
  6. Distinctive:

    Pixels: When the scene is not the reference's size, multiply each position by the scene's size over the reference's (the same factor on …

    line 218
  7. Distinctive:

    Easing: keep each keyframe's easing where it is.

    line 223
  8. Very specific:

    When a seam decision depends on a reference's velocity, path or overlap, the measurement feeds motion-scene-continuity (fetch with …

    line 277

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Motion extraction

picsart_media_track_motion recovers what objects visibly do in a flattened video: their position, rotation and scale on every frame, fitted into scene keyframes. It does not recover the keyframes someone authored. It produces new motion data that reproduces what is on screen.

Looking decides what to track. The tracker measures where, every frame, to a fraction of a pixel. Never write a motion's numbers from a still or from memory: every timing and distance comes from a tracking result. The end points are not enough: a move is matched when it feels the same, so its easing (how it speeds up, slows down, overshoots and settles) is measured too, and built from the numbers.

Cost and receipts

Measure every required window with picsart_media_track_motion and picsart_media_track_motion_diff, applying picsart-video-rules's SPEND rule to their billing contracts. Each replies with a job receipt by default: follow it as picsart-video-rules says. Targets that share a tracking window go in one call, as many as its targets schema allows: one t0/t1 holds for every target, and each must be fully visible at t0. A target that enters later, or is covered before t1, gets a call with its own window.

The method

  1. Look at the first frame of the window with picsart_media_video_stills, and choose what to track. On a flat background, { "auto": true, "max": 9 } finds the distinct objects at t0 and tracks each, ranked by how much they move. Objects that overlap merge into one, so give those a box each instead.
  2. Track. picsart_media_track_motion with the video's source, the window t0/t1 in video seconds, and targets: each one bbox ([x, y, w, h] at t0, in video pixels), points, or auto. Set tolerance for the source (below). For a card, photo or title, a bbox in "mode": "similarity" (position, scale and rotation); "corners" when it skews or tilts in 3D. Leave fps out, so it samples at the video's own rate, and run the window over the whole move: from the frame before it starts to the frame after it settles, so the easing and the settle are in the samples.
  3. Read the result before you trust it. Per target it returns animations (the fitted keyframes with their easing, or an expression), static (the channels that did not move), meta with each channel's maxError, and notes on anything that weakened the track. maxError is the honest reconstruction error: a fit with a large one is an approximation, and you say so.
  4. Work out the easing of each move from the dense samples (below).
  5. Place it on a layer (below), as keyframes with timeMode: "layer".
  6. Check it. Export your render of the same window, track it with the same targets, and compare the two with picsart_media_track_motion_diff and their speed profiles (below).

What the tracker cannot see

It measures where texture goes, frame to frame. Some changes have no texture to follow:

ChangeWhat to do instead
a reveal: a wipe, a mask growingcropped stills of the box, one per frame of the reveal, for its edge; track whatever drives the reveal
a fade or a twinklestills either side and through it; the tracker has no alpha (below)
non-rigid motion: a flap, a bend, a wobbletrack the centre only, and build the shape change as a loop
a small idle loop: bobbing, swaying, breathingtrack over several cycles, at least one and a half: a sway in rotation comes back as a sine expression; build any other loop from the cycles you measured
motion blur on a fast movetrack it again with "estimator": "segmentation", whose centre stays on the smear, and read it with fit_motion.py --min-confidence 0.2; check the smeared positions against cropped stills, one a frame. Each move says how many of its samples were smeared (blurred)

Illustrated templates

Stickers, doodles and cut-out shapes over a photo or a flat background:

  • A sticker with a white border tracks best with "estimator": "segmentation": the border is a clean silhouette, where its flat fill gives features little to hold.

Tolerance, per channel and per unit

tolerance is position in pixels, toleranceScale a ratio (default 0.01) and toleranceRotation degrees (default 0.5). Never reuse a pixel tolerance for scale: 1.5 "pixels" once declared a 57% shrink static.

Sourcetolerance
a compressed reference videoabout 1.5 (it measures to about 0.5–2 px)
your own render0.5 or less

Too tight on a compressed source and the keyframes chase codec noise.

Choosing targets

  • Start where the object is solid. During a fade-in its features are weak, the track dies on the first frame and freezes. Start t0 once it is fully opaque.
  • A tight box inside the object's face. A box over a tilted card in a stack catches its neighbours, and the fit mixes two motions.
  • Opaque objects only, for rotation and scale. A half-transparent object blends with whatever moves behind it, and its measured pose drifts toward that. Ghosts, depth-faded cards and translucent chrome give approximate motion at best; say so.
  • Video playing inside the box: the features follow the content, not the box. Set "estimator": "segmentation" on that target, which tracks the silhouette. Its centre leans toward bright content, so a still element of the same group, tracked instead, is more exact.
  • A rotating object takes a box, never points. A point tracks translation only: a corner turning in place reads about 4 px wrong at 30°. A box in similarity mode compensates.
  • End the window before the object is covered. When it slides behind something, its confidence falls and its pose freezes. Shorten t1 and track again, rather than keep the frozen tail.
  • Coming in from off screen: while the object is clipped by the frame edge, a box around it is biased. Start at its first visible frame with a box on the textured part already in view (a photo's face, not its edge), with "estimator": "features", so the fit follows the content, not the clipped outline. fit_motion.py then marks the move startsBeforeWindow: build the entrance from off screen so it crosses the edge at the measured speed.
  • Clip the window, not the keyframes. A fit over frozen samples emits false keyframes at the tail. Track again with an earlier t1; never prune the keyframes by hand.
  • A track that is covered, then reacquired. After the confidence collapses, the later samples can be a valid track of the object that replaced it (a card becoming a chip). Check them against a still before discarding them.
  • One object, two stretches that need different estimators (a video inside a card, then siblings touching it): track each stretch on its own, with one overlapping frame. The two estimators' reference points differ by about 2 px: measure that offset on the overlap frame and subtract it when you join them, and multiply the second stretch's scale by the overlap frame's scale.
Contentmodeestimator
textured (a photo, a UI card with text)similarityfeatures
flat or solid, or video inside the boxsimilaritysegmentation
stretched in perspectivecornersfeatures
position alone matterscentereither

Reading the result

  • Position is the object's centre, in the video's own pixels. Under a layer's default anchor (the centre of its box, for every content kind but text), that is the layer's transform.position. Text anchors at its origin: place it by its alignment instead.
  • Rotation and scale are relative to t0: 0° and 1.0 there. The tracker cannot know an object's tilt at t0 from one box: add the pose the layer starts at yourself.
  • Rotation is counter-clockwise positive; a scene's rotation is clockwise. A turn the tracker reports as +12° is -12 in the scene. Flip the sign of every rotation value you place.
  • Scale comes as [s, s], because a scene's scale is 2-D.
  • Keyframe times count from t0 unless the call set normalizeTime: "absolute".
  • Easing is a preset or a fitted curve, chosen per segment. The closest of the four presets (linear, ease_in, ease_out, ease_in_out) comes first. When none of them fits within tolerance, the segment moves and it has enough samples, it gets a fitted curve instead: {"kind": "cubic_bezier", "control": [x1, y1, x2, y2]}, the scene's own parametric easing, with a y outside 0–1 for an overshoot. Copy either form into the keyframe as it came. A short segment keeps its closest preset, so a sharp snap or a long settle there reads close, not exact: its maxError says how close; track it denser to give it a curve.
  • An expression instead of keyframes means the channel fit one within tolerance: orbit (the path is a circle) or sine (a rotation that sways back and forth, meta's kind says which). On a compressed reference real paths often are not pure circles, and the keyframes it falls back to are the honest answer, not a failure.
  • The tracker has no alpha. It cannot tell a fade from a move. Judge a fade on stills either side of it, and never invent one to cover a jump in pose.

The feel of each move: easing

The fitted animations give each segment's easing, and the report at resultUrl holds every sample (t, x, y, rotation, scale, confidence). Save that report (as a file, where you run commands) and read each move's timing curve from it:

python3 scripts/fit_motion.py moves track.json [--target card1] [--into sheet.json]

Per move, on each channel that moves, it gives:

  • the target, the channel, from and to;
  • delay (from the window's start) and at (source seconds), then duration, up to the frame it settles, and hold, how long it rests after;
  • curve: linear, ease_in, ease_out, ease_in_out, or spring when it overshoots and swings back, named from where its peak speed comes (peakAt, 0 to 1) and how far above the mean speed it is;
  • cubicBezier, the curve fitted to the samples (a y outside 0–1 is an overshoot), and preset when one of the scene's four presets fits about as well;
  • overshoot, how far past the rest it goes as a share of the travel, swings, how many times it crosses its rest before it settles, and for a spring its stiffness and damping (mass 1);
  • peakSpeed and meanSpeed, and staggerAfter: how long after the previous target's move it starts;
  • keyframes the scene can take as they are: one curve, or for a spring each swing as its own segment, since one cubic-bezier holds only one overshoot.

A move out and back, or one that bounces back and forth, comes back as one move per leg; a spring's swings about its rest, each smaller than the last, stay one move however lightly damped. Where it rests is measured from the frames that hold still after it; only when the track ends while it still moves is the rest worked out from its swings. A rotation that crosses ±180° is read as the short turn it is. A slow move is kept: its start is the last frame within the tolerance of the rest before it. --into replaces every target and channel it measured, so a channel measured still again loses its old move. Track each window of one object under its own target id (card1_in, card1_out): moves of the same id and channel from another report are replaced.

Use the tracker's own fitted keyframes and easings where the scene takes them (a preset or a cubic_bezier); take fit_motion.py's keyframes only where it does not, such as a spring with more than one swing. Either way, the numbers that describe the move come from fit_motion.py. A move it marks startsBeforeWindow, or not settled, was tracked over too short a window: widen it and track again.

Record each move where the use case keeps the spec: target, channel, from → to, duration, delay, curve, overshoot, and the tracking result it came from. Never words alone ("slides in").

Known limits of splitting moves. Samples alone cannot tell these apart; fit_motion.py reads them as below, and its tests pin it. Look at the stills when one matters.

  • A trip whose swings shrink like a spring's, with the track ending before it holds still (0→100→30→80, still moving at the end), reads as one spring. Keep a still second after the motion in the tracked window, and the measured rest splits it.
  • One turn back by less than the travel, then still (0→100→60), reads as one move with an overshoot (67 % here), as an ease with a big overshoot would. A turn back past that (0→100→40) is two moves.
  • A spring that freezes mid-swing and holds there rests where it froze: it may come back as more than one move, and the last one ends where it froze.
  • A window that opens on the frame the spring is released, with the track ending after two turns (still moving), cannot show the spring started from rest: it reads as up to three ease moves. Start the window a few still frames before the release, or track it until it settles, and it is one spring.

Creating similar motion

  • Build each move from its numbers: its from and to, its duration, its curve (the preset, the fitted cubic_bezier, or the spring's keyframes) and its overshoot.
  • Keep the order and the stagger. The moves start in the reference's order, each staggerAfter the one before it, scaled to the scene's time like every keyframe.
  • Match a text animation by its mechanism, not only its timing: a wipe, letters one by one, a line rising, or a mask reveal that follows a moving object (a sticker that wipes the title in as it passes). Read the mechanism from cropped stills of the reveal, one per frame, and track the object that leads it; then build the same mechanism, with the measured timing.

Placing tracked motion on a layer

The result is in the reference's frame and time. A layer needs its own:

  • Time. A keyframe's layer time is its tracked time, plus t0 when times count from t0, plus the scene second where the reference's second 0 plays, minus the layer's start.
  • Pixels. When the scene is not the reference's size, multiply each position by the scene's size over the reference's (the same factor on both axes) and add the offset of where the reference's frame sits in the scene.
  • Pose. Rotation: the layer's starting rotation minus the tracked value. Scale: the layer's starting scale times the tracked value.
  • Easing: keep each keyframe's easing where it is; it shapes the segment that starts at it.

An expression is in the reference's time, and an orbit also in its pixels, so these rules cannot move it: when the scene differs from the reference in size or timing, track that object again as keyframes, or place it at the reference's size and time inside a composition sized like the reference.

Checking a render against the reference

Export your render over the fragment and track it with the same box as the reference: the same part of the object, the same size against it, at the same moment of the move, with the same mode and estimator. The tracker reports the centre of the box you drew, so another box is another point, and a card tracked by its face against one tracked by its corner reads 150 px apart while both move alike.

Then measure the deviation with the starting offset removed:

python3 scripts/fit_motion.py diff ref-track.json our-track.json --offset <ref second 0 minus ours>

Both tracks are read at every sample time of either, over the time both cover. For each channel that changes on either side from the first shared sample (a slow pan too), it gives:

  • startOffset and placement: where ours sits at the first shared sample against the reference;
  • maxDev: the raw deviation;
  • moveDev: the deviation once the starting offset is removed, which is the move itself.

The move passes on moveDev, within the threshold the use case sets. The placement is the layout: report it, and fix it when the layout should match. picsart_media_track_motion_diff gives the raw maxDev, meanDev and worstT for the same pair. A number replaces "it looks about right".

Then check the feel. A linear move can pass the position check against one that eases. Compare the two speed profiles:

python3 scripts/fit_motion.py compare ref-track.json our-track.json --offset <ref second 0 minus ours>

Per move it checks the start, the duration, where the peak speed comes (not when both moves are linear: a linear move has no peak) and how big it is, the overshoot, the swings and the curve's name, and says which differ. speed writes each target's speed over time as CSV, to show the two side by side. Fix the measured differences and check again while each correction makes progress. If the evidence or a tool limit prevents a match, report the remaining difference with its numbers and cause. ASK (with your host's own question tool, such as AskUserQuestion, so the options are buttons the user taps, the recommended one first; only where the host has none, the options numbered in one message; in unattended pace, never ask: write it to the project record's openDecisions with the answer you assumed, as picsart-video-rules says) whether to accept that approximation or change the approach; never label it a pass.

  • Like for like. The same targets, the same boxes, the same window, the same tolerance and the same --min-confidence on both sides. Your render tracked from its own second 0 against the reference from its second 4 is a 4 s offset, not a mismatch.
  • What the deviation says. A constant offset is geometry: a position or a size. A deviation that grows and shrinks along a move is easing. One that peaks at a beat's start or end is timing.

Measuring for a seam

When a seam decision depends on a reference's velocity, path or overlap, the measurement feeds motion-scene-continuity (fetch with picsart_scene_recipe). Tracking answers what moved, where and when; continuity judges what that movement communicates and whether the scene should preserve, replace, interrupt or reset it. A close numeric fit does not by itself prove a readable handoff.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.