Song timing

When to use

At the start of any music video (build the audio map before keying anything), whenever something is keyed to a word or a hit, and whenever a draft looks early or late. Everything else keys to the map this skill builds.

1. Get the stems from the music tool itself

• For a Suno song, use Suno's own stems: the song's menu → Edit → Get Stems / MIDI → Auto split → Extract → Download, with WAV and MIDI ticked (the menu as of October 2026). It gives about ten stems (vocals, backing vocals, drums, percussion, bass, guitar, keyboard, synth, strings, fx) and one MIDI file per stem. It uses credits on the creator's account, so ask first.
• Not a general-purpose stem splitter. The End Of The Show's stems came from one, and so did I Can Make Anything's first audio map, where the piano was buried in a mixed "other" stem. From then on the stems came from Suno.
• Check them against the master: cross-correlate the summed stems with the master mix and confirm a lag of 0. The sum won't null the mix exactly, because they're separations, not multitracks.
• Check where each instrument really is. An instrument can land in a neighboring stem (keyboard vs synth).
• To edit the mix itself (a phrase moved, a vocal ducked), prefer adding a stem's phrase on top of the mix, level-matched, over rebuilding from the summed stems, which sounds different.

2. Time every word on the vocal audio

The onset is the first audible energy of the word's first sound: a fricative at its noise onset, a stop at its release burst, a vowel or sonorant at the first change into it. Never the breath or aspirated "h" before a phrase, and never the previous word's last consonant or a pitch ornament. A word is never readable before its onset; the breath can drive anticipation that isn't readable yet.

Don't trust these for when a word is sung:

• the lyric sheet's times or anything generated from them;
• automatic aligners: within 2 frames only about half the time on Everybody's Got A Home, and held notes dragged whole words 1–1.7 s off;
• speech-transcription word times: 0.5–0.8 s early on Everybody's Got A Home, and up to a syllable early on I Can Make Anything. Use them to confirm which words are sung, never when;
• the stem's MIDI on its own: Suno's vocal MIDI ran ~6.5 frames ahead of its own audio on Everybody's Got A Home (the drum MIDI ~6 frames), and some consonants still started before the MIDI note. So no blanket offset either.

The traps:

• A held note's tail labeled as the next word. The next word starts after the held word's final consonant.
• A melisma's notes aren't words, and the MIDI can split a vibrato into alternating notes that look like syllables.
• Same-pitch syllables merged into one note ("I can", "hun-dred"). Read them on the audio.
• Sung s and f hiss 5–8 frames before the vowel. Decide the onset by the rule above, and be consistent.
• A doubled backing phrase just before the lead line can be mistaken for it.
• Repeated lines aren't copies. Never copy an onset across lines by melodic analogy. Measure each line on its own, then cross-check repeated choruses against each other.

The method that worked:

1. Take a candidate time for each syllable from the MIDI note or an aligner.
2. Place it on an acoustic event of its kind on the vocal stem, using a spectrogram crop: the hiss rise, the burst, an energy dip, a pitch step.
3. Do it twice independently. Where the two agree within ~25 ms, take the mean; where they don't, measure again.
4. Store every word in one file: onset, end, confidence and the reason. It's the single source every scene keys to.
5. When a call comes down to perception, render a short A/B clip with the song and the vocal waveform for the creator rather than describing it.
6. Re-timing a word moves it in the shot too: check the camera at the new time before saving.

3. Beats and hits from the drum stem

• Check the tempo. A song can drift: The End Of The Show's drifted from 123.5 to 127.7 BPM and back. Keep a list of beat times rather than one fixed period unless you've measured that the tempo holds.
• Fit the beat grid to the drum stem's audio onsets, not the drum MIDI. Check the grid sits within ~0.1 beat of the onsets on average. A grid half a beat off lands every snap on the "and".

• Find the hits: onset candidates by spectral flux on the drum stem, then classify each by the bands just before and after it:

  • kick: a rise at 30–110 Hz;
  • snare or clap: noise at 1.2–5 kHz with a 25–110 ms tail (a kick's click has none);
  • hats: a short burst at 7–16 kHz;
  • crash: 5–16 kHz held for 150–500 ms.

  Take the thresholds from the song's own levels. A hit can be two at once (a kick and a clap on 2 and 4). Give each hit a strength from 0 to 1, and check the result bar by bar as a 16-step pattern.

• Map the song's events too: drum stops and re-entries, fills, risers, drops, silences, and prominent instrument notes (a piano line) from that stem's MIDI plus its measured lag.

4. Drive the camera and animation from the map

• Moves start and land on beats, speed up into accents and slow through held notes (shot-design). Not every hit needs a move: a camera that snaps to everything is as bad as one that never lands.
• A slight shake on the kicks. On each kick, a small push toward the subject and a quick roll-and-nudge that dies in about a fifth of a second, on top of a slow sway. Keep it softer on a held framing that has to be read.
• The accent lands on the beat: anticipation before it, the impact at it, follow-through after. Not a slow move that starts on the beat and peaks late.
• Prominent notes get an accent or a push, not just the beat.
• Pace follows the song. A slow song gets slow, eased, beautiful moves.

// A small jolt on every kick, layered on the keyed camera. kicks: sorted kick times in seconds.
function kickJolt(t: number, kicks: number[]): number {
  const last = kicks.findLast((k) => k <= t);
  if (last === undefined || t - last > 0.5) return 0;
  const s = t - last;
  return Math.exp(-11 * s) * Math.sin(52 * s); // two quick wiggles, gone in ~0.2 s
}
// e.g. offset the camera by a few thousandths of its distance to the subject along its right and up axes,
// and roll it a few thousandths of a radian, times kickJolt(t, kicks).

Failure modes

• Keying to the MIDI or a transcription because it's convenient: the words show before they're sung.
• One offset for the whole song instead of per-word onsets.
• Re-timing an approved word from MIDI note starts alone when it was a melisma: check the spectrogram first.
• Keying the beat from the drum MIDI with its lag applied the wrong way: every snap half a beat off.

Verify

• A word-timing clip: render the section with each word's onset marked against the vocal waveform. Step the frames at each onset: anything keyed to the word starts its entrance on the onset's frame, and nothing of it reads before.
• The beat grid against the drum stem: mean distance to the audio onsets under ~0.1 beat.
• The audio lines up: ffprobe shows start_time 0 on both streams, and the clip's frame 0 is the song at the clip's own start (song time 0 for the whole film; the section's start for an excerpt).