ai-review-of-audio

A multimodal model is a useful listener and an unreliable judge. It hears pacing holes, dead air, abrupt edges, mid-sentence cuts, hidden vocals and tonal mismatches that energy-based checks miss; it transcribes lyrics badly, it cannot read absolute levels, and in pairwise tests it prefers whichever clip you played first (or second). The protocol below keeps the first and controls the second.

1. What it is good at, measured in practice

Against a 190-minute radio cut, a model reviewer produced 280 timestamped notes; the objective gate (opendubstage qc) independently caught 38 of the 139 that overlapped its checks. The model was the only ear on: cuts that start or end mid-thought; sung vocals under speech (energy-based masking is not vocal-aware); local rumble and room jumps between adjacent voices (the tonal check averages per class); music chopped mid-phrase behind a ≥ 1.5 s fade; a 60–90 s music interlude read as a pacing hole. It also caught voices buried in a generated instrumental that both Whisper and the generator's own lyric channel missed.

What it is bad at: word and lyric transcription (do not use its text); absolute loudness (it will call −24 LUFS dialogue "quiet" if the previous chunk was louder); consistency across chunks (scores drift ±2); and pairwise preference (see §4).

2. Running a review

• Proxy: audio-only aac 128k in an mp4 container (sound review), or 360p H.264 CRF 30 + 128k AAC (picture + sound). Inline upload; nothing stored server-side.
• Chunks ≤ 240 s (model context and attention both degrade past that); overlap 5 s; convert each note's clip-relative t to absolute film time.
• Brief: role ("senior documentary editor and re-recording mixer"), the kind of piece and its intent, what the narration and sources are, and the categories to report. Say "timestamps are seconds from the start of THIS clip" and "do not transcribe lyrics". Full briefs in references/prompts.md.
• Schema (structured output): summary, pacing_score (1 rushed, 5 ideal, 9 drags), mix_score (1–10), notes[] with t, t_end, category ∈ {pacing, music, dialogue, ambience, picture, praise}, severity 1–3, note, fix. Ask for praise too: it marks what to protect.
• Output: <label>.json plus a Markdown digest per chunk with scores and notes. Keep a .partial.json as chunks complete so a failed run resumes.
• Cost control: the per-chunk summaries are what you read first; drill into notes by severity.

3. Reading the notes

• Sort by severity, then by category. Severity-3 dialogue notes ("cuts off mid-sentence") go to the transcription check; severity-3 music notes ("vocals clash") go to the vocals rule; pacing notes ("15 s of dead air") go to bed coverage.
• Cross-check against opendubstage qc: match each note to the nearest problem of the same kind within ±15 s and print caught / missed per kind. The misses define which listens must stay in the gate.
• Treat the score as a trend within one run, not a number to ship on. Median mix scores of 3–7/10 across chunks were typical for a rough radio cut in practice; the same mix scored 6 after beds and transitions were added.
• Never make a content edit from the model's transcription; verify the words with ASR on the cue.

4. Pairwise A/B: position bias is real

Measured in practice: the same pair of restoration variants, swapped in order, produced the opposite preference. LLM judges have a strong, documented position bias (Zheng et al. 2023 and later audio-judge work). Rules:

1. Blind labels: variants are named A/B (or by random letters), never by method.
2. Run both orders for every pair: (A, B) and (B, A), identical prompt.
3. Count only order-consistent wins: a variant wins a pair only if it is preferred in both orders; otherwise the pair is a tie. Report wins, ties and order-flips separately; a high flip rate (> 30 %) means the pair is below the model's discrimination threshold and a human must decide.
4. Rate before you pick: ask for per-variant scores (intelligibility, naturalness, noise, 1–10) and a one-line reason, then the pick. In practice the model's pick sometimes contradicted its own ratings; treat the ratings as the data and the pick as a tiebreak.
5. Multi-variant sets (4–5 chains): present as a labelled set, randomise the order per call, run ≥ 2 orderings, and keep only variants whose rank is stable within ±1 across orderings.
6. Include a control: the original clip, or two identical clips under different labels. If the model prefers one identical clip to the other with confidence, discard the run.
7. Decision rule: a model A/B shortlists; a blind human A/B on headphones and a TV decides. Nothing ships on a model score alone (NPR §5.11; the texture-floor and WER rules in dialogue-editing-and-restoration apply regardless of the model's opinion).

A/B prompt and result table template: references/prompts.md.

5. Hidden vocals in generated underscores

Generated "instrumental" beds sometimes contain buried vocalisations, hums or words that an ASR model and the generator's own lyric channel report as empty. Before an underscore goes under speech:

1. Render the bed alone at full level, 30–60 s windows covering the whole file.
2. Ask the model: "Does this track contain any human voice, singing, humming, chanting, choir, vocoded voice or spoken words, however faint? Answer none or list each with a timestamp and a confidence 1–5. Do not transcribe words." Run twice with the windows in different orders.
3. Any report of confidence ≥ 3 in both runs: separate the vocals (BS-RoFormer), inspect the vocals stem's energy, and either use the instrumental stem or drop the track.
4. Also run a separation pass on every generated bed regardless (cheap) and flag vocal-stem energy > −40 dB relative to the mix.

6. Prompts that work and prompts that do not

• Good: concrete categories, a required fix field, "be specific and practical", a request for things that work well.
• Bad: "rate this mix" with no brief (scores collapse to 6–7); asking for lyrics or exact words; asking for absolute levels in dB; chunks over 4 minutes; asking a picture+sound review to also judge the radio cut (give it the audio-only pass separately).
• Always pass the intent (timeless, unhurried, a feature documentary, not a slideshow) or the model will optimise for a trailer.

7. Verify