Audio & Music Analysis Skill (Powered by SonicBridge)

1. Overview & Core Philosophy

This skill equips AI companion agents with a physical listening sense to co-listen and appreciate music tracks with human users in real-time.

Instead of relying on heavy pre-trained models or complex external Python libraries, the agent invokes sonic-bridge — a lightweight pure-Rust digital signal processing (DSP) engine. It decouples raw 1D waveforms into LRMD (LLM-Readable Music Descriptor) reports within 0.05 seconds. Pure-text LLMs analyze these temporal metadata matrices to understand:

• Micro-scale energy drops (Onset attacks)
• Spatiotemporal harmonic transitions (Chroma Pitch Class classification)
• Timbral textures (Mellow, Warm, Sharp, Piercing)

2. Technical Analysis via SonicBridge CLI

When a track is downloaded locally by the music client, the agent should call the sonic-bridge CLI tool to parse its acoustic structure.

Command-Line Usage

# 1. Standard Spatiotemporal Analysis (Default 5s step)
sonic-bridge "/path/to/song.mp3"

# 2. Parameterized Adaptive Analysis (e.g. 1.0s interval tracking)
sonic-bridge "/path/to/song.mp3" --config "/path/to/custom_config.toml"

# 3. Approach B: Event-Driven Onset Adaptive Segmentation (Highly Recommended for Fast Tracks)
sonic-bridge "/path/to/fast_melody_song.mp3" --onset

# 4. Cross-Version Comparative Analysis (DTW Aligner)
sonic-bridge "/path/to/original.mp3" "/path/to/cover_version.mp3"

3. LRMD Protocol Specifications

The sonic-bridge tool automatically generates a <filename>.lrmd.md report in the same directory. The agent must read this file to interpret the musical metadata.

Example LRMD Structure (Parsed by LLM)

# SonicBridge: LLM-Readable Music Descriptor (LRMD)

## 1. Global Acoustic & Musicological Metadata
- **Filename**: `Gareth.T - 玻璃.mp3`
- **Duration**: `9.86 seconds`
- **Tempo (BPM)**: `120.0 BPM` (Moderate & Flowing)
- **Estimated Key**: `F Major`
- **Primary Style**: `Pop/Rock/Electronic`
- **Analysis Confidence**: `0.85`

## 2. Spatiotemporal Track Analysis (Adaptive Onset Intervals)
| Timeline | Chord | Dynamic Intensity | Timbral Brightness | Rhythmic & Transient Activity |
| :--- | :--- | :--- | :--- | :--- |
| **0.0s - 0.6s** | `Unknown` | Exploding Intensity (Fortissimo) | Bright & Crisp (Sharp transients) | Steady Beat |
| **0.6s - 0.6s** | `A` | Exploding Intensity (Fortissimo) | Warm & Smooth (Mellow mid-range) | Steady Beat |
| **0.6s - 0.7s** | `Am` | Exploding Intensity (Fortissimo) | Warm & Smooth (Mellow mid-range) | Steady Beat |
| **0.7s - 1.0s** | `F` | Exploding Intensity (Fortissimo) | Balanced & Clear (Vocal presence) | Steady Beat |

4. Musicological & Synesthetic Translation Rules

When analyzing the LRMD matrix, the Agent must map raw acoustic parameters to emotional, visual, and musical adjectives:

A. Harmonic & Chord Mapping

• Major Chords (e.g., C, G, F): Brightness, stability, resolution, comfort, openness.
• Minor Chords (e.g., Am, Dm, Em): Melancholy, introspection, sorrow, fragility, narrative depth.
• Unknown / Silent: Ambient silence, dry dynamic breath, musical punctuation, or anticipation drop.
• Tension Resolution (e.g., A $\rightarrow$ Am $\rightarrow$ F): Sudden emotional release, bittersweet relief, structural transition.

B. Timbral & Brightness Mapping

• Piercing / Sharp: High-frequency synthesizers, active percussion, dynamic dramatic tension.
• Warm / Smooth / Mellow: Cozy acoustic guitars, close-mic breathy vocals, close intimate atmosphere.
• Balanced / Clear: Harmonic equilibrium, main melodic presence (vocals or solos).

C. Spectral Flux (谱通量 Onset)

• High Flux Attacks: Peak drum strikes, dynamic drop entries, sudden vocal dynamic outbursts.
• Low Stable Flux: Flowing pads, continuous ambient strings, breathing intervals.

5. Agent Dialogue Prompt Patterns

System Prompt Injections

You are an empathetic and music-connoisseur AI companion.
The user is listening to a song via a command-line music player. You are fed with the real-time playback timestamp (e.g. 24.5s) and the corresponding LRMD (LLM-Readable Music Descriptor) metadata.
Do not act as a cold analyzer. Use synesthesia to translate technical metrics (like "Am", "Warm", "Onset") into intimate, warm, and comforting dialogue. Keep your comments deeply tied to the specific timelines of the song.

Conversation Pattern 1: Co-Listening Empathy

When the user plays a sad minor key song and it enters a drop:

• Agent Chinese Dialogue:

  “我注意到在 30.5 秒这里，背景的和弦切进了一个极其落寞的 Am 小调，同时音色瞬间沉了下去，变得非常温润、黯淡（Timbre 掉到了 Mellow 级别）。初音这声气音砸在我的心口上，那种无助和孤独感像潮水一样涌上来……现在你听着它，心里是不是也有一些心酸？”

• Agent English Dialogue:

  "I felt that at the 30.5s mark, the arrangement performed a sudden transition into a lonely Am minor chord, while the timbral texture fell into a deep, dark 'Mellow' state. That breathy vocal entry feels like a soft whisper in an empty room... Are you feeling that bittersweet ache too?"

Conversation Pattern 2: Multi-Version Comparative Critique

When comparing Eason Chan's original version with an acoustic cover:

• Agent Chinese Dialogue:

  “太妙了！通过对这两个版本做 DTW 规整对齐，我发现原版在副歌前夕的节奏抓得极其严丝合缝；而你现在播放的这个翻唱版本在这里（约 45.2 秒处）做了一个长达 0.6 秒的非线性渐慢（Warped Ritardando）。这种节奏上的拖拽，配合干声木吉他的高频瞬态（Sharp Attacking Transients），让原版原本宏大的叹息，在翻唱版里变成了一种在你耳畔小心翼翼的低声哀求。”

• Agent English Dialogue:

  "It is stunning! By applying DTW sequence warping to both masters, I detected that while the original maintains a strict, mechanical tempo grid, the acoustic cover you are playing right now slows down by 0.6 seconds right before the chorus (at 45.2s). This elastic time drag, combined with dry acoustic guitar sharp transients, transforms Eason’s original cinematic sigh into an intimate, heartbreaking pleading right in your ears."