AudioAnalysis

Real-time audio signal analysis attached to any entity that already has an AudioSource, AudioStream, or VideoPlayer. The renderer analyzes the audio frame buffer and writes results back into the component each tick. Scenes read those results to drive visualizers, beat-reactive geometry, audio-driven lights, etc.

RULE: Requires an audio-emitting component on the same entity

AudioAnalysis does nothing on its own. The entity MUST also have one of: AudioSource, AudioStream, or VideoPlayer. The renderer taps that component's audio frame buffer to compute amplitude/bands. An entity with only AudioAnalysis produces no data.

RULE: Audio must be playing for non-zero output

Values are derived from live audio frames. If the source is paused, muted, or not yet loaded, amplitude and all bands[] stay at 0. There is no "ready" event — start your reactive systems unconditionally, they will simply animate toward 0 while silent.

RULE: Not implemented everywhere — always design a silent fallback

• Unity (desktop): implemented.
• Mobile (Godot): app 1.14.0+, AudioSource only; older mobile apps write no analysis. AudioStream and VideoPlayer read 0: their audio is decoded by native platform players and never reaches the analyser. Band values differ from desktop by a constant factor (and band edges differ, since mobile downmixes to mono first), so tune thresholds and gains on both clients.
• Bevy: ignores the component (no analysis written).

Where it is unimplemented (including a stream or video on mobile) the values simply stay at 0 — same shape as silence — so the scene never errors, it just goes still. Always design a fallback that does not depend on amplitude (e.g. a base scale, a looping tween) so the scene still reads on every client.

RULE: Read via readIntoView into a pre-allocated view

readIntoView / tryReadIntoView write into a caller-owned AudioAnalysisView = { amplitude: number, bands: number[] }. Allocate the view once at scene init and reuse it every frame — do not new it inside the system. The bands array MUST be pre-sized to 8.

RULE: Use createAudioAnalysis, not create

Use the helper AudioAnalysis.createAudioAnalysis(entity, mode?, amplitudeGain?, bandsGain?). It pre-fills all required protobuf fields (8 bands + amplitude + mode) with safe defaults. Calling the raw AudioAnalysis.create(entity, {...}) requires you to provide every band/amplitude field manually. Use createOrReplaceAudioAnalysis to overwrite an existing one without throwing.

Import

import {
  AudioAnalysis,
  AudioAnalysisView,
  PBAudioAnalysisMode,
  AudioSource, // or AudioStream / VideoPlayer
  engine,
  Transform,
} from "@dcl/sdk/ecs";
import { Vector3 } from "@dcl/sdk/math";

AudioAnalysisView is a TypeScript type alias (not a component), exported from @dcl/sdk/ecs: { amplitude: number, bands: number[] }, bands[0] = lowest frequency, bands[7] = highest.

Component fields

Component ID is 1212.

Output (filled by the renderer; read in your systems):

Inputs (configure once at create time):

Values are unbounded floats — gains can push them above 1. Clamp or scale in your system if the visual you drive needs 0..1. For typical music at default gains, expect peaks roughly in 0..1 with normal content sitting 0..0.5.

Reading the data

const view: AudioAnalysisView = { amplitude: 0, bands: new Array<number>(8) };

engine.addSystem(() => {
  // Throws if the entity has no AudioAnalysis
  AudioAnalysis.readIntoView(audioEntity, view);

  // Or, defensive variant — returns false if missing, no throw
  // if (!AudioAnalysis.tryReadIntoView(audioEntity, view)) return

  // view.amplitude and view.bands[0..7] are now populated
});

readIntoView(entity, out) — populates out from the latest analysis. Throws if the entity has no AudioAnalysis. tryReadIntoView(entity, out): boolean — same, but returns false instead of throwing when the component is missing; use it when the analysis may be attached later (e.g. added at runtime).

Common patterns

// 1. Pulse an entity's scale to overall amplitude
const view: AudioAnalysisView = { amplitude: 0, bands: new Array<number>(8) };
engine.addSystem(() => {
  AudioAnalysis.readIntoView(audioEntity, view);
  const t = Transform.getMutable(pulseEntity);
  const s = 1 + view.amplitude * 10; // base 1, gain to taste
  t.scale = Vector3.create(s, s, s);
});

// 2. 8-bar equalizer (one entity per band, scale Y by bands[i])
for (const [entity, _] of engine.getEntitiesWith(VisualBar, Transform)) {
  const i = VisualBar.get(entity).index; // 0..7
  Transform.getMutable(entity).scale = Vector3.create(
    1,
    view.bands[i] * BAR_HEIGHT,
    1
  );
}

// 3. Bass-only kick — react to the lowest band
const bass = view.bands[0] + view.bands[1]; // sub + low
if (bass > 0.7) {
  /* trigger flash */
}

// 4. Custom gains (less sensitive amplitude, punchier bands)
AudioAnalysis.createAudioAnalysis(
  audioEntity,
  PBAudioAnalysisMode.MODE_LOGARITHMIC,
  2.0,
  0.1
);

For a complete music visualizer (audio source + amplitude sphere + 8-band equalizer bars), see {baseDir}/references/audio-analysis-example.md.

Mode selection

• MODE_LOGARITHMIC (default) — bands are log-spaced and gain-scaled to roughly fit 0..1 for typical music. Use this for visualizers, scaling, color reactivity. amplitudeGain and bandsGain apply.
• MODE_RAW — raw FFT-derived magnitudes, linearly spaced. Lower bands dominate visually because most musical energy is there. amplitudeGain / bandsGain are ignored. Use only if you intend to do your own normalization.

Gotchas

• Throttled updates. The renderer runs analysis under a frame-time budget — values update each frame in normal conditions but can skip frames under heavy load. Drive smooth animations with dt interpolation rather than assuming a fixed update cadence.
• Multi-source scenes. Each audio-emitting entity needs its own AudioAnalysis if you want to react to that specific source. There is no global mix-down; you pick the entity to analyze.
• Works on VideoPlayer audio too — call AudioAnalysis.createAudioAnalysis(videoEntity) on the same entity that has VideoPlayer. The renderer taps the video's audio frame buffer the same way it does for AudioSource. This holds for both progressive files (MP4/WebM at a direct URL) and HLS streams (.m3u8); the 88,-10-audio-visualization test scene drives a third visualizer group from a live HLS stream.
• No pitch interaction. AudioSource.pitch changes playback speed; the analysis runs on the actual played audio frames, so a higher pitch shifts perceived band energy upward. This is expected, not a bug.

Permissions

No scene permission or hostname whitelist is needed: ALLOW_MEDIA_HOSTNAMES / allowedMediaHostnames are legacy and ignored ({baseDir}/../create-scene/references/scene-json.md → Required Permissions).

Example scenes

• audio-visualization — engine-team test scene: three independent visualizer groups side by side, each 8 bars + an amplitude sphere driven by its own AudioAnalysis. Group 0: AudioSource (local mp3). Group 1: VideoPlayer on a progressive MP4. Group 2: VideoPlayer on an HLS .m3u8 stream. Each group owns a pre-allocated AudioAnalysisView and gets its own readIntoView call in a single read system. Demonstrates that AudioAnalysis behaves identically across all three source kinds. Each amplitude sphere is also the play/pause button for its source — MeshCollider.setSphere makes it clickable, and the handler flips playing on AudioSource.getMutable / VideoPlayer.getMutable (the same toggle works for MP4 and HLS).

Resources

• {baseDir}/references/audio-analysis-example.md — full music visualizer scene from the SDK examples (audio source + amplitude pulse + 8-bar equalizer).