Zum Inhalt

Physarum Native — Audio-Reactivity Design Spec

Status: Approved · 2026-06-04 · author: Johannes (brainstormed with Claude) Builds on: the native authoring app + engine. Ports the web audio-reactivity system (src/audio/*, src/engine/Engine.ts computeAudioFactors) to native, so the native app can replace the WKWebView DMG for audio users (cutover prerequisite). Scope decision (brainstorm 2026-06-04): Microphone-only, both platforms, full 10-target routing parity. System/“tab” audio is out of scope (impossible on iOS; macOS-only via ScreenCaptureKit deferred to a fast-follow).

1. Kontext & Ziel

Audio-Reaktivität ist die letzte web-only-Funktion vor dem nativen Cutover (neben dem nun portierten Capture/Export). Gute Nachricht: die native Engine ist schon ~60% verdrahtet — AudioReactive/ AudioRoute/ModTarget/AudioBand existieren (spiegeln die Web-Schemata, alle 25 Presets tragen das Feld, default leer/enabled:false), und die Sim-Shader lesen+applizieren bereits die 6 Agent-seitigen Faktoren (UniformInput.Audio, Slots 16–21, default 1.0). Was fehlt: Audio-Input, der Route- Evaluator, das Engine-Wiring der Faktoren pro Frame, die 4 Post-seitigen Faktoren, Permissions und die Editor-UI.

2. Festgezurrte Entscheidungen (Brainstorming 2026-06-04)

Achse Entscheidung
Quellen Nur Mikrofon, iOS + macOS. System-/Tab-Audio out-of-scope (iOS unmöglich; macOS-only via ScreenCaptureKit = möglicher Fast-Follow).
Targets Volle 10-Target-Parität (6 Agent + 4 Post).
Agent-Faktoren (6) Über den bereits verdrahteten In-Shader-Mechanismus (UniformInput.Audio Slots 16–21) — die Faktoren werden gesetzt statt hart 1.0. moveSpeed/turnSpeed/deposit/sensorDistance/rhythmAmplitude/chaosStrength sind per-Species/per-Agent → müssen im Shader skaliert werden.
Post-Faktoren (4) CPU-seitig auf eine livePreset-Kopie angewandt (exposure *= f etc. für exposure/bloomIntensity/saturation/contrast — globale Post-Skalare). Keine PostParams-Layout- oder Post-Shader-Änderung. Identisch zur Web-Shader-Multiplikation.
Analyse AVAudioEngine-Mic-Tap → vDSP-FFT (Hann, fftSize 1024). 3 Bänder (bass 20–250, mid 250–4000, high 4000–16000 Hz), dB-Floor/Ceil −90/−20 → 0..1, Attack/Release-Smoothing (musikalisch, nicht jittrig). Web-parität.
Routing RouteEvaluator = Port von computeAudioFactors: additive Akkumulation pro Target, effGain = gain × autoGainMul, contribution = level × depth × effGain. AutoGain-EMA mit Web-Konstanten (Target 0.7, Peak-Floor 0.05, Max 20×, EMA-Rate 0.005).
Modulation-Scope Nur Live-View. Capture/Export bleibt deterministisch → rendert mit neutralen Faktoren (kein Audio im Export). (Audio-im-Export bräuchte einen Live-Record-Modus, separat deferred.)
Permission iOS: NSMicrophoneUsageDescription + AVAudioSession(.record/.measurement) + AVAudioApplication.requestRecordPermission. macOS: NSMicrophoneUsageDescription + App-Sandbox-Entitlement com.apple.security.device.audio-input + AVCaptureDevice.requestAccess(.audio). Anfrage bei erstmaligem Enable.
Threading Tap-Closure läuft auf Audio-Render-Thread → keine Allokation/Locks/SwiftUI dort; nur in-place vDSP, dann lock-free Single-Slot-Publish; der Render-Loop pullt die letzten Bänder pro Frame.

3. Komponenten (klare Grenzen, isoliert testbar)

3a. AudioAnalyzer (rein, testbar) — Sources/Audio/AudioAnalyzer.swift

PCM-Float-Buffer → AudioBands { bass, mid, high: Float } (0..1). Hann-Fenster (vorab gebaut), vDSP.FFT (reuse setup), Magnitude, Bucketing in 3 Bänder über Bin-Frequenzen k·sr/N, dB-Mapping (−90..−20 → 0..1), per-Band Attack/Release-One-Pole. Nimmt sampleRate als Eingabe (Hardware-abhängig). Test: synthetisches Sinus-PCM (z.B. 100 Hz → Bass hoch, 8 kHz → High hoch), Stille → ~0, Clamp 0..1.

3b. RouteEvaluator + AudioFactors (rein, testbar) — Sources/Audio/RouteEvaluator.swift

struct AudioFactors: Equatable {           // all default 1.0 (neutral)
    // agent: moveSpeed, turnSpeed, deposit, sensorDistance, rhythmAmplitude, chaosStrength
    // post:  exposure, bloomIntensity, saturation, contrast
    // (one Float per ModTarget)
}
struct AutoGainState { var runningPeak: Float = 0.05 }   // EMA carrier

func evaluateAudioFactors(bands: AudioBands, reactive: AudioReactive,
                          autoGain: inout AutoGainState) -> AudioFactors
Port of web computeAudioFactors: if !reactive.enabled → all 1.0. Else update autoGain.runningPeak EMA toward the frame peak (rate 0.005, floor 0.05), autoGainMul = reactive.autoGain ? clamp(TARGET/peak, 1, MAX) : 1; effGain = reactive.gain × autoGainMul; for each route factors[target] += level(source) × depth × effGain. Test: parity cases — disabled→all 1.0; single bass→moveSpeed route at known level/ depth/gain → exact expected factor; autoGain raises a quiet signal toward target; multiple routes to one target accumulate additively.

3c. AudioInput (I/O) — Sources/Audio/AudioInput.swift

Owns AVAudioEngine, installs the input tap (inputFormat(forBus:), never hardcode SR), runs the AudioAnalyzer on the tap buffer, publishes AudioBands lock-free (e.g. os_unfair_lock-guarded slot or an atomic triple). start()/stop(); permission request; iOS AVAudioSession setup + interruption/ route-change/media-reset observers (reinstall tap on route change, rebuild on reset). macOS: no session, AVCaptureDevice auth. Exposes var latestBands: AudioBands (main/render-thread safe read) and var levelsForMeter: AudioBands for the UI.

3d. Engine-Wiring

  • SimEngine: add a settable audioFactors (the 6 agent factors) used by writeParams instead of the hardcoded 1.0 in UniformInput.Audio. Default neutral. (No shader change — slots 16–21 already consumed.)
  • Post factors: applied CPU-side in the render loop to a livePreset copy before renderer.render(...) (exposure *= f.exposure; bloomIntensity *= f.bloomIntensity; saturation *= f.saturation; contrast *= f.contrast).
  • RenderCoordinator owns an AudioInput. Each draw() (when params.audioReactive.enabled): pull bands → evaluateAudioFactors → sim.audioFactors = agentPart; build livePreset with post factors → renderer.render(sim, preset: livePreset, ...). When disabled (or input stopped): neutral factors, livePreset == params. The AudioInput is started/stopped as enabled toggles.
  • Capture/Export untouched: CaptureService continues to render the preset with neutral factors (no AudioInput), keeping exports deterministic.

3e. UI — Sources/Editor/AudioReactiveSection.swift

Editor section mirroring the web RoutesEditor: - Enable toggle (drives AudioInput.start/stop + permission request on first enable). - AutoGain toggle + Gain slider (0.1–10×, formatted N.NNx). - 3-band live meter (bass/mid/high bars from AudioInput.levelsForMeter, ~15–30 Hz update). - Routes editor: list of routes; add/remove; per route: source (bass/mid/high), target (Menu grouped “Agent behaviour” / “Post effects”), depth slider (−2..+2). Max 16 (schema). Writes into state.edited.audioReactive (so it’s part of the preset, persists/saves like any param, undo-able). - Permission-denied state: a one-line hint + a button to open Settings.

4. Datenfluss (pro Frame, Live)

AVAudioEngine tap ──(audio thread: vDSP → AudioBands)──▶ lock-free slot
                                                              │ (render thread pulls)
RenderCoordinator.draw():  bands ─▶ evaluateAudioFactors(reactive,&autoGain) ─▶ AudioFactors
   ├─ sim.audioFactors = agent(6)         ─▶ writeParams ─▶ sim shaders (slots 16–21)
   └─ livePreset = params; post(4) applied ─▶ renderer.render(sim, livePreset)
User edits routes/gain in the editor → state.edited.audioReactive → flows via the existing MetalView.sync → coordinator.setParams path → params.audioReactive updates → next frame uses it.

5. Increments (Scope in)

  • Inc 4a — Analyse + Routing-Kern (rein, getestet): AudioBands, AudioAnalyzer (vDSP), AudioFactors, AutoGainState, evaluateAudioFactors. Unit-Tests für Web-Parität. Kein I/O, keine Engine-Änderung.
  • Inc 4b — Mic-Input + Engine-Wiring (end-to-end Live): AudioInput (AVAudioEngine, Permission, beide Plattformen, lock-free Publish); SimEngine settable audioFactors; post-Faktoren CPU-seitig im RenderCoordinator; per-Frame-Anwendung; Info.plist NSMicrophoneUsageDescription + macOS audio-input- Entitlement; ein minimaler Enable-Hook zum End-to-end-Test.
  • Inc 4c — Audio-Editor-UI: AudioReactiveSection (Enable/AutoGain/Gain/Meter/Routes) mit Web-Parität, in den Editor eingehängt, schreibt in state.edited.audioReactive.

6. Testing & Verifikation

  • Unit (App-Test-Target, kein GPU/Audio-HW):
  • AudioAnalyzer: synthetisches PCM (Sinus bei 100 Hz/1 kHz/8 kHz) → erwartetes Band dominiert; Stille→~0; Clamp 0..1; Smoothing-Monotonie.
  • evaluateAudioFactors: disabled→alle 1.0; bekannte Single-Route → exakter Faktor; AutoGain hebt leises Signal Richtung Target; Mehrfach-Routes akkumulieren additiv; depth-Vorzeichen.
  • Round-trip von AudioReactive durch Editor-Mutationen (add/remove route, gain).
  • Integration / visuell (Johannes, Mic-HW): Mic-Permission-Prompt (iOS+macOS); laut abgespielte Musik moduliert sichtbar (agent + post); AutoGain glättet; Gain-Slider wirkt; Meter bewegt sich; Enable aus → Animation neutral; Editor-Routes wirken live; Performance (Tap + 60 fps); Capture/Export bleibt audio-frei/deterministisch.
  • Gate: Engine swift test grün; App-Suite grün; macOS + iOS-sim BUILD SUCCEEDED; aufs iPhone deployed.

7. Out-of-scope (später)

System-/Tab-Audio (macOS ScreenCaptureKit; iOS unmöglich) · Audio-im-Export (bräuchte Live-Record-Modus) ·

3 Bänder / 10-Band-Spektrum · per-Route-Smoothing-Kurven-UI · tvOS · Hintergrund-Audio.

8. Offene Risiken / im Plan zu klären

  • Tap-Thread-Sicherheit: keine Allokation/Locks im Tap; lock-free Publish; Render-Loop pullt. Verifizieren, dass kein Audio-Glitch/UI-Stutter.
  • Sample-Rate-Varianz (44.1/48 kHz, Geräte-abhängig) → Band-Bin-Mapping aus der echten inputFormat ableiten, nicht hardcoden; Tap bei Route-Wechsel neu installieren.
  • Pegel-Normalisierung über Geräte (eingebaut/AirPods/USB) sehr unterschiedlich → AutoGain/dB-Floor essenziell; Web-Konstanten als Start, ggf. tunen.
  • iOS-Session-Lifecycle: Interruptions (Anruf/Siri), mediaServicesWereReset → Engine/Tap neu aufbauen.
  • Post-CPU-Multiplikation == Web-Shader-Multiplikation? Bestätigen, dass exposure/bloomIntensity/saturation/contrast im Post-Shader linear als Skalar genutzt werden (dann ist CPU-param *= factor identisch). Falls eines nichtlinear genutzt wird, dieses Target in-shader faktorisieren.
  • AudioFactors→ModTarget-Mapping driftfrei zur nativen ModTarget-Enum halten (eine Quelle).
  • Batterie/Thermik: Tap nur bei enabled; Engine stoppen wenn aus / View unsichtbar.