Zum Inhalt

tvOS T2b — Phone-Mic → ATV Audio Reactivity — Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: superpowers:executing-plans (inline) — interdependent Swift units compiled together; verify with the macOS host suite + 3-platform build at the end.

Goal: Stream mic-derived audio factors from the iOS editor-remote (iPhone/iPad) to the tvOS Ambient display so the ATV visuals react to room audio.

Architecture: Phone computes the 10 audio factors via the existing pure evaluateAudioFactors and streams them as a new RemoteMessage.audioFactors(AudioFactorFrame); the ATV writes them into sim.audioFactors + renderer.audioPostFactors (already read every frame, neutral today). Decoupled: a render-loop source-closure pulls the latest frame (no SwiftUI churn).

Tech Stack: Swift, SwiftUI, Network.framework, Metal, XCTest (macOS host).

Spec: docs/superpowers/specs/2026-06-19-tvos-t2b-phone-mic-design.md

File Structure

  • Create apple/PhysarumApp/Sources/Remote/AudioFactorFrame.swift — Codable wire payload (10 floats) + mapping to/from AudioFactors / UniformInput.Audio; .neutral.
  • Modify Remote/RemoteMessage.swift — add case audioFactors(AudioFactorFrame).
  • Modify Remote/RemoteController.swift — sendAudioFactors(...) + sendAudioFactorsNeutral() + a 2nd RemoteSendGate (≈30 Hz) + a stream AutoGainState.
  • Modify Remote/AmbientHost.swift — .audioFactors → controller.receiveAudioFactors.
  • Modify Ambient/AmbientController.swift — remoteAudio: AudioFactorFrame?, receiveAudioFactors, staleness decay in tick, clear on resumeAutonomousCycle.
  • Modify Player/RenderCoordinator.swift — remoteAudioFactorsSource + applyAudio() remote branch.
  • Modify Player/MetalView.swift — pass-through remoteAudioFactorsSource.
  • Modify TV/TVPlayerView.swift — wire { controller.remoteAudio }.
  • Modify App/RootView.swift (#if os(iOS)) — 30 Hz stream timer + neutral-on-disable + streaming indicator.
  • Tests Tests/AudioFactorFrameTests.swift (new), extend Tests/RemoteMessageTests.swift, Tests/RemoteFramingTests.swift, Tests/AmbientHostApplyTests.swift, add Tests/AmbientControllerAudioTests.swift.

Task 1: AudioFactorFrame wire type + mapping

Files: Create Remote/AudioFactorFrame.swift; Test Tests/AudioFactorFrameTests.swift.

import PhysarumEngine

/// The 10 audio modulation factors (6 agent + 4 post) as a compact Codable wire payload. The phone
/// computes these from its mic + routes via `evaluateAudioFactors`; the ATV applies them directly to
/// `sim.audioFactors` + `renderer.audioPostFactors`. A separate type (not `UniformInput.Audio`, which
/// is a GPU uniform struct) keeps the wire format explicit + Equatable for tests.
struct AudioFactorFrame: Codable, Equatable {
    var moveSpeed, turnSpeed, deposit, sensorDistance, rhythmAmplitude, chaosStrength: Float
    var exposure, bloomIntensity, saturation, contrast: Float
}

extension AudioFactorFrame {
    static let neutral = AudioFactorFrame(
        moveSpeed: 1, turnSpeed: 1, deposit: 1, sensorDistance: 1, rhythmAmplitude: 1, chaosStrength: 1,
        exposure: 1, bloomIntensity: 1, saturation: 1, contrast: 1)

    init(_ f: AudioFactors) {
        let a = f.agentUniform
        self.init(moveSpeed: a.moveSpeedFactor, turnSpeed: a.turnSpeedFactor, deposit: a.depositFactor,
                  sensorDistance: a.sensorDistanceFactor, rhythmAmplitude: a.rhythmAmplitudeFactor,
                  chaosStrength: a.chaosStrengthFactor,
                  exposure: f.postExposure, bloomIntensity: f.postBloomIntensity,
                  saturation: f.postSaturation, contrast: f.postContrast)
    }

    var agentUniform: UniformInput.Audio {
        var a = UniformInput.Audio()
        a.moveSpeedFactor = moveSpeed; a.turnSpeedFactor = turnSpeed; a.depositFactor = deposit
        a.sensorDistanceFactor = sensorDistance; a.rhythmAmplitudeFactor = rhythmAmplitude
        a.chaosStrengthFactor = chaosStrength
        return a
    }
    var postExposure: Float { exposure }
    var postBloomIntensity: Float { bloomIntensity }
    var postSaturation: Float { saturation }
    var postContrast: Float { contrast }
}

Tests: .neutral all 1.0; building from a non-trivial AudioFactors (set a few targets ≠ 1 via subscript) round-trips through agentUniform/postX; JSON Codable round-trip equals original.

Task 2: RemoteMessage.audioFactors case

Files: Remote/RemoteMessage.swift; extend Tests/RemoteMessageTests.swift, Tests/RemoteFramingTests.swift.

Add case audioFactors(AudioFactorFrame) to the enum (Codable/Equatable synthesised). Test: a .audioFactors(frame) JSON-round-trips equal; RemoteFraming.encode + Decoder yields exactly the same message.

Task 3: phone send path

Files: Remote/RemoteController.swift.

Add: 

private var audioGate = RemoteSendGate(minParamInterval: 0.030)   // ≈33 Hz cap; the 30 Hz view timer paces it
private var streamAutoGain = AutoGainState()

/// Stream the current audio reactivity to the ATV as 10 factors (throttled). Recomputed here via the
/// pure `evaluateAudioFactors` (decoupled from the phone's own render loop) using a private autoGain.
func sendAudioFactors(bands: AudioBands, reactive: AudioReactive, tuning: AudioTuning) {
    guard state == .connected, audioGate.allowParams(at: now()) else { return }
    let f = evaluateAudioFactors(bands: bands, reactive: reactive, autoGain: &streamAutoGain, tuning: tuning)
    link?.send(.audioFactors(AudioFactorFrame(f)))
}

/// One neutral frame on a clean stop (audio toggled off) so the ATV relaxes immediately instead of
/// holding the last beat until its staleness timer fires.
func sendAudioFactorsNeutral() {
    guard state == .connected else { return }
    link?.send(.audioFactors(.neutral))
}
(RemoteSendGate already takes a per-instance interval — no refactor needed.)

Task 4: ATV controller — receive + staleness + disconnect

Files: Ambient/AmbientController.swift; Test Tests/AmbientControllerAudioTests.swift.

Add property + init param + logic: 

private(set) var remoteAudio: AudioFactorFrame?   // factors the tvOS renderer applies; nil = neutral
private var audioSilence: Double = .greatestFiniteMagnitude
let audioStaleAfter: Double                       // no frame for this long → relax to neutral
- init(... audioStaleAfter: Double = 0.5); store it. - func receiveAudioFactors(_ frame: AudioFactorFrame) { remoteAudio = frame; audioSilence = 0 } - In tick(_ dt:), after the existing body: if remoteAudio != nil { audioSilence += dt; if audioSilence > audioStaleAfter { remoteAudio = nil } } - In resumeAutonomousCycle(): also remoteAudio = nil (disconnect → neutral immediately).

Tests: receiveAudioFactors sets remoteAudio; tick past audioStaleAfter clears it; a fresh receiveAudioFactors resets the timer; resumeAutonomousCycle() clears it.

Task 5: AmbientHost branch

Files: Remote/AmbientHost.swift; extend Tests/AmbientHostApplyTests.swift.

Add to the switch: case .audioFactors(let frame): controller.receiveAudioFactors(frame). Test: applying .audioFactors(frame) sets controller.remoteAudio == frame.

Task 6: RenderCoordinator apply

Files: Player/RenderCoordinator.swift.

Add var remoteAudioFactorsSource: (() -> AudioFactorFrame?)?. Rewrite applyAudio(): 

private func applyAudio() {
    let ar = audioReactive
    audioInput?.setEnabled(ar.enabled)
    if let input = audioInput, ar.enabled, input.running {
        let f = evaluateAudioFactors(bands: input.latestBands, reactive: ar, autoGain: &autoGain, tuning: audioTuning)
        sim.audioFactors = f.agentUniform
        renderer.audioPostFactors = (f.postExposure, f.postBloomIntensity, f.postSaturation, f.postContrast)
        return
    }
    autoGain = AutoGainState()   // reset so a local re-enable ramps fresh
    if let rf = remoteAudioFactorsSource?() {   // tvOS sink: phone-streamed factors
        sim.audioFactors = rf.agentUniform
        renderer.audioPostFactors = (rf.postExposure, rf.postBloomIntensity, rf.postSaturation, rf.postContrast)
    } else {
        sim.audioFactors = UniformInput.Audio()
        renderer.audioPostFactors = (1, 1, 1, 1)
    }
}
(iOS/macOS leave remoteAudioFactorsSource nil → identical neutral behaviour as before.)

Task 7: MetalView pass-through

Files: Player/MetalView.swift.

Add var remoteAudioFactorsSource: (() -> AudioFactorFrame?)? = nil; in sync: c.remoteAudioFactorsSource = remoteAudioFactorsSource.

Task 8: TVPlayerView wiring

Files: TV/TVPlayerView.swift.

In the MetalView(...) call add: remoteAudioFactorsSource: { controller.remoteAudio }. (Closure defers the read → no 30 Hz SwiftUI re-render; the render loop pulls live.)

Task 9: RootView iOS streaming + indicator

Files: App/RootView.swift (#if os(iOS)).

  • Stored (inside #if os(iOS)): private let audioStreamTimer = Timer.publish(every: 1.0 / 30.0, on: .main, in: .common).autoconnect().
  • In the #if os(iOS) modifier block: 
    .onReceive(audioStreamTimer) { _ in
    guard remote.state == .connected, state.audioReactive.enabled else { return }
    remote.sendAudioFactors(bands: state.audio.latestBands, reactive: state.audioReactive, tuning: state.audioTuning)
}
.onChange(of: state.audioReactive.enabled) { _, on in
    if !on, remote.state == .connected { remote.sendAudioFactorsNeutral() }
}
  • Indicator in appleTVButton: when remote.state == .connected && state.audioReactive.enabled, overlay a small waveform glyph (green) on the appletv.fill icon. Device-verified, not host-tested.

Task 10: Verification

  • cd apple/PhysarumApp && xcodebuild -scheme Physarum -destination 'platform=macOS' test → all green incl. new tests (expect 271 + new).
  • xcodebuild ... -destination 'generic/platform=iOS Simulator' build → BUILD SUCCEEDED.
  • xcodebuild ... -destination 'generic/platform=tvOS Simulator' build → BUILD SUCCEEDED.
  • Adversarial multi-lens review of the diff; fix findings.
  • Device sign-off (Johannes): iPhone+iPad mic → ATV reacts; toggle-off/disconnect → ATV calm; survives crossfade.

Self-Review

  • Spec coverage: ① activation (Task 9) · ② wire (Tasks 1–2) · ③ send path (Task 3) · ④ apply path (Tasks 4–8) · ⑤ edge-cases: stop=neutral frame (Task 3/9), disconnect=clear (Task 4), staleness=decay (Task 4), crossfade-independent (factors applied in applyAudio, untouched by transition) · ⑥ tests (each task) — all covered.
  • Type consistency: AudioFactorFrame fields/agentUniform/postX consistent across Tasks 1/3/6/8; remoteAudioFactorsSource signature identical in Tasks 6/7/8; receiveAudioFactors/remoteAudio consistent in Tasks 4/5/8.
  • Edge refinement vs spec: spec said "ramp to neutral"; implemented as clean-stop neutral frame (smooth, phone factors already relaxed) + snap-to-neutral on stale/disconnect. Honours "never freeze on last value"; simpler + testable. (Spec edge-case wording updated to match.)