Device-Feedback Iteration Implementation Plan¶

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Iterate on the shipped Capture + Audio features from device feedback: (A) audio reacts to change/transients with a default wiring; (B) capture gets save feedback, a redesigned sheet, and PNG/HEIC/Live Photo.

Architecture: (A) Add a per-band transient signal (spectral flux → adaptive baseline → fast-attack/slow-release envelope → peak-normalized) in AudioAnalyzer, carried in AudioBands; AudioRoute gains mode:{level,transient}; evaluateAudioFactors branches on it; the existing factor = 1 + signal·depth·gain is unchanged but now punches. (B) Add PhotoFormat{png,heic,livePhoto}, a CaptureOutput result type, HEIC encoding, a Live Photo path on VideoClipRecorder (paired metadata + key-frame still), and a redesigned CaptureSheet with a thumbnail preview + inline save confirmation + haptic.

Tech Stack: Swift 5 / SwiftUI, Accelerate (vDSP), AVFoundation (AVAssetWriter + metadata tracks), ImageIO (HEIC/PNG), Photos (paired resources), Metal (PhysarumEngine), XCTest.

Spec: docs/superpowers/specs/2026-06-05-device-feedback-iteration-design.md Audio research (parity/tuning): half-wave spectral flux, baseline subtraction, fast-attack/slow-release envelope, peak-norm. Live Photo research: asset-identifier UUID in still (kCGImagePropertyMakerAppleDictionary["17"]) + video (com.apple.quicktime.content.identifier) + a com.apple.quicktime.still-image-time timed metadata track; saved via paired .photo + .pairedVideo resources.

PART A — Increment 6a: Audio Reactivity v2 (transient response)¶

Task 1: Engine — RouteMode + AudioRoute.mode (back-compat)¶

Files: Modify apple/PhysarumEngine/Sources/PhysarumEngine/Model/Types.swift, Model/Preset.swift; Test apple/PhysarumEngine/Tests/PhysarumEngineTests/AudioRouteModeTests.swift (create).

[ ] Step 1: Write the failing test (back-compat decode + new field)

// apple/PhysarumEngine/Tests/PhysarumEngineTests/AudioRouteModeTests.swift
import XCTest
@testable import PhysarumEngine

final class AudioRouteModeTests: XCTestCase {
    func testDecodeWithoutModeDefaultsToLevel() throws {
        // Old preset JSON has no "mode" key on a route.
        let json = #"{"source":"bass","target":"moveSpeed","depth":1.0}"#.data(using: .utf8)!
        let route = try JSONDecoder().decode(AudioRoute.self, from: json)
        XCTAssertEqual(route.mode, .level)
    }
    func testDecodeWithMode() throws {
        let json = #"{"source":"bass","target":"bloomIntensity","depth":1.2,"mode":"transient"}"#.data(using: .utf8)!
        let route = try JSONDecoder().decode(AudioRoute.self, from: json)
        XCTAssertEqual(route.mode, .transient)
    }
    func testRoundTripEncodesMode() throws {
        let r = AudioRoute(source: .high, target: .exposure, depth: 0.6, mode: .transient)
        let data = try JSONEncoder().encode(r)
        let back = try JSONDecoder().decode(AudioRoute.self, from: data)
        XCTAssertEqual(back.mode, .transient)
    }
    func testMemberwiseInitDefaultsToLevel() {
        let r = AudioRoute(source: .bass, target: .moveSpeed, depth: 1)
        XCTAssertEqual(r.mode, .level)
    }
}

[ ] Step 2: Run to verify it fails

Run: cd apple/PhysarumEngine && swift test --filter AudioRouteModeTests Expected: FAIL to compile — AudioRoute.mode / RouteMode undefined.

[ ] Step 3: Add RouteMode to Types.swift

After the AudioBand enum in Types.swift, add:

public enum RouteMode: String, Codable, CaseIterable, Sendable {
    case level      // sustained absolute band level
    case transient  // change/onset signal (idles at 0, spikes on hits)
}

[ ] Step 4: Add mode to AudioRoute with back-compat decode (Preset.swift)

Replace the current AudioRoute struct with:

public struct AudioRoute: Codable, Sendable {
    public var source: AudioBand
    public var target: ModTarget
    public var depth: Float
    public var mode: RouteMode
    public init(source: AudioBand, target: ModTarget, depth: Float, mode: RouteMode = .level) {
        self.source = source; self.target = target; self.depth = depth; self.mode = mode
    }
    // Custom decode: a missing "mode" (old preset JSON) defaults to .level.
    public init(from decoder: Decoder) throws {
        let c = try decoder.container(keyedBy: CodingKeys.self)
        source = try c.decode(AudioBand.self, forKey: .source)
        target = try c.decode(ModTarget.self, forKey: .target)
        depth = try c.decode(Float.self, forKey: .depth)
        mode = try c.decodeIfPresent(RouteMode.self, forKey: .mode) ?? .level
    }
}

[ ] Step 5: Run tests + the full engine suite

Run: cd apple/PhysarumEngine && swift test 2>&1 | tail -5 Expected: AudioRouteModeTests pass + the existing 21 still pass (the 25 preset JSONs decode — they have routes:[], so no route-mode decode, unaffected; and any preset with routes now defaults mode to .level).

[ ] Step 6: Commit

cd "$(git rev-parse --show-toplevel)"
git add apple/PhysarumEngine/Sources/PhysarumEngine/Model/Types.swift apple/PhysarumEngine/Sources/PhysarumEngine/Model/Preset.swift apple/PhysarumEngine/Tests/PhysarumEngineTests/AudioRouteModeTests.swift
git commit -m "feat(engine): AudioRoute.mode (level/transient) with back-compatible decode"

Task 2: AudioBands.transient + evaluator branch¶

Files: Modify apple/PhysarumApp/Sources/Audio/AudioBands.swift, Sources/Audio/AudioFactors.swift; Test apple/PhysarumApp/Tests/AudioFactorsTests.swift (extend).

[ ] Step 1: Extend AudioBands with a transient triple + accessors

Replace AudioBands.swift with:

import Foundation
import PhysarumEngine

/// Three normalized (0..1) band energies (smoothed absolute `level`) plus a
/// per-band `transient` signal (change/onset, idles at 0, spikes on hits).
struct AudioBands: Equatable {
    var bass: Float
    var mid: Float
    var high: Float
    var transientBass: Float = 0
    var transientMid: Float = 0
    var transientHigh: Float = 0
    init(bass: Float, mid: Float, high: Float,
         transientBass: Float = 0, transientMid: Float = 0, transientHigh: Float = 0) {
        self.bass = bass; self.mid = mid; self.high = high
        self.transientBass = transientBass; self.transientMid = transientMid; self.transientHigh = transientHigh
    }
    static let zero = AudioBands(bass: 0, mid: 0, high: 0)
    var peak: Float { max(bass, max(mid, high)) }

    func level(_ band: AudioBand) -> Float {
        band == .bass ? bass : band == .mid ? mid : high
    }
    func transient(_ band: AudioBand) -> Float {
        band == .bass ? transientBass : band == .mid ? transientMid : transientHigh
    }
}

[ ] Step 2: Add the failing evaluator test

Append to AudioFactorsTests.swift:

href="#__codelineno-5-1"> func testTransientModeUsesTransientSignal() { var ag = AutoGainState() // level is high (0.9) but transient is low (0.1) — a .transient route reads 0.1, not 0.9. var bands = AudioBands(bass: 0.9, mid: 0, high: 0) bands.transientBass = 0.1 let r = AudioReactive(enabled: true, gain: 1, autoGain: false, routes: [AudioRoute(source: .bass, target: .moveSpeed, depth: 2, mode: .transient)]) let f = evaluateAudioFactors(bands: bands, reactive: r, autoGain: &ag) XCTAssertEqual(f[.moveSpeed], 1.0 + 0.1 * 2, accuracy: 1e-5) // uses transient 0.1, not level 0.9 } func testLevelModeUnchanged() { var ag = AutoGainState() var bands = AudioBands(bass: 0.5, mid: 0, high: 0) bands.transientBass = 0.9 let r = AudioReactive(enabled: true, gain: 1, autoGain: false, routes: [AudioRoute(source: .bass, target: .moveSpeed, depth: 2, mode: .level)]) let f = evaluateAudioFactors(bands: bands, reactive: r, autoGain: &ag) XCTAssertEqual(f[.moveSpeed], 1.0 + 0.5 * 2, accuracy: 1e-5) // level mode unaffected by transient }

[ ] Step 3: Run, expect fail (mode-aware branch absent; the existing code ignores mode)

Run: cd apple/PhysarumApp && xcodegen generate && xcodebuild -scheme Physarum -destination 'platform=macOS' test -only-testing:PhysarumAppTests/AudioFactorsTests 2>&1 | tail -25 Expected: the two new tests FAIL (the loop currently always reads the level).

[ ] Step 4: Branch on route.mode in evaluateAudioFactors

In AudioFactors.swift, replace the route loop body:

        for route in reactive.routes {
            let level: Float = route.source == .bass ? bands.bass
                             : route.source == .mid ? bands.mid : bands.high
            f[route.target] += level * route.depth * effGain
        }

with:

        for route in reactive.routes {
            let signal = route.mode == .transient
                ? bands.transient(route.source)
                : bands.level(route.source)
            f[route.target] += signal * route.depth * effGain
        }

[ ] Step 5: Run, expect TEST SUCCEEDED (the full AudioFactorsTests)

Run: cd apple/PhysarumApp && xcodebuild -scheme Physarum -destination 'platform=macOS' test -only-testing:PhysarumAppTests/AudioFactorsTests 2>&1 | tail -25 Expected: all pass (existing 8 + 2 new).

[ ] Step 6: Commit

cd "$(git rev-parse --show-toplevel)"
git add apple/PhysarumApp/Sources/Audio/AudioBands.swift apple/PhysarumApp/Sources/Audio/AudioFactors.swift apple/PhysarumApp/Tests/AudioFactorsTests.swift
git commit -m "feat(app): per-band transient in AudioBands + level/transient route branch"

Task 3: AudioAnalyzer — the transient pipeline (spectral flux)¶

Files: Modify apple/PhysarumApp/Sources/Audio/AudioAnalyzer.swift; Test apple/PhysarumApp/Tests/AudioTransientTests.swift (create).

[ ] Step 1: Write the failing test

// apple/PhysarumApp/Tests/AudioTransientTests.swift
import XCTest
@testable import Physarum

final class AudioTransientTests: XCTestCase {
    private func tone(freq: Double, amp: Float, sr: Double, n: Int) -> [Float] {
        (0..<n).map { amp * Float(sin(2.0 * Double.pi * freq * Double($0) / sr)) }
    }
    func testSteadyToneTransientStaysLow_OnsetSpikes() {
        let sr = 48000.0, n = 1024
        let a = AudioAnalyzer(fftSize: n, smoothing: 0)
        // Warm up the baseline with many steady bass-tone frames.
        var last = AudioBands.zero
        for _ in 0..<60 { last = a.analyze(samples: tone(freq: 120, amp: 0.5, sr: sr, n: n), sampleRate: sr) }
        XCTAssertLessThan(last.transientBass, 0.35, "steady tone should idle low")
        // A sudden loud onset (much louder bass) should spike the transient.
        let onset = a.analyze(samples: tone(freq: 120, amp: 1.0, sr: sr, n: n), sampleRate: sr)
        XCTAssertGreaterThan(onset.transientBass, last.transientBass + 0.2, "onset should spike transient")
    }
    func testSilenceTransientIsZero() {
        let sr = 48000.0, n = 1024
        let a = AudioAnalyzer(fftSize: n, smoothing: 0)
        var b = AudioBands.zero
        for _ in 0..<10 { b = a.analyze(samples: [Float](repeating: 0, count: n), sampleRate: sr) }
        XCTAssertLessThan(b.transientBass, 0.05)
        XCTAssertLessThan(b.transientMid, 0.05)
        XCTAssertLessThan(b.transientHigh, 0.05)
    }
}

[ ] Step 2: Run, expect fail (transient always 0 today)

Run: cd apple/PhysarumApp && xcodegen generate && xcodebuild -scheme Physarum -destination 'platform=macOS' test -only-testing:PhysarumAppTests/AudioTransientTests 2>&1 | tail -25 Expected: testSteadyToneTransientStaysLow_OnsetSpikes fails (transientBass is 0, onset doesn't spike).

[ ] Step 3: Add the transient pipeline to AudioAnalyzer

Add per-band bin sums of flux and per-band stateful processing. In AudioAnalyzer, add:

    // Transient state (per band): slow baseline EMA of flux, fast-attack/slow-release
    // envelope of the above-baseline deviation, and a slow rolling peak for AGC.
    private struct BandT { var baseline: Float = 0; var env: Float = 0; var peak: Float = 0.02 }
    private var tBass = BandT(), tMid = BandT(), tHigh = BandT()
    private var prevMags: [Float]?
    private static let peakFloor: Float = 0.02

    /// Sum half-wave-rectified flux over the band's bin range.
    private func bandFlux(_ mags: [Float], _ prev: [Float], sampleRate: Double, range: (Double, Double)) -> Float {
        let binHz = sampleRate / Double(fftSize)
        let lo = max(0, Int((range.0 / binHz).rounded(.down)))
        let hi = min(mags.count - 1, Int((range.1 / binHz).rounded(.up)))
        guard hi >= lo else { return 0 }
        var s: Float = 0
        for i in lo...hi { s += max(0, mags[i] - prev[i]) }
        return s
    }

    /// One band's transient (0..1): baseline-subtract, asymmetric envelope, peak-normalize.
    private func transient(_ flux: Float, _ st: inout BandT, baselineCoef: Float, releaseCoef: Float, peakCoef: Float) -> Float {
        st.baseline += baselineCoef * (flux - st.baseline)
        let dev = max(0, flux - st.baseline)
        st.env = dev > st.env ? dev : releaseCoef * st.env     // instant attack, slow release
        st.peak = max(st.env, peakCoef * st.peak)              // rolling peak, slow decay
        return min(1, max(0, st.env / max(st.peak, Self.peakFloor)))
    }

Then, in analyze(...), compute mags so it's available outside the closure (the current code returns db from the closure; also surface mags). Change the closure to also capture mags, e.g. introduce var magsOut = [Float](repeating: 0, count: halfN) before the closure and assign inside it (magsOut = mags), or restructure to return (db, mags). After the band-level computation (the raw/smoothed block), add:

        // Transient pipeline (uses the linear magnitudes, not dB).
        let hop = Float(max(1, min(samples.count, fftSize))) / Float(sampleRate)
        let baselineCoef = 1 - exp(-hop / 1.5)    // baseline EMA, ~1.5 s window
        let releaseCoef  = exp(-hop / 0.25)        // envelope release, ~250 ms tail
        let peakCoef     = exp(-hop / 3.0)         // rolling-peak decay, ~3 s
        if let prev = prevMags, prev.count == magsOut.count {
            let fB = bandFlux(magsOut, prev, sampleRate: sampleRate, range: BASS_HZ)
            let fM = bandFlux(magsOut, prev, sampleRate: sampleRate, range: MID_HZ)
            let fH = bandFlux(magsOut, prev, sampleRate: sampleRate, range: HIGH_HZ)
            smoothed.transientBass = transient(fB, &tBass, baselineCoef: baselineCoef, releaseCoef: releaseCoef, peakCoef: peakCoef)
            smoothed.transientMid  = transient(fM, &tMid,  baselineCoef: baselineCoef, releaseCoef: releaseCoef, peakCoef: peakCoef)
            smoothed.transientHigh = transient(fH, &tHigh, baselineCoef: baselineCoef, releaseCoef: releaseCoef, peakCoef: peakCoef)
        }
        prevMags = magsOut
        return smoothed

You must make magsOut (the linear scaled magnitudes, mags[i] * scale) available at this point — restructure the closure so it assigns both db and the scaled magnitudes to outer vars. Keep the existing dB/band-level path unchanged.

[ ] Step 4: Run, expect TEST SUCCEEDED

Run: cd apple/PhysarumApp && xcodebuild -scheme Physarum -destination 'platform=macOS' test -only-testing:PhysarumAppTests/AudioTransientTests -only-testing:PhysarumAppTests/AudioAnalyzerTests 2>&1 | tail -20 Expected: AudioTransientTests pass + AudioAnalyzerTests (band levels) still pass. If the onset-spike margin is marginal, the algorithm is correct — adjust the test's amplitudes (e.g. 0.4 steady → 1.0 onset) rather than weakening the pipeline; the steady-low and onset-higher relationship must hold.

[ ] Step 5: Commit

cd "$(git rev-parse --show-toplevel)"
git add apple/PhysarumApp/Sources/Audio/AudioAnalyzer.swift apple/PhysarumApp/Tests/AudioTransientTests.swift
git commit -m "feat(app): per-band spectral-flux transient signal (baseline + attack/release + peak-norm)"

Task 4: Default wiring + route mode UI¶

Files: Modify apple/PhysarumApp/Sources/Model/AppState.swift, Sources/Editor/AudioReactiveSection.swift; Test apple/PhysarumApp/Tests/AudioWiringTests.swift (create).

[ ] Step 1: Write the failing test for the default wiring

// apple/PhysarumApp/Tests/AudioWiringTests.swift
import XCTest
@testable import Physarum
import PhysarumEngine

final class AudioWiringTests: XCTestCase {
    func testDefaultWiringEnablesFiveRoutes() {
        let s = AppState()
        s.applyDefaultAudioWiring()
        let ar = s.edited.audioReactive
        XCTAssertTrue(ar.enabled); XCTAssertTrue(ar.autoGain)
        XCTAssertEqual(ar.routes.count, 5)
        // First route is the "money" route: bass transient → bloom.
        XCTAssertEqual(ar.routes[0].source, .bass)
        XCTAssertEqual(ar.routes[0].target, .bloomIntensity)
        XCTAssertEqual(ar.routes[0].mode, .transient)
        // It pushes undo (so the pre-state is recoverable).
        XCTAssertFalse(s.undoStack.isEmpty)
    }
}

[ ] Step 2: Run, expect fail (applyDefaultAudioWiring undefined)

Run: cd apple/PhysarumApp && xcodegen generate && xcodebuild -scheme Physarum -destination 'platform=macOS' test -only-testing:PhysarumAppTests/AudioWiringTests 2>&1 | tail -20

[ ] Step 3: Add applyDefaultAudioWiring to AppState

In AppState.swift (near rename/moveCustom), add:

func applyDefaultAudioWiring() {
    pushUndo()
    edited.audioReactive = AudioReactive(enabled: true, gain: 1, autoGain: true, routes: [
        AudioRoute(source: .bass, target: .bloomIntensity, depth: 1.2, mode: .transient),
        AudioRoute(source: .bass, target: .moveSpeed,      depth: 0.8, mode: .transient),
        AudioRoute(source: .high, target: .exposure,       depth: 0.6, mode: .transient),
        AudioRoute(source: .mid,  target: .chaosStrength,  depth: 0.7, mode: .transient),
        AudioRoute(source: .mid,  target: .saturation,     depth: 0.4, mode: .level),
    ])
}

[ ] Step 4: Run, expect TEST SUCCEEDED

Run: cd apple/PhysarumApp && xcodebuild -scheme Physarum -destination 'platform=macOS' test -only-testing:PhysarumAppTests/AudioWiringTests 2>&1 | tail -20

[ ] Step 5: Add the "Apply default reactivity" button + per-route mode picker (AudioReactiveSection)

In routesEditor's header HStack (next to the + button), add before the +:

                Button("Default") { state.applyDefaultAudioWiring() }
                    .font(Theme.Fonts.eyebrow).buttonStyle(.plain).foregroundStyle(Theme.Colors.accent)
                    .help("Apply a starter set of routes for instant, punchy reactivity")

Change the + button's new-route to default to transient:

                        state.edited.audioReactive.routes.append(
                            AudioRoute(source: .bass, target: .moveSpeed, depth: 1, mode: .transient))

In routeRow(_:), add a compact mode picker to the row's top HStack (after the Target picker, before the remove button):

                    Picker("", selection: Binding<RouteMode>(
                        get: {
                            guard idx < state.edited.audioReactive.routes.count else { return .level }
                            return state.edited.audioReactive.routes[idx].mode
                        },
                        set: { newValue in
                            guard idx < state.edited.audioReactive.routes.count else { return }
                            state.pushUndo(); state.edited.audioReactive.routes[idx].mode = newValue
                        })) {
                        Text("Lvl").tag(RouteMode.level)
                        Text("Trans").tag(RouteMode.transient)
                    }
                    .pickerStyle(.menu)
                    .accessibilityLabel("Mode")

[ ] Step 6: Regenerate + build (macOS + iOS sim) + app suite

Run:

cd apple/PhysarumApp && xcodegen generate \
  && xcodebuild -scheme Physarum -destination 'platform=macOS' build 2>&1 | tail -5 \
  && xcodebuild -scheme Physarum -destination 'generic/platform=iOS Simulator' build 2>&1 | tail -5 \
  && xcodebuild -scheme Physarum -destination 'platform=macOS' test 2>&1 | tail -10

Expected: BUILD SUCCEEDED (both); TEST SUCCEEDED (suite + the new audio tests).

[ ] Step 7: Commit

cd "$(git rev-parse --show-toplevel)"
git add apple/PhysarumApp/Sources/Model/AppState.swift apple/PhysarumApp/Sources/Editor/AudioReactiveSection.swift apple/PhysarumApp/Tests/AudioWiringTests.swift
git commit -m "feat(app): default audio wiring button + per-route level/transient mode picker"

PART B — Increment 6b: Capture polish¶

Task 5: PhotoFormat + HEIC export¶

Files: Modify apple/PhysarumApp/Sources/Capture/CaptureConfig.swift; Create Sources/Capture/ImageWriter.swift; Modify apple/PhysarumEngine/.../Renderer/Renderer.swift (public cgImage); Modify Sources/Capture/CaptureService.swift, Sources/Capture/CaptureSheet.swift; Test Tests/PhotoFormatTests.swift.

[ ] Step 1: Add PhotoFormat to CaptureConfig + a test

Add to CaptureConfig.swift:

enum PhotoFormat: String, CaseIterable, Identifiable, Equatable {
    case png, heic, livePhoto
    var id: String { rawValue }
    var label: String { self == .png ? "PNG" : self == .heic ? "HEIC" : "Live Photo" }
    var fileExtension: String { self == .heic ? "heic" : self == .livePhoto ? "heic" : "png" }
}

Add a photoFormat field to CaptureConfig: var photoFormat: PhotoFormat = .png. Test Tests/PhotoFormatTests.swift:

import XCTest
@testable import Physarum
final class PhotoFormatTests: XCTestCase {
    func testExtensions() {
        XCTAssertEqual(PhotoFormat.png.fileExtension, "png")
        XCTAssertEqual(PhotoFormat.heic.fileExtension, "heic")
        XCTAssertEqual(PhotoFormat.livePhoto.fileExtension, "heic")
    }
    func testLabels() { XCTAssertEqual(PhotoFormat.livePhoto.label, "Live Photo") }
}

[ ] Step 2: Engine — public cgImage(_:) (refactor the readback out of encodePNG)

In Renderer.swift, add a public method that returns the texture's CGImage (extract from the existing encodePNG readback), and make encodePNG call it:

public func cgImage(_ tex: MTLTexture) -> CGImage? {
    let w = tex.width, h = tex.height
    let bytesPerRow = w * 4
    var raw = [UInt8](repeating: 0, count: bytesPerRow * h)
    tex.getBytes(&raw, bytesPerRow: bytesPerRow, from: MTLRegionMake2D(0, 0, w, h), mipmapLevel: 0)
    let cs = CGColorSpaceCreateDeviceRGB()
    let bm = CGBitmapInfo(rawValue: CGImageAlphaInfo.premultipliedFirst.rawValue | CGBitmapInfo.byteOrder32Little.rawValue)
    guard let provider = CGDataProvider(data: Data(raw) as CFData) else { return nil }
    return CGImage(width: w, height: h, bitsPerComponent: 8, bitsPerPixel: 32, bytesPerRow: bytesPerRow,
                   space: cs, bitmapInfo: bm, provider: provider, decode: nil, shouldInterpolate: false, intent: .defaultIntent)
}

Refactor encodePNG(_:to:) to guard let img = cgImage(tex) else { throw ShaderError.compile("CGImage") } then the existing CGImageDestination png write. Run cd apple/PhysarumEngine && swift test to confirm 21 still pass (savePNG unchanged behaviour).

[ ] Step 3: ImageWriter (app) — PNG / HEIC(+JPEG fallback) with optional Live Photo asset id

// apple/PhysarumApp/Sources/Capture/ImageWriter.swift
import Foundation
import ImageIO
import UniformTypeIdentifiers
import CoreVideo

enum ImageWriter {
    static func writePNG(_ image: CGImage, to url: URL) -> Bool {
        guard let dest = CGImageDestinationCreateWithURL(url as CFURL, UTType.png.identifier as CFString, 1, nil) else { return false }
        CGImageDestinationAddImage(dest, image, nil)
        return CGImageDestinationFinalize(dest)
    }

    /// Write HEIC (fallback to JPEG if the HEIC encoder is unavailable). `assetID`,
    /// when set, embeds the Apple maker-note asset identifier (Live Photo pairing).
    /// Returns the actual file URL written (extension may be .heic or .jpg).
    static func writeStill(_ image: CGImage, baseURL: URL, assetID: String?) -> URL? {
        let heicSupported = (CGImageDestinationCopyTypeIdentifiers() as? [String] ?? []).contains(UTType.heic.identifier)
        let type = heicSupported ? UTType.heic : UTType.jpeg
        let ext = heicSupported ? "heic" : "jpg"
        let url = baseURL.deletingPathExtension().appendingPathExtension(ext)
        guard let dest = CGImageDestinationCreateWithURL(url as CFURL, type.identifier as CFString, 1, nil) else { return nil }
        var opts: [CFString: Any] = [kCGImageDestinationLossyCompressionQuality: 0.9]
        if let id = assetID { opts[kCGImagePropertyMakerAppleDictionary] = ["17": id] }
        CGImageDestinationAddImage(dest, image, opts as CFDictionary)
        return CGImageDestinationFinalize(dest) ? url : nil
    }

    /// CVPixelBuffer (BGRA) → CGImage (for the Live Photo key frame readback).
    static func cgImage(from pb: CVPixelBuffer) -> CGImage? {
        CVPixelBufferLockBaseAddress(pb, .readOnly)
        defer { CVPixelBufferUnlockBaseAddress(pb, .readOnly) }
        let w = CVPixelBufferGetWidth(pb), h = CVPixelBufferGetHeight(pb)
        let bpr = CVPixelBufferGetBytesPerRow(pb)
        guard let base = CVPixelBufferGetBaseAddress(pb) else { return nil }
        let cs = CGColorSpaceCreateDeviceRGB()
        let bm = CGBitmapInfo(rawValue: CGImageAlphaInfo.premultipliedFirst.rawValue | CGBitmapInfo.byteOrder32Little.rawValue)
        guard let ctx = CGContext(data: base, width: w, height: h, bitsPerComponent: 8, bytesPerRow: bpr, space: cs, bitmapInfo: bm.rawValue) else { return nil }
        return ctx.makeImage()
    }
}

[ ] Step 4: CaptureService.captureStill — branch on photoFormat (png / heic)

Replace captureStill's tail (after rendering into target) to use the format. Get the CGImage via renderer.cgImage(target), then:

        guard let img = renderer.cgImage(target) else { throw CaptureError.encode }
        let base = FileManager.default.temporaryDirectory
            .appendingPathComponent(captureFilename(presetName: presetName, date: Date(), ext: "png"))
        switch photoFormat {
        case .png:
            guard ImageWriter.writePNG(img, to: base) else { throw CaptureError.encode }
            return base
        case .heic, .livePhoto:   // livePhoto handled separately in Task 6; .heic writes a HEIC still
            guard let out = ImageWriter.writeStill(img, baseURL: base, assetID: nil) else { throw CaptureError.encode }
            return out
        }

Thread photoFormat from config into captureStill (add a photoFormat: PhotoFormat param, passed from run/config.photoFormat). For now .livePhoto falls through to a HEIC still; Task 6 wires the real Live Photo path.

[ ] Step 5: Add the format picker to the sheet (minimal, pre-redesign)

In CaptureSheet.swift, add @State private var photoFormat: PhotoFormat = .png; thread it into config (CaptureConfig(kind: .still, longEdge: stillLong, photoFormat: photoFormat)); in form's else (photo) branch, add a format picker:

            Picker("Format", selection: $photoFormat) {
                ForEach(PhotoFormat.allCases) { Text($0.label).tag($0) }
            }

[ ] Step 6: Regenerate + build + suite

cd apple/PhysarumApp && xcodegen generate \
  && xcodebuild -scheme Physarum -destination 'platform=macOS' build 2>&1 | tail -5 \
  && xcodebuild -scheme Physarum -destination 'generic/platform=iOS Simulator' build 2>&1 | tail -5 \
  && xcodebuild -scheme Physarum -destination 'platform=macOS' test 2>&1 | tail -8

Expected: BUILD SUCCEEDED (both); TEST SUCCEEDED.

[ ] Step 7: Commit

cd "$(git rev-parse --show-toplevel)"
git add apple/PhysarumApp/Sources/Capture/CaptureConfig.swift apple/PhysarumApp/Sources/Capture/ImageWriter.swift apple/PhysarumEngine/Sources/PhysarumEngine/Renderer/Renderer.swift apple/PhysarumApp/Sources/Capture/CaptureService.swift apple/PhysarumApp/Sources/Capture/CaptureSheet.swift apple/PhysarumApp/Tests/PhotoFormatTests.swift
git commit -m "feat(app): HEIC photo export + format picker (PNG/HEIC) + engine cgImage"

Task 6: Live Photo (paired still + motion)¶

Files: Modify apple/PhysarumApp/Sources/Capture/VideoClipRecorder.swift, Sources/Capture/CaptureService.swift, Sources/Capture/CaptureSaver.swift. Add a CaptureOutput result type. Integration test Tests/LivePhotoMetadataTests.swift.

[ ] Step 1: Add a CaptureOutput result type + thread it through CaptureService.State

In CaptureService.swift, add (top level):

enum CaptureOutput: Equatable {
    case file(URL, isVideo: Bool)
    case livePhoto(still: URL, video: URL)
    var primaryURL: URL { switch self { case .file(let u, _): return u; case .livePhoto(_, let v): return v } }
    var isVideo: Bool { switch self { case .file(_, let v): return v; case .livePhoto: return true } }
}

Change State.done(url: URL, kind: CaptureKind) → State.done(CaptureOutput). Update run/capture to produce a CaptureOutput (still file → .file(url, isVideo: false), video → .file(url, isVideo: true), live photo → .livePhoto(...)), and the Task { @MainActor } to set .done(output). Update the CURRENT (un-redesigned) CaptureSheet.doneView to consume CaptureOutput minimally so it compiles (Task 7 redesigns it):

            case .done(let out):
                doneView(out: out)

with a temporary doneView(out: CaptureOutput) that does: Save (out → saveToPhotos or saveLivePhoto), ShareLink(item: out.primaryURL), Done.

[ ] Step 2: Live Photo recorder path (VideoClipRecorder)

Add to VideoClipRecorder an optional Live Photo configuration:

    struct LivePhoto { let assetID: String; let keyFrameIndex: Int }
    private let livePhoto: LivePhoto?
    private var metaInput: AVAssetWriterInput?
    private var metaAdaptor: AVAssetWriterInputMetadataAdaptor?
    private(set) var keyFrameImage: CGImage?

Add livePhoto: LivePhoto? = nil to the init. In begin(): - before startWriting(), if livePhoto != nil: writer.metadata = [contentIdentifierItem(livePhoto!.assetID)] and add the metadata input:

        if let lp = livePhoto {
            let spec: [String: Any] = [
                kCMMetadataFormatDescriptionMetadataSpecificationKey_Identifier as String: "mdta/com.apple.quicktime.still-image-time",
                kCMMetadataFormatDescriptionMetadataSpecificationKey_DataType as String: "com.apple.metadata.datatype.int8",
            ]
            var desc: CMFormatDescription?
            CMMetadataFormatDescriptionCreateWithMetadataSpecifications(allocator: kCFAllocatorDefault, metadataType: kCMMetadataFormatType_Boxed, metadataSpecifications: [spec] as CFArray, formatDescriptionOut: &desc)
            let mInput = AVAssetWriterInput(mediaType: .metadata, outputSettings: nil, sourceFormatHint: desc)
            mInput.expectsMediaDataInRealTime = false
            if writer.canAdd(mInput) { writer.add(mInput) }
            metaInput = mInput
            metaAdaptor = AVAssetWriterInputMetadataAdaptor(assetWriterInput: mInput)
            _ = lp
        }

After startSession(atSourceTime: .zero), if livePhoto, append the still-image-time group:

        if let lp = livePhoto, let adaptor = metaAdaptor {
            let keyTime = CMTime(value: CMTimeValue(lp.keyFrameIndex), timescale: CMTimeScale(fps))
            let oneFrame = CMTime(value: 1, timescale: CMTimeScale(fps))
            adaptor.append(AVTimedMetadataGroup(items: [stillImageTimeItem()], timeRange: CMTimeRange(start: keyTime, duration: oneFrame)))
        }

Add the metadata-item helpers:

    private func contentIdentifierItem(_ id: String) -> AVMetadataItem {
        let i = AVMutableMetadataItem()
        i.key = "com.apple.quicktime.content.identifier" as NSString
        i.keySpace = AVMetadataKeySpace(rawValue: "mdta")
        i.value = id as NSString
        i.dataType = "com.apple.metadata.datatype.UTF-8"
        return i
    }
    private func stillImageTimeItem() -> AVMetadataItem {
        let i = AVMutableMetadataItem()
        i.key = "com.apple.quicktime.still-image-time" as NSString
        i.keySpace = AVMetadataKeySpace(rawValue: "mdta")
        i.value = 0 as NSNumber
        i.dataType = "com.apple.metadata.datatype.int8"
        return i
    }

In appendFrame(index:pts:), after a successful renderInto(tex) and before adaptor.append, capture the key frame:

            if let lp = livePhoto, index == lp.keyFrameIndex {
                keyFrameImage = ImageWriter.cgImage(from: pb)
            }

In finish(), also metaInput?.markAsFinished() before writer.finishWriting.

[ ] Step 3: CaptureService.captureLivePhoto + CaptureSaver.saveLivePhoto

Add to CaptureService a Live Photo branch (called from captureStill when photoFormat == .livePhoto, OR a dedicated method invoked from run): render a ~2 s HEVC clip via a VideoClipRecorder(livePhoto: LivePhoto(assetID:, keyFrameIndex: frames/2)), then write the HEIC still from recorder.keyFrameImage with the same assetID, returning .livePhoto(still:video:). Concretely add:

    private func captureLivePhoto(device: MTLDevice, renderer: Renderer, sim: SimEngine, preset: PresetParams,
                                  presetName: String, exportW: Int, exportH: Int, cover: Bool) throws -> CaptureOutput {
        let fps = 60, seconds = 2
        let frames = clipFrameCount(durationSeconds: seconds, fps: fps)
        let assetID = UUID().uuidString
        let base = captureFilename(presetName: presetName, date: Date(), ext: "mov")
        let videoURL = FileManager.default.temporaryDirectory.appendingPathComponent(base)
        let recorder = VideoClipRecorder(
            device: device, url: videoURL, width: exportW, height: exportH, fps: fps, codec: .hevc,
            livePhoto: VideoClipRecorder.LivePhoto(assetID: assetID, keyFrameIndex: frames / 2),
            renderInto: { tex in sim.step(); try renderer.renderOffscreen(sim: sim, preset: preset, into: tex, cover: cover) })
        // bridge runClip (reuse the same semaphore pattern as captureVideo)
        let sema = DispatchSemaphore(value: 0); var result: Result<URL, Error>!
        Task {
            do { let out = try await runClip(frameCount: frames, fps: fps, recorder: recorder, renderFrame: { _ in },
                                             isCancelled: { self.cancelled },
                                             progress: { p in Task { @MainActor in if case .running = self.state { self.state = .running(progress: p) } } })
                 result = .success(out) } catch { result = .failure(error) }
            sema.signal()
        }
        sema.wait()
        let movURL = try result!.get()
        guard let key = recorder.keyFrameImage else { throw CaptureError.encode }
        let stillBase = FileManager.default.temporaryDirectory.appendingPathComponent(captureFilename(presetName: presetName, date: Date(), ext: "heic"))
        guard let stillURL = ImageWriter.writeStill(key, baseURL: stillBase, assetID: assetID) else { throw CaptureError.encode }
        return .livePhoto(still: stillURL, video: movURL)
    }

Route photoFormat == .livePhoto in run/captureStill to captureLivePhoto (returning CaptureOutput). Add to CaptureSaver:

    static func saveLivePhoto(still: URL, video: URL) async -> Bool {
        let status = await PHPhotoLibrary.requestAuthorization(for: .addOnly)
        guard status == .authorized || status == .limited else { return false }
        return await withCheckedContinuation { cont in
            PHPhotoLibrary.shared().performChanges {
                let req = PHAssetCreationRequest.forAsset()
                req.addResource(with: .photo, fileURL: still, options: nil)
                req.addResource(with: .pairedVideo, fileURL: video, options: nil)
            } completionHandler: { ok, _ in cont.resume(returning: ok) }
        }
    }

[ ] Step 4: Integration test — the Live Photo .mov carries the metadata

// apple/PhysarumApp/Tests/LivePhotoMetadataTests.swift
import XCTest
import AVFoundation
import Metal
@testable import Physarum
import PhysarumEngine

final class LivePhotoMetadataTests: XCTestCase {
    func testLivePhotoMovHasContentIdentifierAndStillImageTimeTrack() async throws {
        guard MTLCreateSystemDefaultDevice() != nil else { throw XCTSkip("No Metal device") }
        let device = MTLCreateSystemDefaultDevice()!
        let id = "TEST-\(UUID().uuidString)"
        let url = FileManager.default.temporaryDirectory.appendingPathComponent("lp-test.mov")
        let rec = VideoClipRecorder(device: device, url: url, width: 128, height: 128, fps: 30, codec: .hevc,
            livePhoto: VideoClipRecorder.LivePhoto(assetID: id, keyFrameIndex: 5),
            renderInto: { _ in })   // no-op render; we only check the container metadata
        _ = try await runClip(frameCount: 12, fps: 30, recorder: rec, renderFrame: { _ in }, isCancelled: { false }, progress: { _ in })
        let asset = AVURLAsset(url: url)
        let meta = try await asset.load(.metadata)
        let cid = meta.first { $0.key as? String == "com.apple.quicktime.content.identifier" }
        XCTAssertNotNil(cid, "content.identifier missing")
        XCTAssertEqual(try await cid?.load(.stringValue), id)
        let metaTracks = try await asset.loadTracks(withMediaType: .metadata)
        XCTAssertFalse(metaTracks.isEmpty, "still-image-time metadata track missing")
        XCTAssertNotNil(rec.keyFrameImage, "key frame not captured")
        try? FileManager.default.removeItem(at: url)
    }
}

(The renderInto is a no-op here, so the key frame pixel buffer is whatever the pool gives — keyFrameImage is still non-nil. The test verifies the container-level metadata + track, which is the Live Photo correctness gate; full visual fidelity is the device sign-off.)

[ ] Step 5: Regenerate + build (macOS + iOS sim) + suite

cd apple/PhysarumApp && xcodegen generate \
  && xcodebuild -scheme Physarum -destination 'platform=macOS' build 2>&1 | tail -5 \
  && xcodebuild -scheme Physarum -destination 'generic/platform=iOS Simulator' build 2>&1 | tail -5 \
  && xcodebuild -scheme Physarum -destination 'platform=macOS' test 2>&1 | tail -10

Expected: BUILD SUCCEEDED (both); TEST SUCCEEDED (incl. LivePhotoMetadataTests).

[ ] Step 6: Commit

cd "$(git rev-parse --show-toplevel)"
git add apple/PhysarumApp/Sources/Capture/VideoClipRecorder.swift apple/PhysarumApp/Sources/Capture/CaptureService.swift apple/PhysarumApp/Sources/Capture/CaptureSaver.swift apple/PhysarumApp/Sources/Capture/CaptureSheet.swift apple/PhysarumApp/Tests/LivePhotoMetadataTests.swift
git commit -m "feat(app): Live Photo export (paired still+motion metadata) + CaptureOutput result"

Task 7: Capture sheet redesign (preview card + save feedback + haptic)¶

Files: Modify apple/PhysarumApp/Sources/Capture/CaptureSheet.swift.

[ ] Step 1: Redesign the sheet

Rewrite CaptureSheet to: a styled segmented Foto/Video; the form with format/resolution/(video) duration/codec; a centered HUD; and a done preview card — a thumbnail of the capture (load the still URL as an Image; for video/livePhoto load a frame via AVAssetImageGenerator), the filename, and PhysButton actions (ember "Save to Photos" primary, ghost Share + Done). After Save: drive a @State private var saveState: SaveState (.idle/.saving/.saved/.failed), show a spinner then a green "✓ Saved to Photos" + fire a success haptic (iOS UINotificationFeedbackGenerator().notificationOccurred(.success)). Use Theme tokens + PhysButton/PhysSection to match the app. Save routes CaptureOutput: .file(url, isVideo) → CaptureSaver.saveToPhotos; .livePhoto(still, video) → CaptureSaver.saveLivePhoto. Share shares out.primaryURL. Keep .presentationDetents([.medium]) and the adaptive popover/sheet presentation as wired in RootView.

// Sketch of the done preview card (full file rewrite — keep the form/HUD states styled too):
private enum SaveState: Equatable { case idle, saving, saved, failed }
@State private var saveState: SaveState = .idle
@State private var thumb: Image?

@ViewBuilder private func doneCard(_ out: CaptureOutput) -> some View {
    VStack(spacing: Theme.Space.s3) {
        (thumb ?? Image(systemName: "photo"))
            .resizable().aspectRatio(contentMode: .fit)
            .frame(maxHeight: 180)
            .clipShape(RoundedRectangle(cornerRadius: Theme.Radius.md))
            .overlay(RoundedRectangle(cornerRadius: Theme.Radius.md).strokeBorder(Theme.Colors.borderSubtle, lineWidth: 1))
        Text(out.primaryURL.lastPathComponent).font(Theme.Fonts.eyebrow)
            .foregroundStyle(Theme.Colors.fgTertiary).lineLimit(1).truncationMode(.middle)
        switch saveState {
        case .saved: Label("Saved to Photos", systemImage: "checkmark.circle.fill").foregroundStyle(.green).font(Theme.Fonts.label)
        case .failed: Text("Couldn't save — check permissions.").font(Theme.Fonts.eyebrow).foregroundStyle(Theme.Colors.fgTertiary)
        default: EmptyView()
        }
        #if os(iOS)
        PhysButton(title: saveState == .saving ? "Saving…" : "Save to Photos", kind: .ember) { save(out) }
        #endif
        HStack(spacing: Theme.Space.s3) {
            ShareLink(item: out.primaryURL) { Text("Share").frame(maxWidth: .infinity) }
            PhysButton(title: "Done", kind: .ghost) { isPresented = false }
        }
    }
    .task(id: out.primaryURL) { thumb = await makeThumbnail(out) }
}

private func save(_ out: CaptureOutput) {
    saveState = .saving
    Task {
        let ok: Bool
        switch out {
        case .file(let u, let v): ok = await CaptureSaver.saveToPhotos(u, isVideo: v)
        case .livePhoto(let s, let v): ok = await CaptureSaver.saveLivePhoto(still: s, video: v)
        }
        saveState = ok ? .saved : .failed
        #if os(iOS)
        if ok { UINotificationFeedbackGenerator().notificationOccurred(.success) }
        #endif
    }
}

Implement makeThumbnail(_:): for .file(url, isVideo:false) load the image file (if let d = try? Data(contentsOf: url), let ui = UIImage(data: d) … Image(uiImage:) on iOS / NSImage on macOS — use a small cross-platform helper); for video/livePhoto use AVAssetImageGenerator on the video URL at time 0. (A platform image load helper is fine; keep it bounded.)

[ ] Step 2: Regenerate + build (macOS + iOS sim) + suite

cd apple/PhysarumApp && xcodegen generate \
  && xcodebuild -scheme Physarum -destination 'platform=macOS' build 2>&1 | tail -5 \
  && xcodebuild -scheme Physarum -destination 'generic/platform=iOS Simulator' build 2>&1 | tail -5 \
  && xcodebuild -scheme Physarum -destination 'platform=macOS' test 2>&1 | tail -8

Expected: BUILD SUCCEEDED (both); TEST SUCCEEDED.

[ ] Step 3: Commit

cd "$(git rev-parse --show-toplevel)"
git add apple/PhysarumApp/Sources/Capture/CaptureSheet.swift
git commit -m "feat(app): redesigned capture sheet (preview card + inline save confirmation + haptic)"

Task 8: Gate (sim + device)¶

[ ] Step 1: Full suites + sim launch + screenshot

cd apple/PhysarumEngine && swift test 2>&1 | grep -E "Executed [0-9]+ test" | tail -1
cd apple/PhysarumApp && xcodebuild -scheme Physarum -destination 'platform=macOS' test 2>&1 | grep -E "Executed [0-9]+ tests, with|TEST SUCCEEDED" | tail -2
xcrun simctl boot 'iPhone 17 Pro' 2>/dev/null; open -a Simulator
xcodebuild -scheme Physarum -destination 'platform=iOS Simulator,name=iPhone 17 Pro' build 2>&1 | tail -3
APP=$(xcodebuild -scheme Physarum -destination 'platform=iOS Simulator,name=iPhone 17 Pro' -showBuildSettings 2>/dev/null | awk '/ BUILT_PRODUCTS_DIR =/{d=$3}/ FULL_PRODUCT_NAME =/{n=substr($0,index($0,"=")+2)}END{print d"/"n}')
xcrun simctl install booted "$APP" && xcrun simctl launch booted de.jkaindl.physarum; sleep 4
xcrun simctl io booted screenshot /tmp/iter-sim.png

Expected: engine + app suites green; redesigned capture sheet renders.

[ ] Step 2: Headless deploy to the iPhone

UDID=1D819185-3910-5389-A9BE-C377B8010156
cd apple/PhysarumApp && xcodebuild -scheme Physarum -destination "platform=iOS,id=$UDID" -allowProvisioningUpdates build 2>&1 | tail -3
APP=$(xcodebuild -scheme Physarum -destination "platform=iOS,id=$UDID" -showBuildSettings 2>/dev/null | awk '/ BUILT_PRODUCTS_DIR =/{d=$3}/ FULL_PRODUCT_NAME =/{n=substr($0,index($0,"=")+2)}END{print d"/"n}')
xcrun devicectl device install app --device "$UDID" "$APP"

[ ] Step 3: Hand-off note for Johannes

Audio: "Apply default reactivity" → music should visibly punch (kick→bloom/move, hats→exposure, snare→chaos); per-route Lvl/Trans toggle. Capture: PNG/HEIC/Live Photo; Live Photo lands in Photos as an animated Live Photo (not split); the done card shows a thumbnail + "✓ Saved to Photos" + haptic.

Verification Gate (whole iteration)¶

[ ] Engine swift test green (21 + AudioRouteModeTests). App suite green (prior + AudioFactors transient, AudioTransientTests, AudioWiringTests, PhotoFormatTests, LivePhotoMetadataTests).
[ ] macOS + iOS-sim BUILD SUCCEEDED. Sim screenshot; deployed to iPhone JK.
[ ] Johannes: audio punches on the beat with the default wiring; Live Photo + HEIC + PNG; redesigned sheet + save confirmation + haptic.

Self-Review (against the spec)¶

Spec coverage: §3.1 transient pipeline → Task 3 · §3.2 model (RouteMode/AudioRoute.mode/AudioBands.transient) → Tasks 1,2 · §3.3 evaluator branch → Task 2 · §3.4 default wiring → Task 4 · §3.5 route mode UI → Task 4 · §4.1 PhotoFormat → Task 5 · §4.2 service branch + §4.5 HEIC → Task 5 · §4.3 Live Photo recorder + §4.4 saveLivePhoto → Task 6 · §4.5 sheet redesign + save feedback + haptic → Task 7 · gate → Task 8.

Placeholder scan: none — real code/commands. (Task 7's makeThumbnail platform-image helper is described concretely; the implementer writes the small cross-platform load.)

Type consistency: RouteMode/AudioRoute.mode (engine) consistent Tasks 1,2,4. AudioBands.transient(_:)/level(_:) consistent Tasks 2,3. evaluateAudioFactors branch Task 2. PhotoFormat Tasks 5,6. CaptureOutput/State.done(CaptureOutput) consistent Tasks 6,7. VideoClipRecorder.LivePhoto/keyFrameImage consistent Task 6. ImageWriter.writeStill/writePNG/cgImage(from:) consistent Tasks 5,6,7. CaptureSaver.saveLivePhoto Task 6.