Ambient sound tags
The gateway can tag what the room sounds like around a turn (speech, music, typing, a door, a beep) with Apple's built-in SoundAnalysis classifier: 303 classes, shipped with the OS, no model to install and no microphone grant, since the…
Living doc, checked against the code 2026-11-14
The gateway can tag what the room sounds like around a turn (speech, music, typing, a door, a beep) with Apple’s built-in SoundAnalysis classifier: 303 classes, shipped with the OS, no model to install and no microphone grant, since the gateway hands it PCM. The tags are shadow evidence: they land in the session trace and on the Admin surface, and the gateway does not route, arm, refuse or speak on them. Two switches use them, both off by default:
[media] pause_for_request: the sidecar’s media share over the last 10 s is one input to whether the gateway pauses the room’s media throughvulpine-room(roomctl) for a request (crates/voice_gateway/src/media_pause.rs).[ambient] inform_brain: withmode = "shadow", a committed turn of a loadout that sets[ambient] room_note = truecarries aVULPINE_ROOM_NOTEline. Otherwise theambient_noterow traces withcarried: false.
Shape
client mic ─ws──▶ gateway session loop ──┬─▶ STT relay (authoritative)
├─▶ shadow invocation worker (EOU)
└─▶ fox9-ambient-sidecar (this)
│ fox9-pcm-f32le-v1 over stdio
▼
ambient_window trace events
admin events (gateway.ambient)
- Sidecar:
runtime/ambient-sidecar/ambient_sidecar.swift, one Swift file built with CommandLineToolsswiftc(just build-ambient-sidecar), with no Xcode, SwiftPM or Rust↔Swift bridge. It speaks thefox9-pcm-f32le-v1stdio protocol ofparakeet-eou-worker(Fframes in, one JSON line back per feed,Rreset,Zfinalize,Qquit), so the gateway drives it withEouStdioClient. Each response carries the classification windows that closed during that feed:{"t0","t1","speech","top":[{label,score}…]}.speechis always the classifier’s confidence for itsspeechclass, in the top-k or not. - Gateway:
crates/voice_gateway/src/ambient.rs. The session loop fans every mic chunk into a bounded queue, with the shadow invocation detector’s 16 kHz f32 normalization. A pump task feeds the sidecar and collects windows into a small ledger, which the loop drains after each mic frame into the trace (ambient_window,ambient_sidecar) and the Admin event sink. A full queue drops the chunk and counts it: audio never waits on classification. A spawn or feed failure parks the runtime inunavailablefor the session and surfaces once; the mic path keeps flowing. - Config (
[ambient]):mode = off | shadow(default off),worker_bin,window_s(1.5),overlap(0.5 → one window every 0.75 s),top_k(3),min_score(0.1),queue_capacity(64). - Admin: fields
ambient.mode(editable, restart) and the window geometry (read-only) under the audio section. Events on componentgateway.ambientareambient_sidecar_ready(Info, once per session) andambient_sidecar_unavailable(Warning, with the repair action). Labels never pass through the event sink; they live in traces only.
Operating
just build-ambient-sidecar # on a Mac, ~5 s
just ambient-sidecar-roll host=<host> # stage on the gateway host at ~/.local/state/foxline/bin/
# the gateway config:
[ambient]
mode = "shadow"
worker_bin = "<home>/.local/state/foxline/bin/fox9-ambient-sidecar" # an absolute path
Then roll the gateway. To verify, run
grep ambient_sidecar ~/.local/state/foxline/traces/rust-gateway/<session>.jsonl on the gateway host: expect one ready
line with classes: 303, and an ambient_window line every ~0.75 s while audio flows. To replay a raw 16 kHz f32le
capture through the sidecar offline, run runtime/ambient-sidecar/drive_file.py <file> --out windows.jsonl.
The sidecar takes about 90 MB while a session has audio. The gateway spawns it per session and it exits with the session, so nothing stays resident between sessions.