Skip to content
Vulpine
Esc
↑↓navigate↵open⌘Jpreview
On this page

Ambient sound tags

The gateway can tag what the room sounds like around a turn (speech, music, typing, a door, a beep) with Apple's built-in SoundAnalysis classifier: 303 classes, shipped with the OS, no model to install and no microphone grant, since the…

Living doc, checked against the code 2026-11-14

The gateway can tag what the room sounds like around a turn (speech, music, typing, a door, a beep) with Apple’s built-in SoundAnalysis classifier: 303 classes, shipped with the OS, no model to install and no microphone grant, since the gateway hands it PCM. The tags are shadow evidence: they land in the session trace and on the Admin surface, and the gateway does not route, arm, refuse or speak on them. Two switches use them, both off by default:

  • [media] pause_for_request: the sidecar’s media share over the last 10 s is one input to whether the gateway pauses the room’s media through vulpine-room (roomctl) for a request (crates/voice_gateway/src/media_pause.rs).
  • [ambient] inform_brain: with mode = "shadow", a committed turn of a loadout that sets [ambient] room_note = true carries a VULPINE_ROOM_NOTE line. Otherwise the ambient_note row traces with carried: false.

Shape

client mic ─ws──▶ gateway session loop ──┬─▶ STT relay (authoritative)
                                          ├─▶ shadow invocation worker (EOU)
                                          └─▶ fox9-ambient-sidecar  (this)
                                                  │ fox9-pcm-f32le-v1 over stdio
                                                  ▼
                                          ambient_window trace events
                                          admin events (gateway.ambient)
  • Sidecar: runtime/ambient-sidecar/ambient_sidecar.swift, one Swift file built with CommandLineTools swiftc (just build-ambient-sidecar), with no Xcode, SwiftPM or Rust↔Swift bridge. It speaks the fox9-pcm-f32le-v1 stdio protocol of parakeet-eou-worker (F frames in, one JSON line back per feed, R reset, Z finalize, Q quit), so the gateway drives it with EouStdioClient. Each response carries the classification windows that closed during that feed: {"t0","t1","speech","top":[{label,score}…]}. speech is always the classifier’s confidence for its speech class, in the top-k or not.
  • Gateway: crates/voice_gateway/src/ambient.rs. The session loop fans every mic chunk into a bounded queue, with the shadow invocation detector’s 16 kHz f32 normalization. A pump task feeds the sidecar and collects windows into a small ledger, which the loop drains after each mic frame into the trace (ambient_window, ambient_sidecar) and the Admin event sink. A full queue drops the chunk and counts it: audio never waits on classification. A spawn or feed failure parks the runtime in unavailable for the session and surfaces once; the mic path keeps flowing.
  • Config ([ambient]): mode = off | shadow (default off), worker_bin, window_s (1.5), overlap (0.5 → one window every 0.75 s), top_k (3), min_score (0.1), queue_capacity (64).
  • Admin: fields ambient.mode (editable, restart) and the window geometry (read-only) under the audio section. Events on component gateway.ambient are ambient_sidecar_ready (Info, once per session) and ambient_sidecar_unavailable (Warning, with the repair action). Labels never pass through the event sink; they live in traces only.

Operating

just build-ambient-sidecar                 # on a Mac, ~5 s
just ambient-sidecar-roll host=<host>     # stage on the gateway host at ~/.local/state/foxline/bin/
# the gateway config:
[ambient]
mode = "shadow"
worker_bin = "<home>/.local/state/foxline/bin/fox9-ambient-sidecar"   # an absolute path

Then roll the gateway. To verify, run grep ambient_sidecar ~/.local/state/foxline/traces/rust-gateway/<session>.jsonl on the gateway host: expect one ready line with classes: 303, and an ambient_window line every ~0.75 s while audio flows. To replay a raw 16 kHz f32le capture through the sidecar offline, run runtime/ambient-sidecar/drive_file.py <file> --out windows.jsonl.

The sidecar takes about 90 MB while a session has audio. The gateway spawns it per session and it exits with the session, so nothing stays resident between sessions.

Was this page helpful?