---
title: "Ambient sound tags"
description: "The gateway can tag what the room sounds like around a turn (speech, music, typing, a door, a beep) with Apple's built-in SoundAnalysis classifier: 303 classes, shipped with the OS, no model to install and no microphone grant, since the…"
type: "living"
source: "docs/ambient-sidecar.md"
verified: "2026-11-14"
group: "The voice path"
---

> **Living doc**, checked against the code 2026-11-14

The gateway can tag what the room sounds like around a turn (speech, music, typing, a door, a beep) with Apple's
built-in SoundAnalysis classifier: 303 classes, shipped with the OS, no model to install and no microphone grant,
since the gateway hands it PCM. The tags are **shadow evidence**: they land in the session trace and on the Admin
surface, and the gateway does not route, arm, refuse or speak on them. Two switches use them, both off by default:

- `[media] pause_for_request`: the sidecar's media share over the last 10 s is one input to whether the gateway
  pauses the room's media through `vulpine-room` (`roomctl`) for a request (`crates/voice_gateway/src/media_pause.rs`).
- `[ambient] inform_brain`: with `mode = "shadow"`, a committed turn of a loadout that sets `[ambient] room_note =
  true` carries a `VULPINE_ROOM_NOTE` line. Otherwise the `ambient_note` row traces with `carried: false`.


## Shape

```
client mic ─ws──▶ gateway session loop ──┬─▶ STT relay (authoritative)
                                          ├─▶ shadow invocation worker (EOU)
                                          └─▶ fox9-ambient-sidecar  (this)
                                                  │ fox9-pcm-f32le-v1 over stdio
                                                  ▼
                                          ambient_window trace events
                                          admin events (gateway.ambient)
```

- **Sidecar**: `runtime/ambient-sidecar/ambient_sidecar.swift`, one Swift file built with CommandLineTools `swiftc`
  (`just build-ambient-sidecar`), with no Xcode, SwiftPM or Rust↔Swift bridge. It speaks the `fox9-pcm-f32le-v1`
  stdio protocol of `parakeet-eou-worker` (`F` frames in, one JSON line back per feed, `R` reset, `Z` finalize, `Q`
  quit), so the gateway drives it with `EouStdioClient`. Each response carries the classification windows that closed
  during that feed: `{"t0","t1","speech","top":[{label,score}…]}`. `speech` is always the classifier's confidence for
  its `speech` class, in the top-k or not.
- **Gateway**: `crates/voice_gateway/src/ambient.rs`. The session loop fans every mic chunk into a bounded queue,
  with the shadow invocation detector's 16 kHz f32 normalization. A pump task feeds the sidecar and collects windows
  into a small ledger, which the loop drains after each mic frame into the trace (`ambient_window`,
  `ambient_sidecar`) and the Admin event sink. A full queue drops the chunk and counts it: audio never waits on
  classification. A spawn or feed failure parks the runtime in `unavailable` for the session and surfaces once; the
  mic path keeps flowing.
- **Config** (`[ambient]`): `mode = off | shadow` (default off), `worker_bin`, `window_s` (1.5), `overlap` (0.5 →
  one window every 0.75 s), `top_k` (3), `min_score` (0.1), `queue_capacity` (64).
- **Admin**: fields `ambient.mode` (editable, restart) and the window geometry (read-only) under the audio section.
  Events on component `gateway.ambient` are `ambient_sidecar_ready` (Info, once per session) and
  `ambient_sidecar_unavailable` (Warning, with the repair action). Labels never pass through the event sink; they
  live in traces only.


## Operating

```
just build-ambient-sidecar                 # on a Mac, ~5 s
just ambient-sidecar-roll host=<host>     # stage on the gateway host at ~/.local/state/foxline/bin/
# the gateway config:
[ambient]
mode = "shadow"
worker_bin = "<home>/.local/state/foxline/bin/fox9-ambient-sidecar"   # an absolute path
```

Then roll the gateway. To verify, run
`grep ambient_sidecar ~/.local/state/foxline/traces/rust-gateway/<session>.jsonl` on the gateway host: expect one `ready`
line with `classes: 303`, and an `ambient_window` line every \~0.75 s while audio flows. To replay a raw 16 kHz f32le
capture through the sidecar offline, run `runtime/ambient-sidecar/drive_file.py <file> --out windows.jsonl`.

The sidecar takes about 90 MB while a session has audio. The gateway spawns it per session and it exits with the
session, so nothing stays resident between sessions.
