---
title: "Room audio captions"
description: "A small audio language model over the last 30 s of room audio."
type: "living"
source: "runtime/ambient-caption/README.md"
verified: "2026-11-14"
group: "The voice path"
---

> **Living doc**, checked against the code 2026-11-14

It reads the gateway's stdio worker protocol
`fox9-pcm-f32le-v1`: the same 16 kHz mono float32 PCM and the same ops (`F` frame, `R` reset, `Z` flush, `Q` quit) as
the SoundAnalysis sidecar (`runtime/ambient-sidecar`). It cuts the audio
into **30 s windows on a 10 s hop**, asks three fixed prompts per window and writes one receipt line per window. It is
shadow-only: nothing routes a turn on it, and the gateway does not spawn it.


| File | What it is |
| --- | --- |
| `ambient_caption.py` | The worker: the window cutter, the three prompts, the receipt line. `--model fake` (no model; tests and protocol checks) or `--model samsone134m` (the spike's model). |
| `drive_windows.py` | Streams the cut windows of a manifest through the worker, joins each receipt line with the window's known TTS-overlap spans, and writes the aggregate score. |

`tools/ambient/cut_windows.py` cuts the windows: 50 windows of 30 s from the
archived session PCM, resampled to 16 kHz, with the spans where Vulpine's own TTS was playing. Its manifest and PCM live
in `ambient-caption-windows/` under the private data folder (`python3 tools/repo/paths.py path private` prints it),
never in git.

## The prompts

Prompts are lowercase ASCII: Samsone lowercases prompts, and its pruned vocabulary drops uppercase tokens.

| Key | Prompt |
| --- | --- |
| `caption` | `caption the audio` |
| `tv_or_person` | `is the speech in this audio from a tv or speaker, or from a person in the room? answer tv, person or no speech` |
| `sounds` | `list the sounds you hear` |

## Protocol

```text
stdout on start   {"type":"ready","abi":1,"session":1,"protocol":"fox9-pcm-f32le-v1","engine":"fake",
                   "sample_rate_hz":16000,"window_s":30.0,"hop_s":10.0,"prompts":[...],"version":"0.1.0"}
stdin  'F' u32le n  + n bytes f32le PCM  ->  {"type":"stream","session":S,"feed_index":i,"fed_samples":k,"windows":[...]}
stdin  'R'                               ->  {"type":"reset","session":S+1}
stdin  'Z'                               ->  {"type":"finalize","session":S,"windows":[...]}
stdin  'Q'                               ->  {"type":"stopped"}   then exit 0
```

A frame is 4 to 262,144 bytes and a multiple of 4. Anything else answers
`{"type":"error","code":"invalid_frame_size","bytes":n}` and exits 1. `truncated_length`, `truncated_pcm` and
`unknown_op` do the same. A window is emitted on the feed that completes it, so that feed costs three generations.
`Z` emits a trailing window over the last ≤ 30 s once, if audio arrived after the last window's end. That window has
`partial: true` when the stream was shorter than a window.

## Receipt line

One JSON object per window, inside the `windows` array of a `stream` / `finalize` event. `--receipts out.jsonl` also
appends it to that file. The keys:

| Key | Meaning |
| --- | --- |
| `schema` | `fox9.ambient_caption.window.v1` |
| `session` | The protocol session (bumped by `R`). |
| `window_index` | 0-based window number within the session. |
| `t0` | Window start, stream seconds since the last reset. |
| `t1` | Window end, stream seconds. |
| `samples` | Samples in the window (480,000 for a full 30 s at 16 kHz). |
| `partial` | True for a flushed window shorter than 30 s. |
| `rms_dbfs` | The window's level (−120 for digital silence). |
| `model` | `fake` or `samsone134m`. |
| `answers` | `{caption, tv_or_person, sounds}`: the model's text. Household-derived: never in git. |
| `tv_or_person` | `tv`, `person`, `no_speech` or `unclear`, parsed from the `tv_or_person` answer. |
| `wall_ms` | `{caption, tv_or_person, sounds}`: wall time per generation, ms. |
| `wall_ms_total` | The three summed: the window's cost. |
| `peak_rss_mb` | The worker process's peak RSS so far, MB. |

`drive_windows.py` adds `manifest_id`, `tts_overlap_s` and `has_tts` (TTS overlap ≥ `--min-tts-s`, default 1 s) to
each line, and `sa_agree` with `--soundanalysis DIR`. It also writes a summary (`fox9.ambient_caption.summary.v1`) of
aggregates only:

- `n` and `windows_with_tts`;
- the `tv_or_person` confusion, split by `with_tts` / `without_tts`;
- `bleed_cue`, which scores the answer `tv` as a detector of Vulpine's own loudspeaker speech (tp/fp/fn/tn, precision,
  recall);
- `wall_ms` p50/p95/max per prompt and in total;
- `peak_rss_mb`, and `soundanalysis_agreement` when scored.

