Room audio captions
A small audio language model over the last 30 s of room audio.
Living doc, checked against the code 2026-11-14
It reads the gateway’s stdio worker protocol
fox9-pcm-f32le-v1: the same 16 kHz mono float32 PCM and the same ops (F frame, R reset, Z flush, Q quit) as
the SoundAnalysis sidecar (runtime/ambient-sidecar). It cuts the audio
into 30 s windows on a 10 s hop, asks three fixed prompts per window and writes one receipt line per window. It is
shadow-only: nothing routes a turn on it, and the gateway does not spawn it.
| File | What it is |
|---|---|
ambient_caption.py |
The worker: the window cutter, the three prompts, the receipt line. --model fake (no model; tests and protocol checks) or --model samsone134m (the spike’s model). |
drive_windows.py |
Streams the cut windows of a manifest through the worker, joins each receipt line with the window’s known TTS-overlap spans, and writes the aggregate score. |
tools/ambient/cut_windows.py cuts the windows: 50 windows of 30 s from the
archived session PCM, resampled to 16 kHz, with the spans where Vulpine’s own TTS was playing. Its manifest and PCM live
in ambient-caption-windows/ under the private data folder (python3 tools/repo/paths.py path private prints it),
never in git.
The prompts
Prompts are lowercase ASCII: Samsone lowercases prompts, and its pruned vocabulary drops uppercase tokens.
| Key | Prompt |
|---|---|
caption |
caption the audio |
tv_or_person |
is the speech in this audio from a tv or speaker, or from a person in the room? answer tv, person or no speech |
sounds |
list the sounds you hear |
Protocol
stdout on start {"type":"ready","abi":1,"session":1,"protocol":"fox9-pcm-f32le-v1","engine":"fake",
"sample_rate_hz":16000,"window_s":30.0,"hop_s":10.0,"prompts":[...],"version":"0.1.0"}
stdin 'F' u32le n + n bytes f32le PCM -> {"type":"stream","session":S,"feed_index":i,"fed_samples":k,"windows":[...]}
stdin 'R' -> {"type":"reset","session":S+1}
stdin 'Z' -> {"type":"finalize","session":S,"windows":[...]}
stdin 'Q' -> {"type":"stopped"} then exit 0
A frame is 4 to 262,144 bytes and a multiple of 4. Anything else answers
{"type":"error","code":"invalid_frame_size","bytes":n} and exits 1. truncated_length, truncated_pcm and
unknown_op do the same. A window is emitted on the feed that completes it, so that feed costs three generations.
Z emits a trailing window over the last ≤ 30 s once, if audio arrived after the last window’s end. That window has
partial: true when the stream was shorter than a window.
Receipt line
One JSON object per window, inside the windows array of a stream / finalize event. --receipts out.jsonl also
appends it to that file. The keys:
| Key | Meaning |
|---|---|
schema |
fox9.ambient_caption.window.v1 |
session |
The protocol session (bumped by R). |
window_index |
0-based window number within the session. |
t0 |
Window start, stream seconds since the last reset. |
t1 |
Window end, stream seconds. |
samples |
Samples in the window (480,000 for a full 30 s at 16 kHz). |
partial |
True for a flushed window shorter than 30 s. |
rms_dbfs |
The window’s level (−120 for digital silence). |
model |
fake or samsone134m. |
answers |
{caption, tv_or_person, sounds}: the model’s text. Household-derived: never in git. |
tv_or_person |
tv, person, no_speech or unclear, parsed from the tv_or_person answer. |
wall_ms |
{caption, tv_or_person, sounds}: wall time per generation, ms. |
wall_ms_total |
The three summed: the window’s cost. |
peak_rss_mb |
The worker process’s peak RSS so far, MB. |
drive_windows.py adds manifest_id, tts_overlap_s and has_tts (TTS overlap ≥ --min-tts-s, default 1 s) to
each line, and sa_agree with --soundanalysis DIR. It also writes a summary (fox9.ambient_caption.summary.v1) of
aggregates only:
nandwindows_with_tts;- the
tv_or_personconfusion, split bywith_tts/without_tts; bleed_cue, which scores the answertvas a detector of Vulpine’s own loudspeaker speech (tp/fp/fn/tn, precision, recall);wall_msp50/p95/max per prompt and in total;peak_rss_mb, andsoundanalysis_agreementwhen scored.