Skip to content
Vulpine
Esc
↑↓navigate↵open⌘Jpreview
On this page

Room audio captions

A small audio language model over the last 30 s of room audio.

Living doc, checked against the code 2026-11-14

It reads the gateway’s stdio worker protocol fox9-pcm-f32le-v1: the same 16 kHz mono float32 PCM and the same ops (F frame, R reset, Z flush, Q quit) as the SoundAnalysis sidecar (runtime/ambient-sidecar). It cuts the audio into 30 s windows on a 10 s hop, asks three fixed prompts per window and writes one receipt line per window. It is shadow-only: nothing routes a turn on it, and the gateway does not spawn it.

File What it is
ambient_caption.py The worker: the window cutter, the three prompts, the receipt line. --model fake (no model; tests and protocol checks) or --model samsone134m (the spike’s model).
drive_windows.py Streams the cut windows of a manifest through the worker, joins each receipt line with the window’s known TTS-overlap spans, and writes the aggregate score.

tools/ambient/cut_windows.py cuts the windows: 50 windows of 30 s from the archived session PCM, resampled to 16 kHz, with the spans where Vulpine’s own TTS was playing. Its manifest and PCM live in ambient-caption-windows/ under the private data folder (python3 tools/repo/paths.py path private prints it), never in git.

The prompts

Prompts are lowercase ASCII: Samsone lowercases prompts, and its pruned vocabulary drops uppercase tokens.

Key Prompt
caption caption the audio
tv_or_person is the speech in this audio from a tv or speaker, or from a person in the room? answer tv, person or no speech
sounds list the sounds you hear

Protocol

stdout on start   {"type":"ready","abi":1,"session":1,"protocol":"fox9-pcm-f32le-v1","engine":"fake",
                   "sample_rate_hz":16000,"window_s":30.0,"hop_s":10.0,"prompts":[...],"version":"0.1.0"}
stdin  'F' u32le n  + n bytes f32le PCM  ->  {"type":"stream","session":S,"feed_index":i,"fed_samples":k,"windows":[...]}
stdin  'R'                               ->  {"type":"reset","session":S+1}
stdin  'Z'                               ->  {"type":"finalize","session":S,"windows":[...]}
stdin  'Q'                               ->  {"type":"stopped"}   then exit 0

A frame is 4 to 262,144 bytes and a multiple of 4. Anything else answers {"type":"error","code":"invalid_frame_size","bytes":n} and exits 1. truncated_length, truncated_pcm and unknown_op do the same. A window is emitted on the feed that completes it, so that feed costs three generations. Z emits a trailing window over the last ≤ 30 s once, if audio arrived after the last window’s end. That window has partial: true when the stream was shorter than a window.

Receipt line

One JSON object per window, inside the windows array of a stream / finalize event. --receipts out.jsonl also appends it to that file. The keys:

Key Meaning
schema fox9.ambient_caption.window.v1
session The protocol session (bumped by R).
window_index 0-based window number within the session.
t0 Window start, stream seconds since the last reset.
t1 Window end, stream seconds.
samples Samples in the window (480,000 for a full 30 s at 16 kHz).
partial True for a flushed window shorter than 30 s.
rms_dbfs The window’s level (−120 for digital silence).
model fake or samsone134m.
answers {caption, tv_or_person, sounds}: the model’s text. Household-derived: never in git.
tv_or_person tv, person, no_speech or unclear, parsed from the tv_or_person answer.
wall_ms {caption, tv_or_person, sounds}: wall time per generation, ms.
wall_ms_total The three summed: the window’s cost.
peak_rss_mb The worker process’s peak RSS so far, MB.

drive_windows.py adds manifest_id, tts_overlap_s and has_tts (TTS overlap ≥ --min-tts-s, default 1 s) to each line, and sa_agree with --soundanalysis DIR. It also writes a summary (fox9.ambient_caption.summary.v1) of aggregates only:

  • n and windows_with_tts;
  • the tv_or_person confusion, split by with_tts / without_tts;
  • bleed_cue, which scores the answer tv as a detector of Vulpine’s own loudspeaker speech (tp/fp/fn/tn, precision, recall);
  • wall_ms p50/p95/max per prompt and in total;
  • peak_rss_mb, and soundanalysis_agreement when scored.

Was this page helpful?