Skip to content
Vulpine
Esc
↑↓navigate↵open⌘Jpreview
On this page

Room speaker

The Voice PE puck as a fox9 frontend skin (ADR 0033): one Node bridge per device, ESPHome's native API on one side and the gateway's protocol as an ordinary client on the other; one puck turn from the on-device wake to the playback receipt.

Living doc, checked against the code 2026-11-14

satellite-esphome — the Voice PE as a fox9 Frontend Skin

The bridge (ADR 0033) is one Node process per device. On one side is the ESPHome native API (esphome-client 2.x, Noise-encrypted). On the other is the Vulpine gateway, reached as an ordinary protocol-v5 client that is also an owner. The gateway learns nothing ESPHome-specific, and arms the turn from the device’s own wake.

One puck turn through the bridge, from the on-device wake to the playback receipt:

device request(start) ─▶ respondToRequest ─▶ RUN_START, STT_START
device audio 16 kHz   ─▶ 24 kHz, 20 ms PCM16 frames ─▶ gateway (binary) + vad_hint true
gateway commit        ─▶ STT_VAD_END, STT_END ─▶ vad_hint false
first reply chunk     ─▶ TTS_START, announce(url) ─▶ device fetches once (410 after)
gateway reply chunks  ─▶ streamed into the clip until turn_audio_completed (10 s drain fallback)
announce_finished     ─▶ TTS_END, RUN_END ─▶ playback_done{turn_id, playback_id, audible_ms MEASURED}

A clip ends early three ways:

  • a cancel press or the gateway’s audio_reset aborts it;
  • the device hanging up mid-body aborts it (client_closed);
  • a device that takes nothing for 1 s plus the oldest unread chunk’s audio is detached (stale_stream).

Each cuts the response, makes the URL answer 410 and stops buffering. The bridge logs stream_aborted / stale_stream with the run’s generation. At most 64 chunks sit unread in the device’s response; the rest wait in the clip until it reads.

Run

Node ≥ 22.18 runs the TypeScript directly; there is no build step. Install the dependencies from the bun workspace: bun install --filter @fox9/satellite-esphome at the repo root.

node src/main.ts --device <puck-host> --noise-key <base64> \
  --gateway ws://<gateway-host>:8780 --room kitchen --bind <bridge-ip> --port 8862

mDNS resolves the --device name on every connect attempt, esphome-client’s own reconnects included, and logs puck_resolved {hostname, address, ms, fallback} each time. --device-fallback <addr> (env DEVICE_FALLBACK_HOST) is used when the name does not answer within 3 s.

The owner receipt persists at ~/.local/state/fox9-satellite/<device>.owner.json (0600). Every later boot is a resume hello. If the restarted gateway refuses the resume (owner_refused / owner_expired), the bridge moves the receipt aside and admits fresh once (diagnostic owner_resume_refused_fresh_fallback). A pre-protocol-5 placeholder receipt is moved aside as .v4-placeholder. ESPHOME_NOISE_KEY may replace --noise-key. Output is one JSON line per ladder event.

Other options (each has an env twin, see src/main.ts):

  • --sonos <host> plays the reply on a Sonos player;
  • --follow-up question|always|off reopens the mic after a reply;
  • --wake-word picks the on-device model;
  • --capture-dir saves each listening window;
  • --mock-device stands in for the puck for one run.

Without a device or a gateway, node src/stub-gateway.ts 8790 [reply.wav] stands in for the gateway. It answers the first vad_hint false with the WAV or a tone.

On the gateway host the bridge is a launchd agent, com.voidfab.fox9.satellite-<room>, installed by tools/satellite/install_mini_launchd.py. Its wrapper sources ~/.config/vulpine/satellite-<room>.env (0600, shape in tools/satellite/satellite.env.example) and stays inert until DEVICE_HOST is set. Exit codes: 0 when SIGTERM/SIGINT asked for it (one shutdown row); 1 on a failure or when the device’s request stream ends; 75 when the gateway link closes. GatewayLink never redials, so a gateway restart ends the process and launchd reruns the whole startup (owner resume, session, device, ring, wake word).

Beyond one turn

Announcements

Every gateway turn with audio that is not the current run’s reply plays as an announcement (serveAnnouncements): a room readout, a herdr report, a notice, an approvals card. Announcements play in arrival order, through the same stream clip and output as a reply, and never over one (one output lock). A turn the gateway reset (audio_reset) is skipped. Its playback_done goes back when the output finished or at least fetched the clip. Rows: announcement_start, announcement_played.

The follow-up window

The puck only streams inside a request, so after a reply the bridge asks it for one: a 120 ms silent clip announced with startConversation opens a request without a wake. --follow-up (env FOLLOW_UP) picks when:

  • question (the default): only after a reply whose turn_audio_completed said asked_question;
  • always;
  • off.

It runs only after a reply that played (played or played_unacked). The wakeless request caps the mic at 9 s after a question and 4 s otherwise; the wake cap is 15 s. The gateway’s own dialogue window decides what it takes. Rows: follow_up_requested, conversation_requested; the run receipt carries followUp: true.

The button, the dial, the mute switch

The firmware keeps the single press for itself (start / stop a request). The bridge gets double_press, long_press and stop_word (the on-device “Stop” model) as event telemetry and acts by phase (onButton):

phase presses effect
a reply or an announcement sounding double, long, single, stop_word cancel: the output’s own cancel (the Sonos stops now), the gateway’s interrupt, a cancel earcon
listening double, long “over”: vad_hint false, frames keep flowing until the gateway commits
between the commit and the first reply audio double, long, stop_word cancel the pending reply (interrupt; the run ends declined)
idle any nothing; the firmware handles it

With --controls gateway (env VULPINE_SATELLITE_CONTROLS; the default local is the table above), the bridge stops deciding. Every press, every dial click and the mute switch go to the gateway as a typed satellite_control frame (satellite-controls.v1). The gateway maps them by the turn machine’s state:

  • a single press is “over” while the request ear is armed, and a cancel while a reply sounds;
  • a double press starts over;
  • a long press toggles the mute;
  • the dial moves the room’s volume.

The bridge keeps two things. A single press while its output sounds also stops that output (button_forwarded, button_cancel {by: gateway}). The on-device Stop word stays its own cancel. The gateway must accept satellite_control; an older gateway refuses the frame.

The gateway answers with satellite_state {volume?, muted?} on bind and whenever the room volume or the mute settles (SatelliteStateMirror). The Sonos clip volume follows the room’s volume, and the dial then stops moving it here. The puck’s own Mute switch is set to the gateway’s mute. The switch’s report of a position the gateway asked for is not sent back (satellite_mute_echo_suppressed); a hand flip is.

Also:

  • a request whose wake phrase is Stop (stop said with nothing playing) is closed at once (run_ignored);
  • each dial click moves the Sonos clip volume by 3, and the next clip plays at it (nothing without --sonos);
  • the mute switch is only logged (puck_mute);
  • with --sonos, two short earcons play on the puck’s own speaker: a rising pair at the commit and a falling pair at a cancel.

The LED ring

Inside a run the firmware owns the ring. At connect the bridge finds the puck’s “LED Ring” light (led_ring_found / led_ring_missing). It paints only the moments the firmware leaves idle (src/led-ring.ts): teal while an announcement sounds, red when the gateway link is lost, off again at idle. Red stays until a fresh process, which the exit-75 restart provides.

satellitectl and the control socket

The bridge listens on ~/.local/state/foxline/satellite/<device>.sock: one JSON object per line in, one JSON line out (src/control-socket.ts).

op fields does
status device, room, output, wake word, running timers, today’s Stop fires
timers the running timers with seconds left
timer_start seconds (1–86400), name (≤ 40 chars, default timer) the puck counts down and rings on its own speaker
timer_cancel id (optional; all timers without it) cancels on the puck
ring state: idle, announcing, gateway_down paints the ring
wakes hours (1–48, default 24) the wakes per hour since the bridge started
wake_inbox limit (1–300, default 200) the kept wakes, newest first, voice labels joined first
wake_label id, label: real, false or null sets or clears one wake’s label (how admin)
wake_clip id, part (0–15) one 120,000-byte part of a wake’s WAV, base64
stop_fires today’s on-device Stop fires: total, silent, in_context, last_at
liveness puck_link, puck_last_heard, puck_ping_ms, heartbeat_last_row, heartbeat_interval_s, clip_store (open, unproven, held, refused)

node src/satellitectl.ts <socket> '<json>' sends one request, e.g. '{"op":"timer_start","seconds":300,"name":"pasta"}'. The brain’s pi_satellite_timer tool (catalog/pi/extensions/vulpine-tools) speaks the same socket.

The fallback voice (Wyoming)

The bridge has no TTS engine of its own. Every sound the puck plays is gateway audio or one of the bridge’s earcons. So the fallback voice is not configured here: it belongs to the path Home Assistant speaks through.

HA tts.speak (engine fox9-tts) -> tools/wyoming_tts/server.py :10200 -> busy or tts_failed
                                -> --fallback 127.0.0.1:10210 -> runtime/wyoming-kokoro (kokoro, af_heart)

The kokoro server listens on loopback port 10210, beside the Wyoming server on the gateway host. Install it with just models --only wyoming-kokoro --bootstrap, and check it with runtime/wyoming-kokoro/check.py. The one line that points the Wyoming server at it is --fallback 127.0.0.1:10210 in the nix host configuration’s fox9-wyoming-tts module, outside this repo. See runtime/wyoming-kokoro.

For Home Assistant to play an announcement on the puck itself, HA needs a media player for the puck. While this bridge holds the puck’s one voice-assistant subscription, HA’s ESPHome entry stays disabled (tools/satellite/ha_config_entry.py). In that state the fallback voice reaches HA’s other media players, not the puck.

Wake word and device errors

--wake-word (env WAKE_WORD) is activated over the API 2.5 s after the serve loop starts. The puck answers the configuration request only on the connection that holds the voice-assistant subscription. Rows: wake_word_config, wake_word_set or wake_word_missing. The device’s error log is forwarded. “Multiple API Clients” becomes voice_assistant_refused: another client, such as Home Assistant’s ESPHome entry, holds the one voice-assistant subscription.

Every wake, kept

The puck wakes on its own (micro_wake_word on the device) and tells the bridge only through a request, which carries the phrase and no score. So the bridge reads the puck’s log at DEBUG (VULPINE_WAKE_LOG=off reads ERROR only), where micro_wake_word writes Detected '<phrase>' with sliding average probability is X and max probability is Y. src/wake-ledger.ts joins each line to the request that follows (either order, 3 s).

Every wake settles once as a wake_detected row with the score when the log said it, the phase, and action:

  • run;
  • queued_busy: it arrived while a run or a reply was in flight;
  • ignored_stop: the stop model fired in silence;
  • stop_word: a stop with no request;
  • no_request: a detection the puck sent nowhere.

Each wake also writes one owner-only file pair under VULPINE_WAKE_CLIP_DIR (default ~/.local/state/foxline/satellite/wake-clips/<device>; never the system temp folder; skipped when its volume is not mounted). The pair is a JSON sidecar (ts, wake, scores, action, why, generation, outcome, turn id, levels, label) and, for a wake that streamed, the run’s first VULPINE_WAKE_CLIP_SECONDS (6) of channel 0 as a 16 kHz WAV. The store keeps 500 clips or 30 days, oldest dropped first. VULPINE_WAKE_CLIPS=off stops the files; the rows stay. The pre-roll (the sound that made it fire) is not on the API. Wakes per hour: satellitectl.ts <socket> '{"op":"wakes","hours":24}' (since the bridge started) or just apps::puck-wakes <host> (from the sidecars on the bridge’s host).

Labels (src/wake-labels.ts): each sidecar takes real or false.

  • By hand: just apps::puck-wakes <host> 24 --list --unlabeled lists the ids, just apps::puck-wake-label &lt;id> false &lt;host> sets one, and clear undoes it.
  • By voice: say “I wasn’t talking to you” while the puck still listens after a false wake. That labels the gateway’s turn outcome record. Every puck-wakes read first joins those labels from <state>/training/turn-outcomes/ onto the puck wake whose window holds the labeled record’s settle time. A wake that settled no decision (noise only) has no record, so it takes a hand label.
  • A hand label is never overwritten.

The Admin’s Wake inbox (apps/satellite-esphome/src/wake-inbox.ts): the gateway asks this socket for wake_inbox, wake_label and wake_clip, and never opens the clip folder itself. The card lists the wakes with Play and Real / False / Clear, the wakes per hour for the last 24 hours, and the sliding-average score histogram split by label. A label set there goes through the same labelSidecar as a hand label.

The log stream proves itself

esphome-client 2.0.0’s LogLevel has no CONFIG, so its DEBUG (4) is CONFIG on the ESPHome wire (NONE 0, ERROR 1, WARN 2, INFO 3, CONFIG 4, DEBUG 5, VERBOSE 6, VERY_VERBOSE 7). The puck sends a line only when the subscription level is at least the line’s, and Detected ... is a DEBUG line (ESP_LOGD). So src/device-log-heartbeat.ts subscribes at the wire’s DEBUG (5). The stock firmware package sets logger: level: DEBUG.

  • At start it asks the puck for one config dump. The logger’s own Max Level line becomes a device_log_firmware_level row: carries_wake_lines, and the yaml line an OTA would need if it said INFO.
  • Every VULPINE_DEVICE_LOG_HEARTBEAT_S (600; 0 turns it off) it writes a device_log_heartbeat row: the lines since the last row by level, the micro_wake_word lines, the last line’s age, stream (unproven, open, ended) and wake_source. wake_source is device_log, or request_only when the stream ended or the firmware filters DEBUG; the ledger still counts every wake that opened a request.
  • A stream that ends is opened again after 5 s.

node src/device-log-tail.ts <host> [seconds] [level] takes the same wire level names.

Stop fires

The firmware keeps the stop model armed, so it fires with nothing to stop, several times a day. Each fire reaches the bridge as the stop_word event. When the puck had nothing of its own to stop, a request with wake “Stop” follows about 40 ms later. src/stop-fires.ts joins them into one stop_detected row with the bridge’s context at the fire (silent: the bridge idle and no timer it set ringing; in_context otherwise) and what it did (ended_no_turn, cancelled_output, cancelled_reply, left_to_puck, none).

A silent fire’s request ends with no turn and nothing audible. Before it ends, the bridge keeps VULPINE_STOP_CLIP_SECONDS (3; 0 keeps none) of what the puck streamed, in the wake clip store (the ignored_stop wake’s WAV, 0600, the same retention). It is never sent to the gateway; the Wake inbox plays and labels it. Today’s counts are on status and stop_fires. The gateway’s Admin Health row satellite.stop_fires reads them, at most once a minute off the Health poll.

A parked bridge, and how it is found

A blocked synchronous write (for example a macOS consent prompt for a clip folder on an external volume) can park the bridge’s main thread with the process alive: no row, no heartbeat. Three guards catch it:

  • src/volume-gate.ts: a clip folder under /Volumes is touched synchronously only after an async probe proved it (the mount point, mkdir -p 0700, a write check). A probe still running after 5 s is held: one wake_clips_held row with the reason, wakes still logged (wake_clip_failed, sidecar: null), the Wake inbox ops answer an error, and nothing else waits. When the prompt is answered the probe returns and wake_clips_resumed says so. A failed probe (not mounted, denied) is wake_clips_refused, retried at most once a minute. The bridge probes at boot, so a missing grant shows at start, not at the first wake.
  • src/loop-watchdog.ts: a worker thread reads a beat the main thread stamps every second. A beat older than VULPINE_BRIDGE_STALL_S (120; 0 turns it off) writes one bridge_stalled row straight to stdout and SIGKILLs the process. launchd (KeepAlive SuccessfulExit=false) then runs the startup again.
  • src/liveness.ts and the liveness op report when the ESPHome link last heard the puck (esphome-client pings after 30 s of quiet), the heartbeat’s last row and the clip store’s state. The gateway’s Admin Health row satellite.puck_last_heard reads it and warns on a puck quiet for 2 min, a heartbeat two windows late, a held or refused clip store, or a bridge that does not answer.

A face

The puck has no screen, so main.ts declares no avatar verbs (avatar_actions: [], every action refused). A satellite that has a face (a Stack-chan, behind its own bridge) passes face: true to GatewayLink. Its hello declares set_expression, set_state, focus and clear with the eight emotions as its closed list, and the gateway’s avatar_action events reach the link’s avatar_action listener. src/face-rig.ts drives the face state table (packages/voice-client/src/presentation/faceState.ts) from them and draws it as text; attachFace(link, rig, draw) wires the two. The verb mapping and the 8-to-4 fold are in docs/contracts/avatar-action.v1.md (“A face on a satellite”).

The face also declares play_animation with Stack-chan’s hand animations (clap, thinking, rock-paper-scissors; the rig plays each for 3 s, append queues) and the mood feature. The link emits each mood event to its mood listener. attachFace folds it onto the eight emotions (FaceRig.setMood) and redraws the table with no receipt. The table’s last two rows read hands and mood.

BLE room presence

src/ble-presence.ts reads the puck’s BLE scan for the room the operator is in. It needs the BLE firmware variant (tools/satellite/firmware/fox-nine-voice-pe-ble.yaml), where the puck passes its adverts to the bridge (bluetooth_proxy, passive).

  • The bridge resolves the operator’s own phone and watch by their IRKs (VULPINE_BLE_IRKS in the env file, 0600, never argv or a log line).
  • A device is near at a smoothed RSSI of -75 dBm or more for three adverts. It stays near down to -85 dBm and is gone after 60 s unheard.
  • While a device is near, the bridge sends the gateway presence_observation {source: ble, room} once at the flip and again every minute. Only the room crosses.
  • The hello declares presence_observation only when IRKs are set. The gateway reads it as the ble room source.
  • On the stock firmware the reader logs ble_presence_unavailable once and reads nothing. VULPINE_BLE_PRESENCE=off (or --ble-presence off) turns it off.

The face’s receipts

The face bridge reads the gateway hello’s features.

  • To a gateway that offers avatar_receipt, it sends one avatar_receipt per avatar_action that carried an action_id, from the receipt FaceRig.apply computed (GatewayLink.sendAvatarReceipt).
  • To a gateway that offers rig_report and rig_face, it sends the rig’s report (FaceRig.rigReport(): the fold’s id or table, the emotion shown, the decode word text and the mood layer). It sends it after bind, after each mood and after each action, once per change (GatewayLink.reportRig). The Admin’s avatar card then reads “mood: wry → doubtful” under the rig line.

The stub gateway offers features only when a test names them (new StubGateway({ features })).

The Stack-chan bridge, simulator first

The Stack-chan bridge runs against a simulator. The firmware’s own web simulator cannot be its peer: that wasm build leaves out the conversation module and its network is a stub. So the peer is a simulated Stack-chan page that speaks xiaozhi-v1 itself, and the firmware’s own parser judges every frame either side sends.

  • The bridge (src/stackchan/bridge.ts): a face GatewayLink (the satellite_esphome_face hello, no new sender) on the gateway side, a xiaozhi-v1 WebSocket server for one device at /xiaozhi/v1/ on the other. It binds 127.0.0.1 unless --lan and mints the device’s session id itself, so no gateway owner id, token, session, turn or action id reaches the device.

    the gateway the device (xiaozhi-v1)
    user_transcript (final) stt {text}
    the face’s emotion: a held set_expression, else the folded mood, else the phase llm {emotion}, only the eight names
    the reply’s first audio chunk tts start
    the reply’s PCM (audio_chunk) binary frames, one Opus packet of the hello’s 60 ms each, between tts start and tts stop
    each reply sentence (assistant_delta, cut at sentence ends) tts sentence_start {text} at its place in the audio
    turn_audio_completed tts stop after the last frame (a sentence the audio never reached is started first); then playback_done back to the gateway
    play_animation, focus, clear custom {payload: {vulpine_face: {hands}}} and {look: {x, y}}
    error alert {status: error, message, emotion: sad}: the code’s words, never its message or anything hex

    From the device: listen start manual or auto is vad_hint true and listen stop is vad_hint false; abort is the interrupt the puck sends; listen start realtime is refused with an alert (the gateway takes turns), and so is a hello with bad audio_params (then the connection closes, 1008). A frame over 32 KiB closes it (1009); a second device is refused (1013). The gateway’s wire has no marker where one spoken segment ends, so a sentence’s caption starts at an estimate of its place in the reply’s audio (60 ms a character, src/stackchan/captions.ts). With pacing on (the default), device frames leave at that place on a device’s clock, as a real device would hear them.

  • Audio both ways. The codec is libopus 1.6.1 as WebAssembly (npm libopus-wasm 0.4.1, pinned exactly; MIT, libopus BSD-3-Clause): one module that runs under node, Bun and the page with no native install (src/stackchan/opus.ts). The firmware decodes what the SERVER hello names, so the reply is encoded to the bridge hello’s 24 kHz in 60 ms packets (AUDIO, 32 kb/s), the gateway’s own reply rate. Another rate goes through src/stackchan/resample.ts first. Frames leave on the device’s clock with two in hand (frame k at the reply’s start plus (k - 2) x 60 ms). tts stop follows the last one, and an abort drops the rest at once. The mic’s packets (16 kHz, 60 ms, the firmware’s own encoder) are decoded straight to the uplink’s 24 kHz and sent up as the link’s 20 ms PCM16 frames, only between listen start and listen stop. Anything else is dropped and counted. just apps::stackchan-opus-bench measures the codec.

  • The page (src/stackchan/sim/, served by main.ts --sim through src/stackchan/sim-server.ts, a closed file list with the types stripped at serve time): a xiaozhi-v1 client drawing the 320x240 face from applyServerEvent (sim/face.ts). Controls: Connect, Talk (listen start manual, then listen stop), Abort, Head touch (a cancel while it speaks, else Talk), Look (Stack-chan, the default, or Porcelain), Mic tone (Off, the default; Sine sweep; Speech-like chirp, streamed by Talk as the firmware’s mic would) and Frames log. Two meters read the audio: Heard decodes each reply frame for its level, Sent shows the tone going up. Under reduced motion nothing blinks or drifts. Nothing on the page plays a sound or opens a microphone.

  • Run it: just apps::stackchan-sim prints http://127.0.0.1:<port>/; the stub gateway in the same process answers Talk with a scripted face turn. Against a real gateway (nothing deploys this): node apps/satellite-esphome/src/stackchan/main.ts --gateway ws://<gateway>:8780 --room desk [--sim].

Was this page helpful?