Skip to content
Vulpine
Esc
↑↓navigate↵open⌘Jpreview
On this page

Speech adapter failures

What one client is told when its own speech adapters fail: the readiness text_only change for TTS and the typed stt_error {reason, recoverable, session_id} for a lost recogniser, sent to the affected connection only, [stt] on_fault (end_session or text_only, with stt_recovered when the recogniser is back), and the Client Core's typed error {code, requestId}, and the adapter tier: [adapters.supervision] restarts homectl on a jittered backoff, answers a home verb in text while it is down, and shows adapter.<name> rows in Admin Health.

Living doc, checked against the code 2026-11-14

adapter-degradation.v1 — what one client is told when its speech adapters fail

  • Status: live on protocol 5, additive. Every frame here is an optional variant or field; an older client ignores a server event type it does not know.
  • Scope: the events go to the affected connection only. A peer bound to the same gateway is never told.

Speech output (TTS): readiness

A client that declared the readiness feature gets one readiness {owner: "tts", state: "degraded", reason: "text_only", generation} when its TTS worker fails. Its replies continue as text. A client that declared chips also gets the voice_text_only chip.

Speech input (STT): stt_error

Gateway config: [stt] on_fault

This key sets what a session does when its recognizer is lost. It is a validated enum: any other word refuses the config at load.

value recoverable the session
end_session (default) false ends on the client’s next microphone frame (below). Today’s behavior, byte for byte.
text_only true stays up: typed turns and replies carry on, the microphone path refuses with a spoken reason, and the recognizer is rebuilt from the microphone path once the worker is back.

Gateway → client: stt_error

field type rule
type "stt_error"
reason transport_closed | transport_read_failed | duplicate_ready a closed set (SttErrorReason). transport_closed: the recognizer’s connection closed (a crashed or restarted sidecar). transport_read_failed: reading it failed. duplicate_ready: the recognizer announced itself twice on one connection, so its admission was revoked.
recoverable bool false under on_fault = "end_session": the session ends. true under text_only: the session stays up as text-only until the recognizer is back.
session_id string, optional the voice session whose recognizer was lost.
  • When. Once, when the session’s STT reader reports its terminal error. The reader stops, revokes the connection’s PCM admission, queues the error and wakes the session loop. The loop sends stt_error before it handles the next microphone frame. Per-utterance errors from the sidecar (an error message carrying an utterance id) are not terminal and do not send it.
  • Not feature-gated. Every client gets it; an older client ignores it.
  • Beside it, under end_session only. The same error also goes out as error {code: "stt_error", message} for a client that reads only error. Under text_only it does not, because an older client may treat an error frame as fatal. A non-terminal STT error (no typed event) goes out as error {code: "stt_error"} under both arms.
  • Under a dictation hold. The hold drops the recognizer’s error frame (traced dictation_hold_dropped_frame, family stt_error) and still sends the typed stt_error.
  • Trace. stt_error_sent {reason, recoverable, on_fault?} on the affected session’s trace. on_fault is present only under text_only ("text_only").

Then, under end_session (the default)

The client’s next microphone frame finds the recognizer gone (“STT connection is not admitted for PCM”) and the session ends. Its trace runs through gateway_shutdown_phase to owner_finalization. The client reconnects to get a new recognizer.

Then, under text_only

  • The session stays up. A microphone frame that finds the recognizer gone opens the same fault instead of ending the session (traced stt_fault_pcm_refused {opened, error}), so a frame that races the typed event is safe.
  • Typed turns work. text_utterance turns run and their replies are spoken or shown as usual.
  • The microphone refuses, with one spoken reason. While the fault holds, microphone frames are dropped before the recognizer. The first refused frame speaks one gateway line (spoken report stt_unavailable, traced stt_mic_refused {on_fault} and gateway_report_effect {report: "stt_unavailable"}) in the dictation refusals’ shape: turn_started, assistant_delta, audio, turn_completed. There is no new frame type. The line is spoken once per fault, and never twice inside 60 s. The wake fan-out and shadow lanes still hear the microphone.
  • The recognizer comes back without a rebind. The session rebuilds its recognizer, with the same backend, from the microphone path: 2 s after the fault, then on a doubling backoff capped at 30 s (traced stt_reconnect {attempt, ok, error?}). The next frame goes to the rebuilt recognizer. When it accepts the frame, the fault ends (traced stt_recovered {retries, down_ms, refused_frames}), the client gets stt_recovered and speech works again. A failed rebuild, or a frame the rebuilt recognizer refuses, backs off and the microphone keeps refusing. A recognizer that dies again opens a new fault and sends a new stt_error.
  • What is not sent. No readiness change, no chip, and no legacy error {code: "stt_error"}.

Injecting the fault on purpose: [stt] fault_injection

The recognizer runs in-process (CoreML), so a test has no worker to kill. [stt] fault_injection = true admits one owner-local Admin request. The key is a bool, default false, and needs [admin] enabled = true.

request answer
stt_fault_inject {session_id?, reason?} stt_fault_injected {session_id, reason} — the fault is queued on that session’s loop.
  • session_id names the session as its Admin row does; absent, it means the sole live session. reason is an SttErrorReason word (default transport_closed).
  • Refusals are typed Admin errors: stt_fault_injection_disabled (the flag is off), owner_local_required, stt_fault_no_session, stt_fault_session_ambiguous, stt_fault_unknown_session.
  • On its next pass the session replaces its recognizer with one that acts like a dead sidecar connection. It reports ONE terminal error (terminal: reason) and refuses every frame with “STT connection is not admitted for PCM”. Everything after that is the real path above.
  • Trace. stt_fault_injected {session_id, reason, applied} on the session’s trace, before stt_error_sent. applied: false means the session had no recognizer to lose, and nothing else follows.
  • tools/gateway/stt_fault_inject.py sends it on the gateway host (--session, --reason; one JSON line; exit 0 / 1 refused / 3 unreachable). benchmarks/qa/sections/26-stt-fault.toml reads it.

Gateway → client: stt_recovered (under text_only)

field type rule
type "stt_recovered"
session_id string, optional the voice session whose recognizer is back; the same id the fault’s stt_error named.
  • When. Once per fault, when the rebuilt recognizer accepts its first microphone frame. Never under end_session, whose fault ends the session instead.
  • Where. The affected connection only.
  • Not feature-gated. It is an optional event type within protocol 5. An older Client Core drops a type it does not know without throwing or closing.
  • What a client does. It clears the STT banner and mic refusal that stt_error {recoverable: true} raised. It also clears them on the session’s next user_transcript, so a client that misses the event still recovers.

The adapter tier: [adapters.supervision]

The gateway supervises three adapters it spawns (crates/voice_gateway/src/adapter_supervision.rs): homectl (the home bridge’s events-in stream and the home verbs) and two long-running helpers, fleetctl (vulpine-fleet run) and audioctl (vulpine-audio watch). The helpers restart through one supervisor loop (adapter_supervision::keep_alive). A helper’s first stdout line marks it up; an exit is a failure. Neither helper has a verb to refuse, so down only rests it and shows on Health.

key default rule
enabled true false is the rollback: the bridge uses its own restart loop and no verb is refused.
initial_backoff_ms 1000 the first restart’s wait, 10 to 60000.
max_backoff_ms 60000 the cap for the doubling, jitter included.
jitter 0.2 the share of the wait added or taken away at random, 0 to 0.5.
max_restarts 5 the number of failures allowed inside the window before the adapter is down.
window_ms 300000 the window that failures are counted over.
helpers true the fleet adapter and the audio watcher restart through the supervisor. false is their rollback: each goes back to its own linear loop (restart_ms times the attempt, capped at 60 s), with no Health row.
turn_path true the turn path’s three queues are bounded and a dead EOU wake worker respawns within the session (below). false is the rollback: the unbounded queues and a wake worker that stays dead until the next session.
  • States. up: the adapter answered (a hello, or a verb’s receipt). restarting {attempt}: it exited, timed out or gave no receipt; the next start waits 1 s, 2 s, 4 s and so on, up to the 60 s cap. down: more than max_restarts failures inside the window. An answer does not clear the failure window, so an adapter that says hello and dies again still goes down.
  • Degrade to text. A verb on a down adapter inside its backoff is not spawned. Its row is ok: false, reason: "adapter_down", error: "homectl is down; try again later", and the brain’s tool result carries that text at once. After the backoff, one call goes through as the probe, and a receipt marks the adapter up.
  • Trace. adapter_supervision {adapter, state, attempt, reason, backoff_ms}, one row per failure and one per change to up, in adapter-supervision.jsonl in the trace folder.
  • Admin. The Health panel shows one adapter.<name> row (kind connector) per adapter the supervisor has seen. up is ready. restarting is reconnecting with attempting and its attempt count. down is degraded (critical) with exhausted, degraded_capability: "text only: …", the last failure as terminal_reason, and the retry time in repair_action.

The turn path’s queues and the wake worker (turn_path)

  • The recognizer’s jobs: 4. All sessions share one recognizer thread. A recognition waits up to 2 s for room, then is refused stt_queue_full, which the session handles as any recognizer error ([stt] on_fault). A warm pass is skipped when the queue is full.
  • Wyoming TTS commands: 16. A full queue refuses the next sentence (tts_queue_full), which degrades to text as a TTS failure does. The last slot is kept for the reply’s end of text, so a stream is never left open.
  • Wyoming TTS frames: 64. The render loop waits for room before it reads the server’s next event, so a slow session pushes back on the socket and no frame is dropped mid-word. A session that takes nothing for 10 s fails the render (tts_frames_stalled), and the reply degrades to text.
  • The EOU wake worker. A worker that dies mid-session is dropped. The next mic frame starts a fresh one through a per-session supervisor on the budget above (eou_worker): 1 s, 2 s, 4 s between starts, up at its ready line. Past the budget the session hears one invocation error sidecar_down (wake fails open; the Admin’s Errors row names it), and a start is still tried once per cool-down.

In the Client Core

packages/voice-client hands these frames to a skin typed: {type: 'stt_error', sttError: {reason, recoverable, session_id?}} and {type: 'stt_recovered', sttRecovered: {session_id?}}. The wire’s fields keep their names.

The legacy error {code: "stt_error"} and every other gateway error arrive as {type: 'error', message, code?, requestId?}:

  • message is "<code>: <message>", for a consumer that parses it.
  • code carries the wire’s code typed (only a plain lower-case word of at most 64 characters).
  • requestId carries the wire’s optional request_id when the error answers a command by id (the read verb’s read_fetch_* errors, docs/contracts/reading.v1.md).

A client-raised error carries neither code nor requestId, so a skin that branches on code never mistakes a client diagnostic for a gateway refusal. The shared view-model in packages/vulpine-voice-ui-state (“wire notices”) holds the STT banner from stt_error to stt_recovered, and Vigil draws it with the shared chrome.

Was this page helpful?