Speech adapter failures
What one client is told when its own speech adapters fail: the readiness text_only change for TTS and the typed stt_error {reason, recoverable, session_id} for a lost recogniser, sent to the affected connection only, [stt] on_fault (end_session or text_only, with stt_recovered when the recogniser is back), and the Client Core's typed error {code, requestId}, and the adapter tier: [adapters.supervision] restarts homectl on a jittered backoff, answers a home verb in text while it is down, and shows adapter.<name> rows in Admin Health.
Living doc, checked against the code 2026-11-14
adapter-degradation.v1 — what one client is told when its speech adapters fail
- Status: live on protocol 5, additive. Every frame here is an optional variant or field; an older client ignores a server event type it does not know.
- Scope: the events go to the affected connection only. A peer bound to the same gateway is never told.
Speech output (TTS): readiness
A client that declared the readiness feature gets one
readiness {owner: "tts", state: "degraded", reason: "text_only", generation} when its TTS worker fails. Its replies
continue as text. A client that declared chips also gets the voice_text_only chip.
Speech input (STT): stt_error
Gateway config: [stt] on_fault
This key sets what a session does when its recognizer is lost. It is a validated enum: any other word refuses the config at load.
| value | recoverable |
the session |
|---|---|---|
end_session (default) |
false |
ends on the client’s next microphone frame (below). Today’s behavior, byte for byte. |
text_only |
true |
stays up: typed turns and replies carry on, the microphone path refuses with a spoken reason, and the recognizer is rebuilt from the microphone path once the worker is back. |
Gateway → client: stt_error
| field | type | rule |
|---|---|---|
type |
"stt_error" |
|
reason |
transport_closed | transport_read_failed | duplicate_ready |
a closed set (SttErrorReason). transport_closed: the recognizer’s connection closed (a crashed or restarted sidecar). transport_read_failed: reading it failed. duplicate_ready: the recognizer announced itself twice on one connection, so its admission was revoked. |
recoverable |
bool | false under on_fault = "end_session": the session ends. true under text_only: the session stays up as text-only until the recognizer is back. |
session_id |
string, optional | the voice session whose recognizer was lost. |
- When. Once, when the session’s STT reader reports its terminal error. The reader stops, revokes the
connection’s PCM admission, queues the error and wakes the session loop. The loop sends
stt_errorbefore it handles the next microphone frame. Per-utterance errors from the sidecar (anerrormessage carrying an utterance id) are not terminal and do not send it. - Not feature-gated. Every client gets it; an older client ignores it.
- Beside it, under
end_sessiononly. The same error also goes out aserror {code: "stt_error", message}for a client that reads onlyerror. Undertext_onlyit does not, because an older client may treat anerrorframe as fatal. A non-terminal STT error (no typed event) goes out aserror {code: "stt_error"}under both arms. - Under a dictation hold. The hold drops the recognizer’s error frame (traced
dictation_hold_dropped_frame, familystt_error) and still sends the typedstt_error. - Trace.
stt_error_sent {reason, recoverable, on_fault?}on the affected session’s trace.on_faultis present only undertext_only("text_only").
Then, under end_session (the default)
The client’s next microphone frame finds the recognizer gone (“STT connection is not admitted for PCM”) and the
session ends. Its trace runs through gateway_shutdown_phase to owner_finalization. The client reconnects to get a
new recognizer.
Then, under text_only
- The session stays up. A microphone frame that finds the recognizer gone opens the same fault instead of
ending the session (traced
stt_fault_pcm_refused {opened, error}), so a frame that races the typed event is safe. - Typed turns work.
text_utteranceturns run and their replies are spoken or shown as usual. - The microphone refuses, with one spoken reason. While the fault holds, microphone frames are dropped before
the recognizer. The first refused frame speaks one gateway line (spoken report
stt_unavailable, tracedstt_mic_refused {on_fault}andgateway_report_effect {report: "stt_unavailable"}) in the dictation refusals’ shape:turn_started,assistant_delta, audio,turn_completed. There is no new frame type. The line is spoken once per fault, and never twice inside 60 s. The wake fan-out and shadow lanes still hear the microphone. - The recognizer comes back without a rebind. The session rebuilds its recognizer, with the same backend, from the
microphone path: 2 s after the fault, then on a doubling backoff capped at 30 s (traced
stt_reconnect {attempt, ok, error?}). The next frame goes to the rebuilt recognizer. When it accepts the frame, the fault ends (tracedstt_recovered {retries, down_ms, refused_frames}), the client getsstt_recoveredand speech works again. A failed rebuild, or a frame the rebuilt recognizer refuses, backs off and the microphone keeps refusing. A recognizer that dies again opens a new fault and sends a newstt_error. - What is not sent. No
readinesschange, no chip, and no legacyerror {code: "stt_error"}.
Injecting the fault on purpose: [stt] fault_injection
The recognizer runs in-process (CoreML), so a test has no worker to kill. [stt] fault_injection = true admits one
owner-local Admin request. The key is a bool, default false, and needs [admin] enabled = true.
| request | answer |
|---|---|
stt_fault_inject {session_id?, reason?} |
stt_fault_injected {session_id, reason} — the fault is queued on that session’s loop. |
session_idnames the session as its Admin row does; absent, it means the sole live session.reasonis anSttErrorReasonword (defaulttransport_closed).- Refusals are typed Admin errors:
stt_fault_injection_disabled(the flag is off),owner_local_required,stt_fault_no_session,stt_fault_session_ambiguous,stt_fault_unknown_session. - On its next pass the session replaces its recognizer with one that acts like a dead sidecar connection. It reports
ONE terminal error (
terminal: reason) and refuses every frame with “STT connection is not admitted for PCM”. Everything after that is the real path above. - Trace.
stt_fault_injected {session_id, reason, applied}on the session’s trace, beforestt_error_sent.applied: falsemeans the session had no recognizer to lose, and nothing else follows. tools/gateway/stt_fault_inject.pysends it on the gateway host (--session,--reason; one JSON line; exit 0 / 1 refused / 3 unreachable).benchmarks/qa/sections/26-stt-fault.tomlreads it.
Gateway → client: stt_recovered (under text_only)
| field | type | rule |
|---|---|---|
type |
"stt_recovered" |
|
session_id |
string, optional | the voice session whose recognizer is back; the same id the fault’s stt_error named. |
- When. Once per fault, when the rebuilt recognizer accepts its first microphone frame. Never under
end_session, whose fault ends the session instead. - Where. The affected connection only.
- Not feature-gated. It is an optional event type within protocol 5. An older Client Core drops a type it does not know without throwing or closing.
- What a client does. It clears the STT banner and mic refusal that
stt_error {recoverable: true}raised. It also clears them on the session’s nextuser_transcript, so a client that misses the event still recovers.
The adapter tier: [adapters.supervision]
The gateway supervises three adapters it spawns (crates/voice_gateway/src/adapter_supervision.rs): homectl (the
home bridge’s events-in stream and the home verbs) and two long-running helpers, fleetctl (vulpine-fleet run) and
audioctl (vulpine-audio watch). The helpers restart through one supervisor loop
(adapter_supervision::keep_alive). A helper’s first stdout line marks it up; an exit is a failure. Neither helper
has a verb to refuse, so down only rests it and shows on Health.
| key | default | rule |
|---|---|---|
enabled |
true |
false is the rollback: the bridge uses its own restart loop and no verb is refused. |
initial_backoff_ms |
1000 |
the first restart’s wait, 10 to 60000. |
max_backoff_ms |
60000 |
the cap for the doubling, jitter included. |
jitter |
0.2 |
the share of the wait added or taken away at random, 0 to 0.5. |
max_restarts |
5 |
the number of failures allowed inside the window before the adapter is down. |
window_ms |
300000 |
the window that failures are counted over. |
helpers |
true |
the fleet adapter and the audio watcher restart through the supervisor. false is their rollback: each goes back to its own linear loop (restart_ms times the attempt, capped at 60 s), with no Health row. |
turn_path |
true |
the turn path’s three queues are bounded and a dead EOU wake worker respawns within the session (below). false is the rollback: the unbounded queues and a wake worker that stays dead until the next session. |
- States.
up: the adapter answered (ahello, or a verb’s receipt).restarting {attempt}: it exited, timed out or gave no receipt; the next start waits 1 s, 2 s, 4 s and so on, up to the 60 s cap.down: more thanmax_restartsfailures inside the window. An answer does not clear the failure window, so an adapter that says hello and dies again still goesdown. - Degrade to text. A verb on a
downadapter inside its backoff is not spawned. Its row isok: false,reason: "adapter_down",error: "homectl is down; try again later", and the brain’s tool result carries that text at once. After the backoff, one call goes through as the probe, and a receipt marks the adapterup. - Trace.
adapter_supervision {adapter, state, attempt, reason, backoff_ms}, one row per failure and one per change toup, inadapter-supervision.jsonlin the trace folder. - Admin. The Health panel shows one
adapter.<name>row (kindconnector) per adapter the supervisor has seen.upisready.restartingisreconnectingwithattemptingand its attempt count.downisdegraded(critical) withexhausted,degraded_capability: "text only: …", the last failure asterminal_reason, and the retry time inrepair_action.
The turn path’s queues and the wake worker (turn_path)
- The recognizer’s jobs: 4. All sessions share one recognizer thread. A recognition waits up to 2 s for room, then
is refused
stt_queue_full, which the session handles as any recognizer error ([stt] on_fault). A warm pass is skipped when the queue is full. - Wyoming TTS commands: 16. A full queue refuses the next sentence (
tts_queue_full), which degrades to text as a TTS failure does. The last slot is kept for the reply’s end of text, so a stream is never left open. - Wyoming TTS frames: 64. The render loop waits for room before it reads the server’s next event, so a slow
session pushes back on the socket and no frame is dropped mid-word. A session that takes nothing for 10 s fails the
render (
tts_frames_stalled), and the reply degrades to text. - The EOU wake worker. A worker that dies mid-session is dropped. The next mic frame starts a fresh one through a
per-session supervisor on the budget above (
eou_worker): 1 s, 2 s, 4 s between starts,upat its ready line. Past the budget the session hears one invocation errorsidecar_down(wake fails open; the Admin’s Errors row names it), and a start is still tried once per cool-down.
In the Client Core
packages/voice-client hands these frames to a skin typed: {type: 'stt_error', sttError: {reason, recoverable, session_id?}} and {type: 'stt_recovered', sttRecovered: {session_id?}}. The wire’s fields keep their names.
The legacy error {code: "stt_error"} and every other gateway error arrive as
{type: 'error', message, code?, requestId?}:
messageis"<code>: <message>", for a consumer that parses it.codecarries the wire’s code typed (only a plain lower-case word of at most 64 characters).requestIdcarries the wire’s optionalrequest_idwhen the error answers a command by id (the read verb’sread_fetch_*errors,docs/contracts/reading.v1.md).
A client-raised error carries neither code nor requestId, so a skin that branches on code never mistakes a
client diagnostic for a gateway refusal. The shared view-model in packages/vulpine-voice-ui-state (“wire notices”)
holds the STT banner from stt_error to stt_recovered, and Vigil draws it with the shared chrome.