Frontend WebSocket
A connection attaches one durable owner, the frontend instance, so a reply, a resume or a dialog answer never goes to the wrong client.
Living doc, checked against the code 2026-11-14
frontend.v5 — the frontend WebSocket: the hello, owner admission, features as the minor, and every frame
- Status: accepted. The wire it describes is live as protocol 5.
- Stability:
public(docs/contracts/api-surfaces.toml, surfacefrontend). A third party may build a frontend on it. A breaking change is protocol 6 and a new file,frontend.v6.md. - Scope: a frontend (Vulpine.app, Vulpine in a browser, the TUI, a room speaker, Vulpine on iPhone, a script) and
the Voice Gateway, over one
WebSocket. The frontend sends
ClientControlframes and microphone PCM. The gateway sendsServerEventframes and reply PCM. - Schema:
frontend.v5.schema.json, generated from the Rust types incrates/protocol/src/lib.rs, gives every field, its type and whether it is optional. This file gives the behavior the schema cannot say (order, admission, negotiation, compatibility) and indexes every frame. The TypeScript types are generated from the same Rust by ts-rs (packages/protocol/src/generated). Frontend to gateway is#/definitions/ClientControl, gateway to frontend is#/definitions/ServerEvent, and every nested type (FrontendCapabilities,ClientOwnerHello,ClientOwnerReceipt,InteractionModeReceipt, …) is a definition of the same name.
Transport
| what | rule |
|---|---|
| address | ws://127.0.0.1:8780 by default (the top-level bind key, --bind or VULPINE_GATEWAY_BIND). The gateway listens on loopback only. |
| off the host | through the Admin relay over ssh, or reach: vulpine/ws/1 carries these same bytes, and vulpine/voice/1 carries a protocol 5 session as a control stream plus Opus datagrams (ADR 0053). |
| text frames | one JSON object each, a ClientControl or a ServerEvent, told apart by its type member (snake_case). |
| binary frames, frontend to gateway | microphone audio: signed 16-bit little-endian PCM, mono, 24 kHz (48 bytes per millisecond). Never base64 in JSON. Only after capabilities_accepted with binary_audio: true; a text-only client’s PCM is refused once per connection (audio_not_declared) and never reaches the recognizer. |
| binary frames, gateway to frontend | reply audio: each audio_chunk text frame is followed by one binary frame of byte_len bytes of 16-bit little-endian PCM at the chunk’s sample_rate_hz and channels. |
| browsers | the gateway refuses the upgrade (403) for an Origin that is not loopback, the desktop app’s Tauri webview or listed in [frontend] allowed_origins (browser_origin.rs). A native client sends no Origin and is accepted. |
The handshake
-
The gateway speaks first. On upgrade it sends
ServerEvent::Hello:protocol_version(5),binary_audioandfeatures, the names this gateway offers (below). -
The frontend’s hello.
ClientControl::Hellocarriesowner(aClientOwnerHello, next section),client(a free name, read by the Admin as one closed word:glyph,vigil,tui,satellite, …, elseother),capabilities(aFrontendCapabilities) anddebug_traces(read and ignored:traceframes follow the gateway’s owndebug_tracesconfig). Any control other thanhellobefore it is refusedcapabilities_required. A secondhellois refusedcapabilities_locked. -
FrontendCapabilities, the fields a builder sets:field rule protocol_version5. Anything at or above MIN_PROTOCOL_VERSION(5) is accepted; below it isunsupported_protocol_version.audioAudioCapabilities(input_pcm,output_pcm,sample_rates_hzwith 24000 in it, optionalframe_ms,reference_channel,playback_ack), ornullfor a text-onlyuiclient.kindoverlay(default),vigilorsatellite(ADR 0033 §2).rolefull(default),mic,speakerorui. Onlyuimay bind with no audio;speakeranduinever arm a turn.roomoptional label of 1 to 32 characters, never an address ( invalid_room_label).speechoptional SpeechOwnership: who owns wake, echo canceling, STT and TTS. Absent = the gateway owns all.featuresthe feature names the frontend consumes (below). Unknown names are ignored. tools,confirmation_actionsthe frontend tools it runs and the confirmations it can show ( frontend_tool_call, the herdr confirmations).avatar_actions,avatar_vocabularythe avatar verbs and names its rig draws (avatar-action.v1). render_kindsthe render kinds and card block kinds it draws (answer-card.v1, panel-card.v1). reachoptional ClientReach: the reach route this connection rides, as a label (ADR 0053). -
The receipt.
capabilities_acceptedcarries the owner receipt (ClientOwnerReceipt), the admitted input rate (24000) and channels (1),binary_audio(false for a text-only client) andfeatures, the agreed set. -
The session.
start_sessionnames theagentandworkspace, and optionally a loadout, persona, voice, model and interaction mode.session_startedanswers with anInteractionModeReceipt(requested, effective and live mode, policy digest, source, generation), thesession_idand the Braincontinuity_id. The owner admission and this receipt are mandatory: a client that cannot read them cannot speak protocol 5.
Owner admission
-
ClientOwnerHellois{"mode": "fresh"}(mint a new owner) or{"mode": "resume", "owner_id", "resume_token"}(attach an owner this frontend held before). It denies unknown fields. -
ClientOwnerReceiptanswers withowner_id,resume_tokenandlease_generation. Keep the token to resume after a reconnect, and keep it secret like a password: never log or show it. The Rust types redact it fromDebug.vulpine_clientcan persist it in a 0600 file under the vulpine state root. -
A resume succeeds only for an owner that is detached (no live attachment) and was seen within 30 days. The store holds at most 256 owners and 32 attached at once.
-
Refusals are
errorframes, and the connection stays open:code meaning retry? owner_busythe store is busy with this owner yes, after a backoff owner_attachment_capacitytoo many attachments right now yes, after a backoff owner_capacitya fresh hello met a store whose every owner may still be resumed no owner_refusedan unknown owner, a wrong token, or an owner that is attached elsewhere no: send fresh
Features: the minor
Protocol 5 has no minor number on the wire. The features lists are the minor and the only version signal inside
protocol 5: a client asks “does this gateway do X” by looking for X’s name in the gateway’s hello.
- Offered. The gateway’s
hellolists every name it offers (GATEWAY_FEATURESincrates/voice_gateway/src/ws/room_routing.rs; the committed hello fixture,benchmarks/fixtures/hello-capability.v1.json, records it). - Agreed.
capabilities_accepted.featuresis the frontend’s declared names ∩ the offered ones, in the gateway’s order. Only the first 16 names a client declares are read, so declare only what you consume. - Two kinds of name.
- A send feature gates a gateway frame: the gateway sends it only to a client that declared it, and a client
that did not declare it never sees it. Examples:
presence,procedure,mood,chips,session_lifecycle,readiness,playback_pause,direct_verbs,brain_context,orb_style,tts_segment_started. - A read feature says the gateway reads a control. A client checks the offered list before it sends the
control, so an older gateway is never sent one it would refuse. Examples:
render_receipt,rig_report,observe_session,list_sessions,file_fetch,capture_attachment. Some (sidecar_report,media_route_report,screen_context_report,motion_plan_switch,presence_observation,chip_tap) are read only from a client that also declared them. The table isOFFERED_CONTROLS(members:OFFERED_FIELDS) incrates/protocol/src/offered.rs. Its generated TypeScript,packages/protocol/src/wire.ts, is what the Client Core and the satellite bridge import;PROTOCOL_VERSIONis in the same file.
- A send feature gates a gateway frame: the gateway sends it only to a client that declared it, and a client
that did not declare it never sees it. Examples:
Frames
Every type on the wire, one row each. The Rust variant is the tag in UpperCamelCase (start_session is
ClientControl::StartSession). “Feature” names the hello feature that gates the frame, where one does. A frame
family with its own contract is linked; that contract is the authority for its fields and errors.
Frontend to gateway (ClientControl)
Session.
type |
what | gate / contract |
|---|---|---|
hello |
the frontend’s hello: owner, client name, capabilities | first frame, once |
start_session |
bind a voice session: agent, workspace, loadout, persona, voice, model, interaction mode | after capabilities_accepted |
end_session |
close the session | |
switch_model |
ask the Brain to use another model | answered model_changed |
get_state |
ask for the state snapshot (presence, chips, mood, agreed features) |
feature get_state |
list_sessions |
ask for the live sessions on this gateway | read feature list_sessions; answered live_sessions |
Turns and speech.
type |
what | gate / contract |
|---|---|---|
text_utterance |
a typed turn, with an optional utterance_id echoed on its transcript and an optional capture |
capture: read feature capture_attachment, capture.v1 |
vad_hint |
the frontend’s speech hint and echo-residual metrics; a hint, never a commit | |
interrupt |
barge-in: stop the reply | |
speak |
read text aloud through the session’s voice without the Brain (optionally to a room’s speaker) | |
call |
open the call window (no wake between turns until “over and out”) | |
satellite_control |
a satellite’s button, dial or mute switch, as touched | satellite-controls.v1 |
mood_feedback |
the operator’s verdict on a reply’s mood | |
chip_tap |
the operator tapped a chip that offered an action | read feature chip_tap |
Playback evidence.
type |
what | gate / contract |
|---|---|---|
audio_playback_observed |
the frontend’s player is (or stopped) sounding a reply | |
audio_echo_risk_observed |
the post-playback acoustic tail is open or closed | |
audio_playback_reference_observed |
whether the playback reference is calibrated | |
audio_playback_finished |
a reply’s playout finished, with its playout report | |
playback_done |
the speaker that played the reply says so (only with audio.playback_ack) |
ADR 0033 §3 |
resume_playback |
after a drop, a resumed owner asks for a reply’s audio from next_seq on; a phone woken by an announce push asks turn_id: "announce" from 0, the alias of the announcement the gateway holds for it (a line spoken to its phone room; the replayed audio_chunks carry the real announce-… turn id, and the alias never replays a reply: turn_mismatch) |
send feature resume_playback |
playback_received |
the client holds turn_id’s audio before next_seq (one past the last audio_chunk it received in full); the gateway drops those chunks from the owner’s replay ring, so a reply held in full sends no wake after the link goes and one cut off mid-stream keeps only its tail; no answer, traced playback_received |
send feature playback_received (with resume_playback); sent by the writer before it closes |
wake_token |
the phone’s Push to Talk wake token (kind: "pushtotalk", token: lower-case hex, or null to clear), kept for this owner in memory; when a reply is held for the owner after its link is gone the gateway sends one textless push, and the phone re-dials as the same owner and asks resume_playback. Optional apns_environment (sandbox | production) and bundle_id name the build the token belongs to, so one gateway wakes a development build and a TestFlight build alike: the push goes to that environment and to the topic <bundle_id>.voip-ptt when bundle_id is [push] bundle_id or its .dev sibling (any other is refused, bundle_not_allowed); without them, [push] decides. The push body is {"v":1,"kind":"reply"} for a held reply and {"v":1,"kind":"announce"} for a line spoken to the owner’s phone room (a paired phone is a room speaker, driver phone) |
send feature wake_token; needs resume_playback too; never logged |
Dictation and reading.
type |
what | gate / contract |
|---|---|---|
dictation_hold |
hold-to-talk pressed or released | dictation.v1 |
dictation_inserted |
the client’s proof that a dictation_text landed (or was refused) |
dictation.v1 |
dictation_tidy_request |
ask for a guarded tidy before typing | dictation.v1 |
dictation_review |
the operator’s choice on an open review | dictation.v1 |
dictation_erased |
what “scratch that” erased | dictation.v1 |
start_read |
read a page or text aloud with a resumable cursor | reading.v1 |
reading_control |
move or stop the read in flight | reading.v1 |
Tools, confirmations and approvals.
type |
what | gate / contract |
|---|---|---|
invoke_tool |
run one Brain tool directly, through the loadout’s gates | answered tool_receipt; tool.v1 |
frontend_tool_result |
the result of a frontend_tool_call |
|
herdr_pane_focus_confirmation |
the operator’s answer to a pane-focus confirmation | |
herdr_command_confirmation |
the operator’s answer to a herdr command confirmation | |
background_work_settled_ack |
the frontend saw a background_work_settled |
|
approval_redeem |
a signed yes for one signed-only approval request | |
approval_deny |
the operator’s Deny on a signed approval card | |
inject_span |
open or close a span of injected test audio |
Avatar, render and files.
type |
what | gate / contract |
|---|---|---|
avatar_action |
one typed avatar action from the operator, no Brain turn | read feature avatar_action; avatar-action.v1 |
avatar_receipt |
what the rig did with one avatar_action it was sent |
read feature avatar_receipt; avatar-action.v1 |
avatar_redeclare |
the rig’s names changed after bind | read feature avatar_redeclare; avatar-action.v1 |
rig_report |
what the rig draws now: kind, manifest, state, decode path | read feature rig_report; avatar-action.v1 |
motion_plan_switch |
the frontend’s switch for the decision model’s motion plan | read feature motion_plan_switch; decision.v1 |
render_receipt |
what the frontend did with one render: drawn, refused or dropped |
read feature render_receipt; answer-card.v1 |
file_fetch |
the bytes of a file this connection was shown, by SHA-256 | read feature file_fetch; file.v1 |
Observe.
type |
what | gate / contract |
|---|---|---|
observe_session |
a ui client follows another live session read-only |
read feature observe_session |
observed_answer |
an observer answers the observed session’s dialog, as its owner | read feature observed_answer |
Reports from the frontend. Each is traced and kept on the session’s Admin row; none is answered.
type |
what | gate / contract |
|---|---|---|
footprint_report |
the frontend’s own memory footprint, in tenths of a MiB | read feature footprint_report |
sidecar_report |
the frontend’s sidecar window changed state | read feature sidecar_report |
media_route_report |
a microphone or speaker route’s generation and state | read feature media_route_report |
screen_context_report |
the operator’s front app (bundle id, category, idle), as typed Brain input | read feature screen_context_report |
presence_observation |
a room a frontend’s scanner heard the operator in | read feature presence_observation |
Gateway to frontend (ServerEvent)
Handshake and session.
type |
what | gate / contract |
|---|---|---|
hello |
protocol version, binary audio, the offered features | first frame |
capabilities_accepted |
the owner receipt, the admitted audio, the agreed features | answers hello |
session_started |
the interaction-mode receipt, session and continuity ids, model, tools bound | answers start_session; hooks.v1 |
session_ended |
the session closed | |
session_lifecycle |
the typed lifecycle: starting, started, start_rejected, rotating, closing, closed |
send feature session_lifecycle |
models_available |
the models a session may switch to, and the current one | |
model_changed |
the session’s model changed | |
voice_profiles_available |
the personas and voices, and whether the voice is locked | |
readiness |
one owner’s readiness changed | send feature readiness; adapter-degradation.v1 |
stt_error |
this client’s recognizer is gone | adapter-degradation.v1 |
stt_recovered |
the rebuilt recognizer took its first frame | adapter-degradation.v1 |
live_sessions |
the answer to list_sessions |
|
direct_verbs |
the gateway’s closed verbs, once at bind | send feature direct_verbs |
error |
a refusal or failure: code, message, optional request_id and candidates |
see Errors |
trace |
a debug trace row, only when the gateway’s debug_traces config is on |
config, not the hello |
Turns and audio.
type |
what | gate / contract |
|---|---|---|
user_transcript |
a partial or final transcript of the operator, with utterance_id when echoed |
|
turn_started |
a reply turn began (persona, character) | |
assistant_delta |
a piece of the reply text | |
turn_completed |
the reply text is complete | |
turn_audio_completed |
the reply’s audio is complete on the gateway’s side; asked_question |
|
audio_chunk |
the header of the next binary PCM frame: turn, index, bytes, rate, channels | followed by one binary frame |
audio_reset |
drop queued reply audio | |
playback_pause |
pause or resume the reply audio the player holds | send feature playback_pause; dictation.v1 |
playback_resumed |
the answer to resume_playback once the replay is sent |
|
playback_resume_refused |
resume_playback refused, with its closed reason |
|
follow_up |
the gateway’s follow-up window after a reply played out: verdict: "opened" with open_ms (the window’s length, decided in one place on the gateway) or verdict: "refused" with a closed reason word (decision_gate, self_heard, …) when the window heard words and did not take them, or verdict: "taken" at the take (before the reply’s turn_started, which waits for the brain); additive, no transcript rides it |
|
tts_segment_started |
a reply’s TTS segment began (inside observed) |
send feature tts_segment_started; avatar-action.v1 |
tts_segment_shipped |
a segment’s last audio shipped | avatar-action.v1 |
invocation_feedback |
what became of an armed request: armed, acknowledged, canceled, expired, refused, re-armed | |
brain_context |
the session’s Brain context after a turn, counts only | send feature brain_context |
Presence, chips and mood.
type |
what | gate / contract |
|---|---|---|
phase |
the turn phase word | |
presence |
the embodiment presence: one event per real transition | send feature presence |
state |
the snapshot a reconnecting client redraws from | send feature get_state; answers get_state |
procedure |
one radio procedure word and its outcome | send feature procedure |
mood |
the mood the reply carries and the voice it resolved to | send feature mood |
chip |
a chip to show, by a stable text key | send feature chips |
chip_cleared |
a chip went away | send feature chips |
satellite_state |
a satellite’s settled volume and mute | satellite-controls.v1 |
orb_style |
the orb style the operator asked for by voice | send feature orb_style |
Dictation and reading.
type |
what | gate / contract |
|---|---|---|
dictation_partial |
the running transcript of a held dictation | dictation.v1 |
dictation_text |
the settled words of one dictation | dictation.v1 |
dictation_tidy |
the tidy a review asked for, with the guard’s verdict | dictation.v1 |
dictation_review |
a spoken choice on an open review | dictation.v1 |
erase_last_insert |
“scratch that”: erase exactly the proven insert | dictation.v1 |
reading_progress |
where a read stands | reading.v1 |
Tools, dialogs, confirmations and background work.
type |
what | gate / contract |
|---|---|---|
tool_receipt |
the outcome of one invoke_tool |
tool.v1 |
frontend_tools_negotiated |
the frontend tools the session admitted | |
frontend_tool_call |
run one frontend tool; answer with frontend_tool_result |
|
frontend_tool_rejected |
a frontend tool call the gateway refused | |
brain_ui_request |
an extension in the Brain asked the operator a question mid-turn | |
brain_ui_answered |
that question settled | |
herdr_pane_focus_confirmation_requested |
confirm focusing a herdr pane | |
herdr_command_confirmation_requested |
confirm a herdr command | |
herdr_agent_lifecycle |
a herdr agent’s status changed | |
herdr_semantic_wait_settled |
a wait on a herdr agent settled | |
background_work_settled |
a background job settled; ack with background_work_settled_ack |
|
signed_approval |
a signed-only approval request for the desktop app’s signed card |
Avatar, render and files.
type |
what | gate / contract |
|---|---|---|
avatar_action |
one typed avatar action for the rig | avatar-action.v1 |
avatar_forwarded |
what became of an avatar_action sent to another session |
avatar-action.v1 |
render |
one typed visual component for the render panel, only for a declared kind | answer-card.v1, panel-card.v1, visual.v1, file.v1, page.v1 |
file_chunk |
one piece of a fetched file, base64 | file.v1 |
file_fetch_refused |
file_fetch refused, with its closed reason |
file.v1 |
Observe.
type |
what | gate / contract |
|---|---|---|
observe_started |
observe_session was admitted |
|
observed |
one event of the observed session, wrapped | |
observe_ended |
the observation closed: ended or replaced |
|
observe_lagged |
the observer fell behind; skipped events were dropped |
Errors
error {code, message, request_id?, candidates?}. code is a closed snake_case word. message is for a log, never
for parsing. An error never closes the connection by itself. The handshake’s codes:
| code | when |
|---|---|
invalid_json |
a text frame that does not decode as a ClientControl, including a type this gateway does not know |
capabilities_required |
a control before hello |
capabilities_locked |
a second hello on one connection |
unsupported_protocol_version |
capabilities.protocol_version below 5 |
incompatible_audio_capabilities |
a role other than ui with no PCM in and out, or playback_ack from a client that is not a satellite or speaker |
unsupported_input_sample_rate |
sample_rates_hz without 24000 |
invalid_room_label |
a room label that is empty or over 32 characters |
incompatible_speech_ownership |
a speech block the gateway cannot honor |
owner_busy, owner_attachment_capacity, owner_capacity, owner_refused |
owner admission (above) |
audio_not_declared |
PCM from a client whose hello declared none (sent once per connection) |
satellite_off |
a kind: satellite client turned off in the Admin: its hello is refused before owner admission, and a bound one is sent this code and then closed (the one code that closes; satellite-controls.v1) |
Each frame family’s own codes (read_fetch_*, observe_*, the dictation and avatar refusals) are in its contract or
in frontend-interfaces.
Compatibility
- Additive within 5. A new optional field, a new frame
type, a new feature name, or a new word in an enum that keeps a catch-all arm never moves the protocol number. - A reader is lenient. A frontend ignores members and
types it does not know. The schema is the writer’s view of this build (itsx-vulpine.reading). A reader stays as lenient as the Rust types are. - A writer is careful. A frontend sends a gated control only when the gateway’s
hellooffered its feature. An older gateway answers a control it does not know withinvalid_jsonand carries on. - Breaking. Removing or retyping a field, renaming a
type, or making an optional field required is protocol 6:PROTOCOL_VERSIONmoves,frontend.v6.mdandfrontend.v6.schema.jsonare written, andMIN_PROTOCOL_VERSIONsays how long 5 is still accepted.
See also
- JSON Schemas: this wire’s generated schema and how it is drift-tested.
- admin.v6: the owner’s Admin socket, which is not this wire.
- frontend-interfaces: the story of every frame added within 5.
- API surfaces: every interface with its stability label.
- ADR 0033 (the satellite frontend skin: kinds, roles, speech ownership) and ADR 0059 §5 (semver and the hello’s contract table).