Skip to content
Vulpine
Esc
↑↓navigate↵open⌘Jpreview
On this page

Frontend WebSocket

A connection attaches one durable owner, the frontend instance, so a reply, a resume or a dialog answer never goes to the wrong client.

Living doc, checked against the code 2026-11-14

frontend.v5 — the frontend WebSocket: the hello, owner admission, features as the minor, and every frame

  • Status: accepted. The wire it describes is live as protocol 5.
  • Stability: public (docs/contracts/api-surfaces.toml, surface frontend). A third party may build a frontend on it. A breaking change is protocol 6 and a new file, frontend.v6.md.
  • Scope: a frontend (Vulpine.app, Vulpine in a browser, the TUI, a room speaker, Vulpine on iPhone, a script) and the Voice Gateway, over one WebSocket. The frontend sends ClientControl frames and microphone PCM. The gateway sends ServerEvent frames and reply PCM.
  • Schema: frontend.v5.schema.json, generated from the Rust types in crates/protocol/src/lib.rs, gives every field, its type and whether it is optional. This file gives the behavior the schema cannot say (order, admission, negotiation, compatibility) and indexes every frame. The TypeScript types are generated from the same Rust by ts-rs (packages/protocol/src/generated). Frontend to gateway is #/definitions/ClientControl, gateway to frontend is #/definitions/ServerEvent, and every nested type (FrontendCapabilities, ClientOwnerHello, ClientOwnerReceipt, InteractionModeReceipt, …) is a definition of the same name.

Transport

what rule
address ws://127.0.0.1:8780 by default (the top-level bind key, --bind or VULPINE_GATEWAY_BIND). The gateway listens on loopback only.
off the host through the Admin relay over ssh, or reach: vulpine/ws/1 carries these same bytes, and vulpine/voice/1 carries a protocol 5 session as a control stream plus Opus datagrams (ADR 0053).
text frames one JSON object each, a ClientControl or a ServerEvent, told apart by its type member (snake_case).
binary frames, frontend to gateway microphone audio: signed 16-bit little-endian PCM, mono, 24 kHz (48 bytes per millisecond). Never base64 in JSON. Only after capabilities_accepted with binary_audio: true; a text-only client’s PCM is refused once per connection (audio_not_declared) and never reaches the recognizer.
binary frames, gateway to frontend reply audio: each audio_chunk text frame is followed by one binary frame of byte_len bytes of 16-bit little-endian PCM at the chunk’s sample_rate_hz and channels.
browsers the gateway refuses the upgrade (403) for an Origin that is not loopback, the desktop app’s Tauri webview or listed in [frontend] allowed_origins (browser_origin.rs). A native client sends no Origin and is accepted.

The handshake

  1. The gateway speaks first. On upgrade it sends ServerEvent::Hello: protocol_version (5), binary_audio and features, the names this gateway offers (below).

  2. The frontend’s hello. ClientControl::Hello carries owner (a ClientOwnerHello, next section), client (a free name, read by the Admin as one closed word: glyph, vigil, tui, satellite, …, else other), capabilities (a FrontendCapabilities) and debug_traces (read and ignored: trace frames follow the gateway’s own debug_traces config). Any control other than hello before it is refused capabilities_required. A second hello is refused capabilities_locked.

  3. FrontendCapabilities, the fields a builder sets:

    field rule
    protocol_version 5. Anything at or above MIN_PROTOCOL_VERSION (5) is accepted; below it is unsupported_protocol_version.
    audio AudioCapabilities (input_pcm, output_pcm, sample_rates_hz with 24000 in it, optional frame_ms, reference_channel, playback_ack), or null for a text-only ui client.
    kind overlay (default), vigil or satellite (ADR 0033 §2).
    role full (default), mic, speaker or ui. Only ui may bind with no audio; speaker and ui never arm a turn.
    room optional label of 1 to 32 characters, never an address (invalid_room_label).
    speech optional SpeechOwnership: who owns wake, echo canceling, STT and TTS. Absent = the gateway owns all.
    features the feature names the frontend consumes (below). Unknown names are ignored.
    tools, confirmation_actions the frontend tools it runs and the confirmations it can show (frontend_tool_call, the herdr confirmations).
    avatar_actions, avatar_vocabulary the avatar verbs and names its rig draws (avatar-action.v1).
    render_kinds the render kinds and card block kinds it draws (answer-card.v1, panel-card.v1).
    reach optional ClientReach: the reach route this connection rides, as a label (ADR 0053).
  4. The receipt. capabilities_accepted carries the owner receipt (ClientOwnerReceipt), the admitted input rate (24000) and channels (1), binary_audio (false for a text-only client) and features, the agreed set.

  5. The session. start_session names the agent and workspace, and optionally a loadout, persona, voice, model and interaction mode. session_started answers with an InteractionModeReceipt (requested, effective and live mode, policy digest, source, generation), the session_id and the Brain continuity_id. The owner admission and this receipt are mandatory: a client that cannot read them cannot speak protocol 5.

Owner admission

  • ClientOwnerHello is {"mode": "fresh"} (mint a new owner) or {"mode": "resume", "owner_id", "resume_token"} (attach an owner this frontend held before). It denies unknown fields.

  • ClientOwnerReceipt answers with owner_id, resume_token and lease_generation. Keep the token to resume after a reconnect, and keep it secret like a password: never log or show it. The Rust types redact it from Debug. vulpine_client can persist it in a 0600 file under the vulpine state root.

  • A resume succeeds only for an owner that is detached (no live attachment) and was seen within 30 days. The store holds at most 256 owners and 32 attached at once.

  • Refusals are error frames, and the connection stays open:

    code meaning retry?
    owner_busy the store is busy with this owner yes, after a backoff
    owner_attachment_capacity too many attachments right now yes, after a backoff
    owner_capacity a fresh hello met a store whose every owner may still be resumed no
    owner_refused an unknown owner, a wrong token, or an owner that is attached elsewhere no: send fresh

Features: the minor

Protocol 5 has no minor number on the wire. The features lists are the minor and the only version signal inside protocol 5: a client asks “does this gateway do X” by looking for X’s name in the gateway’s hello.

  • Offered. The gateway’s hello lists every name it offers (GATEWAY_FEATURES in crates/voice_gateway/src/ws/room_routing.rs; the committed hello fixture, benchmarks/fixtures/hello-capability.v1.json, records it).
  • Agreed. capabilities_accepted.features is the frontend’s declared names ∩ the offered ones, in the gateway’s order. Only the first 16 names a client declares are read, so declare only what you consume.
  • Two kinds of name.
    • A send feature gates a gateway frame: the gateway sends it only to a client that declared it, and a client that did not declare it never sees it. Examples: presence, procedure, mood, chips, session_lifecycle, readiness, playback_pause, direct_verbs, brain_context, orb_style, tts_segment_started.
    • A read feature says the gateway reads a control. A client checks the offered list before it sends the control, so an older gateway is never sent one it would refuse. Examples: render_receipt, rig_report, observe_session, list_sessions, file_fetch, capture_attachment. Some (sidecar_report, media_route_report, screen_context_report, motion_plan_switch, presence_observation, chip_tap) are read only from a client that also declared them. The table is OFFERED_CONTROLS (members: OFFERED_FIELDS) in crates/protocol/src/offered.rs. Its generated TypeScript, packages/protocol/src/wire.ts, is what the Client Core and the satellite bridge import; PROTOCOL_VERSION is in the same file.

Frames

Every type on the wire, one row each. The Rust variant is the tag in UpperCamelCase (start_session is ClientControl::StartSession). “Feature” names the hello feature that gates the frame, where one does. A frame family with its own contract is linked; that contract is the authority for its fields and errors.

Frontend to gateway (ClientControl)

Session.

type what gate / contract
hello the frontend’s hello: owner, client name, capabilities first frame, once
start_session bind a voice session: agent, workspace, loadout, persona, voice, model, interaction mode after capabilities_accepted
end_session close the session
switch_model ask the Brain to use another model answered model_changed
get_state ask for the state snapshot (presence, chips, mood, agreed features) feature get_state
list_sessions ask for the live sessions on this gateway read feature list_sessions; answered live_sessions

Turns and speech.

type what gate / contract
text_utterance a typed turn, with an optional utterance_id echoed on its transcript and an optional capture capture: read feature capture_attachment, capture.v1
vad_hint the frontend’s speech hint and echo-residual metrics; a hint, never a commit
interrupt barge-in: stop the reply
speak read text aloud through the session’s voice without the Brain (optionally to a room’s speaker)
call open the call window (no wake between turns until “over and out”)
satellite_control a satellite’s button, dial or mute switch, as touched satellite-controls.v1
mood_feedback the operator’s verdict on a reply’s mood
chip_tap the operator tapped a chip that offered an action read feature chip_tap

Playback evidence.

type what gate / contract
audio_playback_observed the frontend’s player is (or stopped) sounding a reply
audio_echo_risk_observed the post-playback acoustic tail is open or closed
audio_playback_reference_observed whether the playback reference is calibrated
audio_playback_finished a reply’s playout finished, with its playout report
playback_done the speaker that played the reply says so (only with audio.playback_ack) ADR 0033 §3
resume_playback after a drop, a resumed owner asks for a reply’s audio from next_seq on; a phone woken by an announce push asks turn_id: "announce" from 0, the alias of the announcement the gateway holds for it (a line spoken to its phone room; the replayed audio_chunks carry the real announce-… turn id, and the alias never replays a reply: turn_mismatch) send feature resume_playback
playback_received the client holds turn_id’s audio before next_seq (one past the last audio_chunk it received in full); the gateway drops those chunks from the owner’s replay ring, so a reply held in full sends no wake after the link goes and one cut off mid-stream keeps only its tail; no answer, traced playback_received send feature playback_received (with resume_playback); sent by the writer before it closes
wake_token the phone’s Push to Talk wake token (kind: "pushtotalk", token: lower-case hex, or null to clear), kept for this owner in memory; when a reply is held for the owner after its link is gone the gateway sends one textless push, and the phone re-dials as the same owner and asks resume_playback. Optional apns_environment (sandbox | production) and bundle_id name the build the token belongs to, so one gateway wakes a development build and a TestFlight build alike: the push goes to that environment and to the topic <bundle_id>.voip-ptt when bundle_id is [push] bundle_id or its .dev sibling (any other is refused, bundle_not_allowed); without them, [push] decides. The push body is {"v":1,"kind":"reply"} for a held reply and {"v":1,"kind":"announce"} for a line spoken to the owner’s phone room (a paired phone is a room speaker, driver phone) send feature wake_token; needs resume_playback too; never logged

Dictation and reading.

type what gate / contract
dictation_hold hold-to-talk pressed or released dictation.v1
dictation_inserted the client’s proof that a dictation_text landed (or was refused) dictation.v1
dictation_tidy_request ask for a guarded tidy before typing dictation.v1
dictation_review the operator’s choice on an open review dictation.v1
dictation_erased what “scratch that” erased dictation.v1
start_read read a page or text aloud with a resumable cursor reading.v1
reading_control move or stop the read in flight reading.v1

Tools, confirmations and approvals.

type what gate / contract
invoke_tool run one Brain tool directly, through the loadout’s gates answered tool_receipt; tool.v1
frontend_tool_result the result of a frontend_tool_call
herdr_pane_focus_confirmation the operator’s answer to a pane-focus confirmation
herdr_command_confirmation the operator’s answer to a herdr command confirmation
background_work_settled_ack the frontend saw a background_work_settled
approval_redeem a signed yes for one signed-only approval request
approval_deny the operator’s Deny on a signed approval card
inject_span open or close a span of injected test audio

Avatar, render and files.

type what gate / contract
avatar_action one typed avatar action from the operator, no Brain turn read feature avatar_action; avatar-action.v1
avatar_receipt what the rig did with one avatar_action it was sent read feature avatar_receipt; avatar-action.v1
avatar_redeclare the rig’s names changed after bind read feature avatar_redeclare; avatar-action.v1
rig_report what the rig draws now: kind, manifest, state, decode path read feature rig_report; avatar-action.v1
motion_plan_switch the frontend’s switch for the decision model’s motion plan read feature motion_plan_switch; decision.v1
render_receipt what the frontend did with one render: drawn, refused or dropped read feature render_receipt; answer-card.v1
file_fetch the bytes of a file this connection was shown, by SHA-256 read feature file_fetch; file.v1

Observe.

type what gate / contract
observe_session a ui client follows another live session read-only read feature observe_session
observed_answer an observer answers the observed session’s dialog, as its owner read feature observed_answer

Reports from the frontend. Each is traced and kept on the session’s Admin row; none is answered.

type what gate / contract
footprint_report the frontend’s own memory footprint, in tenths of a MiB read feature footprint_report
sidecar_report the frontend’s sidecar window changed state read feature sidecar_report
media_route_report a microphone or speaker route’s generation and state read feature media_route_report
screen_context_report the operator’s front app (bundle id, category, idle), as typed Brain input read feature screen_context_report
presence_observation a room a frontend’s scanner heard the operator in read feature presence_observation

Gateway to frontend (ServerEvent)

Handshake and session.

type what gate / contract
hello protocol version, binary audio, the offered features first frame
capabilities_accepted the owner receipt, the admitted audio, the agreed features answers hello
session_started the interaction-mode receipt, session and continuity ids, model, tools bound answers start_session; hooks.v1
session_ended the session closed
session_lifecycle the typed lifecycle: starting, started, start_rejected, rotating, closing, closed send feature session_lifecycle
models_available the models a session may switch to, and the current one
model_changed the session’s model changed
voice_profiles_available the personas and voices, and whether the voice is locked
readiness one owner’s readiness changed send feature readiness; adapter-degradation.v1
stt_error this client’s recognizer is gone adapter-degradation.v1
stt_recovered the rebuilt recognizer took its first frame adapter-degradation.v1
live_sessions the answer to list_sessions
direct_verbs the gateway’s closed verbs, once at bind send feature direct_verbs
error a refusal or failure: code, message, optional request_id and candidates see Errors
trace a debug trace row, only when the gateway’s debug_traces config is on config, not the hello

Turns and audio.

type what gate / contract
user_transcript a partial or final transcript of the operator, with utterance_id when echoed
turn_started a reply turn began (persona, character)
assistant_delta a piece of the reply text
turn_completed the reply text is complete
turn_audio_completed the reply’s audio is complete on the gateway’s side; asked_question
audio_chunk the header of the next binary PCM frame: turn, index, bytes, rate, channels followed by one binary frame
audio_reset drop queued reply audio
playback_pause pause or resume the reply audio the player holds send feature playback_pause; dictation.v1
playback_resumed the answer to resume_playback once the replay is sent
playback_resume_refused resume_playback refused, with its closed reason
follow_up the gateway’s follow-up window after a reply played out: verdict: "opened" with open_ms (the window’s length, decided in one place on the gateway) or verdict: "refused" with a closed reason word (decision_gate, self_heard, …) when the window heard words and did not take them, or verdict: "taken" at the take (before the reply’s turn_started, which waits for the brain); additive, no transcript rides it
tts_segment_started a reply’s TTS segment began (inside observed) send feature tts_segment_started; avatar-action.v1
tts_segment_shipped a segment’s last audio shipped avatar-action.v1
invocation_feedback what became of an armed request: armed, acknowledged, canceled, expired, refused, re-armed
brain_context the session’s Brain context after a turn, counts only send feature brain_context

Presence, chips and mood.

type what gate / contract
phase the turn phase word
presence the embodiment presence: one event per real transition send feature presence
state the snapshot a reconnecting client redraws from send feature get_state; answers get_state
procedure one radio procedure word and its outcome send feature procedure
mood the mood the reply carries and the voice it resolved to send feature mood
chip a chip to show, by a stable text key send feature chips
chip_cleared a chip went away send feature chips
satellite_state a satellite’s settled volume and mute satellite-controls.v1
orb_style the orb style the operator asked for by voice send feature orb_style

Dictation and reading.

type what gate / contract
dictation_partial the running transcript of a held dictation dictation.v1
dictation_text the settled words of one dictation dictation.v1
dictation_tidy the tidy a review asked for, with the guard’s verdict dictation.v1
dictation_review a spoken choice on an open review dictation.v1
erase_last_insert “scratch that”: erase exactly the proven insert dictation.v1
reading_progress where a read stands reading.v1

Tools, dialogs, confirmations and background work.

type what gate / contract
tool_receipt the outcome of one invoke_tool tool.v1
frontend_tools_negotiated the frontend tools the session admitted
frontend_tool_call run one frontend tool; answer with frontend_tool_result
frontend_tool_rejected a frontend tool call the gateway refused
brain_ui_request an extension in the Brain asked the operator a question mid-turn
brain_ui_answered that question settled
herdr_pane_focus_confirmation_requested confirm focusing a herdr pane
herdr_command_confirmation_requested confirm a herdr command
herdr_agent_lifecycle a herdr agent’s status changed
herdr_semantic_wait_settled a wait on a herdr agent settled
background_work_settled a background job settled; ack with background_work_settled_ack
signed_approval a signed-only approval request for the desktop app’s signed card

Avatar, render and files.

type what gate / contract
avatar_action one typed avatar action for the rig avatar-action.v1
avatar_forwarded what became of an avatar_action sent to another session avatar-action.v1
render one typed visual component for the render panel, only for a declared kind answer-card.v1, panel-card.v1, visual.v1, file.v1, page.v1
file_chunk one piece of a fetched file, base64 file.v1
file_fetch_refused file_fetch refused, with its closed reason file.v1

Observe.

type what gate / contract
observe_started observe_session was admitted
observed one event of the observed session, wrapped
observe_ended the observation closed: ended or replaced
observe_lagged the observer fell behind; skipped events were dropped

Errors

error {code, message, request_id?, candidates?}. code is a closed snake_case word. message is for a log, never for parsing. An error never closes the connection by itself. The handshake’s codes:

code when
invalid_json a text frame that does not decode as a ClientControl, including a type this gateway does not know
capabilities_required a control before hello
capabilities_locked a second hello on one connection
unsupported_protocol_version capabilities.protocol_version below 5
incompatible_audio_capabilities a role other than ui with no PCM in and out, or playback_ack from a client that is not a satellite or speaker
unsupported_input_sample_rate sample_rates_hz without 24000
invalid_room_label a room label that is empty or over 32 characters
incompatible_speech_ownership a speech block the gateway cannot honor
owner_busy, owner_attachment_capacity, owner_capacity, owner_refused owner admission (above)
audio_not_declared PCM from a client whose hello declared none (sent once per connection)
satellite_off a kind: satellite client turned off in the Admin: its hello is refused before owner admission, and a bound one is sent this code and then closed (the one code that closes; satellite-controls.v1)

Each frame family’s own codes (read_fetch_*, observe_*, the dictation and avatar refusals) are in its contract or in frontend-interfaces.

Compatibility

  • Additive within 5. A new optional field, a new frame type, a new feature name, or a new word in an enum that keeps a catch-all arm never moves the protocol number.
  • A reader is lenient. A frontend ignores members and types it does not know. The schema is the writer’s view of this build (its x-vulpine.reading). A reader stays as lenient as the Rust types are.
  • A writer is careful. A frontend sends a gated control only when the gateway’s hello offered its feature. An older gateway answers a control it does not know with invalid_json and carries on.
  • Breaking. Removing or retyping a field, renaming a type, or making an optional field required is protocol 6: PROTOCOL_VERSION moves, frontend.v6.md and frontend.v6.schema.json are written, and MIN_PROTOCOL_VERSION says how long 5 is still accepted.

See also

  • JSON Schemas: this wire’s generated schema and how it is drift-tested.
  • admin.v6: the owner’s Admin socket, which is not this wire.
  • frontend-interfaces: the story of every frame added within 5.
  • API surfaces: every interface with its stability label.
  • ADR 0033 (the satellite frontend skin: kinds, roles, speech ownership) and ADR 0059 §5 (semver and the hello’s contract table).

Was this page helpful?