Skip to content
Vulpine
Esc
↑↓navigate↵open⌘Jpreview
On this page

Architecture

One voice turn as the code runs it on the Mac mini: the Rust Voice Gateway as the sole gateway and brain path, with its STT, turn machine, speculative brain dispatch and streamed TTS.

Living doc, checked against the code 2026-11-14

Vulpine follows PiBot’s low-latency worker design, not HTTP microservices. The Rust Voice Gateway is the only gateway and the only path to the Brain. fox9 and foxline identifiers below are compatibility names. Installing it is docs/install.md; running and rolling the gateway is its README.

The parts, and where they run:

Vulpine.app, Vigil and the room speaker’s bridge are clients of the one gateway WebSocket. Vulpine Gateway.app reaches the gateway only through its owner-local Admin socket.

One voice turn:

A turn takes one of two paths after the commit. A verb turn is a request the gateway matches against its own short list of verbs (what you can say): it starts no brain turn. An agent turn is everything else, and goes to the brain. Both wait for the commit, the gateway being sure you have finished: 978 ms p50 over 218 live turns (benchmarks/results/brain-ttft-cloud-vs-local-2026-11-06.json). After it, a verb’s match costs microseconds (benchmarks/results/page-verb-cost-20261107.md) and the voice worker’s first audio about 140 ms (benchmarks/results/tts-idle-ttfa-2026-11-07.json). An agent turn waits 1.4 s p50 (3.8 s p90) for a hosted model’s first token and 441 ms more for first audio; speech end to audible is 3.0 s p50 for an agent turn.

The STT box is the ANE authority lane, which the installed config turns on ([shadow_stt] mode = "authority"). When mode is not authority, the loadout’s own STT adapter takes its place. Without speculation, the committed text goes to Pi as an ordinary prompt.

The gateway never calls the Brain’s LLM servers; Pi does. The decision model is optional and off by default ([decision_shadow], below): the turn never waits for it.

The gateway binds each warm Brain process to a BrainIdentity of eleven fields (crates/voice_gateway/src/brain/launch.rs): the owner, agent, workspace, loadout, model, persona and persona prompt, the frontend capability profile’s hash, the tool-policy hash (the loadout’s allowed Pi tools plus the negotiated frontend tools), the admitted project root’s id, and the launch hash (command, arguments and working directory). The warm-Brain pool is keyed by the whole identity plus the launch (brain/pool.rs). A Brain is reused only when every field matches. The Pi session is named foxline-voice-<agent>-<loadout>-<model hash>-<capability hash>-<conversation>. The capability hash covers the owner, the frontend and tool-policy hashes, the persona and its prompt, the project root and the launch.

Runtime defaults

  • Brain: Pi over RPC. Tools, voice-only extensions and MCP servers are Pi’s. The operator loadout loads MCP through local:mcp from catalog/agents/operator/.vulpine/mcp/default.json (moraine, fox9docs and homeha). A server entry’s toolOverrides replaces a tool’s description and narrows its schema in the voice prompt (catalog/pi/extensions/mcp/README.md).
  • TTS: Qwen3-TTS through the qwen3-worker backend. With VULPINE_TTS_HTTP_URL set, the qwentts.cpp tts-server (127.0.0.1:8850, installed by just models) does the synthesis (TTS adapter, below).
  • STT: the loadouts name parakeet-silero (a parakeet.cpp worker with Silero VAD behind a local WebSocket). The installed config sets [shadow_stt] mode = "authority", which makes the CoreML Parakeet v3 lane (coreml-parakeet-v3, on the ANE) the session STT instead.
  • Either one can be your own: a TTS or STT implementation plugs in as a hardpoint (hardpoint.v1).

The system map is docs/architecture/fox9-system.html, with a static docs/architecture/fox9-system.svg.

Product boundaries

  • Voice Gateway: the Rust runtime for voice sessions, frame routing, adapters, the Pi RPC lifecycle, frontend tools, Avatar Actions and tracing.
  • Vigil: the browser skin. It maps the gateway protocol to a radio-codec call. Its screens are Vigil Frames over a presentation layer shared with Glyph.
  • Vulpine.app: the Mac client (apps/glyph, a Tauri app whose surfaces are moving to GPUI). Glyph is its look.
  • Room speaker: the Voice PE puck, bridged by apps/satellite-esphome as an ordinary gateway client.
  • The Admin: the GPUI control plane (apps/admin-shadow, shipped as Vulpine Gateway.app) over the gateway’s owner-local Admin socket, not the voice WebSocket.
  • Integrations: home, room and fleet adapters run outside the gateway and speak Vulpine’s vocabulary over two contracts (integrations).
  • Brain: the Pi RPC process. It owns reasoning, model and provider access, tool use, skills, extensions, prompt templates and session behavior.
  • Agent: a Pi-backed working identity rooted in a workspace and its Pi configuration.
  • Persona: the voice and UI identity a frontend presents. A bundled demo persona may carry metadata and instructions.
  • Media and runtime outputs: generated or copyrighted media (voice references, fillers, avatar assets, game sound) and runtime outputs (traces, recordings) stay out of git, on the machine that made them.

Rust Voice Gateway protocol

Frontends talk to the gateway over WebSocket:

  • JSON text messages carry control and event frames such as hello, start_session, end_session, vad_hint, audio_playback_observed, audio_echo_risk_observed, herdr_pane_focus_confirmation, interrupt, and errors.
  • Binary messages carry PCM audio. Audio is never base64 in JSON.
  • A frontend declares its capabilities before the session starts. The profile binds the warm-Brain identity, frontend tool negotiation, Avatar Actions and confirmation capabilities.
  • Debug trace streaming is off unless configured; server-side JSONL traces are the default.

Trace event names include mic_frame_received, stt_partial, stt_final, brain_request_start, brain_first_token, tts_request_start, tts_audio_start, frontend_audio_play_observed, barge_in_received and tts_cancel_sent.

Turn management

The gateway owns turn state. Frontend VAD and noise-gate messages are hints; they never commit a turn by themselves. STT and VAD evidence drives the turn.user_started, turn.user_committed and turn.interrupted frames.

The turn machine (TurnState in crates/voice_gateway/src/turn.rs) has three states:

[turn] in the gateway config sets the turn strategy:

  • max_utterance_duration_ms
  • barge_in_confirmation_window_ms
  • the tiered verb resolver keys (verb_tier_shadow, verb_tier_route_relaxed, verb_confirm_*)
  • commit_guards (shadow by default: a guard writes a commit_guard trace row and changes no turn)
  • duplicate-commit arbitration (arbitrate_duplicate_commits, arbitration_window_ms, arbitration_takeover_ms, arbitration_cohorts)

Utterance-ending silence is not a [turn] key: the STT’s end-of-utterance evidence decides it.

With [speculation] enabled, a stable interim transcript goes to Pi before the final commits. The commit then promotes that generation or reforks the Pi session with the committed text. A final that resolves to a tier-1/2 gateway verb cancels it (docs/speculation.md).

Barge-in stops gateway output at once with a turn.interrupted frame; later STT and VAD evidence confirms where your speech began. Client Core plays assistant audio in receive order on one timeline and binds each span to its turn. Only audio actually playing can authorize an interruption. Implausible render or capture timing fails closed to echo-risk quarantine. Each STT cohort carries an id, so a late final from a retired cohort never changes a newer turn, and a transport without cohort ids is refused.

The decision front door

The Decision model is optional and off by default. It is meant to settle more spoken turns before the Brain does; today it settles one kind. With [decision_shadow] on, the gateway asks it on the first partials, beside the turn, and never waits for it. A verb is still matched only by the gateway’s own grammar at the commit. The one answer that acts is the decision gate ([decision_gate]): in the follow-up window after a reply, a line said without the callsign that the model reads as not addressed to Vulpine is dropped instead of going to the Brain. A turn started with the callsign is never gated. Every other answer is traced only (docs/contracts/decision.v1.md, crates/voice_gateway/src/decision_shadow.rs, crates/voice_gateway/src/decision_gate.rs, crates/decidectl/src/lib.rs).

Each entry sits behind its own breaker. The whole cascade answers under one deadline, the session’s follow-up window; running out of it abstains. The gate abstains (the line goes to the Brain as before) whenever no answer was in by the final, or the answering model is not the one its lock pins, and it turns itself off after three Missed labels on its drops in seven days (said as “that was for you”, or the Admin’s Missed button).

Approvals

An agent that wants to act asks the Voice Gateway first, on the owner-local approval socket. Nothing on the socket runs anything (crates/voice_gateway/src/approval_queue.rs, ADR 0040).

create, update and delete are never pre-authorized, and delete never becomes trusted. A DANGER command is the operator’s every time.

Configuration

The gateway reads ~/.config/vulpine/config.toml on every platform and writes it with defaults on first run (default_config_path in crates/voice_gateway/src/config.rs). --config (or VULPINE_GATEWAY_CONFIG) names another file. Command-line flags override the config file. --bind, --trace-dir and --debug-traces also read VULPINE_GATEWAY_* variables. --env-file exports a KEY=VALUE file before anything else runs. The environment variables are named VULPINE_*; vulpine-env.toml lists each one.

Generate the JSON schema with:

just gateway --print-config-schema

.vulpine loadouts

Project-local voice configuration lives under .vulpine/; Pi’s own defaults stay under .pi/:

  • .vulpine/ is the gateway’s voice-session configuration: loadouts, voice-only extensions, adapter choices, lifecycle policy, frontend tool requirements, and any project-root coordinate a closed gateway tool needs ([tools] herdr_attention_project_root, below).
  • .pi/ is Pi’s Brain configuration: Pi defaults, provider and model behavior, tools, skills, prompt templates and session semantics.

The gateway resolves the named loadout from .vulpine/loadouts/<name>.toml, or the default from .vulpine/loadout.toml. It then resolves voice-only extensions from bundled builtin: references or project-local .vulpine/extensions/ paths. A workspace with only .foxline/ is refused at bind; rename it: mv .foxline .vulpine (refuse_legacy_only_workspace in crates/voice_gateway/src/loadout.rs).

docs/foxline-loadouts.md has the schema.

Bundled loadouts are examples and shared defaults. A project-local .vulpine loadout overrides them for that workspace: voice constraints, frontend tool requirements, adapter selection and voice-only extensions. Personas are not per workspace: they live in the Catalog (see Agent and Persona switching).

Pi RPC Brain adapter

The Brain adapter launches only pi --mode rpc, or a process named by the loadout’s [pi] command / [brain] background_command that speaks Pi’s RPC lines (runtime/k-brain, runtime/fm-brain). The selected .vulpine loadout sets the launch arguments:

  • Pi profile, config and session directory fields become explicit Pi arguments.
  • Allowed Pi tools become --tools.
  • Resolved voice-only extensions become explicit --extension flags.
  • --no-context-files is on by default, from the gateway config.

The adapter maps Pi RPC JSONL events into canonical Brain frames:

  • prompt send -> brain.request_start
  • the first text delta is traced as brain_first_token
  • text deltas -> brain.text_delta
  • tool start/end -> brain.tool_call and brain.tool_result
  • agent done -> brain.done
  • Pi errors or rejected prompts -> brain.error

A warm Brain can be prewarmed, shut down when idle, restarted and shut down fully.

The Herdr attention tool’s project root comes from the loadout, never from Pi or a frontend. A loadout that allows the brain_herdr_attention Pi tool must name [tools] herdr_attention_project_root; either one without the other fails the bind (brain/pool.rs). The root is a relative path: non-empty, at most 1,024 bytes, no control characters, never absolute. It resolves from the selected workspace and is refused when it passes through a symlink (ProjectRootAuthority::admit, crates/voice_gateway/src/herdr_attention.rs). Its opaque id is the Brain identity’s project root id.

TTS adapter

The gateway exposes TTS through an adapter trait. The built-in backend is qwen3-worker; a TTS of your own binds as a hardpoint instead (hardpoint.tts.v1: a Wyoming server on loopback, checked at bind, with qwen3-worker as its fallback). qwen3-worker speaks the runtime/qwen3_tts_worker/qwen3_tts_worker.py binary protocol:

  • input speak, cancel and shutdown frames use a 9-byte <type,u32 request_id,u32 payload_len> header.
  • worker audio_start, audio_chunk, audio_done and error frames map to canonical TTS frames.
  • PCM chunks also emit binary audio frames for the frontend.
  • request start, first audio and cancel events are traced.

The gateway hands the worker its voice through inherited descriptors, never a path. The persona WAV and transcript are /dev/fd/3 and /dev/fd/4; the Python worker’s source, embedded in the gateway binary, is /dev/fd/5. Every session’s TTS setup requires VULPINE_QWEN_UV: a root-owned, non-writable absolute executable whose parent folders are root-owned and non-writable too. PATH cannot choose the worker’s bytes.

The worker has two synthesis modes behind the same frame protocol. With VULPINE_TTS_HTTP_URL set (the setting for the TTS server just models installs), it health-checks the qwentts.cpp tts-server, registers the sealed reference as a named voice (VULPINE_TTS_HTTP_VOICE, default operator) plus the mood bank, and streams PCM from the server over HTTP. It loads no model itself. In this mode the worker is the gateway binary itself (--vulpine-tts-worker-v1, the same frames, no Python) at the server’s 24 kHz; VULPINE_TTS_NATIVE_WORKER=0 keeps the Python worker. Without VULPINE_TTS_HTTP_URL, the Python worker loads Qwen3-TTS through MLX on macOS. docs/voice-operator-reference.md covers the live voice.

rust-candle, cpp-ggml, elevenlabs and openai are reserved backend names for future adapters; none is active.

STT adapter

The gateway exposes STT through an adapter trait. Loadouts can name parakeet-silero, native-parakeet-silero or in-process-parakeet, or an STT of your own as a hardpoint (hardpoint.stt.v1). The installed session STT is the CoreML authority lane below. parakeet-silero speaks the Parakeet + Silero WebSocket protocol; its server is not in this repo:

  • binary messages carry PCM audio to the STT service.
  • word and interim messages map to canonical STT partial frames.
  • final messages map to VAD speech-stop evidence plus canonical STT final frames.
  • status messages such as speech start/end map to VAD evidence frames.
  • errors map to canonical STT error frames.

native-parakeet-silero is a fail-closed path for a deployment that supplies VULPINE_STT_NATIVE_WORKER, VULPINE_STT_NATIVE_MODEL, VULPINE_STT_NATIVE_VAD and VULPINE_STT_ARTIFACT_MANIFEST. The gateway checks the manifest against the files’ bytes and sends nothing until the service’s first frame, within five seconds, is a matching ready. This repo supplies none of them, so the path is inactive.

[shadow_stt] runs a second recognizer beside the session STT. With mode = "authority" and backend = "coreml-parakeet-v3" (the installed setting), the CoreML Parakeet v3 lane in crates/coreml_stt becomes the session STT. One recognizer thread serves every session. A segmenting adapter per session closes a final after authority_eou_hold_ms of silence, and emits partials only when authority_partial_interval_ms is non-zero. The loadout’s relay adapter then runs as the shadow only when authority_relay_shadow is on.

The turn manager takes these VAD and STT frames as its turn evidence.

When a session’s recognizer is lost, the client gets the typed stt_error {reason, recoverable} (docs/contracts/adapter-degradation.v1.md), and [stt] on_fault decides the rest:

  • end_session (the default) ends the session on the next microphone frame.
  • text_only keeps it. Typed turns and replies carry on, and the microphone is refused with one spoken line. The gateway rebuilds the recognizer on a 2 s to 30 s backoff until one takes a frame, then sends the client stt_recovered (ws/stt_fault.rs; trace rows stt_mic_refused, stt_reconnect, stt_recovered, stt_fault_pcm_refused).

Frontend clients

Vulpine.app, Vigil, the room speaker’s bridge, the terminal skin and any new client connect through the same gateway WebSocket protocol. The gateway emits only semantic events and Avatar Actions; each client maps them to its own presentation. The LED scanner is a Glyph Avatar, not a separate client.

Vigil advertises the narrow herdr.pane.focus Confirmation Capability:

  • A Pi extension submits only its live tool-call ID and a typed pane_id, through a private, token-authenticated gateway Unix socket. The gateway matches both against the active Brain operation.
  • Client Core shows the gateway’s challenge without answering it. Vigil renders Cancel and Focus buttons.
  • Only the gateway validates the action, capability, pane, session, nonce and 30-second expiry, refuses a replay, and calls Herdr pane.focus over the deployment’s Herdr socket. Pi and the frontend never run that mutation.
  • No other Herdr command, key, close, worktree or layout operation is part of this surface.

Agent and Persona switching

A gateway session names its Pi-backed agent and its voice and presentation persona separately. Both come in start_session, beside workspace, loadout, voice and model (crates/protocol/src/lib.rs).

  • The persona resolves in one place (resolve_session_persona, crates/voice_gateway/src/ws/persona.rs): an explicit persona first (empty or blank counts as unset), then [frontend] default_persona in the gateway config, then the agent’s name.
  • A persona’s package lives in the runtime root’s Catalog, catalog/personas/<id>, and supplies the voice reference and the persona prompt. A workspace that holds a persona folder is refused at bind, with the move named (refuse_workspace_local_personas in crates/voice_gateway/src/loadout.rs).
  • The persona and its prompt are part of the warm-Brain identity (.with_persona_context(persona, persona_prompt) in crates/voice_gateway/src/ws/connection/text/start_session/bind.rs). A different persona or prompt binds a different warm Pi session for the same agent, workspace and loadout. A voice change alone keeps the identity.
  • A different agent changes more: the workspace with its .pi and .vulpine configuration, the loadout, the tool policy and the extensions.
  • No frame switches the agent or the persona inside a session. A second start_session is refused with session_already_started (crates/voice_gateway/src/ws/wire.rs). Client Core switches by ending the session, reconnecting and starting a new one (switchPersona and switchPersonaVoice in packages/voice-client/src/index.ts). switchCharacter also moves the agent when the character names an agent workspace and no workspace override is set. switch_model is the in-session control; the model is part of the identity too.

Avatar Actions

Avatar Actions are semantic presentation requests, not sprite commands. A frontend advertises the action names it supports in its capability profile. The gateway routes only those, as avatar_action server events:

  • set_state
  • set_expression
  • focus
  • play_animation
  • clear

Each client maps these actions to its own visual language.

Frontend tools

Frontend tools are negotiated when the session starts. The effective tool surface is:

(loadout.required_frontend + loadout.optional_frontend) ∩ frontend.advertised_tools

A missing required frontend tool fails the session start with a clear error. An optional tool is available only when the frontend advertises it. The gateway rejects a tool call outside the negotiated surface before it reaches the frontend.

The gateway emits frontend_tools_negotiated, frontend_tool_call and frontend_tool_rejected, and accepts frontend_tool_result control messages. Pi reaches them through the voice-only frontend-tools extension, which is a placeholder today.

Why worker processes instead of HTTP services

  • Lower latency: PCM can stream as soon as the worker emits it.
  • Better interruption/barge-in: cancellation is a direct worker frame.
  • Fewer ports and health checks.
  • One runtime server owns character state, session state, brain, and TTS lifecycle.

Python boundary

Python is a compatibility and development boundary. It remains for:

  • runtime/qwen3_tts_worker/qwen3_tts_worker.py, the MLX path and the fallback; with a TTS server the gateway runs its own Rust worker instead;
  • installer and extraction scripts under tools/;
  • the transcript helper service during asset installation.

Model provisioning

Models are fetched by an explicit provisioning step, never at gateway start. Release binaries do not embed large model weights. Each model is pinned by sha256 in tools/install/models.toml; what a fresh install downloads, and its size, is in docs/install.md.

Important models:

  • Qwen3-TTS (live): the 0.6B Base Q8_0 GGUF served by the qwentts.cpp tts-server.
  • Qwen3-TTS (MLX fallback): mlx-community/Qwen3-TTS-12Hz-0.6B-Base-6bit, the worker’s default without VULPINE_TTS_HTTP_URL.
  • CoreML Parakeet v3: the bundle named by [shadow_stt] bundle_dir for the authority lane.
  • Parakeet: mudler/parakeet-cpp-gguf.
  • VAD: ggml-org/whisper-vad Silero GGML.

Repo layout

  • crates/protocol/ - Rust source of truth for the gateway’s protocol types.
  • crates/voice_gateway/ - Rust Voice Gateway runtime core.
  • crates/overlay_stdio/ - the roll, run as the gateway binary’s overlay-stdio subcommand.
  • crates/vulpine_cli/ - the vulpine command (vulpine models, the pack verifier).
  • crates/coreml_stt/ - CoreML Parakeet v3 recognizer for the ANE STT lane.
  • crates/decidectl/ - decision.v1 client the gateway asks the decision model through.
  • crates/head_mlxc/ - head_r9 run from Rust over mlx-c; an adapter outside the gateway’s Cargo graph.
  • crates/aec_daf/ - Rust port of LocalVQE’s DAF front-end (bulk-delay prealignment, Kalman filter, GRU step-size controller): subtractive echo cancellation on the wake-worker fanout.
  • crates/evidence_harness/ - evidence-replay, offline replay of recorded capture streams for acoustic experiments; reports are aggregate-only.
  • crates/provider_publisher/ - bounded native provider admission and journaled publication, with no execution or host acquisition.
  • crates/vulpine_client/ - the shell-neutral Rust client of the gateway WebSocket.
  • crates/vulpine_env/ - the environment-variable registry, generated from vulpine-env.toml.
  • crates/admin_core/ - the Admin’s UI-free core: the owner-socket client, projections and host-local reads.
  • crates/admin_markers/ - the owner-socket rules the gateway and the Admin share (markers, socket path, frame names, bounds), and the config and state roots.
  • crates/vulpine_fleet/, crates/fleet_probe/ - the fleet adapter, and the probe binary it runs on each fleet host.
  • crates/vulpine_provider/ - whether an upstream model provider is degraded.
  • crates/vulpine_room/ - one interface for making a room say or play something.
  • crates/vulpine_wechat/ - the WeChat send channel’s control plane.
  • crates/vigil_serve/ - Vigil’s installed build served as a static page on loopback.
  • crates/host_footprint/, crates/glyph_shader_check/ - the Glyph’s memory footprint read through libproc, and its WGSL shaders checked with naga.
  • packages/protocol/ - generated TypeScript protocol package (@voidfab/vulpine-protocol).
  • packages/voice-client/ - browser-clean TypeScript Client Core (@voidfab/vulpine-voice-client).
  • packages/vulpine-voice-ui/ - shared React voice chrome (connect control, device rows, captions, confirmations).
  • packages/vulpine-voice-ui-state/ - the voice chrome’s framework-free view-model.
  • apps/vigil/ - Vigil, Vulpine in a browser (a web app).
  • apps/glyph/ - Vulpine.app, the Mac client, in its Glyph look (a Tauri ambient overlay: swappable chassis around Orb, Cast, LED and SVG Visual Themes).
  • apps/admin-shadow/ - the Admin, shipped as Vulpine Gateway.app.
  • apps/satellite-esphome/ - the room speaker bridge for the Voice PE (ESPHome native API to the gateway WebSocket).
  • apps/tui/ - the terminal skin: pi-tui over Client Core, text only.
  • apps/phone/ - Vulpine on iPhone, in development (GPUI).
  • apps/site/ - Vulpine’s public site (static Astro pages).
  • runtime/ - the runtime helpers the gateway, a launchd job or a roll spawns: the worker adapters, the adapter CLIs (homectl, roomctl, visionctl, callctl, screenctl), the brain tools’ scripts, the launchd jobs and the roll; runtime-tools.toml lists them.
  • services/ - the remaining Parakeet STT helpers.
  • tools/ - operator and repo tooling by seam: install and extraction utilities, evaluation, docs and verification, meetctl.
  • scripts/ - user-facing shell entrypoints.
  • catalog/agents/ - bundled demo agent and persona metadata only; generated voice and avatar assets stay local.
  • docs/ - architecture, benchmark, extraction and loadout notes.

See also

Was this page helpful?