---
title: "Architecture"
description: "One voice turn as the code runs it on the Mac mini: the Rust Voice Gateway as the sole gateway and brain path, with its STT, turn machine, speculative brain dispatch and streamed TTS."
type: "living"
source: "ARCHITECTURE.md"
verified: "2026-11-14"
group: "The voice path"
---

> **Living doc**, checked against the code 2026-11-14

Vulpine follows PiBot's low-latency worker design, not HTTP microservices. The Rust Voice Gateway is the only
gateway and the only path to the Brain. `fox9` and `foxline` identifiers below are compatibility names. Installing
it is [docs/install.md](/docs/install-and-operate/install); running and rolling the gateway is
[its README](/docs/install-and-operate/voice-gateway).

The parts, and where they run:

```mermaid
flowchart LR
    subgraph desk["Your Mac or a browser"]
        Glyph["Vulpine.app (the Glyph look)"]
        Vigil["Vulpine in a browser (Vigil)"]
    end
    subgraph host["The gateway host (a Mac)"]
        AdminApp["Vulpine Gateway.app (the Admin)"]
        GW["Voice Gateway :8780"]
        STT["Speech recognition (CoreML Parakeet v3)"]
        DEC["Decision model (optional, off by default)"]
        BRAIN["Pi brain (pi --mode rpc)"]
        TTS["TTS server (qwentts.cpp)"]
        DOCS["Docs site :7806 (optional)"]
    end
    subgraph homeserver["Home server (optional)"]
        HOUSE["Home Assistant, Frigate, speakers"]
    end
    Puck["Room speaker (Voice PE)"] --> Sat["Room speaker bridge"]
    Glyph --> GW
    Vigil --> GW
    Sat --> GW
    AdminApp -. "Admin socket" .-> GW
    GW --> STT
    GW -. "decision.v1, beside the turn" .-> DEC
    GW --> BRAIN
    GW --> TTS
    BRAIN -- "MCP" --> DOCS
    GW -- "adapter CLIs (homectl, roomctl)" --> HOUSE
```

Vulpine.app, Vigil and the room speaker's bridge are clients of the one gateway WebSocket. Vulpine Gateway.app
reaches the gateway only through its owner-local Admin socket.

One voice turn:

```mermaid
---
config:
  sequence:
    wrap: true
    width: 110
    actorMargin: 10
    messageFontSize: 13
---
sequenceDiagram
    participant F as Frontend (Vulpine.app, Vigil, room speaker bridge)
    participant G as Gateway (ws.rs)
    participant S as STT (CoreML Parakeet v3)
    participant D as Decision model (optional)
    participant T as Turn machine (turn.rs)
    participant B as Pi brain (RPC)
    participant W as TTS worker
    participant Q as tts-server (qwentts.cpp)
    F->>G: mic PCM (WebSocket binary)
    G->>S: PCM frames
    S-->>T: speech start and partials
    G-->>D: first partials (decision.v1, never waited for)
    T->>B: speculative prompt off a stable partial (output held)
    S-->>T: final
    Note over T: commit (turn.user_committed)
    alt a verb turn: the gateway's own grammar matches
        T->>G: run the verb (no brain turn; a held speculation is cancelled)
        G->>W: the verb's reply, if it speaks
    else an agent turn: everything else
        T->>B: promote, or refork with the committed text
        B->>B: tools, extensions and MCP servers
        B-->>G: text deltas (brain.text_delta)
        G->>W: speak frame (binary, stdin)
    end
    W->>Q: streaming synthesis (HTTP)
    Q-->>W: PCM
    W-->>G: audio_chunk frames
    G-->>F: assistant PCM (WebSocket binary)
    F-->>G: audio_playback_observed
    opt barge-in while the assistant speaks
        F->>G: interrupt
        G->>W: cancel frame
    end
```

A turn takes one of two paths after the commit. A **verb turn** is a request the gateway matches against its own
short list of verbs ([what you can say](/docs/install-and-operate/voice-commands)): it starts no brain turn. An **agent turn** is
everything else, and goes to the brain. Both wait for the commit, the gateway being sure you have finished: 978 ms
p50 over 218 live turns (`benchmarks/results/brain-ttft-cloud-vs-local-2026-11-06.json`). After it, a verb's match
costs microseconds (`benchmarks/results/page-verb-cost-20261107.md`) and the voice worker's first audio about 140 ms
(`benchmarks/results/tts-idle-ttfa-2026-11-07.json`). An agent turn waits 1.4 s p50 (3.8 s p90) for a hosted
model's first token and 441 ms more for first audio; speech end to audible is 3.0 s p50 for an agent turn.

The STT box is the ANE authority lane, which the installed config turns on (`[shadow_stt] mode = "authority"`).
When `mode` is not `authority`, the loadout's own STT adapter takes its place. Without speculation, the committed
text goes to Pi as an ordinary prompt.

The gateway never calls the Brain's LLM servers; Pi does. The decision model is optional and off by default
(`[decision_shadow]`, below): the turn never waits for it.

The gateway binds each warm Brain process to a `BrainIdentity` of eleven fields
(`crates/voice_gateway/src/brain/launch.rs`): the owner, agent, workspace, loadout, model, persona and persona
prompt, the frontend capability profile's hash, the tool-policy hash (the loadout's allowed Pi tools plus the
negotiated frontend tools), the admitted project root's id, and the launch hash (command, arguments and working
directory). The warm-Brain pool is keyed by the whole identity plus the launch (`brain/pool.rs`). A Brain is reused
only when every field matches. The Pi session is named
`foxline-voice-<agent>-<loadout>-<model hash>-<capability hash>-<conversation>`. The capability hash covers the
owner, the frontend and tool-policy hashes, the persona and its prompt, the project root and the launch.

## Runtime defaults

- Brain: Pi over RPC. Tools, voice-only extensions and MCP servers are Pi's. The operator loadout loads MCP through
  `local:mcp` from `catalog/agents/operator/.vulpine/mcp/default.json` (`moraine`, `fox9docs` and `homeha`). A
  server entry's `toolOverrides` replaces a tool's description and narrows its schema in the voice prompt
  (`catalog/pi/extensions/mcp/README.md`).
- TTS: Qwen3-TTS through the `qwen3-worker` backend. With `VULPINE_TTS_HTTP_URL` set, the qwentts.cpp
  `tts-server` (127.0.0.1:8850, installed by `just models`) does the synthesis (TTS adapter, below).
- STT: the loadouts name `parakeet-silero` (a parakeet.cpp worker with Silero VAD behind a local WebSocket). The
  installed config sets `[shadow_stt] mode = "authority"`, which makes the CoreML Parakeet v3 lane
  (`coreml-parakeet-v3`, on the ANE) the session STT instead.
- Either one can be your own: a TTS or STT implementation plugs in as a hardpoint ([hardpoint.v1](/docs/contracts/hardpoint.v1)).

The system map is `docs/architecture/fox9-system.html`, with a static `docs/architecture/fox9-system.svg`.

## Product boundaries

- Voice Gateway: the Rust runtime for voice sessions, frame routing, adapters, the Pi RPC lifecycle, frontend tools,
  Avatar Actions and tracing.
- Vigil: the browser skin. It maps the gateway protocol to a radio-codec call. Its screens are Vigil Frames over a
  presentation layer shared with Glyph.
- Vulpine.app: the Mac client (`apps/glyph`, a Tauri app whose surfaces are moving to GPUI). Glyph is its look.
- Room speaker: the Voice PE puck, bridged by `apps/satellite-esphome` as an ordinary gateway client.
- The Admin: the GPUI control plane (`apps/admin-shadow`, shipped as Vulpine Gateway.app) over the gateway's
  owner-local Admin socket, not the voice WebSocket.
- Integrations: home, room and fleet adapters run outside the gateway and speak Vulpine's vocabulary over two
  contracts ([integrations](/docs/adapters-and-integrations/overview)).
- Brain: the Pi RPC process. It owns reasoning, model and provider access, tool use, skills, extensions, prompt
  templates and session behavior.
- Agent: a Pi-backed working identity rooted in a workspace and its Pi configuration.
- Persona: the voice and UI identity a frontend presents. A bundled demo persona may carry metadata and instructions.
- Media and runtime outputs: generated or copyrighted media (voice references, fillers, avatar assets, game sound)
  and runtime outputs (traces, recordings) stay out of git, on the machine that made them.

## Rust Voice Gateway protocol

Frontends talk to the gateway over WebSocket:

- JSON text messages carry control and event frames such as `hello`, `start_session`, `end_session`, `vad_hint`,
  `audio_playback_observed`, `audio_echo_risk_observed`, `herdr_pane_focus_confirmation`, `interrupt`, and errors.
- Binary messages carry PCM audio. Audio is never base64 in JSON.
- A frontend declares its capabilities before the session starts. The profile binds the warm-Brain identity, frontend
  tool negotiation, Avatar Actions and confirmation capabilities.
- Debug trace streaming is off unless configured; server-side JSONL traces are the default.

Trace event names include `mic_frame_received`, `stt_partial`, `stt_final`, `brain_request_start`,
`brain_first_token`, `tts_request_start`, `tts_audio_start`, `frontend_audio_play_observed`, `barge_in_received` and
`tts_cancel_sent`.

## Turn management

The gateway owns turn state. Frontend VAD and noise-gate messages are hints; they never commit a turn by
themselves. STT and VAD evidence drives the `turn.user_started`, `turn.user_committed` and `turn.interrupted`
frames.

The turn machine (`TurnState` in `crates/voice_gateway/src/turn.rs`) has three states:

```mermaid
stateDiagram-v2
    [*] --> Idle
    Idle --> Listening: session started
    Listening --> UserSpeaking: speech start or first partial
    UserSpeaking --> Listening: speech stop
    UserSpeaking --> Listening: final commits (turn.user_committed)
    UserSpeaking --> Listening: STT error, loop guard or wake alias
    UserSpeaking --> Listening: max utterance (turn.interrupted)
    Listening --> Listening: final or armed request commits
    Listening --> Idle: over and out, session ended or shutdown
    UserSpeaking --> Idle: over and out, session ended or shutdown
```

`[turn]` in the gateway config sets the turn strategy:

- `max_utterance_duration_ms`
- `barge_in_confirmation_window_ms`
- the tiered verb resolver keys (`verb_tier_shadow`, `verb_tier_route_relaxed`, `verb_confirm_*`)
- `commit_guards` (`shadow` by default: a guard writes a `commit_guard` trace row and changes no turn)
- duplicate-commit arbitration (`arbitrate_duplicate_commits`, `arbitration_window_ms`, `arbitration_takeover_ms`, `arbitration_cohorts`)

Utterance-ending silence is not a `[turn]` key: the STT's end-of-utterance evidence decides it.

With `[speculation] enabled`, a stable interim transcript goes to Pi before the final commits. The commit then
promotes that generation or reforks the Pi session with the committed text. A final that resolves to a tier-1/2
gateway verb cancels it (`docs/speculation.md`).

Barge-in stops gateway output at once with a `turn.interrupted` frame; later STT and VAD evidence confirms where your
speech began. Client Core plays assistant audio in receive order on one timeline and binds each span to its turn.
Only audio actually playing can authorize an interruption. Implausible render or capture timing fails closed to
echo-risk quarantine. Each STT cohort carries an id, so a late final from a retired cohort never changes a newer
turn, and a transport without cohort ids is refused.


## The decision front door

The Decision model is optional and off by default. It is meant to settle more spoken turns before the Brain does;
today it settles one kind. With `[decision_shadow]` on, the gateway asks it on the first partials, beside the turn,
and never waits for it. A verb is still matched only by the gateway's own grammar at the commit. The one answer that
acts is the decision gate (`[decision_gate]`): in the follow-up window after a reply, a line said without the
callsign that the model reads as not addressed to Vulpine is dropped instead of going to the Brain. A turn started
with the callsign is never gated. Every other answer is traced only (`docs/contracts/decision.v1.md`,
`crates/voice_gateway/src/decision_shadow.rs`, `crates/voice_gateway/src/decision_gate.rs`,
`crates/decidectl/src/lib.rs`).

```mermaid
flowchart TD
    P["Partial transcript (up to 'partials', default 2)"] --> SH["[decision_shadow] (off by default)"]
    SH --> LORA["lora: kev.serve :8871 (decision.v1)"]
    LORA -- "answered" --> TR["decision_shadow trace row (stable verdict)"]
    LORA -- "rejected (unreachable, timeout, 5xx, bad reply): its breaker opens" --> HEAD["head: frozen-encoder head_r9 :8872"]
    HEAD --> TR
    HEAD -- "rejected, or the deadline passes" --> NONE["abstain"]
    TR -. "a follow-up without the callsign" .-> GATE{"[decision_gate]: addressed?"}
    GATE -- "no" --> DROP["dropped, nothing sent"]
    GATE -- "yes, or abstains" --> BR
    NONE --> BR["Pi Brain: the fallback"]
```

Each entry sits behind its own breaker. The whole cascade answers under one deadline, the session's follow-up
window; running out of it abstains. The gate abstains (the line goes to the Brain as before) whenever no answer was
in by the final, or the answering model is not the one its lock pins, and it turns itself off after three Missed labels on its drops
in seven days (said as "that was for you", or the Admin's Missed button).


## Approvals

An agent that wants to act asks the Voice Gateway first, on the owner-local approval socket. Nothing on the socket
runs anything (`crates/voice_gateway/src/approval_queue.rs`, ADR 0040).

```mermaid
flowchart TD
    ASK["Agent request on approval.sock (exec, cp, create, update, delete, grant, batch)"] --> DANGER{"DANGER exec?"}
    DANGER -- "yes" --> CARD
    DANGER -- "no" --> PRE{"policy auto glob, 24-hour grant or learned trust?"}
    PRE -- "yes" --> YES["answered at once"]
    PRE -- "no" --> CARD["one spoken card for all pending requests"]
    CARD --> ANS{"operator answers"}
    ANS -- "yes" --> OK["approved"]
    ANS -- "yes, remember" --> GRANT["approved, and a sealed 24-hour grant (never for DANGER)"]
    ANS -- "no or no answer" --> NO["refused; a no resets learned trust"]
    OK -. "teaches" .-> LT["learned trust"]
    LT -. "learned_trust_revoked" .-> CARD
```

create, update and delete are never pre-authorized, and delete never becomes trusted. A DANGER command is the
operator's every time.

## Configuration

The gateway reads `~/.config/vulpine/config.toml` on every platform and writes it with defaults on first run
(`default_config_path` in `crates/voice_gateway/src/config.rs`). `--config` (or `VULPINE_GATEWAY_CONFIG`) names
another file. Command-line flags override the config file. `--bind`, `--trace-dir` and `--debug-traces` also read
`VULPINE_GATEWAY_*` variables. `--env-file` exports a `KEY=VALUE` file before anything else runs. The environment
variables are named `VULPINE_*`; `vulpine-env.toml` lists each one.


Generate the JSON schema with:

```bash
just gateway --print-config-schema
```

## `.vulpine` loadouts

Project-local voice configuration lives under `.vulpine/`; Pi's own defaults stay under `.pi/`:

- `.vulpine/` is the gateway's voice-session configuration: loadouts, voice-only extensions, adapter choices,
  lifecycle policy, frontend tool requirements, and any project-root coordinate a closed gateway tool needs
  (`[tools] herdr_attention_project_root`, below).
- `.pi/` is Pi's Brain configuration: Pi defaults, provider and model behavior, tools, skills, prompt templates and
  session semantics.

The gateway resolves the named loadout from `.vulpine/loadouts/<name>.toml`, or the default from
`.vulpine/loadout.toml`. It then resolves voice-only extensions from bundled `builtin:` references or project-local
`.vulpine/extensions/` paths. A workspace with only `.foxline/` is refused at bind; rename it:
`mv .foxline .vulpine` (`refuse_legacy_only_workspace` in `crates/voice_gateway/src/loadout.rs`).

`docs/foxline-loadouts.md` has the schema.

Bundled loadouts are examples and shared defaults. A project-local `.vulpine` loadout overrides them for that
workspace: voice constraints, frontend tool requirements, adapter selection and voice-only extensions. Personas are
not per workspace: they live in the Catalog (see Agent and Persona switching).

## Pi RPC Brain adapter

The Brain adapter launches only `pi --mode rpc`, or a process named by the loadout's `[pi] command` /
`[brain] background_command` that speaks Pi's RPC lines (`runtime/k-brain`, `runtime/fm-brain`). The selected
`.vulpine` loadout sets the launch arguments:

- Pi profile, config and session directory fields become explicit Pi arguments.
- Allowed Pi tools become `--tools`.
- Resolved voice-only extensions become explicit `--extension` flags.
- `--no-context-files` is on by default, from the gateway config.

The adapter maps Pi RPC JSONL events into canonical Brain frames:

- prompt send -> `brain.request_start`
- the first text delta is traced as `brain_first_token`
- text deltas -> `brain.text_delta`
- tool start/end -> `brain.tool_call` and `brain.tool_result`
- agent done -> `brain.done`
- Pi errors or rejected prompts -> `brain.error`

A warm Brain can be prewarmed, shut down when idle, restarted and shut down fully.

The Herdr attention tool's project root comes from the loadout, never from Pi or a frontend. A loadout that allows
the `brain_herdr_attention` Pi tool must name `[tools] herdr_attention_project_root`; either one without the other
fails the bind (`brain/pool.rs`). The root is a relative path: non-empty, at most 1,024 bytes, no control
characters, never absolute. It resolves from the selected workspace and is refused when it passes through a symlink
(`ProjectRootAuthority::admit`, `crates/voice_gateway/src/herdr_attention.rs`). Its opaque id is the Brain
identity's project root id.

## TTS adapter

The gateway exposes TTS through an adapter trait. The built-in backend is `qwen3-worker`; a TTS of your own binds as
a hardpoint instead ([hardpoint.tts.v1](/docs/contracts/hardpoint.tts.v1): a Wyoming server on loopback, checked at
bind, with `qwen3-worker` as its fallback). `qwen3-worker` speaks the
`runtime/qwen3_tts_worker/qwen3_tts_worker.py` binary protocol:

- input `speak`, `cancel` and `shutdown` frames use a 9-byte `<type,u32 request_id,u32 payload_len>` header.
- worker `audio_start`, `audio_chunk`, `audio_done` and `error` frames map to canonical TTS frames.
- PCM chunks also emit binary audio frames for the frontend.
- request start, first audio and cancel events are traced.

The gateway hands the worker its voice through inherited descriptors, never a path. The persona WAV and transcript
are `/dev/fd/3` and `/dev/fd/4`; the Python worker's source, embedded in the gateway binary, is `/dev/fd/5`. Every
session's TTS setup requires `VULPINE_QWEN_UV`: a root-owned, non-writable absolute executable whose parent folders are root-owned and
non-writable too. `PATH` cannot choose the worker's bytes.


The worker has two synthesis modes behind the same frame protocol. With `VULPINE_TTS_HTTP_URL` set (the setting
for the TTS server `just models` installs), it health-checks the qwentts.cpp `tts-server`, registers the sealed
reference as a named voice (`VULPINE_TTS_HTTP_VOICE`, default `operator`) plus the mood bank, and streams PCM from
the server over HTTP. It loads no model itself. In this mode the worker is the gateway binary itself
(`--vulpine-tts-worker-v1`, the same frames, no Python) at the server's 24 kHz; `VULPINE_TTS_NATIVE_WORKER=0` keeps
the Python worker. Without `VULPINE_TTS_HTTP_URL`, the Python worker loads Qwen3-TTS through MLX on macOS.
`docs/voice-operator-reference.md` covers the live voice.

`rust-candle`, `cpp-ggml`, `elevenlabs` and `openai` are reserved backend names for future adapters; none is active.

## STT adapter

The gateway exposes STT through an adapter trait. Loadouts can name `parakeet-silero`, `native-parakeet-silero` or
`in-process-parakeet`, or an STT of your own as a hardpoint
([hardpoint.stt.v1](/docs/contracts/hardpoint.stt.v1)). The installed session STT is the CoreML authority lane
below. `parakeet-silero` speaks the
Parakeet + Silero WebSocket protocol; its server is not in this repo:

- binary messages carry PCM audio to the STT service.
- `word` and `interim` messages map to canonical STT partial frames.
- `final` messages map to VAD speech-stop evidence plus canonical STT final frames.
- status messages such as speech start/end map to VAD evidence frames.
- errors map to canonical STT error frames.

`native-parakeet-silero` is a fail-closed path for a deployment that supplies `VULPINE_STT_NATIVE_WORKER`,
`VULPINE_STT_NATIVE_MODEL`, `VULPINE_STT_NATIVE_VAD` and `VULPINE_STT_ARTIFACT_MANIFEST`. The gateway checks the
manifest against the files' bytes and sends nothing until the service's first frame, within five seconds, is a
matching `ready`. This repo supplies none of them, so the path is inactive.


`[shadow_stt]` runs a second recognizer beside the session STT. With `mode = "authority"` and
`backend = "coreml-parakeet-v3"` (the installed setting), the CoreML Parakeet v3 lane in `crates/coreml_stt` becomes
the session STT. One recognizer thread serves every session. A segmenting adapter per session closes a final after
`authority_eou_hold_ms` of silence, and emits partials only when `authority_partial_interval_ms` is non-zero. The
loadout's relay adapter then runs as the shadow only when `authority_relay_shadow` is on.

The turn manager takes these VAD and STT frames as its turn evidence.

When a session's recognizer is lost, the client gets the typed `stt_error {reason, recoverable}`
(`docs/contracts/adapter-degradation.v1.md`), and `[stt] on_fault` decides the rest:

- `end_session` (the default) ends the session on the next microphone frame.
- `text_only` keeps it. Typed turns and replies carry on, and the microphone is refused with one spoken line. The
  gateway rebuilds the recognizer on a 2 s to 30 s backoff until one takes a frame, then sends the client
  `stt_recovered` (`ws/stt_fault.rs`; trace rows `stt_mic_refused`, `stt_reconnect`, `stt_recovered`,
  `stt_fault_pcm_refused`).

## Frontend clients

Vulpine.app, Vigil, the room speaker's bridge, the terminal skin and any new client connect through the same
gateway WebSocket protocol.
The gateway emits only semantic events and Avatar Actions; each client maps them to its own presentation. The LED
scanner is a Glyph Avatar, not a separate client.


Vigil advertises the narrow `herdr.pane.focus` Confirmation Capability:

- A Pi extension submits only its live tool-call ID and a typed `pane_id`, through a private, token-authenticated
  gateway Unix socket. The gateway matches both against the active Brain operation.
- Client Core shows the gateway's challenge without answering it. Vigil renders Cancel and Focus buttons.
- Only the gateway validates the action, capability, pane, session, nonce and 30-second expiry, refuses a replay,
  and calls Herdr `pane.focus` over the deployment's Herdr socket. Pi and the frontend never run that mutation.
- No other Herdr command, key, close, worktree or layout operation is part of this surface.

## Agent and Persona switching

A gateway session names its Pi-backed `agent` and its voice and presentation `persona` separately. Both come in
`start_session`, beside `workspace`, `loadout`, `voice` and `model` (`crates/protocol/src/lib.rs`).

- The persona resolves in one place (`resolve_session_persona`, `crates/voice_gateway/src/ws/persona.rs`): an
  explicit `persona` first (empty or blank counts as unset), then `[frontend] default_persona` in the gateway
  config, then the agent's name.
- A persona's package lives in the runtime root's Catalog, `catalog/personas/<id>`, and supplies the voice reference
  and the persona prompt. A workspace that holds a persona folder is refused at bind, with the move named
  (`refuse_workspace_local_personas` in `crates/voice_gateway/src/loadout.rs`).
- The persona and its prompt are part of the warm-Brain identity
  (`.with_persona_context(persona, persona_prompt)` in
  `crates/voice_gateway/src/ws/connection/text/start_session/bind.rs`). A different persona or prompt binds a
  different warm Pi session for the same agent, workspace and loadout. A voice change alone keeps the identity.
- A different agent changes more: the workspace with its `.pi` and `.vulpine` configuration, the loadout, the tool
  policy and the extensions.
- No frame switches the agent or the persona inside a session. A second `start_session` is refused with
  `session_already_started` (`crates/voice_gateway/src/ws/wire.rs`). Client Core switches by ending the session,
  reconnecting and starting a new one (`switchPersona` and `switchPersonaVoice` in
  `packages/voice-client/src/index.ts`). `switchCharacter` also moves the agent when the character names an agent
  workspace and no workspace override is set. `switch_model` is the in-session control; the model is part of the
  identity too.

## Avatar Actions

Avatar Actions are semantic presentation requests, not sprite commands. A frontend advertises the action names it
supports in its capability profile. The gateway routes only those, as `avatar_action` server events:

- `set_state`
- `set_expression`
- `focus`
- `play_animation`
- `clear`

Each client maps these actions to its own visual language.

## Frontend tools

Frontend tools are negotiated when the session starts. The effective tool surface is:

```text
(loadout.required_frontend + loadout.optional_frontend) ∩ frontend.advertised_tools
```

A missing required frontend tool fails the session start with a clear error. An optional tool is available only
when the frontend advertises it. The gateway rejects a tool call outside the negotiated surface before it reaches the
frontend.

The gateway emits `frontend_tools_negotiated`, `frontend_tool_call` and `frontend_tool_rejected`, and accepts
`frontend_tool_result` control messages. Pi reaches them through the voice-only frontend-tools extension, which is a
placeholder today.

## Why worker processes instead of HTTP services

- Lower latency: PCM can stream as soon as the worker emits it.
- Better interruption/barge-in: cancellation is a direct worker frame.
- Fewer ports and health checks.
- One runtime server owns character state, session state, brain, and TTS lifecycle.

## Python boundary

Python is a compatibility and development boundary. It remains for:

- `runtime/qwen3_tts_worker/qwen3_tts_worker.py`, the MLX path and the fallback; with a TTS server the gateway runs
  its own Rust worker instead;
- installer and extraction scripts under `tools/`;
- the transcript helper service during asset installation.


## Model provisioning

Models are fetched by an explicit provisioning step, never at gateway start. Release binaries do not embed large
model weights. Each model is pinned by sha256 in `tools/install/models.toml`; what a fresh install downloads, and
its size, is in [docs/install.md](/docs/install-and-operate/install).

Important models:

- Qwen3-TTS (live): the 0.6B Base Q8_0 GGUF served by the qwentts.cpp `tts-server`.
- Qwen3-TTS (MLX fallback): `mlx-community/Qwen3-TTS-12Hz-0.6B-Base-6bit`, the worker's default without
  `VULPINE_TTS_HTTP_URL`.
- CoreML Parakeet v3: the bundle named by `[shadow_stt] bundle_dir` for the authority lane.
- Parakeet: `mudler/parakeet-cpp-gguf`.
- VAD: `ggml-org/whisper-vad` Silero GGML.


## Repo layout

- `crates/protocol/` - Rust source of truth for the gateway's protocol types.
- `crates/voice_gateway/` - Rust Voice Gateway runtime core.
- `crates/overlay_stdio/` - the roll, run as the gateway binary's `overlay-stdio` subcommand.
- `crates/vulpine_cli/` - the `vulpine` command (`vulpine models`, the pack verifier).
- `crates/coreml_stt/` - CoreML Parakeet v3 recognizer for the ANE STT lane.
- `crates/decidectl/` - decision.v1 client the gateway asks the decision model through.
- `crates/head_mlxc/` - head_r9 run from Rust over mlx-c; an adapter outside the gateway's Cargo graph.
- `crates/aec_daf/` - Rust port of LocalVQE's DAF front-end (bulk-delay prealignment, Kalman filter, GRU step-size
  controller): subtractive echo cancellation on the wake-worker fanout.
- `crates/evidence_harness/` - `evidence-replay`, offline replay of recorded capture streams for acoustic
  experiments; reports are aggregate-only.
- `crates/provider_publisher/` - bounded native provider admission and journaled publication, with no execution or
  host acquisition.
- `crates/vulpine_client/` - the shell-neutral Rust client of the gateway WebSocket.
- `crates/vulpine_env/` - the environment-variable registry, generated from `vulpine-env.toml`.
- `crates/admin_core/` - the Admin's UI-free core: the owner-socket client, projections and host-local reads.
- `crates/admin_markers/` - the owner-socket rules the gateway and the Admin share (markers, socket path, frame
  names, bounds), and the config and state roots.
- `crates/vulpine_fleet/`, `crates/fleet_probe/` - the fleet adapter, and the probe binary it runs on each fleet host.
- `crates/vulpine_provider/` - whether an upstream model provider is degraded.
- `crates/vulpine_room/` - one interface for making a room say or play something.
- `crates/vulpine_wechat/` - the WeChat send channel's control plane.
- `crates/vigil_serve/` - Vigil's installed build served as a static page on loopback.
- `crates/host_footprint/`, `crates/glyph_shader_check/` - the Glyph's memory footprint read through libproc, and
  its WGSL shaders checked with naga.
- `packages/protocol/` - generated TypeScript protocol package (`@voidfab/vulpine-protocol`).
- `packages/voice-client/` - browser-clean TypeScript Client Core (`@voidfab/vulpine-voice-client`).
- `packages/vulpine-voice-ui/` - shared React voice chrome (connect control, device rows, captions, confirmations).
- `packages/vulpine-voice-ui-state/` - the voice chrome's framework-free view-model.
- `apps/vigil/` - Vigil, Vulpine in a browser (a web app).
- `apps/glyph/` - Vulpine.app, the Mac client, in its Glyph look (a Tauri ambient overlay: swappable chassis around
  Orb, Cast, LED and SVG Visual Themes).
- `apps/admin-shadow/` - the Admin, shipped as Vulpine Gateway.app.
- `apps/satellite-esphome/` - the room speaker bridge for the Voice PE (ESPHome native API to the gateway WebSocket).
- `apps/tui/` - the terminal skin: pi-tui over Client Core, text only.
- `apps/phone/` - Vulpine on iPhone, in development (GPUI).
- `apps/site/` - Vulpine's public site (static Astro pages).
- `runtime/` - the runtime helpers the gateway, a launchd job or a roll spawns: the worker adapters, the adapter
  CLIs (homectl, roomctl, visionctl, callctl, screenctl), the brain tools' scripts, the launchd jobs and the roll;
  `runtime-tools.toml` lists them.
- `services/` - the remaining Parakeet STT helpers.
- `tools/` - operator and repo tooling by seam: install and extraction utilities, evaluation, docs and verification,
  meetctl.
- `scripts/` - user-facing shell entrypoints.
- `catalog/agents/` - bundled demo agent and persona metadata only; generated voice and avatar assets stay local.
- `docs/` - architecture, benchmark, extraction and loadout notes.


## See also

- [CONTEXT.md](/docs/the-voice-path/glossary) - the glossary behind the names above.
- docs/architecture/README.md - the system map.
- [docs/key-decisions.md](/docs/install-and-operate/key-decisions) - the decisions that shape it, in plain words.
- [docs/speculation.md](/docs/the-voice-path/speculation) - the speculative turn in full.
- [docs/contracts/decision.v1.md](/docs/contracts/decision.v1) - the decision wire.
