Avatar actions
The avatar wire: the five verbs (set_expression, play_animation, set_state, focus, clear) with closed, bounded arguments (intensity, hold_ms, onset, queue, a source ladder), the vocabulary a rig publishes at bind (an empty declared set refuses everything), the four refusal reasons, and the routed/refused counts the Admin shows per frontend. No producer yet.
Living doc, checked against the code 2026-11-14
avatar-action.v1 — the five avatar verbs, their closed arguments and the rig’s vocabulary
- Status: live. The producers are the tool-receipt reactor (always on), the local lexicon (on by default), the
decision model’s
motion_planquestion once the decision model is set up ([decision_shadow] enabled, off by default), and the operator’svulpine-glyphctl avatar <json>(“Who produces actions” below). This page fixes the shape a producer fills and a rig renders. - Doctrine: avatar presence. Vulpine is a modular protocol with anchor points. A rig is puppeteered by closed questions with closed answers: the producer names a verb and arguments from a set the rig declared. It never sends rig weights, bone angles, keyframes or freeform text.
- Relates:
decision.v1(the producer’s wire),answer-card.v1(the panel channel beside speech),avatar-puppet.v1(the rig side: how a rig draws picks, the override table, the gaze split, the director’s dwell and apex cap, the gated typed questions).
The rule in one line
The gateway relays an avatar action to a bound frontend only when the frontend declared the verb and, when the frontend published a vocabulary, declared the argument. Every argument is bounded. Every refusal has a closed reason, is counted on the frontend’s Admin row and is traced.
The five verbs (gateway → client avatar_action)
The client receives {"type": "avatar_action", "action": {…}}. The action object is one of these. Unknown keys are
refused. Every verb may carry source (below).
verb (type) |
arguments | rule |
|---|---|---|
set_expression |
expression, intensity, hold_ms, onset, rate?, at_ms? |
expression: 1..=64 chars, a name from the rig’s set_expression vocabulary when it published one. intensity: 0..=1, default 1. hold_ms: 0..=30 000, default 4 000; 0 holds until clear; otherwise the client releases to its reactive layer when the time is up. onset: immediate (default) or segment:<n> (below). rate, at_ms: below. |
play_animation |
animation, intensity, queue |
animation: 1..=64 chars, a name from the rig’s play_animation vocabulary when it published one (the older key name still parses). intensity: 0..=1, default 1. queue: append | replace (default) | interrupt. |
set_state |
state |
closed: idle | listening | thinking | speaking | interrupted | reacting. The rig maps each to its own looks. |
focus |
target, x?, y?, rate?, at_ms? |
target closed: user | away | down | up | point. x, y: -1..=1 (a point in the rig’s frame, 0,0 the center); meaningful with point only. rate ({x, y}), at_ms: below. |
clear |
none | release every held expression and queued animation now. |
onset: "segment:<n>" (n 0..=4096) asks the client to apply the action when the reply’s n-th TTS segment starts
playing (the n-th RequestStart of the reply’s TTS queue, counted from 0), not when the frame arrives. A client
that cannot tell segments apart applies it at once. No producer sends a segment onset; the operator’s
vulpine-glyphctl avatar can. The rig beside the session hears the segments start (below, “Live segment onsets”).
rate and at_ms: a producer that decides at 2 to 5 Hz sends where a value is and how fast it moves, and the rig
draws every frame between its decisions.
set_expression’srateis the intensity’s change a second, a number in -4..=4.focus’srateis the point’s,{x, y}, each -4..=4 frame units a second (meaningful withpointonly).at_msis when the value held: a unix ms (the producer’s decision), absent meaning the frame’s arrival.- The rig draws
value + rate × (now − at)inside the value’s bounds (0..=1, -1..=1) for at most 1 000 ms afterat(AVATAR_RATE_HORIZON_MS), then holds it. A decision that never comes again cannot run the value away. The rig asks for frames only while the value moves. - A rate past the bound is
out_of_range. A wrong shape (a number onfocus,{x, y}onset_expression, a missing axis, anat_msthat is not a whole unix ms) isunknown_argument. On the other verbs both keys are unknown.
intensity scales the rig’s own preset (a rig decides what 0.3 of smile looks like); it is never a blend weight.
Examples
{"type": "set_expression", "expression": "happy", "intensity": 0.6, "hold_ms": 4000, "onset": "segment:2", "source": "llm_directive"}
{"type": "play_animation", "animation": "attention", "queue": "interrupt", "source": "contextual_fallback"}
{"type": "set_state", "state": "thinking"}
{"type": "focus", "target": "point", "x": -0.4, "y": 0.1}
{"type": "clear"}
The short forms take the defaults: {"type": "set_expression", "expression": "alert"},
{"type": "play_animation", "name": "speaking"}, {"type": "set_state", "state": "speaking"},
{"type": "focus", "target": "user"}.
source — where the pick came from
Optional on every verb. The ladder, strongest first:
source |
meaning |
|---|---|
llm_directive |
the model answered the typed question for this reply |
requested_emotion |
the operator or a tool asked for this look explicitly |
contextual_fallback |
a rule over the turn’s context picked it (a tool outcome, the phase) |
text_fallback |
a local lexicon over the reply’s words picked it |
The Admin shows the source of the last action. A frontend may use it to arbitrate (below).
The vocabulary a rig publishes at bind
FrontendCapabilities (the hello) carries two fields for avatars:
avatar_actions: [verb, …]— the verbs the frontend renders. An empty list means NONE: every action is refusedundeclared. The satellite puck declares[]and renders none. A satellite with a face declares its verbs (“A face on a satellite” below).avatar_vocabulary(optional, appended) — the names the rig can draw, per verb:{"set_expression": [names…], "play_animation": [names…]}. A verb whose list is absent is checked by name only; a list that is present is closed (an empty list refuses every argument). A list keeps its first 128 names; a name is 1..=64 characters and others are ignored.set_stateandfocusneed no vocabulary: their arguments are closed by this contract.
Vigil’s clip rig vulpi (vulpi v4, apps/vigil/src/portrait/rigs/vulpi-v4.json) has ten clips: idle, thinking,
roam, working, sleeping, attention, notification, error, juggling and affection (attention is the
completion pose). Vigil declares:
"avatar_actions": ["set_expression", "set_state", "play_animation", "clear"],
"avatar_vocabulary": {"set_expression": [], "play_animation": ["idle", "thinking", "roam", "working", "sleeping", "attention", "notification", "error", "juggling", "affection"], "presence_draws_tools": true}
The empty set_expression list refuses every expression until a Live2D model re-declares its own names. The
set_state mapping (thinking → the thinking clip, reacting → attention, the rest → idle) stays in the
rig: presets live in the frontend, and the gateway carries names.
Vigil reads the clip list from the rig’s own table (rigClipNames): the clip states in the table’s order, then each
reaction clip whose file is not already a state’s. Fallback names such as sweeping borrow another clip and are not
declared. The names a producer may pick, the options of the motion_plan question and the names the Admin lists are
these.
The rig report
A frontend that draws a rig tells the gateway what it draws now, so the Admin shows why the fox is still. The client
frame rig_report (a ClientControl variant) carries:
| field | words |
|---|---|
kind |
portrait (the sprite faces), clip (a whole-body clip rig) or face (a face-state table; sent only to a gateway that offers rig_face) |
manifest_id |
the rig manifest’s id, e.g. vulpi v4 (≤ 64 chars, no control characters) |
state |
the state (or requested clip) drawn now (same bound) |
decode |
img (the browser plays the clip), img_under_pass (the <img> still shown under the ASCII pass while its frames decode; sent only to a gateway whose hello offers rig_decode_under_pass, else it reads img), still (reduced motion), the pass’s frame decoder (image_decoder, snapshots, atlas), or for a face table and text (below) |
decode_ms |
the drawn clip’s decode time, for a frame decoder (≤ 600 000) |
held_bytes |
the bytes the drawn clip’s decoded frames hold, while decoded frames are drawn (≤ 4 GiB); the Admin’s rig line shows it as “4.9 MB held at 25%” |
scale_pct |
the snapshot scale the frames are kept at, in percent (1 to 100), with held_bytes |
drawn_as |
what a face rig drew when its art folds the state (sleepy drawn as sad) |
A client sends the frame only to a gateway whose hello offers rig_report. Vigil sends it on each change and after
each bind. The gateway keeps the latest on the session’s Admin row (AdminPresentationSession.avatar_rig, with
at_ms) and clears it at bind. It traces avatar_rig when the kind, manifest or decode path changes, and drops an
out-of-bounds report as avatar_rig_ignored {reason}. Nothing is answered. The Admin’s avatar card shows “rig: vulpi
v4 · clip · thinking · snapshots decoded in 569 ms”.
Refusals
| reason | when |
|---|---|
unknown_action |
the type is not one of the five verbs |
undeclared |
the frontend did not declare the verb (including an empty avatar_actions) |
unknown_argument |
an unknown key; a closed value outside its set (state, target, queue, onset); a missing or empty name; or a name outside the rig’s published vocabulary |
out_of_range |
intensity outside 0..=1, hold_ms over 30 000, x/y outside -1..=1, a segment over 4096, a name over 64 chars |
A refused action is not sent. The gateway traces avatar_action_rejected {verb, reason} (never the argument) and
counts it on the frontend’s Admin row.
Who produces actions
Actions come from the gateway’s producers, off the spoken path, and never from a required brain tool call: picking a face costs no model round before the reply.
Every producer picks only from what the bound frontend declared: set_state and focus from this contract’s closed
sets (never focus: point, which needs coordinates), set_expression and play_animation from the rig’s published
list. A verb declared with no list gets nothing, because a producer never guesses a name. A rig that declared nothing
(the Voice PE puck, Vulpine.app) gets nothing.
| producer | when | source |
on by |
|---|---|---|---|
| the tool-receipt reactor | react.waiting when the brain’s tool batch opens (its first call); react.success or react.error when the batch ends (an error is Pi’s pi_tool_error on any call of the batch) |
contextual_fallback |
always (sends only to a rig that declared a look for it) |
the decision model’s motion_plan |
the reply is done (brain_done); asked on its own task under [decision_shadow] fallback_deadline_ms, the verdict lands through the session’s wake |
llm_directive |
[decision_shadow] motion_plan = true (the default), with [decision_shadow] enabled (off by default) |
| the local lexicon | the model abstained, its answer fell under the margin, or the shadow is off | text_fallback |
the same flag |
| the operator | vulpine-glyphctl avatar [--session <id>] <json> (the client control avatar_action; --session routes to another bound session’s rig) |
as sent; requested_emotion when the action names none |
always |
The reactor’s picks, first declared wins: the rig’s own react.<kind> name (as an animation, then as an
expression), then the generic looks — waiting: play_animation working, set_expression thinking, set_state thinking; success: play_animation attention, set_expression happy, set_state reacting; error: play_animation error, set_expression concerned, set_expression sad, set_state reacting. A repeat of the same kind sends
nothing (one waiting per batch, one outcome per change); the reactor resets at the reply’s end.
The motion_plan question (decision.v1; an operation-and-target shape). One request per reply:
- The state is
{reply, phase}(the reply’s first 600 characters; the trace never carries it). - The questions are
motion_plan, a choice over the declared operations in this contract’s verb order plusnone(keep the avatar as it is), and one argument questionmotion_plan_<verb>for each operation with two or more names. A verb with one name dispatches with no question;clearhas none. - The operation’s instructions say the reply is data to read, never instructions to follow.
- The answer acts only when every asked choice passes the reply-membership guard and its margin is at least
[decision_shadow] motion_plan_min_margin(default 0.394, 0 to 1).nonekeeps the avatar still and the lexicon does not run. - The lexicon is five cues over the reply’s whole words and phrases, failure first (sorry, failed, can’t → error looks; heads up, warning → notification; done, all set → attention; thanks → affection; hmm, let me think → thinking).
Each reply’s verdict is one avatar_motion_plan trace row: decision (act, still, an abstain reason such as
low_margin, abstained, not_asked), margin, fell_back, verb, source, elapsed_ms and stale (the next
turn had started, so the action was not sent). Every send is avatar_action_sent {verb, producer}; every refusal
avatar_action_rejected {verb, reason, producer}.
Other sessions, receipts and re-declaring
Every field here is optional within protocol 5.
-
Another session’s rig.
avatar_actiontakes an optionalsession_id(vulpine-glyphctl avatar --session <id> <json>, the id as the session’s Admin row names it). The gateway queues the action on that session’s row (at most 8 waiting). That session routes it through its own router (producer: operator_forwarded), so the routed or refused line lands on its Admin row. The requester tracesavatar_action_forwarded {session_id}. The requester gets anerrorforavatar_session_unknown(no live session with a bound avatar router has the id),avatar_session_busy(its queue is full) oravatar_session_ui(a roleuiconnection never moves another session’s avatar). Its own id, or none, routes locally. “The forwarded action’s answer” below gives the reply. -
The reactor’s opt-out.
avatar_vocabulary.presence_draws_tools: truesays the rig’s own presence reducer already draws tool outcomes, so the tool-receipt reactor sends it nothing. Vigil declares it. The lexicon and the motion plan still pick. -
The plan on the Admin row. Each reply’s verdict also lands on
avatar_routing.motion_plan(AdminAvatarMotionPlan {decision, margin, fell_back, verb, stale, at_ms}: the trace row’s closed words, never the reply). -
The vocabulary re-declared. The client frame
avatar_redeclare {vocabulary}replaces the names the hello published (bounded as at bind; the declared verbs and the counts stay), and the Admin row lists the new ones. It is ignored (avatar_redeclare_ignored) before the capabilities are declared, and tracedavatar_redeclaredwith the list sizes. The gateway’s hello offersavatar_redeclare. Vigil sends it when the loaded theme.json’s clip names differ from the committed table, so the producers pick a new fox’s names. A loaded Live2D model re-declares its Emote names asset_expressionand no clips; leaving it puts the committed table back. -
The per-action wire receipt. Every
avatar_actionthe gateway sends carries anaction_id(avatar-<n>, minted per router; the newest 32 unreceipted ids are kept). The Client Core answers each with the client frameavatar_receipt {action_id, verb, outcome: drawn | refused, reason?}(reasons:undeclared,unknown_argument,no_rig: no rig on screen), only to a gateway whose hello offersavatar_receipt. The gateway counts it onavatar_routing.receipts(AdminAvatarReceipts {drawn, refused, last_verb, last_outcome, last_reason}) and tracesavatar_receipt(the id, verb and closed words, never the argument). An id it did not send (or already receipted), a verb that is not the sent one, or a reason on the wrong outcome is tracedavatar_receipt_ignored {reason}and dropped. Nothing is answered.
What the Admin shows (the avatar card, under the counts): receipts: drawn 3 · refused 1 · last refused play_animation (no_rig) (a warning on a refusal), plan: low_margin 0.21 · fell back · play_animation 4s ago, and
the rig line’s img under the pass (frames decoding).
Precedence
When several layers want the face at once, the higher wins within a channel and the lower resumes when it releases:
idle < presence (the phase) < mood < brain AvatarAction < reflex (a tool failure)
Every request auto-releases after its hold_ms unless it holds (0). Each rig applies the order itself; the Client
Core does not arbitrate.
The receipt (Client Core)
The Client Core computes one receipt per action it receives, {outcome: applied | unsupported, verb, argument}
(avatarReceipt, in packages/voice-client/src/presentation/avatarReceipt.ts), and writes it to its trace as
avatar_action_received. It also sends it back on the wire as avatar_receipt (see “Other sessions, receipts and
re-declaring”), so the gateway learns what the frontend drew.
What the Admin shows
The Presentation panel’s “Avatar actions” card, per bound frontend: the declared verbs, the published vocabulary,
the routed and refused counts since the frontend bound, and the last action with its verb, argument, source,
outcome, refusal reason and age. The row field is AdminPresentationSession.avatar_routing, present once the
frontend declared its capabilities.
The last action’s source names which producer picked it (contextual_fallback the reactor, llm_directive the
model, text_fallback the lexicon, requested_emotion the operator’s vulpine-glyphctl avatar). An operator action
the rig refused shows there as the refused line with its reason, the check vulpine-glyphctl avatar relies on (it
prints only that the action was sent).
Frontends today
| frontend | declares | renders |
|---|---|---|
Vigil (apps/vigil/src/services/gatewayClient.ts) |
set_state, play_animation, clear and the vulpi names (ten, read from the rig’s table: apps/vigil/src/portrait/avatarRequest.ts); reports its rig (rig_report); also set_expression, published empty (none) until a Live2D model loads and re-declares its Emote names |
the clip rig plays the routed clip over the presence-driven state for the clip’s own length (a loop for 4 s), set_state mapped in the rig, clear drops it; the Live2D portrait (apps/vigil/src/live2d) wears a routed set_expression (its intensity, hold_ms, 0 until the next) with the exp3’s own fade, and clear returns it to rest; the portrait faces (Vigil’s default station) draw nothing for it |
the satellite (apps/satellite-esphome/src/gateway-link.ts) |
[] (the Voice PE puck has no screen); with face: true set_expression, set_state, focus, clear and the eight emotions, play_animation with the three hand animations, and the mood feature |
the puck none; a face satellite the face state table (apps/satellite-esphome/src/face-rig.ts, drawn as text by the fixture) |
Vulpine.app (the Glyph look; the Client Core’s avatarActions option, unset) |
[] |
none |
the conversation window’s 3D fox (apps/convo-gpui, a role ui observer beside Vulpine.app) |
the five verbs, nine expressions and eight animations (apps/fox-gpui/src/directions.rs; the hello fixture’s convo_gpui_fox sender); the portrait director’s 22 faces and alert while the Avatar look is offered |
the faces, gestures, states and gazes over the presence (“The rig beside the session” below) |
Vigil’s Live2D portrait reports its rig as kind portrait, manifest live2d <model>.
A face on a satellite
A satellite with a screen (a Stack-chan on the desk; the Voice PE puck has none, ADR 0033) is a rig like any other: it
declares its verbs and names, and the gateway’s router routes to it by that declaration, whatever the frontend’s kind.
No gateway code is specific to it. The face is the Stack-chan firmware’s face state table, in
packages/voice-client/src/presentation/faceState.ts:
- The table (
FaceState):mouth.open,eyes.{left,right}.{open, gazeX, gazeY}(gaze -1..=1, 0,0 the user, +y up),breath,emotion,theme.{primary, secondary}. Every look a face draws reads this one struct. - The emotions in a stable wire order (the index is the firmware’s number):
neutral0,angry1,sad2,happy3,sleepy4,doubtful5,cold6,hot7. - The capabilities the verbs write through: Face (
setColor,setEmotion,setEyeOpen,setMouthOpen), Motion (lookAt(x, y),lookAway; no poses or servo torque), Lighting (lightOn,lightOff,lightBlink). - The fold for limited art: a pack with fewer drawings maps each emotion onto one it has and the rest onto its
default. The demo pack draws four (
normal,happy,sad,angry):neutralto normal,sleepyandcoldto sad,hotto happy,doubtfulto the default (normal). The rig still declares all eight; the producer never learns of the fold.
The face’s declaration (faceDeclaration(); the hello fixture’s satellite_esphome_face sender), without the hands
that “The face round 2” below adds:
"avatar_actions": ["set_expression", "set_state", "focus", "clear"],
"avatar_vocabulary": {"set_expression": ["neutral", "angry", "sad", "happy", "sleepy", "doubtful", "cold", "hot"]}
How the rig (apps/satellite-esphome/src/face-rig.ts) draws each verb:
set_expressionsets the emotion and holds it forhold_ms(0 untilclear), then releases to the phase’s emotion.set_stateapplies the rig’s phase presets (thinkingisdoubtfulwith a blinking status light,reactingishappy,speakingopens the mouth,listeninglights the status light) under a held expression, which outranks the phase.focusmoves both eyes’ gaze (user0,0,up/down0.6 either way,awaya glance right,pointthe given x, y).clearreleases the held look and looks back at the user.
A segment onset applies at once (the bridge cannot tell TTS segments apart). With these names the producers pick for
a face: the reactor’s set_expression happy on success and sad on an error (concerned is not declared), and
set_state thinking while a tool batch runs. The motion_plan question offers the four verbs and the eight names.
What the Admin shows: the face satellite’s Presentation row reads “satellite full”, and its avatar card lists the
four verbs and “set_expression: neutral, angry, sad, happy, sleepy, doubtful, cold, hot”, from the existing
AdminPresentationSession.avatar_routing.vocabulary.set_expression.
The face round 2
- The hands. Stack-chan’s
RobotUI.setHandAnimationis the face’splay_animation, with the closed listclap,thinking,rock-paper-scissors(FACE_HAND_ANIMATIONS;noneis the hands at rest, never a declared name). The rig plays one at a time for 3 s (FACE_HAND_ANIMATION_MS).queue: appendwaits behind the one playing (at most 4 wait; a further one is dropped).replace(the default) andinterruptstop what plays and drop the queue.clearstops the hands and drops the queue.intensityis kept in the table; the bridge has no blend. The lexicon’s “let me think” cue picksplay_animation thinking, and themotion_planquestion offers the five verbs and the three hand names. - The mood. A face declares the
moodfeature (features: ["presence", "mood"]; the puck declares["presence"]), so the gateway sends it each reply’smoodevent.MOOD_EMOTION_FOLD(packages/voice-client/src/presentation/faceState.ts) folds the ten moods onto the eight emotions: steady and earnest toneutral, wry and sassy todoubtful, reflective tosad, hushed and dreamy tosleepy, menace toangry, playful and lively tohappy. The mood sits over the phase’s look and under a held expression; a held look releases to the mood.unknown(a tag the palette lacks) or any other word drops the layer. A mood is not an action: no receipt, nothing counted on the Admin row.
The full declaration:
"avatar_actions": ["set_expression", "play_animation", "set_state", "focus", "clear"],
"avatar_vocabulary": {"set_expression": ["neutral", "angry", "sad", "happy", "sleepy", "doubtful", "cold", "hot"],
"play_animation": ["clap", "thinking", "rock-paper-scissors"]},
"features": ["presence", "mood"]
What the Admin shows: the same card. The declared verbs include play_animation, and the vocabulary line reads
“set_expression: neutral, … hot · play_animation: clap, thinking, rock-paper-scissors” (from
avatar_routing.vocabulary.play_animation).
The rig report for a face. rig_report has the kind face, the decode words table (drawn from the face-state
table) or text (the fixture’s text face), and drawn_as when the art folds the state onto another drawing. A
client sends a face report only to a gateway that offers rig_face. The fold’s id is the manifest and the emotion is
the state. The Admin’s rig line reads “rig: stackchan-demo · face · sleepy drawn as sad · text”.
The face’s receipts and its mood
- The receipt. The face satellite sends
avatar_receiptfor eachavatar_actionthat carried anaction_id:drawnfor an applied action,refusedwithunknown_argument(a name outside its list) orundeclared. It sends it only to a gateway whose hello offersavatar_receipt. The Admin row’s receipt counts include what a face drew. - The mood on the rig report.
rig_reportandAdminAvatarRigcarry an optionalmood:{name, folds_to}, the gateway’s mood the face took and the emotion its fold turned it into (AvatarRigMood; both ≤ 64 chars). The gateway drops a report with a bad mood asbad_mood. The mood is absent when the face has no mood layer. A face with no fold names its manifesttable. - The bridge sends the rig report after bind, after each mood and after each action, once per change, only to a
gateway that offers
rig_reportandrig_face. The state is the emotion the face shows, so a held expression shows as the state whilemoodnames the layer under it.
What the Admin shows: the avatar card has a mood line under the rig line, “mood: wry → doubtful”; no line when the row carries no mood. The rig line reads “rig: table · face · doubtful · text” for the text face, and the receipt counts include the face’s receipts.
The forwarded action’s answer
The requester of vulpine-glyphctl avatar --session <id> is answered once per forwarded action with
avatar_forwarded {session_id, outcome, action_id?, receipt?, reason?}:
outcome: routed: that session’s router sent it to its frontend.receiptis the frontend’s owndrawnorrefused(withreason:undeclared,unknown_argument,no_rig), absent when no receipt came within the gateway’s wait (AVATAR_FORWARD_WAIT, 5 s).outcome: refused: that session’s router refused it;reasonis the router’s closed word.outcome: queued: the wait ran out before that session took it.outcome: gone: the session went away first.
The queue refusals stay error frames at once (avatar_session_unknown, avatar_session_busy,
avatar_session_ui). The wait runs off the requester’s loop, and the answer is sent at its next wake. The answer
carries words and ids only, never the argument. vulpine-glyphctl prints the line and exits 1 on a refusal. The named
session’s Admin row counts the action and its receipt.
The rig beside the session
The native 3D fox lives in Vulpine.app’s conversation window (apps/convo-gpui), a role ui observer of
Vulpine.app’s session. Vulpine.app declares no avatar verbs ([]), so the window supplies the rig.
-
The window declares the rig. With
--stage-directions on(the default), the window’s observe hello carries the fox’s declaration (the hello fixture’sconvo_gpui_foxsender):"avatar_actions": ["set_expression", "play_animation", "set_state", "focus", "clear"], "avatar_vocabulary": {"set_expression": ["neutral", "happy", "sad", "concerned", "surprised", "curious", "thinking", "sleepy", "alert"], "play_animation": ["attention", "notification", "error", "working", "thinking", "sleeping", "waking", "affection"]}presence_draws_toolsis not declared: the tool-receipt reactor’s picks reach the fox. -
The followed session adopts it. When the window follows a session (
observe_session), it offers its declaration on that session’s row. A session whose own frontend declared no verb adopts the offer: its router becomes the window’s declaration (bounded as at a bind, a fresh ledger), and its Admin row shows the window’s verbs and names. A session whose frontend declared any verb (Vigil, a face satellite) never adopts: its own rig wins. When the window follows another session or goes away, the session drops the rig and its own router is back ([]). Tracedavatar_rig_offered {offered}on the window’s connection andavatar_follower_rig {adopted, renewed, verbs, expressions, animations}on the session’s. -
Every producer reaches it. The reactor, the
motion_planquestion, the lexicon and the operator’svulpine-glyphctl avatarroute through the adopted router and are checked against the fox’s names. A routed action is offered to the session’s observers asobserved{avatar_action, action_id}and not sent to the session’s own frontend. A refusal is answered and counted (vulpine-glyphctl avatarreadsavatar_action_refused). Only an adopted rig’s actions reach observers;avatar_actionis not in the observe forwarded set. -
The Avatar look’s faces. While the window offers the Avatar look (Vulpine.app’s Look → Portrait → Avatar, a Labs setting, starts it with
--fox-look avatar), its hello declares the portrait director’s 22 faces and the fox’s ownalertasset_expression. The portrait draws each as itself. The 3D fox folds the thirteen it has no pose of onto the nearest of its nine: the warm faces onto happy, hurt onto sad, angry and annoyed onto concerned, skeptical onto curious, shocked and afraid onto surprised, bored onto sleepy. -
The window draws them (
apps/fox-gpui/src/directions.rs, on the 3D fox):set_expressionholds one of nine faces, each a pose of its own over the mood (never folded onto a mood),intensityscaling the face’s preset, forhold_ms(0 untilclear).play_animationplays the fox’s gesture for 3 s (replaceandinterruptdrop the queue,appendwaits, at most 4).set_stateholds the state’s pose until the presence moves on (interruptedis idle,reactingattention).focusholds a gaze (pointat x, y).clearlets go of everything.
Precedence: presence < state < gesture for the mood; the face and the gaze are their own channels. A
segment:<n>onset waits for the reply’s n-th TTS segment when the gateway sends segment starts, else applies at once (“Live segment onsets” below). The fox window’s Stage button (“Stage directions”) turns the directions off and on. A strip under the fox names what they show (face happy · gaze). -
Limits. The declaration is made at bind, so a closed fox window still declares it, and its actions draw nothing until the window opens. The 2D clips draw no directions.
What the Admin shows: Vulpine.app’s row, while the window follows it, lists the fox’s five verbs and its names
(avatar_routing.vocabulary) and counts the relayed actions routed and refused since the adoption, with the last
action’s verb, argument and source; when the window leaves, the row reads [] again with its own counts. The
window’s own row lists the same declaration (it is its hello’s) and counts nothing.
Live segment onsets, rates and the window’s receipts
A segment:<n> onset on the fox fires when that segment is heard, the fox draws rate and at_ms (above), and the
window’s receipts count on Vulpine.app’s row.
- The segment starts. The session numbers each reply segment it hands the TTS worker (the reply’s n-th, from 0),
whether or not a rig is adopted. When the segment’s first audio chunk ships to the session’s frontend, the session
offers its observers
tts_segment_started {turn_id, n, starts_in_ms}insideobserved.starts_in_msis how much shipped audio still plays before the segment, so the rig applies the onset when the segment is heard. Ids and numbers only, never text; tracedtts_segment_started {request_id, n, starts_in_ms}. A segment canceled or failed before its audio is dropped. - Who is sent it. The gateway’s hello offers the feature
tts_segment_started. Only an observer whose own hello declared it gets the frame; the conversation window declares it with its rig. The session’s own frontend gets none. Against a gateway that does not offer it, the fox applies asegment:<n>onset at once. - The fox (
apps/fox-gpui/src/directions.rs): a reply’sturn_startedrestarts the count and drops faces still waiting.tts_segment_startedschedules the segment’s startstarts_in_msafter its arrival, and the waiting faces take the stage when it plays. The strip under the fox names a waiting face first (next: happy at segment 2), and the Stage button’s tooltip says which rule is in force (“shows when its sentence is heard”, or “shows at once (the gateway sends no segment starts)”). - The window’s receipts. The window answers each relayed action that carried an
action_idwithavatar_receipt {action_id, verb, outcome, reason?}:drawn;refusedundeclared(a verb it does not draw),unknown_argument(a name or value it does not take) orno_rig(the fox window is closed, or Stage is off). It is the only control the window sends beyond the bind, the probe and the observe request, and it acts on nothing. A receipt for an action the followed session relayed to this window credits that session’s row, with the same checks as the session’s own receipts, tracedavatar_receipt {relayed: true}oravatar_receipt_ignored {reason, relayed: true}. Avulpine-glyphctl avatar --sessionrequester is not answered from a relayed receipt.
The motion plan on the fox. After each finished reply the decision model is asked one motion_plan question over
the fox’s adopted vocabulary (five operations plus none, nine faces, eight gestures, the six states, four gaze
targets). A pick whose margin clears motion_plan_min_margin is relayed to the fox as an immediate action. Below it,
the lexicon falls back to its first declared look (for the fox, a gesture: error, notification, attention, affection
or thinking). The Admin’s plan row shows each verdict.
The switch. The config is the station’s ceiling, and the frontend has a switch under it. The client frame
motion_plan_switch {on} (behind the hello feature motion_plan_switch) turns the plan off for that session and on
again. With motion_plan = false it stays off whatever the switch says. A frame from a client that did not declare
the feature is traced motion_plan_switch_ignored {reason: not_declared} and dropped. An accepted one is traced
motion_plan_switch {on, state}. It is never answered. Vulpine.app’s switch is Operations > Look > Fox follows
replies (shown while Labs is on; on by default), sent on each change and after every bind. The Admin row’s
avatar_routing.motion_plan_state (AdminMotionPlanState: on, off_by_switch, off_by_config, or unknown)
is set at bind and on each switch, and the avatar card draws it (“motion plan: off (the Glyph’s “Fox follows
replies” switch)“).
What the Admin shows: Vulpine.app’s row’s receipt counts (receipts: drawn 3 · refused 1 · last refused set_expression (unknown_argument)) include what the fox drew or refused while the window follows it.
The words follow the voice
A reply drawn as the brain streams it runs ahead of the voice. The rule: words are painted when they are heard; a picture is painted when its tool answers.
- The gateway offers each spoken segment’s start and length to every observer whose hello declared
tts_segment_started, whether or not a rig is adopted. - The window’s reducer (
crates/convo_feed/src/feed/pace.rs) shows a reply up to a segment’stext_endonce the segment is heard (its start’s arrival plusstarts_in_ms). It paces only after it has seen a start: a reply with no start 2,000 ms after its first text is shown whole. What the voice skips is shown 1,500 ms after the last audio ends (10,000 ms after the last start when a length never came). A barge-in shows the rest at once. - A panel paints at its render.
Word timings for the mouth
The 3D fox’s mouth (its visemes) lays the reply’s words out from the speech’s start at a syllable estimate, which on
its own drifts from the voice over a long reply. Two optional additions let it re-anchor on every segment. Both are
numbers only (the observer already has the words from assistant_delta). Both go only where tts_segment_started
goes: inside observed, to an observer whose hello declared that feature, whether or not a rig is adopted. A window
that does not know the frame ignores it.
- The segment’s place.
tts_segment_startedcarriestext_startandtext_end(optional): the segment’s[start, end)in the reply’s streamed text, the turn’sassistant_deltas joined after the mood tag is stripped, counted in chars (Unicode scalar values). The session keeps up to 64 KiB of a reply’s text and finds each segment’s text in it after the last segment’s end. A segment not found verbatim (a citation token stripped from it, a readout’s line) is sent without them. Traced astext_charson thetts_segment_startedrow. - The segment’s length.
tts_segment_shipped {turn_id, n, audio_ms}(an appended event) is sent when the worker settles the segment’s request as completed, once every byte of its audio has shipped.audio_msis the PCM shipped for it (all its chunks, rounded to the ms). It comes after the start, because the audio streams. A segment canceled, failed or superseded is sent no length. Tracedtts_segment_shipped {request_id, n, audio_ms}. Segment 0’s length usually comes while it plays; a later segment’s usually comes before it is heard. - The fox (
apps/fox-gpui/src/lipsync.rs): a segment with a place re-lays its words from when it is heard (starts_in_msafter its frame). Once its length comes, the estimate’s timeline for those words is stretched or squeezed so the last one ends with the audio (each word keeps its share, syllable-weighted, pauses included). A word of an earlier segment still running when a segment starts is cut there. Without the fields (an older gateway, a window that joined mid-reply, a segment not found) the estimate is the whole of it.
The Admin shows nothing for the timings.