Dictation
Hold-to-talk dictation on the ws: the dictation_hold control, the dictation_partial and dictation_text frames and who types them, the client's insert proof (dictation_inserted) and the erase_last_insert the spoken 'scratch that' resolves to, with its spoken refusals.
Living doc, checked against the code 2026-11-14
dictation.v1 — hold-to-talk, the typed words, and “scratch that”
- Status: live on protocol 5, additive. Every frame here is an optional variant or field. An older client ignores the server frames it does not know, and an older gateway refuses the controls as unknown.
- Relates to the spoken verb
type this(turn/phrases.rs), which types through the gateway’s screen route. This contract adds a second way in (the held key) and one way back (the client’s proof).
Why
Hold a key, speak, and the raw transcript lands in the focused field through Vulpine.app, with no tool calls. The client marks the capture as dictation, then proves where it landed, so “scratch that” erases exactly that.
The hold
Client → gateway: dictation_hold
| field | type | rule |
|---|---|---|
type |
"dictation_hold" |
|
state |
pressed | released |
one of each per hold; a second pressed under a hold is ignored (traced already_held) |
literal |
bool, default true |
true types the words as spoken; false runs the word-preserving cleanup (turn/dictation_cleanup.rs) first. Read at pressed. |
focus |
{ pid?: number, bundle_id?: string }, optional |
the frontmost app at key-down (t2t’s focus identity). Traced (focus_bundle_id, focus_pid); the gateway never types on its account. At the settle it is checked against the screen agent (below). |
Every session admits the hold, a session bound to no loadout included. It is refused as dictation_hold_disabled
only when the bound loadout says [dictation] hold_to_talk = false (a closed section; a misspelt key refuses the
bind), and session_required before start_session.
While held
- Microphone PCM goes to the recognizer directly: no armed-STT gate, no enrolment filter, no wake fan-out, no shadow lanes, no pipeline frame. The partials gate opens for the hold and is put back at the settle.
- The recognizer’s frames bypass the turn machine: no alarm or card answer, no speculation, no decision shadow, no
verb resolution, no commit. VAD edges and late wake detections are dropped (traced
dictation_hold_dropped_frame). - A reply not yet sounding waits (the presence hold, call id
dictation_hold), with no presence-bound expiry while the key is down. - A reply line already sounding at the press pauses and resumes at the same point at the release (below). For a
client that did not declare
playback_pause, it plays on. - The running transcript goes back on every partial:
| field | type | rule |
|---|---|---|
type |
"dictation_partial" |
|
request_id |
string | the hold’s key, the same on its dictation_text |
text |
string | the WHOLE transcript so far (settled segments, then the open one), never a delta |
session_id |
string, optional | as user_transcript carries it |
Pausing a sounding line
The gateway offers the feature playback_pause in the hello and pauses only for a client that declared it too
(Vulpine.app). At the press, while shipped audio is still audible, the gateway:
- sends
playback_pause { state: "paused", reason: "dictation_hold" }. The client suspends its output clock: what is scheduled keeps its place, no lease finishes, stall checks do not call the clock dead, and audio still on the wire queues; - keeps the rest of that line back, and parks the playout horizon ahead, so nothing reads the line as finished (no echo-reference stale row, no read segment due, no phase rest);
- traces
reply_line_paused { request_id, tts_request_id, remaining_ms }anddictation_hold { paused_line: true }.
At the settle (any reason), before the held audio ships, the gateway:
- sends
playback_pause { state: "resumed", reason, paused_ms }; - puts the horizon back to now plus what was still to sound at the press (plus anything shipped meanwhile; a cut under the hold stays a cut);
- shifts the heard timeline by the paused span, so a later cut reads only what the room heard;
- traces
reply_line_resumed { paused_ms, remaining_ms, restored, reason }.
The client resumes from the same point and sends audio_playback_finished when the line has drained. A route that
moved to a room under the hold still sends the paused line’s audio to the client. end_session under the hold
resumes first; a closed connection only restores the accounting. The client resumes by itself 160 s after a pause
nobody resumed, and a rebuilt output is never paused.
| field | type | rule |
|---|---|---|
type |
"playback_pause" |
server → client |
state |
paused | resumed |
closed |
reason |
string | dictation_hold |
paused_ms |
number, optional | on resumed: how long the line was paused |
Admin: the hold’s row in the Dictation window says paused a reply line N s (AdminDictationHold.paused_line_ms).
At release
released flushes the recognizer. With no segment open after the last final, the words settle at once; otherwise
the gateway waits up to 1.5 s for that final, then settles on what it heard. The gateway releases a stuck key at
150 s (Vulpine.app ends a lost key-up at 120 s). A hold open at end_session or connection close is dropped, never
typed (dictation_hold_dropped).
The settled words are one GatewayReport::Dictation { origin: hold }, the spoken verb’s report and handler. It
reaches the client as:
| field | type | rule |
|---|---|---|
type |
"dictation_text" |
|
request_id |
string | the hold’s key |
text |
string | the words, after the cleanup when literal = false |
literal |
bool | |
source |
hold | verb |
who types: hold = the client, at its own caret; verb = the gateway’s screen route already typed them, the client must NOT insert |
The focus check. When the hold carried a focus with a bundle_id and the gateway has a screen route, the
settle asks the screen agent for the focused app once (dictation_focus_check { verdict, pressed_bundle_id, now_bundle_id }). A different app refuses the insert (target_changed_app): no dictation_text, nothing typed,
traced dictation_refused { reason: target_changed_app }, an Admin dictation row, and the spoken line “I didn’t type
that: the app changed while you held the key.” A focus that cannot be read (no route, no grant, no bundle on either
side) delivers the words unchecked. Bundle ids are compared first, then pids when either side has no bundle id (only
the socket read carries the agent’s pid).
The settle reads the focus over the screen agent’s socket when it is up ([screen] agent_socket, local reader): ONE
line, {"focus": true, "max_chars": 1, "insert_check": true}, where insert_check skips the selection walk. Without
a socket answer it spawns the tool’s focus; a remote reader goes over ssh. A read is reused for 2 s, only by a hold
whose key went down before it. The row carries path: socket | spawn | cached, ms, family and now_pid. The
socket path takes under 50 ms; the spawned tool 326–620 ms.
The check at the moment of typing. The client passes the hold’s key-down focus to the screen agent’s write
(socket: expect_bundle, expect_pid; CLI: --expect-bundle <id>, --expect-pid <n>; the nested
expect {bundle_id, pid} means the same, and a flat field wins). Before the first keystroke and before the pasteboard
is touched, the agent re-reads the app that owns the focused element. Every given field must match; with only a pid,
a relaunched app counts as changed. A mismatch refuses with reason: target_changed_app, expected {bundle_id?, pid?} and actual {bundle_id?, pid?}: nothing is typed and the clipboard is untouched. A write that named an app
carries focus_checked_at_write: true, refused or not. An older agent ignores the flat fields. Code:
runtime/screen/insert_policy.swift (checkFocusAtWrite, mergedExpectation) and screenctl.swift::focusWrite.
Nothing is spoken for a held dictation: the operator is at the keyboard. Empty words end the hold with
dictation_hold_empty and no report. Trace rows carry counts, reasons and ids, never the words: dictation_hold,
dictation_partial, dictation_final,
dictation_hold_settled { reason, gate_restored, partials_gate_after, armed_window_open },
dictation_report { source, literal, words }, dictation_delivered.
Admin. GetEvents carries an optional dictation window (AdminDictationWindow), counted since the gateway
started:
- holds by mode (
literal | stream | cleanup), partials and the laststable_chars; - deliveries, and the client’s insert receipts (accepted, text-provable, by binding);
- “scratch that” erases asked and done, refusals by reason word, and review choices;
- the latest 20 holds (request id, key-down bundle id, partials, outcome, proof binding, scratch outcome).
A hold also carries partial_chars (the latest partial’s length in Unicode scalar values) and stream_state
(open | frozen | recovered | off, then reconciled at the release); the window carries last_partial_chars. It
holds ids, bundle ids, counts and reason words only, and the validator refuses a label with a space
(crates/voice_gateway/src/dictation_stats.rs).
Before either path, a lone trailing “You” / “you.” is dropped when the text before it already ends a sentence
(a pause-ending decode reads silence that way). Inside a sentence (“Thank you.”) it stays
(turn/dictation_cleanup.rs::strip_tail_artefact).
Styles for literal = false. The loadout’s [dictation.styles] maps a bundle-id prefix to
coding | email | chat | plain. The longest configured prefix wins, then a built-in table (terminals and editors
coding, Mail / Outlook email, Slack / Messages / Discord chat), then plain. dictation_styles::style_for
picks the style; prompt_rules() gives the tidy prompt’s style line (removal and formatting only). A
dictation_tidy_request resolves its style in this order:
- the request’s own
styleword; [dictation.styles]for the app the reviewed hold’s key went down in (the hold’sfocus.bundle_id, or the settle’s read when the hold carried none);- the built-in table;
plain.
The model call’s system prompt ends with that style’s prompt_rules() line. The trace row dictation_tidy_request
carries style (code | email | chat | clean) and style_source (request | config | built_in | plain). Vulpine.app
may send the style it resolved itself; a request without one gets the gateway’s.
The caret join (turn/dictation_stream.rs::caret_repair, beside caret_join). The settle’s insert_check read
returns family and up to 64 characters each side of the caret (caret_before, caret_after; never from a secure
field, never traced). Before dictation_text goes out:
- a word right after the caret gets a trailing space;
- a sentence that goes on after the caret drops the dictation’s closing full stop;
- the words join the character before the caret with a space when one is needed;
- in a terminal (
family = terminal) the first insert on a line gets no leading space and no capital (“first” = no earlier hold landed in the same app inside the scratch window).
A read without caret windows outside a terminal is not repaired. The trace row dictation_caret_repair carries the
decisions and the windows’ lengths, and path: release, stream or verb:
stream. A hold that types ahead reads the field once, over the socket, at its first append (the firstdictation_partialwithstable_charsabove 0); an unread field streams unrepaired. From then on, a needed leading space is part oftextand counted instable_chars. At the release the whole repair runs over the settled words,stream.typed_charscounts the leading space, and anappendwhose typed prefix the repair cut becomes areplace.verb. The spokentype thisroute reads the field once (socket only) before its screen write and applies the same repair (capitaliseunless the verb saidliterally), so “Please [send the report.] to Sam” types “send the report “.
The stream (behind a flag)
Off unless the bound loadout also sets [dictation] stream = true. Only a literal hold streams, because the
cleanup pass may rewrite earlier words. A cleanup hold is traced dictation_stream {state: off, reason: not_literal}.
The stabilizer (turn/dictation_stream.rs) keeps the prefix two running transcripts agree on, cut to a word
boundary.
Two optional fields, both absent when the hold does not stream:
| frame | field | rule |
|---|---|---|
dictation_partial |
stable_chars |
the client may type text’s first stable_chars Unicode scalar values (code points, not UTF-16 units) ahead of the release. It never shrinks and is never taken back mid-stream; type only the part past what was already typed. |
dictation_text |
stream: { typed_chars, kind } |
the release’s batch reconcile. typed_chars is the last stable_chars sent. append: text starts with what was typed — type text after the first typed_chars. replace: it does not — replace the typed_chars typed characters with the whole text in one write. Absent when nothing was typed ahead (stable_chars stayed 0): type text as without a stream. |
A partial that contradicts the typed prefix freezes the stream: stable_chars holds until two transcripts agree
again on a word boundary past it. A freeze that never recovers ends in a replace. Trace rows carry counts only:
dictation_stream {state: open | frozen | recovered | off},
dictation_stream_reconcile {kind, typed_chars, final_chars, freezes, recoveries}. A client that ignores
stable_chars and stream types text at release.
Review mode (client opt-in)
Off unless Vulpine.app sets global_shortcuts.hold_to_talk_literal = false. A review-mode hold is still sent
literal: true, because the review needs the raw words. Its dictation_text is NOT typed at release: the client
shows the raw words, asks for a tidy, and types only on the operator’s choice. An older gateway refuses these
controls, and the client then offers “plain only”.
Client → gateway: dictation_tidy_request
| field | type | rule |
|---|---|---|
type |
"dictation_tidy_request" |
|
request_id |
string | the dictation_text being reviewed |
text |
string | its raw words |
style |
string, optional | clean (default), email, chat, code; anything else is clean |
Refused like the hold (session_required, dictation_hold_disabled). It opens the review: the turn machine’s
review_armed gate goes up (traced review_armed), and a second request replaces the first.
Gateway → client: dictation_tidy
| field | type | rule |
|---|---|---|
type |
"dictation_tidy" |
one per request |
request_id |
string | |
text |
string | the tidy when guard.passed, else the raw words |
guard.kept_word_ratio |
number 0..1 | Sotto’s multiset share of raw words kept |
guard.deletions_only |
bool | Mindtalk’s rule: every tidy word is a raw word, in raw order |
guard.passed |
bool | false = the client offers the raw words only (“plain only”) |
guard.reason, guard.source |
string, optional | why it refused; rules or model |
style |
string, optional | the style word the tidy applied (clean, email, chat, code), after the request → [dictation.styles] → built-in order; the same word the dictation_tidy_request trace row carries |
Who tidies (ws/dictation_tidy.rs, guards from tools/dictation/tidy_guards.py): a dictation with no cue phrase
(“no wait”, “scratch that”, “I mean” …), no stutter and no repeated opening never reaches a model; it gets the
word-preserving pass (turn/dictation_cleanup.rs, source: rules). One that trips the gate goes to the brain lane
with NO tools (brain/tidy.rs: no session, its own system prompt) as data inside nonce delimiters, bounded at 6 s
and off the session loop. The answer is unwrapped and judged: an assistant phrase the raw words lack, collapse
(< 0.3 of the words), growth (> 1.2×), a kept-word ratio < 0.7 at 40+ words, and the polish guard (new_word,
glued, reordered, too_little_kept). A refusal, timeout or model error is passed: false. Trace rows:
dictation_tidy_request {words, chars, style, gate, gate_reasons},
dictation_tidy {source, passed, reason, kept_word_ratio, deletions_only, words_in, words_out, ms}, never the words.
The choice: dictation_review (both directions)
| field | type | rule |
|---|---|---|
type |
"dictation_review" |
|
request_id |
string | |
choice |
send | plain | scratch | timeout |
send types the tidy (the raw words when plain only), plain the raw words, scratch nothing; timeout (client → gateway only) = 10 s passed, nothing typed |
Gateway → client: while review_armed, these closed phrases (whole utterance after the wake, “please” dropped)
become GatewayReport::DictationReview (ahead of the scratch verb, behind the home undo): send it / send that /
send it now → send, say it plain / say that plain / keep it plain → plain, scratch that /
scratch this / scratch it / delete that / delete this → scratch. The choice is sent, nothing is spoken, no
brain turn runs, and the review closes. Client → gateway: the client closes the review (a key, Return, Shift+Return
or Esc, or its timeout), traced dictation_review {choice, via, matched, open_ms}. A review nobody closes disarms at
15 s. Outside a review the phrases keep their other meanings. A choice that types goes through
dictation_inserted, so “scratch that” can erase it.
Client-side gestures (no wire change)
On the Globe key (globectl), two presses within 400 ms open a hands-free hold: the client keeps dictation_hold
open with the key up and sends released on the next tap, on Esc, or after 2.5 s with no new words (6 s before the
first). Esc during any hold sends released and drops the dictation_text that follows. The first tap of a
double-tap is an empty hold (dictation_hold_empty).
The insert proof and “scratch that”
Client → gateway: dictation_inserted
The screen agent’s write receipt (its proof field), relayed whole. The gateway keeps the latest per request_id.
Whether the words are still untouched is the agent’s call at erase time, never a stored flag.
| field | type | rule |
|---|---|---|
type |
"dictation_inserted" |
|
request_id |
string | the dictation_text it answers; any other id is counted and dropped |
proof.receipt_id |
string | the agent’s receipt id, opaque to the gateway; handed back on erase_last_insert |
proof.location, proof.length |
number | the inserted range in the field’s own units (UTF-16 offsets) |
proof.chars |
number | what landed |
proof.text_provable |
bool | the agent can read the range back and compare it |
proof.binding |
range | epoch |
how the agent proves the insert untouched: the range read back, or its input epoch alone |
The refused insert
The client pins the key-down focus on every screen-agent write (expect_pid, expect_bundle, plus the nested
expect {pid, bundle_id}), so the agent refuses at the moment of typing (target_changed_app). The client relays
the refusal on the same control, with no proof:
| field | type | rule |
|---|---|---|
type |
"dictation_inserted" |
|
request_id |
string | the dictation_text it answers |
refused.reason |
string | the agent’s reason word (target_changed_app, secure_field, …; insert_failed when unreadable) |
refused.expected, refused.actual |
string, optional | the key-down and the frontmost bundle ids |
The gateway half. proof and refused are both optional on ClientControl::DictationInserted. The client
sends refused only to a gateway that offers dictation_insert_refused in hello.features; otherwise it shows
“APP CHANGED — NOT TYPED” with no Scratch that and logs it locally. On a refused, the gateway:
- traces
dictation_insert_refused {request_id, reason, expected, actual, had_proof, accepted}(reason words and bundle ids only, each cut to 128 characters; an empty reason isinsert_failed); - keeps no proof, so “scratch that” answers “There’s nothing to scratch.” A later
prooffor the samerequest_idstill lands: the latest receipt wins; - counts it in the Admin dictation window under
refusalsasinsert_<reason>(a write-timeinsert_target_changed_appapart from the settle’starget_changed_app), and sets the hold’soutcometo that key. A refusal for anotherrequest_idis traced withaccepted: falseand not counted.
A frame with both fields counts as refused. A frame with neither is traced dictation_inserted_empty and dropped.
Gateway → client: erase_last_insert
| field | type | rule |
|---|---|---|
type |
"erase_last_insert" |
|
request_id |
string | the dictation to erase |
receipt_id |
string | the agent’s receipt from the last proof |
allow_backspace |
bool | false until the operator rules on the backspace fallback |
Sent only when a spoken “scratch that” resolves to an erase (below). Every other case is refused with a spoken reason
and nothing is sent. The client asks the agent ({"erase_last": true, receipt_id, allow_backspace}) and answers:
Client → gateway: dictation_erased
| field | type | rule |
|---|---|---|
type |
"dictation_erased" |
|
request_id |
string | |
outcome |
erased | refused | unverified | interrupted | failed |
the agent’s outcome word |
reason, method, chars, binding_intact, binding_reason |
optional | the agent’s own detail; traced, never spoken |
erased is silent (a scratched chip) and invalidates the receipt, so a second “scratch that” finds nothing.
refused means the field was edited or the focus moved; it is spoken as “That text has changed; I left it.” and
closes the receipt. unverified, interrupted and failed are spoken as their own lines and leave the receipt open.
A client-started erase (no new frame): Vulpine.app’s Scratch that button and
vulpine-glyphctl dictation erase-last ask the agent directly (same request, same binding check, same 120 s window)
and send dictation_erased with no erase_last_insert before it. The gateway matches it by request_id against the
last proof (traced dictation_erased {accepted}). Vulpine.app relays only erased, so a button never makes Vulpine speak.
A refusal stays on the client’s chip and the receipt stays open.
The verb
Closed phrase set, whole utterance after the wake: scratch that, scratch this, delete that, delete this.
undo that counts only while a dictation receipt is live and the house has nothing to put back. “Scratch that itch,
please” is dictation or a request, never the verb.
Resolution, in the ws loop (crate::dictation_insert::resolve_scratch, then the agent’s answer):
| state | outcome | dictation_scratch.outcome |
spoken |
|---|---|---|---|
| a proof returned, within the window, not yet scratched | erase_last_insert { request_id, receipt_id, allow_backspace: false }; the agent checks the binding |
erase_requested, then the answer’s row below |
nothing until the agent answers; erased = a chip |
the agent answers erased |
the receipt closes | erased |
nothing (a scratched chip) |
the agent answers refused (binding not intact: caret_moved, range_edited, focus_moved, …) |
refusal edited_since; the receipt closes |
changed |
“That text has changed; I left it.” |
the agent answers unverified |
refusal unverified |
unverified |
“I couldn’t check the text, so I left it.” |
the agent answers interrupted / failed |
refusal failed |
failed |
“I couldn’t scratch that.” |
| a dictation delivered but no proof yet, the window passed, already scratched, or nothing delivered | refusal nothing_to_scratch |
nothing |
“There’s nothing to scratch.” |
The trace row dictation_scratch {phrase, outcome, request_id?, receipt_id?, refusal?, binding_reason?, settled?}
counts spoken asks only: one row when the verb is heard, one more with settled: true when the agent answers. A
client-started erase writes dictation_erased and no dictation_scratch row. The
gateway’s record of the last insert is the proof by reference: {request_id, receipt_id, location, length, chars, text_provable, binding}. The pid, the AX element, the window and the inserted string stay inside the screen agent
(InsertReceipt). Its range erase, eraseRange in runtime/screen/insert_policy.swift, selects the range, proves it
reads code-unit equal to the record, writes nothing over it and reads back.
The verb is armed (scratch_armed) from a delivery until the 120 s window ends, through an erase or a refused
answer. Outside the window, and on a gateway with no dictation route, “scratch that” is the armed-request cancel
phrase.