Skip to content
Vulpine
Esc
↑↓navigate↵open⌘Jpreview
On this page

Dictation

Hold-to-talk dictation on the ws: the dictation_hold control, the dictation_partial and dictation_text frames and who types them, the client's insert proof (dictation_inserted) and the erase_last_insert the spoken 'scratch that' resolves to, with its spoken refusals.

Living doc, checked against the code 2026-11-14

dictation.v1 — hold-to-talk, the typed words, and “scratch that”

  • Status: live on protocol 5, additive. Every frame here is an optional variant or field. An older client ignores the server frames it does not know, and an older gateway refuses the controls as unknown.
  • Relates to the spoken verb type this (turn/phrases.rs), which types through the gateway’s screen route. This contract adds a second way in (the held key) and one way back (the client’s proof).

Why

Hold a key, speak, and the raw transcript lands in the focused field through Vulpine.app, with no tool calls. The client marks the capture as dictation, then proves where it landed, so “scratch that” erases exactly that.

The hold

Client → gateway: dictation_hold

field type rule
type "dictation_hold"
state pressed | released one of each per hold; a second pressed under a hold is ignored (traced already_held)
literal bool, default true true types the words as spoken; false runs the word-preserving cleanup (turn/dictation_cleanup.rs) first. Read at pressed.
focus { pid?: number, bundle_id?: string }, optional the frontmost app at key-down (t2t’s focus identity). Traced (focus_bundle_id, focus_pid); the gateway never types on its account. At the settle it is checked against the screen agent (below).

Every session admits the hold, a session bound to no loadout included. It is refused as dictation_hold_disabled only when the bound loadout says [dictation] hold_to_talk = false (a closed section; a misspelt key refuses the bind), and session_required before start_session.

While held

  • Microphone PCM goes to the recognizer directly: no armed-STT gate, no enrolment filter, no wake fan-out, no shadow lanes, no pipeline frame. The partials gate opens for the hold and is put back at the settle.
  • The recognizer’s frames bypass the turn machine: no alarm or card answer, no speculation, no decision shadow, no verb resolution, no commit. VAD edges and late wake detections are dropped (traced dictation_hold_dropped_frame).
  • A reply not yet sounding waits (the presence hold, call id dictation_hold), with no presence-bound expiry while the key is down.
  • A reply line already sounding at the press pauses and resumes at the same point at the release (below). For a client that did not declare playback_pause, it plays on.
  • The running transcript goes back on every partial:
field type rule
type "dictation_partial"
request_id string the hold’s key, the same on its dictation_text
text string the WHOLE transcript so far (settled segments, then the open one), never a delta
session_id string, optional as user_transcript carries it

Pausing a sounding line

The gateway offers the feature playback_pause in the hello and pauses only for a client that declared it too (Vulpine.app). At the press, while shipped audio is still audible, the gateway:

  • sends playback_pause { state: "paused", reason: "dictation_hold" }. The client suspends its output clock: what is scheduled keeps its place, no lease finishes, stall checks do not call the clock dead, and audio still on the wire queues;
  • keeps the rest of that line back, and parks the playout horizon ahead, so nothing reads the line as finished (no echo-reference stale row, no read segment due, no phase rest);
  • traces reply_line_paused { request_id, tts_request_id, remaining_ms } and dictation_hold { paused_line: true }.

At the settle (any reason), before the held audio ships, the gateway:

  • sends playback_pause { state: "resumed", reason, paused_ms };
  • puts the horizon back to now plus what was still to sound at the press (plus anything shipped meanwhile; a cut under the hold stays a cut);
  • shifts the heard timeline by the paused span, so a later cut reads only what the room heard;
  • traces reply_line_resumed { paused_ms, remaining_ms, restored, reason }.

The client resumes from the same point and sends audio_playback_finished when the line has drained. A route that moved to a room under the hold still sends the paused line’s audio to the client. end_session under the hold resumes first; a closed connection only restores the accounting. The client resumes by itself 160 s after a pause nobody resumed, and a rebuilt output is never paused.

field type rule
type "playback_pause" server → client
state paused | resumed closed
reason string dictation_hold
paused_ms number, optional on resumed: how long the line was paused

Admin: the hold’s row in the Dictation window says paused a reply line N s (AdminDictationHold.paused_line_ms).

At release

released flushes the recognizer. With no segment open after the last final, the words settle at once; otherwise the gateway waits up to 1.5 s for that final, then settles on what it heard. The gateway releases a stuck key at 150 s (Vulpine.app ends a lost key-up at 120 s). A hold open at end_session or connection close is dropped, never typed (dictation_hold_dropped).

The settled words are one GatewayReport::Dictation { origin: hold }, the spoken verb’s report and handler. It reaches the client as:

field type rule
type "dictation_text"
request_id string the hold’s key
text string the words, after the cleanup when literal = false
literal bool
source hold | verb who types: hold = the client, at its own caret; verb = the gateway’s screen route already typed them, the client must NOT insert

The focus check. When the hold carried a focus with a bundle_id and the gateway has a screen route, the settle asks the screen agent for the focused app once (dictation_focus_check { verdict, pressed_bundle_id, now_bundle_id }). A different app refuses the insert (target_changed_app): no dictation_text, nothing typed, traced dictation_refused { reason: target_changed_app }, an Admin dictation row, and the spoken line “I didn’t type that: the app changed while you held the key.” A focus that cannot be read (no route, no grant, no bundle on either side) delivers the words unchecked. Bundle ids are compared first, then pids when either side has no bundle id (only the socket read carries the agent’s pid).

The settle reads the focus over the screen agent’s socket when it is up ([screen] agent_socket, local reader): ONE line, {"focus": true, "max_chars": 1, "insert_check": true}, where insert_check skips the selection walk. Without a socket answer it spawns the tool’s focus; a remote reader goes over ssh. A read is reused for 2 s, only by a hold whose key went down before it. The row carries path: socket | spawn | cached, ms, family and now_pid. The socket path takes under 50 ms; the spawned tool 326–620 ms.

The check at the moment of typing. The client passes the hold’s key-down focus to the screen agent’s write (socket: expect_bundle, expect_pid; CLI: --expect-bundle <id>, --expect-pid <n>; the nested expect {bundle_id, pid} means the same, and a flat field wins). Before the first keystroke and before the pasteboard is touched, the agent re-reads the app that owns the focused element. Every given field must match; with only a pid, a relaunched app counts as changed. A mismatch refuses with reason: target_changed_app, expected {bundle_id?, pid?} and actual {bundle_id?, pid?}: nothing is typed and the clipboard is untouched. A write that named an app carries focus_checked_at_write: true, refused or not. An older agent ignores the flat fields. Code: runtime/screen/insert_policy.swift (checkFocusAtWrite, mergedExpectation) and screenctl.swift::focusWrite.

Nothing is spoken for a held dictation: the operator is at the keyboard. Empty words end the hold with dictation_hold_empty and no report. Trace rows carry counts, reasons and ids, never the words: dictation_hold, dictation_partial, dictation_final, dictation_hold_settled { reason, gate_restored, partials_gate_after, armed_window_open }, dictation_report { source, literal, words }, dictation_delivered.

Admin. GetEvents carries an optional dictation window (AdminDictationWindow), counted since the gateway started:

  • holds by mode (literal | stream | cleanup), partials and the last stable_chars;
  • deliveries, and the client’s insert receipts (accepted, text-provable, by binding);
  • “scratch that” erases asked and done, refusals by reason word, and review choices;
  • the latest 20 holds (request id, key-down bundle id, partials, outcome, proof binding, scratch outcome).

A hold also carries partial_chars (the latest partial’s length in Unicode scalar values) and stream_state (open | frozen | recovered | off, then reconciled at the release); the window carries last_partial_chars. It holds ids, bundle ids, counts and reason words only, and the validator refuses a label with a space (crates/voice_gateway/src/dictation_stats.rs).

Before either path, a lone trailing “You” / “you.” is dropped when the text before it already ends a sentence (a pause-ending decode reads silence that way). Inside a sentence (“Thank you.”) it stays (turn/dictation_cleanup.rs::strip_tail_artefact).

Styles for literal = false. The loadout’s [dictation.styles] maps a bundle-id prefix to coding | email | chat | plain. The longest configured prefix wins, then a built-in table (terminals and editors coding, Mail / Outlook email, Slack / Messages / Discord chat), then plain. dictation_styles::style_for picks the style; prompt_rules() gives the tidy prompt’s style line (removal and formatting only). A dictation_tidy_request resolves its style in this order:

  1. the request’s own style word;
  2. [dictation.styles] for the app the reviewed hold’s key went down in (the hold’s focus.bundle_id, or the settle’s read when the hold carried none);
  3. the built-in table;
  4. plain.

The model call’s system prompt ends with that style’s prompt_rules() line. The trace row dictation_tidy_request carries style (code | email | chat | clean) and style_source (request | config | built_in | plain). Vulpine.app may send the style it resolved itself; a request without one gets the gateway’s.

The caret join (turn/dictation_stream.rs::caret_repair, beside caret_join). The settle’s insert_check read returns family and up to 64 characters each side of the caret (caret_before, caret_after; never from a secure field, never traced). Before dictation_text goes out:

  • a word right after the caret gets a trailing space;
  • a sentence that goes on after the caret drops the dictation’s closing full stop;
  • the words join the character before the caret with a space when one is needed;
  • in a terminal (family = terminal) the first insert on a line gets no leading space and no capital (“first” = no earlier hold landed in the same app inside the scratch window).

A read without caret windows outside a terminal is not repaired. The trace row dictation_caret_repair carries the decisions and the windows’ lengths, and path: release, stream or verb:

  • stream. A hold that types ahead reads the field once, over the socket, at its first append (the first dictation_partial with stable_chars above 0); an unread field streams unrepaired. From then on, a needed leading space is part of text and counted in stable_chars. At the release the whole repair runs over the settled words, stream.typed_chars counts the leading space, and an append whose typed prefix the repair cut becomes a replace.
  • verb. The spoken type this route reads the field once (socket only) before its screen write and applies the same repair (capitalise unless the verb said literally), so “Please [send the report.] to Sam” types “send the report “.

The stream (behind a flag)

Off unless the bound loadout also sets [dictation] stream = true. Only a literal hold streams, because the cleanup pass may rewrite earlier words. A cleanup hold is traced dictation_stream {state: off, reason: not_literal}. The stabilizer (turn/dictation_stream.rs) keeps the prefix two running transcripts agree on, cut to a word boundary.

Two optional fields, both absent when the hold does not stream:

frame field rule
dictation_partial stable_chars the client may type text’s first stable_chars Unicode scalar values (code points, not UTF-16 units) ahead of the release. It never shrinks and is never taken back mid-stream; type only the part past what was already typed.
dictation_text stream: { typed_chars, kind } the release’s batch reconcile. typed_chars is the last stable_chars sent. append: text starts with what was typed — type text after the first typed_chars. replace: it does not — replace the typed_chars typed characters with the whole text in one write. Absent when nothing was typed ahead (stable_chars stayed 0): type text as without a stream.

A partial that contradicts the typed prefix freezes the stream: stable_chars holds until two transcripts agree again on a word boundary past it. A freeze that never recovers ends in a replace. Trace rows carry counts only: dictation_stream {state: open | frozen | recovered | off}, dictation_stream_reconcile {kind, typed_chars, final_chars, freezes, recoveries}. A client that ignores stable_chars and stream types text at release.

Review mode (client opt-in)

Off unless Vulpine.app sets global_shortcuts.hold_to_talk_literal = false. A review-mode hold is still sent literal: true, because the review needs the raw words. Its dictation_text is NOT typed at release: the client shows the raw words, asks for a tidy, and types only on the operator’s choice. An older gateway refuses these controls, and the client then offers “plain only”.

Client → gateway: dictation_tidy_request

field type rule
type "dictation_tidy_request"
request_id string the dictation_text being reviewed
text string its raw words
style string, optional clean (default), email, chat, code; anything else is clean

Refused like the hold (session_required, dictation_hold_disabled). It opens the review: the turn machine’s review_armed gate goes up (traced review_armed), and a second request replaces the first.

Gateway → client: dictation_tidy

field type rule
type "dictation_tidy" one per request
request_id string
text string the tidy when guard.passed, else the raw words
guard.kept_word_ratio number 0..1 Sotto’s multiset share of raw words kept
guard.deletions_only bool Mindtalk’s rule: every tidy word is a raw word, in raw order
guard.passed bool false = the client offers the raw words only (“plain only”)
guard.reason, guard.source string, optional why it refused; rules or model
style string, optional the style word the tidy applied (clean, email, chat, code), after the request → [dictation.styles] → built-in order; the same word the dictation_tidy_request trace row carries

Who tidies (ws/dictation_tidy.rs, guards from tools/dictation/tidy_guards.py): a dictation with no cue phrase (“no wait”, “scratch that”, “I mean” …), no stutter and no repeated opening never reaches a model; it gets the word-preserving pass (turn/dictation_cleanup.rs, source: rules). One that trips the gate goes to the brain lane with NO tools (brain/tidy.rs: no session, its own system prompt) as data inside nonce delimiters, bounded at 6 s and off the session loop. The answer is unwrapped and judged: an assistant phrase the raw words lack, collapse (< 0.3 of the words), growth (> 1.2×), a kept-word ratio < 0.7 at 40+ words, and the polish guard (new_word, glued, reordered, too_little_kept). A refusal, timeout or model error is passed: false. Trace rows: dictation_tidy_request {words, chars, style, gate, gate_reasons}, dictation_tidy {source, passed, reason, kept_word_ratio, deletions_only, words_in, words_out, ms}, never the words.

The choice: dictation_review (both directions)

field type rule
type "dictation_review"
request_id string
choice send | plain | scratch | timeout send types the tidy (the raw words when plain only), plain the raw words, scratch nothing; timeout (client → gateway only) = 10 s passed, nothing typed

Gateway → client: while review_armed, these closed phrases (whole utterance after the wake, “please” dropped) become GatewayReport::DictationReview (ahead of the scratch verb, behind the home undo): send it / send that / send it now → send, say it plain / say that plain / keep it plain → plain, scratch that / scratch this / scratch it / delete that / delete this → scratch. The choice is sent, nothing is spoken, no brain turn runs, and the review closes. Client → gateway: the client closes the review (a key, Return, Shift+Return or Esc, or its timeout), traced dictation_review {choice, via, matched, open_ms}. A review nobody closes disarms at 15 s. Outside a review the phrases keep their other meanings. A choice that types goes through dictation_inserted, so “scratch that” can erase it.

Client-side gestures (no wire change)

On the Globe key (globectl), two presses within 400 ms open a hands-free hold: the client keeps dictation_hold open with the key up and sends released on the next tap, on Esc, or after 2.5 s with no new words (6 s before the first). Esc during any hold sends released and drops the dictation_text that follows. The first tap of a double-tap is an empty hold (dictation_hold_empty).

The insert proof and “scratch that”

Client → gateway: dictation_inserted

The screen agent’s write receipt (its proof field), relayed whole. The gateway keeps the latest per request_id. Whether the words are still untouched is the agent’s call at erase time, never a stored flag.

field type rule
type "dictation_inserted"
request_id string the dictation_text it answers; any other id is counted and dropped
proof.receipt_id string the agent’s receipt id, opaque to the gateway; handed back on erase_last_insert
proof.location, proof.length number the inserted range in the field’s own units (UTF-16 offsets)
proof.chars number what landed
proof.text_provable bool the agent can read the range back and compare it
proof.binding range | epoch how the agent proves the insert untouched: the range read back, or its input epoch alone

The refused insert

The client pins the key-down focus on every screen-agent write (expect_pid, expect_bundle, plus the nested expect {pid, bundle_id}), so the agent refuses at the moment of typing (target_changed_app). The client relays the refusal on the same control, with no proof:

field type rule
type "dictation_inserted"
request_id string the dictation_text it answers
refused.reason string the agent’s reason word (target_changed_app, secure_field, …; insert_failed when unreadable)
refused.expected, refused.actual string, optional the key-down and the frontmost bundle ids

The gateway half. proof and refused are both optional on ClientControl::DictationInserted. The client sends refused only to a gateway that offers dictation_insert_refused in hello.features; otherwise it shows “APP CHANGED — NOT TYPED” with no Scratch that and logs it locally. On a refused, the gateway:

  • traces dictation_insert_refused {request_id, reason, expected, actual, had_proof, accepted} (reason words and bundle ids only, each cut to 128 characters; an empty reason is insert_failed);
  • keeps no proof, so “scratch that” answers “There’s nothing to scratch.” A later proof for the same request_id still lands: the latest receipt wins;
  • counts it in the Admin dictation window under refusals as insert_<reason> (a write-time insert_target_changed_app apart from the settle’s target_changed_app), and sets the hold’s outcome to that key. A refusal for another request_id is traced with accepted: false and not counted.

A frame with both fields counts as refused. A frame with neither is traced dictation_inserted_empty and dropped.

Gateway → client: erase_last_insert

field type rule
type "erase_last_insert"
request_id string the dictation to erase
receipt_id string the agent’s receipt from the last proof
allow_backspace bool false until the operator rules on the backspace fallback

Sent only when a spoken “scratch that” resolves to an erase (below). Every other case is refused with a spoken reason and nothing is sent. The client asks the agent ({"erase_last": true, receipt_id, allow_backspace}) and answers:

Client → gateway: dictation_erased

field type rule
type "dictation_erased"
request_id string
outcome erased | refused | unverified | interrupted | failed the agent’s outcome word
reason, method, chars, binding_intact, binding_reason optional the agent’s own detail; traced, never spoken

erased is silent (a scratched chip) and invalidates the receipt, so a second “scratch that” finds nothing. refused means the field was edited or the focus moved; it is spoken as “That text has changed; I left it.” and closes the receipt. unverified, interrupted and failed are spoken as their own lines and leave the receipt open.

A client-started erase (no new frame): Vulpine.app’s Scratch that button and vulpine-glyphctl dictation erase-last ask the agent directly (same request, same binding check, same 120 s window) and send dictation_erased with no erase_last_insert before it. The gateway matches it by request_id against the last proof (traced dictation_erased {accepted}). Vulpine.app relays only erased, so a button never makes Vulpine speak. A refusal stays on the client’s chip and the receipt stays open.

The verb

Closed phrase set, whole utterance after the wake: scratch that, scratch this, delete that, delete this. undo that counts only while a dictation receipt is live and the house has nothing to put back. “Scratch that itch, please” is dictation or a request, never the verb.

Resolution, in the ws loop (crate::dictation_insert::resolve_scratch, then the agent’s answer):

state outcome dictation_scratch.outcome spoken
a proof returned, within the window, not yet scratched erase_last_insert { request_id, receipt_id, allow_backspace: false }; the agent checks the binding erase_requested, then the answer’s row below nothing until the agent answers; erased = a chip
the agent answers erased the receipt closes erased nothing (a scratched chip)
the agent answers refused (binding not intact: caret_moved, range_edited, focus_moved, …) refusal edited_since; the receipt closes changed “That text has changed; I left it.”
the agent answers unverified refusal unverified unverified “I couldn’t check the text, so I left it.”
the agent answers interrupted / failed refusal failed failed “I couldn’t scratch that.”
a dictation delivered but no proof yet, the window passed, already scratched, or nothing delivered refusal nothing_to_scratch nothing “There’s nothing to scratch.”

The trace row dictation_scratch {phrase, outcome, request_id?, receipt_id?, refusal?, binding_reason?, settled?} counts spoken asks only: one row when the verb is heard, one more with settled: true when the agent answers. A client-started erase writes dictation_erased and no dictation_scratch row. The gateway’s record of the last insert is the proof by reference: {request_id, receipt_id, location, length, chars, text_provable, binding}. The pid, the AX element, the window and the inserted string stay inside the screen agent (InsertReceipt). Its range erase, eraseRange in runtime/screen/insert_policy.swift, selects the range, proves it reads code-unit equal to the record, writes nothing over it and reads back.

The verb is armed (scratch_armed) from a delivery until the 120 s window ends, through an erase or a refused answer. Outside the window, and on a gateway with no dictation route, “scratch that” is the armed-request cancel phrase.

Was this page helpful?