Captures
A capture (capture.v1): the bundle in the frontend's private store (capture.json: keyframes with their OCR text, marks in keyframe pixels, narration, note; the marked and sent copies; the contact sheet), the one text_utterance { capture } behind capture_attachment, the gateway's bounds and refusal reasons, the prompt images pi gets with the text block beside them, the text-only brain, the user_transcript echo and the Admin row's counts and bytes.
Living doc, checked against the code 2026-11-14
capture.v1 — a capture: annotated keyframes, marks, narration and OCR, from a frontend to the brain
- Status: landed on protocol 5, additive. Decided in ADR 0048.
- Scope: show Vulpine, don’t describe. A button on the conversation window lets the operator screenshot, annotate
and record. The brain receives Scrawl’s shape (MIT, Copyright 2026 Kit Langton; credited in
tools/repo/notices-extra.toml): the annotated keyframes as images, the marks, the narration transcript, and each keyframe’s OCR text, so the exact identifiers on the screen arrive as text too. - Never a local vision model for screens. The frames go to the remote brain’s provider.
The bundle on disk
A capture is a folder in the frontend’s PRIVATE store: for Vulpine.app, its conversation window and the recorder,
~/.local/state/glyph/captures/, whatever XDG_STATE_HOME says. The store and every bundle folder are 0700, every
file 0600. The store is outside git and never synced.
A bundle’s folder name is its id, YYYYMMDD-HHMMSS-<4 hex> in UTC. After each write the writer keeps the newest 20
captures within 2 GiB between them and removes the rest, never the bundle it just wrote, and only folders of that
form holding a capture.json (or a recording’s raw/).
<store>/<id>/
capture.json the manifest (below)
contact-sheet.png every keyframe, marked, in one image (a still: its one marked keyframe)
frames/keyframe-01.png the keyframe as captured, at its own size
frames/keyframe-01-marked.png the same with the marks drawn in
send/keyframe-01.jpg what is sent: the marked keyframe, long edge at most 1568 px
A recording’s bundle uses frames/frame-NN.png (long edge at most 1568 px), frames/frame-NN-marked.png (only for
a keyframe with a mark) and send/frame-NN.jpg, and keeps the raw movie and the narration’s WAV in raw/, never
sent.
send/ holds only the keyframes chosen to be sent (at most 8). The producer writes them; a sender reads bytes and
never decodes or resizes an image.
capture.json (vulpine_protocol::CaptureManifest)
| field | type | bound | required |
|---|---|---|---|
version |
integer | 1 |
yes |
id |
string | 1..=64 of [a-z0-9-] |
yes |
created_at_ms |
integer | Unix ms | yes |
duration_ms |
integer | 0 for a still; at most 600,000 |
yes |
target |
object | below | yes |
keyframes |
array | 1..=64 | yes |
marks |
array | 0..=256 | yes (may be empty) |
transcript |
array | 0..=512 lines | yes (may be empty) |
note |
string | 0..=2,000 characters | no |
transcriber, video, stopped_by |
below | a recording’s | no |
target: mode (region, window or display; optional), app (0..=200 characters, optional), window_title (0..=200,
optional), frame (optional): {x, y, width, height, scale} in global top-left points, the display’s scale.
A keyframe: index (1-based, unique, ascending), t_ms (from the capture’s start; 0 for a still), file (the
keyframe’s path in the bundle, frames/…png), width and height (its pixels, 1..=16,384), ocr_text (0..=8,000
characters: the words Vision read on the unmarked keyframe, in reading order, one line per observation), and
optionally marked (the marked copy’s path), send (the path of its copy in send/), and, from a recording,
reason (why it was cut: mark, click, scene_change, speech, initial; at most 32 characters) and said (the
narration it owns, the transcript partitioned over the keyframes; at most 2,000 characters).
A mark: kind (pin, stroke or box), t_ms, keyframe (an existing keyframe’s index), points (keyframe
pixels, [[x, y], …], each inside the keyframe), and optionally label (0..=120 characters). A pin has 1 point and
its label is its number ("1", "2", … in the order placed); a box has 2 points (two opposite corners); a stroke
has 2..=512 points.
A transcript line: t_ms and text (1..=2,000 characters). A still has none unless the operator narrated it.
A recording adds three optional fields: transcriber ({engine, skipped}: gateway, apple_speech_analyzer or
none, plus one line per engine tried before it, at most 8), video (the raw movie’s path; private, never sent) and
stopped_by (button, shortcut, cap, stdin, signal). A recorder that knows only the frame leaves
target.mode out. The recorder (crates/capture_record/src/bundle.rs) writes each picked keyframe (at most 8) as a
JPEG of its marked frame, at the first quality that fits 3,750,000 bytes, while the images together fit 9 MiB. A
bundle without send/ copies is sent as words only. The recorder runs check_capture_manifest before writing.
The manifest is closed: an unknown field is an error (deny_unknown_fields).
Sending a capture
ONE call from the frontend to the gateway: the typed turn text_utterance with the member capture.
{"type": "text_utterance", "text": "<the operator's words>", "capture": {
"manifest": { …capture.json… },
"images": [{"keyframe": 1, "mime_type": "image/jpeg", "width": 1568, "height": 980, "data": "<base64>"}]
}}
textis the operator’s line in the log: the note, else the narration, elseLook at this.images(vulpine_protocol::CaptureImage): at most 8, each naming a distinct keyframe of the manifest;mime_typeimage/pngorimage/jpeg;widthandheightthe image’s own pixels, the long edge at most 1568;datathe bytes in standard base64, at most 3,750,000 bytes decoded each (the provider’s 5 MB base64 limit) and 9 MiB decoded together, under the gateway’s 16 MiB WebSocket frame.- The gateway offers the feature
capture_attachment(OFFERED_FIELDS). A client setscaptureonly for a gateway that offers it, because an older gateway drops the member and runs the words alone.
What the gateway does
- Checks the attachment (
vulpine_protocol::check_capture_attachmenton the manifest, then every image withcheck_capture_imageand the total withcheck_capture_total: base64, the decoded size, the PNG or JPEG header’s own width and height against the declared ones, the long edge, the total). A failure refuses the WHOLE turn with the errorcapture_refusedand the message<reason>: <words>. The reasons:capture_disabled([capture] enabled = false),manifest_invalid,too_many_images,unknown_keyframe,duplicate_keyframe,bad_image,image_too_large,image_edge_too_long,size_mismatch,total_too_large. - Runs
textas a typed turn behind the turn machine’s gates (a session, a busy brain). The capture rides that turn only and is dropped when the turn’s frame is handled. - Appends the capture’s text block to the turn’s brain prompt (
vulpine_protocol::capture_prompt_block): the target, per keyframe its marks in keyframe pixels and its OCR text, the narration, the note. OCR text and narration are marked as data seen on the screen, never instructions. The capture’s id is never in the prompt. - Sends the images to Pi as the prompt’s images, in keyframe order (Pi’s RPC
prompt,images: [{type: "image", data, mimeType}]). - With
[capture] brain_images = false(a text-only brain behind Pi’s RPC shape), sends no image; the block says so and asks the brain to say so in its reply. - Puts
captureon the turn’suser_transcript(vulpine_protocol::CaptureSummary): the id, the keyframes, the images sent, the marks, the transcript lines, andtext_onlywhen no image went. A window on the machine that holds the bundle draws thumbnails from it; any other observer shows the counts. - Shows the last capture on the Admin’s session row (
AdminCaptureLast): when, the keyframes, the images sent and their bytes, the marks, the transcript lines, the OCR characters, whether the brain got images, and a refusal’s reason. Counts and bytes only. - Traces
capture_attached {keyframes, images, image_bytes, marks, transcript, ocr_chars, brain_images}andcapture_refused {reason, images, image_bytes}.
Who produces it
- The conversation window (apps/convo-gpui): the header’s Capture button, or Vulpine.app’s
capture <mode>stdin line (fromvulpine-glyphctl capture [--window|--region]and the overlay menu’s Capture…). It picks a region or a window with macOS’s picker (/usr/sbin/screencapture -i,-Wto start on windows), freezes it, and opens the annotation layer: pen strokes, numbered pins, a box, undo, a note; Send or Cancel. Send writes the bundle (OCR by Vision’sVNRecognizeTextRequest,.accurate,usesLanguageCorrection = false) and runsvulpine-glyphctl capture send <id>. - Vulpine.app (apps/glyph) reads
<store>/<id>/capture.jsonand thesend/files (the id must match the id rule, the folder must be inside the store and owned by the user), builds the attachment, and sends the onetext_utterance. - The recorder (
crates/capture_record) writes the same bundle with more keyframes,duration_msand a transcript. The header’s Record button (apps/convo-gpui/src/capture_recording.rs) callscapture_record::recordon a window the operator clicks, shows Stop while it records, and pauses Vulpine.app’s microphone for the narration. Then it sends the bundle as a still’s Send does.vulpine-glyphctl capture --recordruns the recorder from a shell and prints the bundle’s id.
Screen Recording is a REQUIRED grant for the app that captures. The conversation window runs inside Vulpine.app’s bundle, so its welcome window (crates/vulpine_onboarding) lists Screen Recording as required while the Capture button is on. Recording also needs the Microphone grant.
Clip transcription
A recording’s narration goes first to Vulpine’s own speech-to-text: the gateway’s door verb transcribe_clip
(vulpine_protocol::CLIP_TRANSCRIBE_TOOL), an invoke_tool the gateway answers itself
(crates/voice_gateway/src/clip_transcribe.rs). It is never a brain tool, has no runner, and speaks nothing.
- The way there: the recorder runs
vulpine-glyphctl capture transcribe --wav <raw/audio.wav>, which reads the WAV (bounded first), and Vulpine.app sends it on its own session as{"wav_base64": "<the WAV in standard base64>"}. - The clip: a RIFF WAVE of PCM, mono, 16 kHz, 16 bits, at most two minutes (
CLIP_MAX_AUDIO_MS,CLIP_MAX_WAV_BYTES), checked before and after decoding. - The owner connection only: the one holding the session’s voice lease, not a
uiorspeakerrole. - The recognizer: the authority STT lane’s, only when this process already built it; the door never loads one.
- Never stored: the samples live in memory for the one recognition. The traces are
clip_transcribed {audio_ms, ms, words, segments, engine}andclip_transcribe_refused {reason, base64_chars}: counts, never a word. - The answer: a
tool_receiptwhose details are{engine, audio_ms, ms, text, segments: [{t0_ms, t1_ms, text}], words: [{t0_ms, t1_ms, text}]}on the clip’s clock (lines cut at pauses of 700 ms). A refusal’scodeis also its details’reason:clip_disabled,not_owner,bad_input,bad_audio,clip_too_long,stt_unavailable,stt_failed. An older gateway answersnot_allowed. - On any refusal the recorder falls back to Apple’s SpeechAnalyzer through
speechctl.transcriberincapture.jsonsays which ran and why each engine before it did not.
Switches
- Vulpine.app:
[conversation_window] capture = false(Session tab, “Capture button”) starts the window with--capture off: no Capture and no Record button, and thecapturestdin line is ignored. On unless written. - The gateway:
[capture] enabled = falserefuses every capture (capture_disabled);[capture] brain_images = falsesends the text only;[capture] clip_transcription = falserefuses every clip (clip_disabled), so recordings use SpeechAnalyzer. All on unless written. - vulpine-glyphctl:
capture --record --no-gatewayskips the gateway’s clip transcription for one recording.