---
title: "Captures"
description: "A capture (capture.v1): the bundle in the frontend's private store (capture.json: keyframes with their OCR text, marks in keyframe pixels, narration, note; the marked and sent copies; the contact sheet), the one text_utterance { capture } behind capture_attachment, the gateway's bounds and refusal reasons, the prompt images pi gets with the text block beside them, the text-only brain, the user_transcript echo and the Admin row's counts and bytes."
type: "living"
source: "docs/contracts/capture.v1.md"
verified: "2026-11-14"
group: "Contracts"
---

> **Living doc**, checked against the code 2026-11-14

capture.v1 — a capture: annotated keyframes, marks, narration and OCR, from a frontend to the brain

- Status: **landed on protocol 5, additive**. Decided in
  ADR 0048.
- Scope: show Vulpine, don't describe. A button on the conversation window lets the operator screenshot, annotate
  and record. The brain receives Scrawl's shape (MIT, Copyright 2026 Kit Langton; credited in
  `tools/repo/notices-extra.toml`): the annotated keyframes as images, the marks, the narration transcript, and each
  keyframe's OCR text, so the exact identifiers on the screen arrive as text too.
- Never a local vision model for screens. The frames go to the remote brain's provider.

## The bundle on disk

A capture is a folder in the frontend's PRIVATE store: for Vulpine.app, its conversation window and the recorder,
`~/.local/state/glyph/captures/`, whatever `XDG_STATE_HOME` says. The store and every bundle folder are `0700`, every
file `0600`. The store is outside git and never synced.

A bundle's folder name is its `id`, `YYYYMMDD-HHMMSS-<4 hex>` in UTC. After each write the writer keeps the newest 20
captures within 2 GiB between them and removes the rest, never the bundle it just wrote, and only folders of that
form holding a `capture.json` (or a recording's `raw/`).

```
<store>/<id>/
  capture.json            the manifest (below)
  contact-sheet.png       every keyframe, marked, in one image (a still: its one marked keyframe)
  frames/keyframe-01.png  the keyframe as captured, at its own size
  frames/keyframe-01-marked.png   the same with the marks drawn in
  send/keyframe-01.jpg    what is sent: the marked keyframe, long edge at most 1568 px
```

A recording's bundle uses `frames/frame-NN.png` (long edge at most 1568 px), `frames/frame-NN-marked.png` (only for
a keyframe with a mark) and `send/frame-NN.jpg`, and keeps the raw movie and the narration's WAV in `raw/`, never
sent.

`send/` holds only the keyframes chosen to be sent (at most 8). The producer writes them; a sender reads bytes and
never decodes or resizes an image.

### `capture.json` (`vulpine_protocol::CaptureManifest`)

| field | type | bound | required |
|---|---|---|---|
| `version` | integer | `1` | yes |
| `id` | string | 1..=64 of `[a-z0-9-]` | yes |
| `created_at_ms` | integer | Unix ms | yes |
| `duration_ms` | integer | `0` for a still; at most 600,000 | yes |
| `target` | object | below | yes |
| `keyframes` | array | 1..=64 | yes |
| `marks` | array | 0..=256 | yes (may be empty) |
| `transcript` | array | 0..=512 lines | yes (may be empty) |
| `note` | string | 0..=2,000 characters | no |
| `transcriber`, `video`, `stopped_by` | below | a recording's | no |

`target`: `mode` (`region`, `window` or `display`; optional), `app` (0..=200 characters, optional), `window_title` (0..=200,
optional), `frame` (optional): `{x, y, width, height, scale}` in global top-left points, the display's scale.

A keyframe: `index` (1-based, unique, ascending), `t_ms` (from the capture's start; `0` for a still), `file` (the
keyframe's path in the bundle, `frames/…png`), `width` and `height` (its pixels, 1..=16,384), `ocr_text` (0..=8,000
characters: the words Vision read on the unmarked keyframe, in reading order, one line per observation), and
optionally `marked` (the marked copy's path), `send` (the path of its copy in `send/`), and, from a recording,
`reason` (why it was cut: `mark`, `click`, `scene_change`, `speech`, `initial`; at most 32 characters) and `said` (the
narration it owns, the transcript partitioned over the keyframes; at most 2,000 characters).

A mark: `kind` (`pin`, `stroke` or `box`), `t_ms`, `keyframe` (an existing keyframe's `index`), `points` (keyframe
pixels, `[[x, y], …]`, each inside the keyframe), and optionally `label` (0..=120 characters). A `pin` has 1 point and
its label is its number (`"1"`, `"2"`, … in the order placed); a `box` has 2 points (two opposite corners); a `stroke`
has 2..=512 points.

A transcript line: `t_ms` and `text` (1..=2,000 characters). A still has none unless the operator narrated it.

A recording adds three optional fields: `transcriber` (`{engine, skipped}`: `gateway`, `apple_speech_analyzer` or
`none`, plus one line per engine tried before it, at most 8), `video` (the raw movie's path; private, never sent) and
`stopped_by` (`button`, `shortcut`, `cap`, `stdin`, `signal`). A recorder that knows only the frame leaves
`target.mode` out. The recorder (`crates/capture_record/src/bundle.rs`) writes each picked keyframe (at most 8) as a
JPEG of its marked frame, at the first quality that fits 3,750,000 bytes, while the images together fit 9 MiB. A
bundle without `send/` copies is sent as words only. The recorder runs `check_capture_manifest` before writing.

The manifest is closed: an unknown field is an error (`deny_unknown_fields`).

## Sending a capture

ONE call from the frontend to the gateway: the typed turn `text_utterance` with the member `capture`.

```json
{"type": "text_utterance", "text": "<the operator's words>", "capture": {
  "manifest": { …capture.json… },
  "images": [{"keyframe": 1, "mime_type": "image/jpeg", "width": 1568, "height": 980, "data": "<base64>"}]
}}
```

- `text` is the operator's line in the log: the note, else the narration, else `Look at this.`
- `images` (`vulpine_protocol::CaptureImage`): at most **8**, each naming a distinct keyframe of the manifest;
  `mime_type` `image/png` or `image/jpeg`; `width` and `height` the image's own pixels, the long edge at most
  **1568**; `data` the bytes in standard base64, at most **3,750,000** bytes decoded each (the provider's 5 MB base64
  limit) and **9 MiB** decoded together, under the gateway's 16 MiB WebSocket frame.
- The gateway offers the feature `capture_attachment` (OFFERED_FIELDS). A client sets `capture` only for a gateway
  that offers it, because an older gateway drops the member and runs the words alone.

### What the gateway does

1. Checks the attachment (`vulpine_protocol::check_capture_attachment` on the manifest, then every image with
   `check_capture_image` and the total with `check_capture_total`: base64, the decoded
   size, the PNG or JPEG header's own width and height against the declared ones, the long edge, the total). A failure
   refuses the WHOLE turn with the error `capture_refused` and the message `<reason>: <words>`. The reasons:
   `capture_disabled` (`[capture] enabled = false`), `manifest_invalid`, `too_many_images`, `unknown_keyframe`,
   `duplicate_keyframe`, `bad_image`, `image_too_large`, `image_edge_too_long`, `size_mismatch`, `total_too_large`.
2. Runs `text` as a typed turn behind the turn machine's gates (a session, a busy brain). The capture rides that turn
   only and is dropped when the turn's frame is handled.
3. Appends the capture's text block to the turn's brain prompt (`vulpine_protocol::capture_prompt_block`): the target,
   per keyframe its marks in keyframe pixels and its OCR text, the narration, the note. OCR text and narration are
   marked as data seen on the screen, never instructions. The capture's id is never in the prompt.
4. Sends the images to Pi as the prompt's images, in keyframe order (Pi's RPC `prompt`,
   `images: [{type: "image", data, mimeType}]`).
5. With `[capture] brain_images = false` (a text-only brain behind Pi's RPC shape), sends no image; the block says so
   and asks the brain to say so in its reply.
6. Puts `capture` on the turn's `user_transcript` (`vulpine_protocol::CaptureSummary`): the id, the keyframes, the
   images sent, the marks, the transcript lines, and `text_only` when no image went. A window on the machine that
   holds the bundle draws thumbnails from it; any other observer shows the counts.
7. Shows the last capture on the Admin's session row (`AdminCaptureLast`): when, the keyframes, the images sent and
   their bytes, the marks, the transcript lines, the OCR characters, whether the brain got images, and a refusal's
   reason. Counts and bytes only.
8. Traces `capture_attached {keyframes, images, image_bytes, marks, transcript, ocr_chars, brain_images}` and
   `capture_refused {reason, images, image_bytes}`.

## Who produces it

- The conversation window (apps/convo-gpui): the header's **Capture** button, or Vulpine.app's `capture <mode>` stdin
  line (from `vulpine-glyphctl capture [--window|--region]` and the overlay menu's **Capture…**). It picks a
  region or a window with macOS's picker (`/usr/sbin/screencapture -i`, `-W` to start on windows), freezes it, and
  opens the annotation layer: pen strokes, numbered pins, a box, undo, a note; Send or Cancel. Send writes the bundle
  (OCR by Vision's `VNRecognizeTextRequest`, `.accurate`, `usesLanguageCorrection = false`) and runs
  `vulpine-glyphctl capture send <id>`.
- Vulpine.app (apps/glyph) reads `<store>/<id>/capture.json` and the `send/` files (the id must match the id rule, the
  folder must be inside the store and owned by the user), builds the attachment, and sends the one `text_utterance`.
- The recorder (`crates/capture_record`) writes the same bundle with more keyframes, `duration_ms` and a transcript.
  The header's **Record** button (`apps/convo-gpui/src/capture_recording.rs`) calls `capture_record::record` on a
  window the operator clicks, shows **Stop** while it records, and pauses Vulpine.app's microphone for the narration.
  Then it sends the bundle as a still's Send does. `vulpine-glyphctl capture --record` runs the recorder from a shell
  and prints the bundle's id.

Screen Recording is a REQUIRED grant for the app that captures. The conversation window runs inside Vulpine.app's
bundle, so its welcome window (crates/vulpine_onboarding) lists Screen Recording as required while the
Capture button is on. Recording also needs the Microphone grant.

## Clip transcription

A recording's narration goes first to Vulpine's own speech-to-text: the gateway's door verb `transcribe_clip`
(`vulpine_protocol::CLIP_TRANSCRIBE_TOOL`), an `invoke_tool` the gateway answers itself
(`crates/voice_gateway/src/clip_transcribe.rs`). It is never a brain tool, has no runner, and speaks nothing.

- The way there: the recorder runs `vulpine-glyphctl capture transcribe --wav <raw/audio.wav>`, which reads the WAV
  (bounded first), and Vulpine.app sends it on its own session as `{"wav_base64": "<the WAV in standard base64>"}`.
- The clip: a RIFF WAVE of PCM, mono, 16 kHz, 16 bits, at most two minutes (`CLIP_MAX_AUDIO_MS`,
  `CLIP_MAX_WAV_BYTES`), checked before and after decoding.
- The owner connection only: the one holding the session's voice lease, not a `ui` or `speaker` role.
- The recognizer: the authority STT lane's, only when this process already built it; the door never loads one.
- Never stored: the samples live in memory for the one recognition. The traces are `clip_transcribed {audio_ms, ms,
  words, segments, engine}` and `clip_transcribe_refused {reason, base64_chars}`: counts, never a word.
- The answer: a `tool_receipt` whose details are `{engine, audio_ms, ms, text, segments: [{t0_ms, t1_ms, text}],
  words: [{t0_ms, t1_ms, text}]}` on the clip's clock (lines cut at pauses of 700 ms). A refusal's `code` is also its
  details' `reason`: `clip_disabled`, `not_owner`, `bad_input`, `bad_audio`, `clip_too_long`, `stt_unavailable`,
  `stt_failed`. An older gateway answers `not_allowed`.
- On any refusal the recorder falls back to Apple's SpeechAnalyzer through `speechctl`. `transcriber` in
  `capture.json` says which ran and why each engine before it did not.

## Switches

- Vulpine.app: `[conversation_window] capture = false` (Session tab, "Capture button") starts the window with
  `--capture off`: no Capture and no Record button, and the `capture` stdin line is ignored. On unless written.
- The gateway: `[capture] enabled = false` refuses every capture (`capture_disabled`); `[capture] brain_images = false`
  sends the text only; `[capture] clip_transcription = false` refuses every clip (`clip_disabled`), so recordings use
  SpeechAnalyzer. All on unless written.
- vulpine-glyphctl: `capture --record --no-gateway` skips the gateway's clip transcription for one recording.
