Skip to content
Vulpine
Esc
↑↓navigate↵open⌘Jpreview
On this page

Script visuals

A script visual (visual.v1, pi_show_visual): the body (title, summary, script), the Luau ui tree a frontend draws (layout, text, stats, tables, charts, tabs, toggles, sliders, buttons, canvases, 3D scenes, Shadertoy shaders), the sandbox's rules and budgets, the gateway's dry run and its refusal with the line to fix, who gets the visual whole and who its card, and the conversation window's GPU pictures at rest at 0 frames.

Living doc, checked against the code 2026-11-14

visual.v1 — a script visual: a sandboxed Luau program a frontend runs and draws

  • Status: live on protocol 5, additive (ADR 0047). The kind visual.v1 and its body VisualSpec are in the protocol crate. The brain sends a visual with pi_show_visual. The gateway dry-runs every script before relay. The conversation window (apps/convo-gpui) runs and draws it.
  • Scope: the visual response channel’s second tier, beside panel-card.v1. card.v1 stays the closed, speakable default for a plain chart or a status. A visual is for what a card cannot say: a chart that switches metrics, a slider that reshapes a curve, a small simulation, a 3D layout, a shader.
  • The pattern follows zeron’s show_visual (MIT). The code is Vulpine’s.

What the brain sends

pi_show_visual {title, summary, script}. The tool sends the gateway’s show_render mutation line with kind visual.v1 and the body below. summary is the one line the reply says aloud: by card.v1’s rule, the words never depend on the panel. A frontend that cannot run the script shows it too.

VisualSpec, the body of visual.v1

field type bound required
title string 1..=80 characters, plain, no hex token (card.v1’s title rule) yes
summary string 1..=160 characters, plain (card.v1’s summary rule) yes
script string 1..=6,144 bytes, no NUL (vulpine_protocol::VISUAL_SCRIPT_MAX) yes

The body is closed (deny_unknown_fields). The whole JSON body rides one 8 KiB mutation line, so the tool refuses a body over 7 KiB (body_too_large).

The script

A Luau program: Lua 5.1 plus Luau’s +=, continue and if expressions. It builds its data at the top level and returns a view function or a node. The view returns a tree of ui.* nodes. A control’s handler assigns to the script’s own state (local s = ui.state { ... }), and the frontend runs the view again.

local s = ui.state { metric = "p95" }
local regions = { "us-east", "eu-west" }
local p95, p99 = { 182, 140 }, { 240, 210 }
return function()
  local values = s.metric == "p95" and p95 or p99
  return ui.column { gap = 12,
    ui.tabs { options = { "p95", "p99" }, value = s.metric, on_change = function(v) s.metric = v end },
    ui.chart { type = "bar", label = "Latency " .. s.metric, unit = "ms", x = regions,
      series = { { name = s.metric, values = values } } },
  }
end

Nodes are props tables. Children are the array part. A bare string is text, nil and false are skipped, and a plain list of nodes (what ui.map returns) is flattened. An unknown prop, a wrong type or a value out of bounds is an error, never a guess. The Bounds table below holds every bound. Every picture (chart, canvas, shader, scene) needs a label, read by a screen reader.

node props
ui.column, ui.row gap, align (start, center, end, stretch), justify (row: start, center, end, between), wrap
ui.grid columns (1..=6), gap
ui.text children strings and ui.span { ..., weight, tone, mono }; size (xs, sm, body, lg, xl), weight, tone, align, mono
ui.stat label, value, unit, delta, tone
ui.table columns (≤ 8), rows (≤ 50, each as wide as columns)
ui.chart type (bar, line, area), label, x, series (≤ 6 of { name, values }, ≤ 120 values), unit, stacked, height
ui.rule, ui.space size (space)
ui.tabs options (≤ 8), value, on_change(option)
ui.toggle label, value, on_change(on)
ui.slider label, value, min, max, step, unit, on_change(value)
ui.button label, primary, on_click()
ui.canvas label, height, draw(g, w, h, t), animate, on_click(x, y); g:line, g:rect, g:circle, g:polyline, g:polygon, g:text, g.pointer
ui.scene label, height, draw(s, t), orbit, spin, camera { yaw, pitch, zoom }, animate; s:box, s:sphere, s:cylinder, s:line, s:grid
ui.shader label, height, code (a Shadertoy mainImage, ≤ 6 KiB), inputs { name = number } (≤ 16), animate, buffers { a..d = { code, channels } }, channels (≤ 4 buffer names)

Any node also takes grow (take a row’s spare width), width and key. Colors are theme tokens (text, muted, faint, accent, ok, warn, bad, line, surface, chart1..chart6). A canvas and a scene may also name a physical #rrggbb. Layout nodes take no fills or borders. Helpers: ui.state, ui.map, ui.range, ui.scale, ui.clamp, ui.lerp, ui.sum, ui.fmt, ui.series(i).

Bounds

Every bound the parser enforces, one row a bound. The source is catalog/pi/extensions/vulpine-tools/visual-bounds.json, and pi_show_visual’s description is rendered from it. A value past a bound is refused with its line, except where a row says clamped.

node prop bound
call title 1..80 characters
call summary 1..160 characters (the line you say aloud)
script bytes at most 6144 (6 KiB)
view return one node (wrap a list in ui.column)
view nodes at most 600 (a text child counts)
view depth at most 16 levels under the root
view handlers at most 64 (every on_change, on_click and draw)
view canvases at most 4
view shaders at most 2
view scenes at most 2
view children nodes, text or lists of them (nil and false skipped); a text child at most 2000 characters
any node grow true or false (take a row’s spare width)
any node width 8..2000 pt
any node key any value (ignored)
ui.column/ui.row gap 0..64 pt
ui.column/ui.row align start|center|end|stretch
ui.column/ui.row justify start|center|end|between
ui.column/ui.row wrap true or false
ui.grid columns 1..6 (default 2)
ui.grid gap 0..64 pt
ui.text children strings and ui.span, at most 2000 characters in all
ui.text size xs|sm|body|lg|xl
ui.text weight a number 100..900, never a word like “bold”
ui.text tone a theme token: text, muted, faint, accent, ok, warn, bad, line, surface, chart1..chart6
ui.text align start|center|end
ui.text mono true or false
ui.span weight a number 100..900
ui.span tone a theme token
ui.span mono true or false
ui.stat label 1..60 characters, required
ui.stat value 1..40 characters (a number is formatted), required
ui.stat unit at most 16 characters
ui.stat delta at most 24 characters
ui.stat tone a theme token
ui.table columns 1..8 names of at most 40 characters, required
ui.table rows at most 50, each exactly one cell per column, a cell at most 120 characters, required
ui.chart type bar|line|area (default bar)
ui.chart label 1..60 characters, required
ui.chart x one label per value, each at most 16 characters
ui.chart series 1..6 of { name (1..60 characters), values (1..120 finite numbers, as many in every series) } and no other field, required
ui.chart unit at most 16 characters
ui.chart stacked true or false
ui.chart height 40..640 pt (default 180)
ui.space size 0..200 pt (default 12)
ui.tabs options 1..8 of at most 40 characters, required
ui.tabs value one of options (default the first)
ui.tabs on_change function(option), required
ui.toggle label 1..60 characters, required
ui.toggle value true or false
ui.toggle on_change function(on), required
ui.slider label 1..60 characters, required
ui.slider min/max numbers, max above min (default 0 and 1)
ui.slider step 0 or more (0 is continuous)
ui.slider value min..max (default min)
ui.slider unit at most 16 characters
ui.slider on_change function(value), required
ui.button label 1..60 characters, required
ui.button primary true or false
ui.button on_click function(), required
ui.canvas label 1..60 characters, required
ui.canvas height 40..640 pt (default 200)
ui.canvas draw function(g, w, h, t), required
ui.canvas animate true or false
ui.canvas on_click function(x, y)
ui.shader label 1..60 characters, required
ui.shader height 40..640 pt (default 240)
ui.shader code a Shadertoy mainImage, at most 6144 bytes, required
ui.shader inputs at most 16 of name = number, each a GLSL name that is not a built-in (iTime, iResolution, …)
ui.shader animate true or false (default: whether a pass reads the clock or its own last frame)
ui.shader buffers named a, b, c, d, each { code, channels }
ui.shader channels at most 4 names of buffers the shader defines, read as iChannel0..3
shader buffer code a mainImage, at most 6144 bytes, required
shader buffer channels at most 4 buffer names the shader defines (its own: feedback)
ui.scene label 1..60 characters, required
ui.scene height 40..640 pt (default 280)
ui.scene draw function(s, t), required
ui.scene orbit true or false (default true)
ui.scene spin -10..10 radians a second (default 0)
ui.scene camera { yaw -10..10, pitch -1.5..1.5, zoom 0.1..10 } (other keys ignored; a zoom far above 1 can cut objects off the picture)
ui.scene animate true or false (call draw every frame)
canvas pen commands at most 5000 a draw
canvas pen coordinates numbers as arguments: g:line(x1, y1, x2, y2, o), g:rect(x, y, w, h, o), g:circle(x, y, r, o), g:text(x, y, text, o); finite, clamped to -1000000..1000000
canvas pen options one table { name = value } as the last argument, only the names its command takes
g:polyline/g:polygon points { x1, y1, x2, y2, … } or { { x, y }, … }, at most 2000
g:text text at most 200 characters, the third argument
g:line/g:polyline/g:text color a theme token or #rrggbb
g:rect/g:circle/g:polygon fill/stroke a theme token or #rrggbb (neither named: filled in the text color)
g:line/g:rect/g:circle/g:polyline/g:polygon width 0.25..32 (clamped)
g:rect radius 0..10000 (clamped)
g:text size 6..96 (clamped; under 8 reads as noise)
g:text align start|center|end
g:text baseline top|middle|bottom
g:text weight a number 100..900 (clamped), never a word
scene builder primitives at most 2000 a draw (a grid counts its lines)
scene builder call form s:box { … }, s:sphere { … }, s:cylinder { … } and s:grid { … } take one table; s:line(a, b, o) takes two { x, y, z } points as arguments, never s:line { a = …, b = … }
s:box/s:sphere/s:cylinder at { x, y, z } (or x =, y =, z =), each finite and under 100000
s:box size { w, h, d } or one number, each above 0
s:sphere/s:cylinder radius above 0
s:cylinder height above 0
s:grid size/step above 0, at most 200 lines a side
s:box/s:sphere/s:cylinder/s:line/s:grid color a theme token or #rrggbb
s:line width 0.5..16 (clamped)
budget setup/view/handler 250 ms each
budget canvas or scene draw 60 ms each
budget dry run 1500 ms in all
budget heap 64 MiB

A shader is GLSL 4.50 wrapped by visual_sandbox::shader::program with Shadertoy’s inputs: iResolution, iTime, iTimeDelta, iFrame, iMouse (origin bottom-left), and the named inputs as floats.

The loop guard. Every GPUI window draws on one GPU device, and the GPU never preempts inside a draw. One pixel stuck in for(;;){} would freeze every window until the OS reset the GPU. visual_sandbox::shader::guard runs on naga’s module before validation, in ANGLE’s ensure_loop_forward_progress shape.

  • A loop proven finite is left as it is. It has a 32-bit integer variable written only by the loop’s literal step, starting at a literal, compared with a literal before anything else in the body, for at most 4096 iterations (for (int i = 0; i < 64; i++)).
  • Every other loop is accepted and guarded: for(;;), while (true), do ... while, a float loop, a loop bounded by iFrame or an input. One private counter serves the whole invocation. Each guarded loop breaks at its top once the pixel has run 4096 guarded iterations (LOOP_GUARD_CAP, eight times a 512-step raymarch). Nested loops add up, never multiply.
  • A loop that never ends draws what it had (black for for(;;){}).

The frontend compiles that guarded module (ShaderSource::Naga), never the GLSL again, so what the dry run validated is what draws.

The frame deadline. surface3d’s producer watches a shader view’s frames (Scene3d::deadline). Each frame’s on_submitted_work_done marks it finished. A frame the GPU has not finished after 2 s (shadertoy::FRAME_DEADLINE) stops the view on its last picture. The view’s words say which frame and how long, the log names the view, and Stats::missed_deadline is set.

Buffer passes (Shadertoy’s Buffer A to D). buffers names up to four buffers a..d. Each is its own mainImage (code). Every frame draws them into offscreen pictures the size of the shader’s, in the order a, b, c, d, before the image (code). A pass reads up to four buffers through channels as iChannel0..3: texture(iChannel0, fragCoord / iResolution.xy) or texelFetch(iChannel0, ivec2(fragCoord), 0). iChannelResolution[i] is the picture’s size. A buffer drawn earlier in the frame is read as drawn now. The pass’s own buffer and later ones are read as the last frame left them, so a buffer that reads itself is feedback (a trail, a simulation). These are errors at the script’s line: a channel naming a buffer the shader does not define, a fifth channel, a buffer named other than a..d, a prop a buffer does not take. A shader with feedback, or whose passes read the clock, moves unless animate says otherwise.

ui.shader { label = "A dot with a fading trail", channels = { "a" },
  buffers = { a = { channels = { "a" }, code = [[
void mainImage(out vec4 c, in vec2 p) {
  vec2 uv = p / iResolution.xy;
  vec2 at = 0.5 + 0.3 * vec2(cos(iTime), sin(iTime));
  c = max(texture(iChannel0, uv) * 0.96, vec4(smoothstep(0.05, 0.0, length(uv - at))));
}
]] } },
  code = [[ void mainImage(out vec4 c, in vec2 p) { c = texture(iChannel0, p / iResolution.xy); } ]] }

Not supported: textures, sound and the keyboard as channels, glTF models, a scene whose draw runs every frame.

The sandbox

crates/visual_sandbox runs every script, on the gateway and on every frontend, with the same rules:

  • No I/O. Luau with the table, string, math, utf8 and bit32 libraries only. There is no require, loadstring, getfenv/setfenv, collectgarbage, coroutines or debug library. Globals are read-only (Luau’s sandbox) and print is quiet.
  • Budgets. 64 MiB of heap; 250 ms for the setup, a view or a handler; 60 ms for a canvas or scene draw. They are enforced at Luau’s safepoints. Once a budget is spent every safepoint fails, so a pcall cannot catch its way back into work. A call that caught the error and returned anyway still fails (“time budget”).
  • No naive search. Luau’s plain string.find (a plain call, or a pattern with no special character) loops inside its C library, never reaches a safepoint, and costs the haystack’s length times the needle’s. The sandbox’s string.find does that search in linear time (Rust’s two-way search, Knuth-Morris-Pratt for bytes that are not UTF-8) and gives Luau’s own answers. A pattern search stays Luau’s, because its matcher calls the interrupt at every step.
  • A hard abort on the frontend. The safepoints do not see any other call that spends its time inside the C library. So a frontend runs each visual on its own thread (visual_sandbox::worker) and waits at most the call’s budget plus a 40 ms grace. Past that the visual is stopped (“stopped: time budget”). Every later call answers the same at once, and the thread finishes its call on its own. At most four such threads may run before a new script is refused.
  • The same data everywhere. math.random is seeded from the script’s bytes before every call.

The dry run, and the answer

The gateway’s mutation listener runs visual_sandbox::check on a blocking thread before the line reaches the session. The dry run is bounded at 1.5 s in all and covers:

  • the setup and the view;
  • every canvas and scene draw;
  • every shader, through naga’s GLSL front end (the one wgpu compiles it with) and the loop guard;
  • each control once, rendering again after each.

With buffers, every pass is checked and an error names its pass (ui.shader buffer b: shader line 2: ...). A guarded loop is not an error: the shader passes like any other, and the report’s guarded_loops counts it. A failure is answered at once and no frontend sees the script:

{"status": "render_shown", "kind": "visual.v1", "outcome": "refused", "reason": "script_error",
 "waited_ms": 3, "error": "visual:14: ui.chart has no prop \"colour\" (it takes: type, label, ...)"}

visual:14 is the script’s line 14. A shader’s error names its own line (shader line 3: ...). The brain fixes that line and calls again in the same turn. A visual that passed goes through the session’s render router like any render (drawn, refused, dropped, or sent unconfirmed). The Admin’s gateway.visuals Health row counts the dry runs: passed, refused, answered from the cache, the slowest pass in ms and the shader loops the guard bounds. It never shows the script.

The cache. The last 64 dry runs are kept by the SHA-256 of the script’s bytes. A script sent again unchanged is answered without running it, from the brain or the preview door. A pass and a script error are kept. A run the time budget stopped is not.

The switch: [frontend] visual_scripts (on by default). Off, every visual is refused with visual_off and the error says to send a card.v1 instead.

Who gets what

frontend declares gets
the conversation window (apps/convo-gpui, an observer) visual.v1 in its hello the visual whole; it runs the script and draws it
Vulpine.app’s Glyph look, Vigil (they draw cards) card.v1, not visual.v1 the visual’s card projection (visual_card: its title, its summary as the one block, a note that it draws in full in the conversation window); the brain’s answer says projected_to: card.v1
a frontend that draws neither nothing nothing (dropped, undeclared)

The session’s observers are offered the visual whole even when the session’s own frontend got its card (RenderSend::observed). An observer that did not declare visual.v1 is sent the card (ws::observe::observer_frame). The panel kit (packages/vulpine-voice-ui-state) draws a visual.v1 frame as its card if one arrives. It never declares the kind (KIT_UNDECLARED_FRAMES), so the gateway never counts a card as a drawn visual.

Drawing it (the conversation window)

  • The tree in the card’s look (apps/native-cards/src/visual.rs): stacks, text, stats and tables natively, a chart as card.v1’s chart block, controls as labeled chips (a slider steps), a canvas painted from its commands with its words as text. Every node is labeled.
  • Off the main thread (src/visual_run.rs). Each visual’s script runs on its own thread. The window waits 12 ms for a load or a control; a script answers in about 1 ms. A later answer draws the card with “running the script…”. The window keeps drawing and asks again each frame, until the answer or its deadline (“stopped: time budget”). A canvas or scene draw waits up to its 60 ms budget plus the grace.
  • A shader runs on the kit’s surface3d::shadertoy:
    • Compile. wgpu’s GLSL front end compiles it under a validation scope with a 2 s timeout. Each buffer pass has its own scope, so an error names its pass.
    • Buffer passes. They are drawn into half-float pictures before the image (surface3d/shadertoy_passes.rs). Every pass is smaller on a lower rung, and the rung’s budget is the whole frame’s GPU time.
    • The quality ladder. Full, two-thirds, half and quarter resolution, stepped by the GPU’s own time from timestamp queries. Two-thirds and half come up through FSR1 (surface3d/upscale.rs), and quarter is blitted. A new rung is drawn once it has held 12 frames. A shader still over the cheapest rung’s budget is held still.
    • The low-power rule. On battery it runs a rung down, except that a shader too light for an upscale to pay stays full. In Low Power Mode it runs the cheapest rung. Hot, it stops on its last picture.
    • A device lost while it drew stops it for good.
  • A scene on surface3d::prims: instanced boxes, spheres and cylinders and a line list, lit, with depth, an orbit camera the reader drags and an optional turntable spin.
  • 0 frames at rest. A shader that reads the clock, a spinning scene and an animating canvas play for 8 s after they open and after a control, and again while the pointer is over the picture. Otherwise each draws once and rests. Under Reduce Motion nothing plays.
  • At most four GPU pictures at once. A newer visual moves the oldest’s to placeholders.
  • The control: the header’s Visuals toggle (“Script visuals”), or --script-visuals on|off (on unless named). Off, every visual shows its card.

Was this page helpful?