Release and rollback
How to cut a gateway release, roll it to the gateway host and roll it back. just release folds the changelog, tags, builds and writes a receipt. just rollback <N> prints how to go back.
Living doc, checked against the code 2026-11-14
Neither recipe rolls anything: the roll is its own step.
How it works
The version. A release is a tag v<N>. N is one more than the highest v<N> tag. While no tag exists, N is one
more than the live stamp: the VULPINE_OVERLAY_VERSION the last roll wrote. release.py reads it, read-only, with the
installed gateway binary’s overlay-stdio status over ssh, or you pass it with --live N. The receipt’s
version_source says which. The gateway host’s overlay number is the deploy counter: tag v504 is the binary that
just overlay-roll-internal 504 503 ships. --version N overrides the derived number; the tool refuses a number at
or behind the last tag. Other tags (fox9-<sha>-<slug>-<n>, the Admin deploys) are ignored.
The two deploy mechanisms. The gateway host’s agent is com.voidfab.fox9.gateway. Two mechanisms can own it, and
both use the same version numbers:
| Mechanism | Who writes the LaunchAgent | How a roll restarts it | Used today |
|---|---|---|---|
overlay (just overlay-roll / overlay-roll-internal: the gateway binary’s overlay-stdio subcommand, crates/overlay_stdio) |
the roll writes the plist from overlay.env (write_agent_plist) and stamps VULPINE_OVERLAY_VERSION in it |
bootout, a bounded wait for the old pid and the Admin socket, then bootstrap (five tries) | yes |
nix (modules/foxline.nix in the nix host configuration) |
nix-darwin, read-only (0444), marked VULPINE_AGENT_OWNER=<nix owner>; it execs the stable binary with --env-file gateway.env |
the roll stages the binary, config and gateway.env, then kickstart -k (start_nix_owned) |
no: the module declares no gateway |
The first line of just overlay-status names the owner: the nix owner, or
agent owner: overlay_stdio (plist written by the roll). The roll chooses the same way (agent_is_nix_owned), so the
release and roll commands are the same for both; only the restart differs. The version number does not identify the
binary. The build identity does: foxline-voice-gateway@0.1.0+<sha>, printed when the roll is ready and written as
the first gateway_build row of every session trace.
The roll runs from the binary it ships. just overlay-roll-internal copies the signed binary to the gateway host
and runs its own overlay-stdio roll --internal there (just/steps/overlay-roll-internal-rust.steps). just overlay-status,
just overlay and just overlay-roll also run the installed binary’s subcommand; a binary without it exits 78.
Between the copy and the roll, the steps file runs the shipped binary’s roll-notice. On an idle session the live
gateway says “Updating myself, back in a minute.” (at most 8 s, never a refusal). The marker it leaves makes the new
gateway say “All systems back.” once its Health rows are green ([announcements] roll,
crates/voice_gateway/src/roll_announce.rs).
The roll is the commit you build. build.rs stamps the binary from git status: <sha>-dirty when a tracked
file differs. mm item update writes .memento/**, which is tracked, so a tracker append during the build stamps
-dirty. The order is commit → build → roll, then tracker appends, docs and memory.
Check the commit before the roll. build.rs re-stamps only when HEAD moves, so a refused commit followed by a
build ships the previous commit’s stamp. release.py checks for this:
- It refuses a dirty tree before folding.
- It checks the tree is clean again after the release commit.
- After the build, it refuses an artifact that lacks the tagged commit’s short sha or holds
<sha>-dirty(stamp_matches_commitin the receipt).
Run it
Release
- Start with the tree clean and on the commit you mean to ship:
git status --short | wc -lprints 0. Readgit log --oneline -1. just release --dry-run. It prints the version and where it came from, the fold (on a scratch copy), the tag and build commands it would run, and the receipt it would write (under a scratch prefix). It writes nothing in the checkout:git status --porcelainandgit tag -lare hashed before and after, and the receipt shows both (checkout_unchanged: true).just release. In order:tools/docs/changelog_fold.pyfoldschangelog.d/. This is the same fold asjust changelog-fold, called rather than copied.- Everything under
## Unreleasedbecomes## v<N> (<date>), and## Unreleasedstays, empty. - It commits
chore(release): v<N> - changelog folded. A release with no fragments adds only the heading, so this commit runs withHK_SKIP_STEPS=changelog-fragments. - It tags
v<N>(annotated). - It runs
just build-gateway. - It checks the stamp.
- It writes
benchmarks/results/rolls/roll-v<N>-<date>.json. - It prints the roll command and the archive command.
- Roll:
just overlay-roll-internal <N> <N-1>. Compare the roll’sbinary: identity=line with the tag’s commit.- Signing. The machine you roll from does not sign, unless it is the signing Mac (below). The build host signs
with its Developer ID (
tools/repo/remote_release.py: the host’sVULPINE_CODESIGN_IDENTITY, else its one Developer ID Application identity of the team inapps/glyph/src-tauri/bundle-identity.json;--identifier com.voidfab.fox9.gateway --timestamp=none). Never write the identity into the repo. - Attended. The host’s key is behind Touch ID. The roll prints one
attended-sign:line naming the command to run in Terminal at the signing Mac, and waitsVULPINE_SIGN_WAIT_MIN(20) minutes. No tool unlocks the keychain or asks for a password. When nothing is signed, the roll exits 5 before anything ships. - To check a binary first:
python3 tools/repo/remote_release.py sign --check --binary "${CARGO_TARGET_DIR:-target}/release/vulpine-voice-gateway". - A roll on the signing Mac signs there only on an explicit signal:
VULPINE_SIGN_HOST=local(=remoteforbids it), else the marker file~/.config/vulpine/sign-hostholding that Mac’spaths.pyhost label, else a configured host name equal to the Mac’s hostname, LocalHostName or ComputerName. - Every ssh and rsync to the host has a 10 s connect timeout.
just overlay-dry-run <N> <N-1>rehearses the roll on the host and changes nothing.
- Signing. The machine you roll from does not sign, unless it is the signing Mac (below). The build host signs
with its Developer ID (
- Archive the binary so a later rollback can use it; the internal roll skips the volume’s versioned archive. The
receipt’s
archive_commandis the line to run:scp target/release/vulpine-voice-gateway <gateway host>:<archive-dir>/foxline-voice-gateway-stdio-v<N>. - After the roll, commit the receipt and the tracker appends. Push the tag with the branch:
git push origin v<N>, with the output logged, never piped.
Rollback
just rollback <N>, where N is the live version (just overlay-status). It names the highest tag below N and that tag’s commit. It prints:just overlay-roll <prev> <N>, which rolls the archivedfoxline-voice-gateway-stdio-v<prev>from the volume;- the
shasumto run on the gateway host first, with the expected sha256 fromroll-v<prev>-*.jsonwhen that receipt exists.
- If that binary was never archived, rebuild the tag in a worktree and roll it under a new deploy number,
just overlay-roll-internal <N+1> <N>, never asv<prev>again. One number must never name two binaries. - The config does not roll back. The staged config and
overlay.envstay as the vroll left them, and a key v added can stop v at parse. Before rolling, read git diff v<prev> v<N> -- crates/voice_gateway/src/config.rsandjust overlay-config-check. If a roll’s bootstrap fails, read~/.local/state/foxline/overlay-agent.logon the gateway host for a parse error before retrying. - The gateway host’s checkout (loadouts, pi-config, the brain’s tools) does not roll back with the binary. After a
revert, run
just mini-sync-checkout. - The brain’s Pi follows nix, not the roll. Where
/run/current-system/sw/binexists, the roll stages a/nix/store/<hash>-<pkg>/bin/<exe>[brain] pi_commandas/run/current-system/sw/bin/<exe>when the system profile has that command, and collapses thePATHentries the profile covers. Any otheroverlay.envvalue of that shape, such asFOXLINE_QWEN_UV, becomes the resolved store path, because the gateway refuses a symlinkedVULPINE_QWEN_UV(tts_adapter_failed). Adarwin-rebuild switchthat movespichanges the brain’s Pi at its next start; a gateway rollback does not. To pin a version, setVULPINE_FOLLOW_SYSTEM_PROFILE=0in the volumeoverlay.envand do a normal roll. Step 5 of the roll’s dry run lists what moves:just overlay-dry-run <N> <N-1> --normal.
Config
| Flag | Default | Meaning |
|---|---|---|
--dry-run |
off | Write nothing in the checkout. The fold, the receipt and the folded changelog go under --scratch (default: a new temp dir). |
--version N |
derived | Release number. Refused when it is at or behind the last tag or the live stamp. |
--live N |
read over ssh | The live stamp, used only while no v<N> tag exists. |
--host ALIAS |
tools/repo/paths.py host mini |
The gateway host, used in the roll commands and the ssh read. |
--date YYYY-MM-DD |
today | Section heading and receipt name. |
--build-cmd, --artefact |
just build-gateway, target/release/vulpine-voice-gateway |
Overridable for tests. |
The receipt (fox9.roll-receipt.v1) holds:
version,tag,version_source,previous,commit,dry_run;changelog.fragments_foldedandchangelog.section;build: the command, the artifact, itssha256, itsbytes,stamp_matches_commit;host,roll_command,archive_command,rolled: false;who,build_host,when(UTC);checkout_before,checkout_after,checkout_unchanged.
Every client refused after a roll from master: protocol lockstep
Symptom. The roll is up (listener ready on :8780, the identity line names your commit), but every client fails
at hello: Vulpine.app, Vigil in a browser and the room speakers. The gateway answers a hello whose protocol_version is too low
with error {code: unsupported_protocol_version} (validate_frontend_capabilities in
crates/voice_gateway/src/ws/wire.rs). Client Core closes with protocol_handshake_failed: incompatible gateway hello when the gateway’s hello is not its own version. The room speaker’s bridge closes the same way.
Cause. The client wire protocol is pinned in both directions. Master is protocol 5: PROTOCOL_VERSION = 5 and
MIN_PROTOCOL_VERSION = 5 in crates/protocol/src/lib.rs. Protocol 5 requires owner admission (hello.owner; a
bind without an owner fails with owner_required), so a protocol-4 client cannot be admitted. Each client carries
the version it was built with: Vulpine.app and Vigil through Client Core (packages/voice-client, an
exact-equality check on the gateway’s hello), and the room speaker’s bridge through
apps/satellite-esphome/src/gateway-link.ts, which takes PROTOCOL_VERSION = 5 from packages/protocol/src/wire.ts.
What ships with the roll. A roll whose binary changes the protocol version is a lockstep deploy. Rebuild every client from the same tree and ship it in the same sitting as the gateway:
- Vulpine.app on every Mac that runs it (launch a new bundle on the gateway host from Finder the first time; a launchd first launch is killed by the launch constraint);
- Vigil;
- the bridge for each room speaker (
apps/satellite-esphome).
The Admin in Vulpine Gateway.app speaks its own version over its socket (ADMIN_PROTOCOL_VERSION, 6) and is not part
of this check. Inside protocol 5, every wire addition is an optional field or variant that older protocol-5 clients
ignore. A roll that keeps the version needs no client rebuild. After the roll, vulpine-glyphctl owner on each Mac
that runs Vulpine.app shows attached. To back out instead, roll the gateway back; a rollback across a version change needs the old clients too.
Failure modes
- No
v<N>tag and no ssh to the gateway host. The tool refuses. Pass--live Nfromjust overlay-status, or--version N. - A dirty tree. The tool refuses before anything moves. Commit or remove the stray change. Untracked files do not
count, because
build.rsignores them too. - The release commit is refused by a hook. Nothing is tagged. Fix the hook’s complaint, then reset the fold with
git checkout -- CHANGELOG.md changelog.d. - The build fails, or the stamp does not match. The tag exists but nothing was built or receipted. Do not roll.
Fix the build, then either rebuild and run the stamp check by hand, or delete the tag (
git tag -d v<N>) and release again. - Rollback finds no earlier tag. This holds until two releases exist. Read the gateway host’s volume archive
(
foxline-voice-gateway-stdio-v<N>) and roll by hand withjust overlay-roll <old> <live>. - The old gateway does not let go (exit 3,
roll refused (old_gateway_lingered)). After the bootout the roll waits up to 30 s (OLD_GATEWAY_EXIT_BOUND_S) for the old pid to exit and for nothing to answer on the Admin socket. Past that it replaces nothing, bootstraps the old plist again (the old binary keeps serving) and exits 3. Find why the old process hung (a stalled disk,sample <pid>) before rolling again. The gateway’s own Admin bind retries a lost race for 10 s (ADMIN_SELF_PROBE_RETRY_WINDOW). It writesadmin_listener_selfprobe_retryrows on the Admin timeline, then oneadmin_listener_selfprobe_failedif it gives up.