Skip to content
Vulpine
Esc
↑↓navigate↵open⌘Jpreview
On this page

Release and rollback

How to cut a gateway release, roll it to the gateway host and roll it back. just release folds the changelog, tags, builds and writes a receipt. just rollback <N> prints how to go back.

Living doc, checked against the code 2026-11-14

Neither recipe rolls anything: the roll is its own step.

How it works

The version. A release is a tag v<N>. N is one more than the highest v<N> tag. While no tag exists, N is one more than the live stamp: the VULPINE_OVERLAY_VERSION the last roll wrote. release.py reads it, read-only, with the installed gateway binary’s overlay-stdio status over ssh, or you pass it with --live N. The receipt’s version_source says which. The gateway host’s overlay number is the deploy counter: tag v504 is the binary that just overlay-roll-internal 504 503 ships. --version N overrides the derived number; the tool refuses a number at or behind the last tag. Other tags (fox9-<sha>-<slug>-<n>, the Admin deploys) are ignored.

The two deploy mechanisms. The gateway host’s agent is com.voidfab.fox9.gateway. Two mechanisms can own it, and both use the same version numbers:

Mechanism Who writes the LaunchAgent How a roll restarts it Used today
overlay (just overlay-roll / overlay-roll-internal: the gateway binary’s overlay-stdio subcommand, crates/overlay_stdio) the roll writes the plist from overlay.env (write_agent_plist) and stamps VULPINE_OVERLAY_VERSION in it bootout, a bounded wait for the old pid and the Admin socket, then bootstrap (five tries) yes
nix (modules/foxline.nix in the nix host configuration) nix-darwin, read-only (0444), marked VULPINE_AGENT_OWNER=<nix owner>; it execs the stable binary with --env-file gateway.env the roll stages the binary, config and gateway.env, then kickstart -k (start_nix_owned) no: the module declares no gateway

The first line of just overlay-status names the owner: the nix owner, or agent owner: overlay_stdio (plist written by the roll). The roll chooses the same way (agent_is_nix_owned), so the release and roll commands are the same for both; only the restart differs. The version number does not identify the binary. The build identity does: foxline-voice-gateway@0.1.0+<sha>, printed when the roll is ready and written as the first gateway_build row of every session trace.

The roll runs from the binary it ships. just overlay-roll-internal copies the signed binary to the gateway host and runs its own overlay-stdio roll --internal there (just/steps/overlay-roll-internal-rust.steps). just overlay-status, just overlay and just overlay-roll also run the installed binary’s subcommand; a binary without it exits 78.

Between the copy and the roll, the steps file runs the shipped binary’s roll-notice. On an idle session the live gateway says “Updating myself, back in a minute.” (at most 8 s, never a refusal). The marker it leaves makes the new gateway say “All systems back.” once its Health rows are green ([announcements] roll, crates/voice_gateway/src/roll_announce.rs).

The roll is the commit you build. build.rs stamps the binary from git status: <sha>-dirty when a tracked file differs. mm item update writes .memento/**, which is tracked, so a tracker append during the build stamps -dirty. The order is commit → build → roll, then tracker appends, docs and memory.

Check the commit before the roll. build.rs re-stamps only when HEAD moves, so a refused commit followed by a build ships the previous commit’s stamp. release.py checks for this:

  • It refuses a dirty tree before folding.
  • It checks the tree is clean again after the release commit.
  • After the build, it refuses an artifact that lacks the tagged commit’s short sha or holds <sha>-dirty (stamp_matches_commit in the receipt).

Run it

Release

  1. Start with the tree clean and on the commit you mean to ship: git status --short | wc -l prints 0. Read git log --oneline -1.
  2. just release --dry-run. It prints the version and where it came from, the fold (on a scratch copy), the tag and build commands it would run, and the receipt it would write (under a scratch prefix). It writes nothing in the checkout: git status --porcelain and git tag -l are hashed before and after, and the receipt shows both (checkout_unchanged: true).
  3. just release. In order:
    • tools/docs/changelog_fold.py folds changelog.d/. This is the same fold as just changelog-fold, called rather than copied.
    • Everything under ## Unreleased becomes ## v<N> (<date>), and ## Unreleased stays, empty.
    • It commits chore(release): v<N> - changelog folded. A release with no fragments adds only the heading, so this commit runs with HK_SKIP_STEPS=changelog-fragments.
    • It tags v<N> (annotated).
    • It runs just build-gateway.
    • It checks the stamp.
    • It writes benchmarks/results/rolls/roll-v<N>-<date>.json.
    • It prints the roll command and the archive command.
  4. Roll: just overlay-roll-internal <N> <N-1>. Compare the roll’s binary: identity= line with the tag’s commit.
    • Signing. The machine you roll from does not sign, unless it is the signing Mac (below). The build host signs with its Developer ID (tools/repo/remote_release.py: the host’s VULPINE_CODESIGN_IDENTITY, else its one Developer ID Application identity of the team in apps/glyph/src-tauri/bundle-identity.json; --identifier com.voidfab.fox9.gateway --timestamp=none). Never write the identity into the repo.
    • Attended. The host’s key is behind Touch ID. The roll prints one attended-sign: line naming the command to run in Terminal at the signing Mac, and waits VULPINE_SIGN_WAIT_MIN (20) minutes. No tool unlocks the keychain or asks for a password. When nothing is signed, the roll exits 5 before anything ships.
    • To check a binary first: python3 tools/repo/remote_release.py sign --check --binary "${CARGO_TARGET_DIR:-target}/release/vulpine-voice-gateway".
    • A roll on the signing Mac signs there only on an explicit signal: VULPINE_SIGN_HOST=local (=remote forbids it), else the marker file ~/.config/vulpine/sign-host holding that Mac’s paths.py host label, else a configured host name equal to the Mac’s hostname, LocalHostName or ComputerName.
    • Every ssh and rsync to the host has a 10 s connect timeout. just overlay-dry-run <N> <N-1> rehearses the roll on the host and changes nothing.
  5. Archive the binary so a later rollback can use it; the internal roll skips the volume’s versioned archive. The receipt’s archive_command is the line to run: scp target/release/vulpine-voice-gateway <gateway host>:<archive-dir>/foxline-voice-gateway-stdio-v<N>.
  6. After the roll, commit the receipt and the tracker appends. Push the tag with the branch: git push origin v<N>, with the output logged, never piped.

Rollback

  1. just rollback <N>, where N is the live version (just overlay-status). It names the highest tag below N and that tag’s commit. It prints:
    • just overlay-roll <prev> <N>, which rolls the archived foxline-voice-gateway-stdio-v<prev> from the volume;
    • the shasum to run on the gateway host first, with the expected sha256 from roll-v<prev>-*.json when that receipt exists.
  2. If that binary was never archived, rebuild the tag in a worktree and roll it under a new deploy number, just overlay-roll-internal <N+1> <N>, never as v<prev> again. One number must never name two binaries.
  3. The config does not roll back. The staged config and overlay.env stay as the v roll left them, and a key v added can stop v at parse. Before rolling, read git diff v<prev> v<N> -- crates/voice_gateway/src/config.rs and just overlay-config-check. If a roll’s bootstrap fails, read ~/.local/state/foxline/overlay-agent.log on the gateway host for a parse error before retrying.
  4. The gateway host’s checkout (loadouts, pi-config, the brain’s tools) does not roll back with the binary. After a revert, run just mini-sync-checkout.
  5. The brain’s Pi follows nix, not the roll. Where /run/current-system/sw/bin exists, the roll stages a /nix/store/<hash>-<pkg>/bin/<exe> [brain] pi_command as /run/current-system/sw/bin/<exe> when the system profile has that command, and collapses the PATH entries the profile covers. Any other overlay.env value of that shape, such as FOXLINE_QWEN_UV, becomes the resolved store path, because the gateway refuses a symlinked VULPINE_QWEN_UV (tts_adapter_failed). A darwin-rebuild switch that moves pi changes the brain’s Pi at its next start; a gateway rollback does not. To pin a version, set VULPINE_FOLLOW_SYSTEM_PROFILE=0 in the volume overlay.env and do a normal roll. Step 5 of the roll’s dry run lists what moves: just overlay-dry-run <N> <N-1> --normal.

Config

Flag Default Meaning
--dry-run off Write nothing in the checkout. The fold, the receipt and the folded changelog go under --scratch (default: a new temp dir).
--version N derived Release number. Refused when it is at or behind the last tag or the live stamp.
--live N read over ssh The live stamp, used only while no v<N> tag exists.
--host ALIAS tools/repo/paths.py host mini The gateway host, used in the roll commands and the ssh read.
--date YYYY-MM-DD today Section heading and receipt name.
--build-cmd, --artefact just build-gateway, target/release/vulpine-voice-gateway Overridable for tests.

The receipt (fox9.roll-receipt.v1) holds:

  • version, tag, version_source, previous, commit, dry_run;
  • changelog.fragments_folded and changelog.section;
  • build: the command, the artifact, its sha256, its bytes, stamp_matches_commit;
  • host, roll_command, archive_command, rolled: false;
  • who, build_host, when (UTC);
  • checkout_before, checkout_after, checkout_unchanged.

Every client refused after a roll from master: protocol lockstep

Symptom. The roll is up (listener ready on :8780, the identity line names your commit), but every client fails at hello: Vulpine.app, Vigil in a browser and the room speakers. The gateway answers a hello whose protocol_version is too low with error {code: unsupported_protocol_version} (validate_frontend_capabilities in crates/voice_gateway/src/ws/wire.rs). Client Core closes with protocol_handshake_failed: incompatible gateway hello when the gateway’s hello is not its own version. The room speaker’s bridge closes the same way.

Cause. The client wire protocol is pinned in both directions. Master is protocol 5: PROTOCOL_VERSION = 5 and MIN_PROTOCOL_VERSION = 5 in crates/protocol/src/lib.rs. Protocol 5 requires owner admission (hello.owner; a bind without an owner fails with owner_required), so a protocol-4 client cannot be admitted. Each client carries the version it was built with: Vulpine.app and Vigil through Client Core (packages/voice-client, an exact-equality check on the gateway’s hello), and the room speaker’s bridge through apps/satellite-esphome/src/gateway-link.ts, which takes PROTOCOL_VERSION = 5 from packages/protocol/src/wire.ts.

What ships with the roll. A roll whose binary changes the protocol version is a lockstep deploy. Rebuild every client from the same tree and ship it in the same sitting as the gateway:

  • Vulpine.app on every Mac that runs it (launch a new bundle on the gateway host from Finder the first time; a launchd first launch is killed by the launch constraint);
  • Vigil;
  • the bridge for each room speaker (apps/satellite-esphome).

The Admin in Vulpine Gateway.app speaks its own version over its socket (ADMIN_PROTOCOL_VERSION, 6) and is not part of this check. Inside protocol 5, every wire addition is an optional field or variant that older protocol-5 clients ignore. A roll that keeps the version needs no client rebuild. After the roll, vulpine-glyphctl owner on each Mac that runs Vulpine.app shows attached. To back out instead, roll the gateway back; a rollback across a version change needs the old clients too.

Failure modes

  • No v<N> tag and no ssh to the gateway host. The tool refuses. Pass --live N from just overlay-status, or --version N.
  • A dirty tree. The tool refuses before anything moves. Commit or remove the stray change. Untracked files do not count, because build.rs ignores them too.
  • The release commit is refused by a hook. Nothing is tagged. Fix the hook’s complaint, then reset the fold with git checkout -- CHANGELOG.md changelog.d.
  • The build fails, or the stamp does not match. The tag exists but nothing was built or receipted. Do not roll. Fix the build, then either rebuild and run the stamp check by hand, or delete the tag (git tag -d v<N>) and release again.
  • Rollback finds no earlier tag. This holds until two releases exist. Read the gateway host’s volume archive (foxline-voice-gateway-stdio-v<N>) and roll by hand with just overlay-roll <old> <live>.
  • The old gateway does not let go (exit 3, roll refused (old_gateway_lingered)). After the bootout the roll waits up to 30 s (OLD_GATEWAY_EXIT_BOUND_S) for the old pid to exit and for nothing to answer on the Admin socket. Past that it replaces nothing, bootstraps the old plist again (the old binary keeps serving) and exits 3. Find why the old process hung (a stalled disk, sample <pid>) before rolling again. The gateway’s own Admin bind retries a lost race for 10 s (ADMIN_SELF_PROBE_RETRY_WINDOW). It writes admin_listener_selfprobe_retry rows on the Admin timeline, then one admin_listener_selfprobe_failed if it gives up.

Was this page helpful?