Skip to content
Vulpine
Esc
↑↓navigate↵open⌘Jpreview
On this page

Project memory retrieval

How the brain looks up project memory: bounded, citation-bearing project evidence added before a turn from an index built by tools/retrieval/build_retrieval_index.py, marked untrusted, failing closed when the index is missing or malformed, without giving voice input tools, shell access or write authority (ADR 0007).

Living doc, checked against the code 2026-11-14

Vulpine can add citation-bearing project evidence before a Pi turn without giving voice input any tool, shell, recursive filesystem or write authority (ADR 0007). The gateway reads an index that tools/retrieval/build_retrieval_index.py builds offline: a BM25 JSON index, or the SQLite FTS5 store. The evidence is marked untrusted, and a missing or malformed index fails closed.

Runtime boundary

  1. tools/retrieval/build_retrieval_index.py reads only records admitted by catalog/retrieval/sources.json.
  2. An operator supplies the named roots. A declared path must stay beneath its root after symlink resolution.
  3. The only command adapters are fixed Argo project metadata, fixed Memento item listing and fixed Memento search. Manifest data cannot select an executable or an argument vector.
  4. The builder writes an atomic, mode-0600 JSON index. Unchanged records keep their updated_at values.
  5. The gateway opens only the configured index, or the SQLite FTS5 store when store_path is set.
    • The JSON engine ranks with in-process BM25 (k1 1.5, b 0.75) plus authority boosts.
    • The store takes up to 200 FTS5 bm25() candidates and re-scores them with the same authority boosts. It ranks in the FTS index first (the best 400 rowids) and reads only those rows. When a score tie crosses that window’s edge, or restricted rows leave it short, the full join answers instead, so the candidates never change.
    • The reader maps the store (mmap_size 256 MiB) and keeps a 64 MiB page cache. On the gateway host a read after a quiet gap takes 17-30 ms (2026-10-29).
    • Both engines then filter visibility, stale records and superseded coordinates the same way.
  6. When VULPINE_EVIDENCE_EMBEDDING_DIR (or its old name, FOXLINE_EVIDENCE_EMBEDDING_DIR) names a Model2Vec model, the evidence floor drops a head record that is weak on both BM25 and cosine. At most top_k complete records are admitted under max_context_bytes. Evidence keeps its source coordinate, update time, authority and content hash.
  7. Missing or malformed indexes fail closed. For project-state questions, Pi is told that source evidence is unavailable, so it is not invited to guess.

The gateway places evidence in nonce-delimited JSON Lines, each with its coordinate, visibility, authority, update time, content hash and a stable citation token. A Pi system rule says the evidence is untrusted quoted data and requires citation tokens for project-fact claims.

Generated indexes and benchmark traces stay local under benchmarks/traces/ or the gateway’s private state directory; they are never committed. The brain tool brain_work_status reads live Memento items through mm serve, and its answer overrides retrieved evidence for current item status.

Canonical sources

Source Bounded interface Authority Freshness
Vulpine’s glossary, changelog, an ADR, a handoff, a plan Exact admitted Markdown files glossary, project history, architecture decision, receipt, current plan 7–90 days by source
Vulpine’s tracker Canonical repo-local .memento/items inventory through fixed mm item list; admitted Markdown bodies remain path-bounded issue tracker 24 hours
The nix host configuration’s deployment digest Exact admitted external Markdown file declarative deployment receipt 7 days
Argo Fixed argo project info JSON call (its repos member) Argo registry 24 hours
Memento Fixed mm search query foxline JSON call under a named root derived search receipt 24 hours

Memento semantic-only matches are not promoted as exact project evidence. The index records an explicit no-exact-evidence receipt when the lexical score is zero.

[retrieval] config

Key Default Meaning
enabled false Add evidence to brain turns.
index_path retrieval-index.json The JSON index. It serves every turn when store_path is unset, and it is the per-turn fallback when the store fails.
store_path unset The ADR 0031 SQLite FTS5 store. The gateway opens it once at startup and serves turns from it.
top_k 4 Most evidence records per turn (1 to 100).
max_context_bytes 8192 Byte budget for the evidence block (256 to 1,048,576).
allowed_visibilities ["private"] Record visibilities a turn may see.
operator_memory_dir unset Operator-memory directory. Only the Admin health projection reads it.
timeline_path unset The gateway’s own writable recall-by-clock timeline, a separate file from store_path, created owner-only (0600 in a 0700 directory) on first use. Setting it turns on the “keep that” verb and lets pi_recall_window answer, for example ~/.local/state/foxline/timeline.sqlite.

Recall by clock. With timeline_path set, the session holds its room lines in memory only, in a consent ring of at most 15 minutes: the operator’s committed requests, Vulpine’s spoken segments, and an unaddressed final as a length ((N chars, not addressed), never its words). A spoken “keep that” writes the held lines from the last 15 minutes to the timeline in one transaction and says how many it kept (trace keep_that); nothing is written without it. The verb is a closed phrase set: keep / remember / save + that / this, “keep that one”, “… what we just said”, and “keep / save the last few minutes”. The brain tool pi_recall_window (a clock time or a range, session any or current) reads a kept window back over the gateway mutation socket (trace recall_window). Unset timeline_path to turn it off.

Meeting minutes. The brain tool pi_meeting_minutes (call_id, or last: true) reads one stored call’s segments, written to the same timeline by the meeting inlet (meeting.v1). The gateway runs one tool-free prompt on the brain lane and returns typed minutes over the mutation socket: summary, decisions, open questions, doubts and speakers (the contract’s “Minutes” section has the rules). The model has [brain] minutes_deadline_secs (default 45) before the call answers timeout, and a forgotten or opted-out call is refused (trace meeting_minutes). The typed tool door serves both tools too: vulpine-glyphctl meet minutes and vulpine-glyphctl recall.

Each retrieval_end trace names the engine (json or sqlite) and the store_generation. On the store it also carries:

  • candidates_ms: the candidate read alone. duration_ms minus it is the re-score, the evidence floor and the assembly.
  • rank_fallback: the full join answered.
  • the candidate read’s split: rank_ms (the FTS index’s rank) and rows_ms (the ranked rows’ reads, their sort and any fallback).

The Admin’s gateway.retrieval row shows the last turn’s last_ms, rank_ms and rows_ms. If the store did not open, or its query fails, the JSON index serves that turn and the trace records a retrieval_store_fallback event with reason not_open or query_failed. Unset store_path to go back to the JSON index.

Build and query

python3 tools/retrieval/build_retrieval_index.py \
  --manifest catalog/retrieval/sources.json \
  --output "$HOME/Library/Application Support/Foxline/state/retrieval/index.json" \
  --root "foxline=$PWD" \
  --root "macmini_nix=<path to the nix host configuration>" \
  --root "memento=<path to the memento checkout>" \
  --sqlite "$HOME/Library/Application Support/Foxline/state/retrieval/memory.sqlite"

cargo run -p vulpine-voice-gateway --bin vulpine-retrieval-query -- \
  --index "$HOME/Library/Application Support/Foxline/state/retrieval/index.json" \
  --query "Where did we leave Fox9?"

--sqlite is optional. It writes or updates the store in place and adds a new generation only when a record changed. runtime/operator_memory/retrieval_store.py does the same from an existing index. vulpine-retrieval-query reads the JSON index only.

Was this page helpful?