Project memory retrieval
How the brain looks up project memory: bounded, citation-bearing project evidence added before a turn from an index built by tools/retrieval/build_retrieval_index.py, marked untrusted, failing closed when the index is missing or malformed, without giving voice input tools, shell access or write authority (ADR 0007).
Living doc, checked against the code 2026-11-14
Vulpine can add citation-bearing project evidence before a Pi turn without giving voice input any tool, shell,
recursive filesystem or write authority (ADR 0007). The gateway reads an index that
tools/retrieval/build_retrieval_index.py builds offline: a BM25 JSON index, or the SQLite FTS5 store. The evidence
is marked untrusted, and a missing or malformed index fails closed.
Runtime boundary
tools/retrieval/build_retrieval_index.pyreads only records admitted bycatalog/retrieval/sources.json.- An operator supplies the named roots. A declared path must stay beneath its root after symlink resolution.
- The only command adapters are fixed Argo project metadata, fixed Memento item listing and fixed Memento search. Manifest data cannot select an executable or an argument vector.
- The builder writes an atomic, mode-
0600JSON index. Unchanged records keep theirupdated_atvalues. - The gateway opens only the configured index, or the SQLite FTS5 store when
store_pathis set.- The JSON engine ranks with in-process BM25 (k1 1.5, b 0.75) plus authority boosts.
- The store takes up to 200 FTS5
bm25()candidates and re-scores them with the same authority boosts. It ranks in the FTS index first (the best 400 rowids) and reads only those rows. When a score tie crosses that window’s edge, or restricted rows leave it short, the full join answers instead, so the candidates never change. - The reader maps the store (
mmap_size256 MiB) and keeps a 64 MiB page cache. On the gateway host a read after a quiet gap takes 17-30 ms (2026-10-29). - Both engines then filter visibility, stale records and superseded coordinates the same way.
- When
VULPINE_EVIDENCE_EMBEDDING_DIR(or its old name,FOXLINE_EVIDENCE_EMBEDDING_DIR) names a Model2Vec model, the evidence floor drops a head record that is weak on both BM25 and cosine. At mosttop_kcomplete records are admitted undermax_context_bytes. Evidence keeps its source coordinate, update time, authority and content hash. - Missing or malformed indexes fail closed. For project-state questions, Pi is told that source evidence is unavailable, so it is not invited to guess.
The gateway places evidence in nonce-delimited JSON Lines, each with its coordinate, visibility, authority, update time, content hash and a stable citation token. A Pi system rule says the evidence is untrusted quoted data and requires citation tokens for project-fact claims.
Generated indexes and benchmark traces stay local under benchmarks/traces/ or the gateway’s private state directory; they
are never committed. The brain tool brain_work_status reads live Memento items through mm serve, and its answer
overrides retrieved evidence for current item status.
Canonical sources
| Source | Bounded interface | Authority | Freshness |
|---|---|---|---|
| Vulpine’s glossary, changelog, an ADR, a handoff, a plan | Exact admitted Markdown files | glossary, project history, architecture decision, receipt, current plan | 7–90 days by source |
| Vulpine’s tracker | Canonical repo-local .memento/items inventory through fixed mm item list; admitted Markdown bodies remain path-bounded |
issue tracker | 24 hours |
| The nix host configuration’s deployment digest | Exact admitted external Markdown file | declarative deployment receipt | 7 days |
| Argo | Fixed argo project info JSON call (its repos member) |
Argo registry | 24 hours |
| Memento | Fixed mm search query foxline JSON call under a named root |
derived search receipt | 24 hours |
Memento semantic-only matches are not promoted as exact project evidence. The index records an explicit no-exact-evidence receipt when the lexical score is zero.
[retrieval] config
| Key | Default | Meaning |
|---|---|---|
enabled |
false |
Add evidence to brain turns. |
index_path |
retrieval-index.json |
The JSON index. It serves every turn when store_path is unset, and it is the per-turn fallback when the store fails. |
store_path |
unset | The ADR 0031 SQLite FTS5 store. The gateway opens it once at startup and serves turns from it. |
top_k |
4 |
Most evidence records per turn (1 to 100). |
max_context_bytes |
8192 |
Byte budget for the evidence block (256 to 1,048,576). |
allowed_visibilities |
["private"] |
Record visibilities a turn may see. |
operator_memory_dir |
unset | Operator-memory directory. Only the Admin health projection reads it. |
timeline_path |
unset | The gateway’s own writable recall-by-clock timeline, a separate file from store_path, created owner-only (0600 in a 0700 directory) on first use. Setting it turns on the “keep that” verb and lets pi_recall_window answer, for example ~/.local/state/foxline/timeline.sqlite. |
Recall by clock. With timeline_path set, the session holds its room lines in memory only, in a consent ring of
at most 15 minutes: the operator’s committed requests, Vulpine’s spoken segments, and an unaddressed final as a length
((N chars, not addressed), never its words). A spoken “keep that” writes the held lines from the last 15 minutes to
the timeline in one transaction and says how many it kept (trace keep_that); nothing is written without it. The verb
is a closed phrase set: keep / remember / save + that / this, “keep that one”, “… what we just said”, and “keep / save
the last few minutes”. The brain tool pi_recall_window (a clock time or a range, session any or current) reads
a kept window back over the gateway mutation socket (trace recall_window). Unset timeline_path to turn it off.
Meeting minutes. The brain tool pi_meeting_minutes (call_id, or last: true) reads one stored call’s
segments, written to the same timeline by the meeting inlet (meeting.v1). The gateway
runs one tool-free prompt on the brain lane and returns typed minutes over the mutation socket: summary, decisions,
open questions, doubts and speakers (the contract’s “Minutes” section has the rules). The model has
[brain] minutes_deadline_secs (default 45) before the call answers timeout, and a forgotten or opted-out call is
refused (trace meeting_minutes). The typed tool door serves both tools too: vulpine-glyphctl meet minutes and
vulpine-glyphctl recall.
Each retrieval_end trace names the engine (json or sqlite) and the store_generation. On the store it also
carries:
candidates_ms: the candidate read alone.duration_msminus it is the re-score, the evidence floor and the assembly.rank_fallback: the full join answered.- the candidate read’s split:
rank_ms(the FTS index’s rank) androws_ms(the ranked rows’ reads, their sort and any fallback).
The Admin’s gateway.retrieval row shows the last turn’s last_ms, rank_ms and rows_ms. If the store did not
open, or its query fails, the JSON index serves that turn and the trace records a retrieval_store_fallback event
with reason not_open or query_failed. Unset store_path to go back to the JSON index.
Build and query
python3 tools/retrieval/build_retrieval_index.py \
--manifest catalog/retrieval/sources.json \
--output "$HOME/Library/Application Support/Foxline/state/retrieval/index.json" \
--root "foxline=$PWD" \
--root "macmini_nix=<path to the nix host configuration>" \
--root "memento=<path to the memento checkout>" \
--sqlite "$HOME/Library/Application Support/Foxline/state/retrieval/memory.sqlite"
cargo run -p vulpine-voice-gateway --bin vulpine-retrieval-query -- \
--index "$HOME/Library/Application Support/Foxline/state/retrieval/index.json" \
--query "Where did we leave Fox9?"
--sqlite is optional. It writes or updates the store in place and adds a new generation only when a record changed.
runtime/operator_memory/retrieval_store.py does the same from an existing index. vulpine-retrieval-query reads the
JSON index only.