Speech recognition
The speech to text that runs on the Apple Neural Engine: a Rust runner for Parakeet TDT 0.6B v3 through CoreML, the encoder on the Neural Engine and the decode loop on the CPU.
Living doc, checked against the code 2026-11-14
coreml_stt — Parakeet TDT 0.6B v3 on the Apple Neural Engine
A clean-room Rust runner for Parakeet TDT 0.6B v3 through Apple CoreML. The acoustic encoder runs on the Neural Engine and the transducer decode loop on the CPU. It uses no GPU ops, so STT does not contend with TTS and end-of-utterance detection for the GPU during barge-in.
use coreml_stt::{ComputeUnits, Recognizer};
let mut rec = Recognizer::load(bundle_dir, ComputeUnits::CpuAndNeuralEngine)?;
let out = rec.transcribe(&pcm_16k_mono_f32)?;
println!("{}", out.text);
for t in &out.tokens {
println!("{:.2}-{:.2} {:?}", t.start_s, t.end_s, t.text);
}
CLI:
fox9-coreml-stt --bundle <dir> transcribe <wav...> [--json]
fox9-coreml-stt --bundle <dir> serve
fox9-coreml-stt --bundle <dir> partials <wav...> [--interval-ms 500] [--first-tick-ms N] [--texts] [--json]
fox9-coreml-stt --bundle <dir> placement
fox9-coreml-stt --bundle <dir> bench --clip <wav> --warm 5
--compute-units cpu_only|cpu_and_neural_engine|cpu_and_gpu|all goes before
the command.
transcribe --no-seam-gap-repairturns gap repair off.--silence-aligned-windowsstarts each window at the quietest frame near the hop.--zero-token-retryre-decodes a window that reads nothing over speech as two halves (off by default).partialsreplays a clip as an open segment and commits words the waycommit.rsdoes.--first-tick-msplaces the first tick. The live segmenter ticks first at pre-roll + its 600 ms minimum segment.--textsadds three things to the--jsonrow: each tick’s committed prefix, the first non-empty one (the first partial a router reads) and the final text.
serve (src/serve.rs) loads the bundle once, then decodes JSON-line
requests on stdin: {id, wav_path | pcm_f32le_path, sample_rate}.
- A
{"op":"ready","protocol":1,"load_ms":…}line comes first. - Each request gets one line, in order:
{id, text, words[{w, t0, t1}], decode_ms, audio_s}, or{id, error{kind, message}}withkindone ofbad_request,unsupported_sample_rate,audio_unreadable,decode_failed. {"op":"quit"}or end of input ends it.
The decode is transcribe’s, with gap repair on. meetctl (tools/meet)
decodes a call’s windows on one serve child per run by default.
Bundle it expects
Recognizer::load takes either the bundle root (containing parakeet-v3/) or
parakeet-v3/ itself:
| file | in | out |
|---|---|---|
preprocessor.mlmodelc |
audio_signal [1,240000], audio_length [1] |
mel [1,128,1501], mel_length [1] |
encoder.mlmodelc |
mel, mel_length |
encoder [1,1024,188], encoder_length [1] |
decoder.mlmodelc |
targets [1,1], target_length [1], h_in/c_in [2,1,640] |
decoder [1,640,1], h_out, c_out |
joint-decision.mlmodelc |
encoder_step [1,1024,1], decoder_step [1,640,1] |
token_id, token_prob, duration |
parakeet_vocab.json |
— | {"<id>": "<piece>"} × 8192 |
Set VULPINE_COREML_STT_ENCODER to swap the encoder file (e.g.
encoder-lut6.mlmodelc) without a rebuild.
The vocabulary has 8192 entries and no explicit blank token: blank is index
8192 (Vocab::blank_id returns len). A len - 1 blank silently corrupts
every decode.
The front end is CoreML too. preprocessor.mlmodelc maps raw 16 kHz
audio straight to the encoder’s [1,128,1501] mel input, so there is no
hand-written mel to disagree with the encoder’s training. The upstream repo’s
MelEncoder.mlmodelc is a fused mel+encoder (595 MB), not a front end; this
crate does not use it.
Weights license — attribution required
The models are not part of this crate and are not vendored. They come from
FluidInference/parakeet-tdt-0.6b-v3-coreml
under CC-BY-4.0, converted from
nvidia/parakeet-tdt-0.6b-v3,
also CC-BY-4.0.
Speech recognition uses Parakeet TDT 0.6B v3 by NVIDIA, licensed CC-BY-4.0, converted to CoreML by Fluid Inference.
That attribution must travel with anything that ships or serves these weights.
The upstream README’s prose says “Apache 2.0”, but its model-card front
matter, its HF tag and the base model say cc-by-4.0. Treat CC-BY-4.0 as
binding. The crate’s own code is MIT, like the rest of the workspace.
Measured numbers
On an M3 MacBook Air (16 GB, cpu_and_neural_engine, the int8 Encoder_v2),
1376 of the encoder’s 3398 ops run on the ANE and none on the GPU. Pooled WER
over 140 reference clips is 10.07 %, and peak host RSS is 0.074 GB.
Timestamps
TokenSpan::end_s is start_s + duration × 80 ms, the model’s own
predicted duration for that token. A span ends where the token ends, so blank
frames are left out; it does not run to the next emission. One encoder frame
is 80 ms (10 ms mel hop × 8 subsampling).
Long form
The encoder’s window is a fixed 240 000 samples (15 s). Longer audio is cut
into 15 s windows overlapping by 1 s. At each seam the two windows are
stitched on the longest token run in the overlap whose clocks agree within
0.5 s, so a word straddling the cut survives exactly once. With no agreeing
run, the earlier window owns audio up to the midpoint of the overlap. Gap
repair is on by default: any gap over 1.5 s between tokens is decoded again in
a fresh window centered on it. chunk.rs holds the rules.
Caveats
- Host RSS is not the whole cost. 0.074 GB is the host process, without CoreML/ANE weight residency. Budget ~0.7 GB for the system, and budget the first-compile peak, not the steady state.
- The first ever load pays a full ANE compile. CoreML caches compiled
models per executable name under
~/Library/Caches/<executable>/com.apple.e5rt.e5bundlecache/<os build>/, so a rename or an OS update re-pays it. Pre-warm at deploy, never at the first utterance. - macOS only; the library is a no-op stub elsewhere.