ADR-045: On-device retrieval-model support policy (prefix, MRL, sparse, licensing)

Date: 2026-07-06 Status: Proposed Author: Rômulo Febasi

Scopes SYN-307 (retrieval-model options, v1.3+). Extends ADR-008 (BGE-M3 via fastembed + sqlite-vec) and ADR-042 (chunked embeddings + reranker). Motivated by a 2026 model-landscape pass (HuggingFace MTEB, trending sub-500M multilingual encoders) against Synapse’s hard local-first + permissive-licence constraints.

Context

The default embedding backend is BGE-M3 (dense, 1024-d, ~2.2 GB fp32 / ~2.5 GB with the ORT runtime; int8 onboarding option, SYN-237), served through fastembed on the shared ort copy (ADR-008/038). A field pass surfaced newer sub-500M multilingual encoders and index-shrinking techniques that could cut footprint and cold-start without a new native backend, but each carries a license, quality, or determinism catch that has to be gated, not assumed:

• multilingual-e5-small/base (MIT, ~120–280 MB): strong for the size, but E5 is asymmetric: it requires query: / passage: prefixes, and fastembed does not auto-prefix them. Shipping E5 through the current symmetric embed path would silently degrade quality, a real code change (an asymmetric embed seam), not a config swap.
• EmbeddingGemma-300M (best-in-class sub-500M multilingual on MTEB, ~300 MB) , but Gemma-licensed and gated on HuggingFace: not a drop-in permissive dep, and auto-download behind a gate is a distribution problem.
• Qwen3-Embedding-0.6B: competitive, but no maintained ONNX export yet, so it does not run on the ort/fastembed substrate without a backend we have ruled out adding.
• Matryoshka / MRL truncation: a dense vector trained with MRL can be truncated (e.g. 1024→256) for a ~4× smaller index at a small recall cost. BGE-M3 is not MRL-trained, so this only unlocks with an MRL-capable model.
• BGE-M3 sparse output: the model already emits a learned-sparse (SPLADE- like) representation that Synapse currently discards; it is the one honest learned-sparse leg available under a permissive licence (SPLADE proper is non-commercial: rejected). Adding it is a retrieval-quality spike, not a model swap.

Decision (proposed: evidence-gated)

**Adopt a licence-and-determinism-gated policy for on-device retrieval models; add options only behind a benchmark win and only when they respect local-first

• permissive-licence + deterministic constraints. No default change without a measured win.**

1. Permissive-licence gate. A shippable default/opt-in model must be permissively licensed (MIT/Apache-class) and auto-downloadable without a gated-repo token. Reject SPLADE (non-commercial). Gemma-licensed models (EmbeddingGemma) are opt-in-only and never the auto-downloaded default: the licence + gating make them a documented alternative, not the spine.
2. ONNX/ort substrate gate. A candidate must run on the existing ort/fastembed copy (ADR-008/038). No second native backend for a retrieval model without its own ADR. Qwen3-Embedding is deferred until a maintained ONNX export exists.
3. Asymmetric embed seam for E5-class models. Supporting E5 (multilingual-e5-*) requires a real feature: an asymmetric embed path that prefixes query: on the query and passage: on stored chunks. Ship E5 only with that seam and a persona-bench --features real win: never as a bare fastembed enum swap (which would degrade silently).
4. MRL truncation is a model property, not a global knob. Offer dimension truncation only for an MRL-trained model, as a per-model bench-gated option (index size vs recall@10). BGE-M3 is exempt (not MRL-trained).
5. BGE-M3 sparse leg = a separate spike, its own ADR. Wiring the discarded sparse output into a fourth retrieval leg (or replacing FTS/BM25) is a retrieval-quality experiment gated by the ADR-042 discipline: tracked, not bundled here.

Every model addition is disposable-index only (a model swap is a reindex, never a source-of-truth change) and benchmark-gated (ship on a measured recall@1/footprint win with no recall@10 regression).

Consequences

• Improves: a clear, defensible path to a smaller/faster footprint (E5-small ~120 MB vs BGE-M3 ~2.2 GB) and a smaller index (MRL): if the bench earns it. Keeps the door open to best-in-class encoders without betraying local-first or permissive-licence promises.
• Worsens / limits: more model plumbing (asymmetric seam, per-model dims); the most capable sub-500M option (EmbeddingGemma) stays opt-in behind a licence gate, so it cannot be the friction-free default; Qwen3 is blocked on upstream ONNX.
• Open: whether E5-small’s quality at a fraction of the footprint justifies the asymmetric seam; the sparse-leg fusion weight (composes ADR-044); whether to make a smaller model the default or keep BGE-M3 default with smaller models opt-in (bench + footprint call).

Alternatives considered

• Swap the default to a sub-500M model now. Rejected as premature: no bench on Synapse’s own persona-bench, and E5 needs the prefix seam first, a swap today would be an unmeasured quality regression.
• Add a second native backend (e.g. Candle) for Qwen3. Rejected: a native ML backend is a heavy dep requiring its own ADR; the ort/fastembed substrate is the ADR-008 decision, and Qwen3 will likely land an ONNX export.
• Adopt SPLADE for learned-sparse. Rejected: non-commercial licence. The BGE-M3 sparse output is the permissive alternative already in the binary.
• Ship EmbeddingGemma as the default. Rejected: Gemma licence + gated repo make auto-download a distribution/licence problem; it stays a documented opt-in, not the spine.