Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

ADR-008: Embedding backend, vector storage and NER backend for v0.5

Date: 2026-06-02 Status: Accepted (locks the v0.5 brief decisions; consumed by SYN-81 + SYN-83 + SYN-164; supersedes the v0.4-era placeholder reservation) Author: Rômulo Febasi

Context

Slot 008 was reserved at the start of the v0.1 cycle for “the embedding backend choice” with the explicit caveat that it would be filled “after a benchmark spike, when v0.5 lands”. The v0.5 brief (docs/researchs/2026-06-02-v0.5-semantic-layer-and-second-brain.md) decided the four load-bearing questions on 2026-06-02:

  1. Embedding inference stays on-device. The local-first invariant from BR-1.4 and BR-12 binds; Ollama as a runner is a configurable alternative, not the default.
  2. Vector storage lives in the same .synapse/index.db to keep BR-1.5 (“one workspace, one source of truth”) intact.
  3. Hybrid ranking uses RRF in v0.5: no labels yet, no place for convex combination.
  4. Entity linking at ingestion is mandatory (BR-13). It needs a small zero-shot NER model so that pt-BR-grammatical content like “reunião com a Carina sobre a flag do Remote Config” does not require the user to pre-create every proper noun as a slug.

The remaining unknown (and the thing the spike had to settle) was whether the chosen tooling actually composes:

  • Does sqlite-vec integrate with a rusqlite-backed connection without breaking the bundled SQLite that ADR-003 mandates?
  • Does fastembed-rs 5.x produce usable embeddings on a CPU-only laptop, and at what latency?
  • Does the Carina/Firebase canonical query (brief §1.1) actually surface the right notes when the corpus is small enough to read by eye?

The spike at spike-syn-89/ answered all three with numbers. This ADR records the answers, locks the tooling choices, and registers the issues SYN-81 + SYN-83 + SYN-164 have to navigate when production wiring begins.

Decision

Adopt the following stack for the v0.5 semantic layer. Each row of the table is the default; the opt-down column is a workspace-level config override (per §1.3 of the v0.5 brief) for users who explicitly value smaller footprint over recall.

SlotDefault (best)Opt-downCrate
Embedding inferenceBGE-M3 multilingual quantisedBGE-small-EN-v1.5-Qfastembed 5.x
Vector storagesqlite-vec vec0 virtual table(none: single backend)sqlite-vec 0.1.x
NER (entity linking)GLiNER-multi-base-QGLiNER-base-en-Qalso via fastembed-rs (ships GLiNER as ner_model)
Hybrid fusionRRF with k = 60(none: v0.5 ships only RRF; convex combination is the v0.6 polish)implemented in core::search

The configuration knob lives in <workspace>/.synapse/config.toml:

[enrichment]
language        = "pt-BR"            # or "en-US"
embedding_model = "bge-m3"           # or "bge-small-en-v1.5-q"
ner_model       = "gliner-multi"     # or "gliner-base-en"
mode            = "heuristic+ner"    # or "heuristic", "off"

Why these and not the alternatives

CandidateWhy rejected as default
Ollama as embedding runner~700 MB binary + ~280 MB model = ~1 GB on disk, plus a separate daemon the user has to start. Violates “one binary, one workspace” UX even though it does not violate BR-1 strictly. Kept as configurable alternative (provider = "ollama") so a user who already runs Ollama for chat can reuse it.
candle (pure Rust ML)Strong on flexibility (any HF model, no ONNX conversion) but slower per-query than ort on CPU. For an ingestion-time hot path the throughput matters more than the flexibility.
OpenAI / Voyage / Cohere embedding APIsBest raw quality but each write becomes a network call that leaves the workspace. Breaks BR-12 privacy posture without an explicit opt-in. Kept as stretch SYN-165 behind provider = "openai" | "voyage".
lance for vector storageFull ANN + HNSW out of the box, columnar Arrow-native storage. But ~50 MB of extra code in the binary and a separate mental model from “everything lives in .synapse/index.db”. The spike confirmed sqlite-vec is fast enough for the v0.5 ceiling of 10k entities.
Hand-rolled flat scan + SIMD cosineTempting (no extra dep). Rejected because sqlite-vec ships rank fusion + chunked storage + future HNSW for ~1.5 MB of native code; reinventing the storage layer just to avoid one dep is the wrong axis to optimise on.
Full Microsoft-style GraphRAG with community summaries3–5× ingestion cost ([GraphRAG vs Vector RAG 2026]). Requires LLM calls during ingestion which breaks local-first. The brief substitutes the lightweight heuristic + GLiNER combo and reserves the GraphRAG-class polish for v0.6+.

Spike findings (the numbers behind the choices)

Ran 2026-06-02 on aarch64-apple-darwin (Apple M1 Pro, macOS 25.4), cargo run --release, model: BGE-small-EN-v1.5-Q (the lightest variant: the spike did not need the full BGE-M3 download to validate the integration; the production-default BGE-M3 numbers are cited from the public MTEB report referenced below).

[fastembed]   model loaded in            11.12 s    (cold start, first-time download)
[fastembed]   embedded 8 samples in       69.48 ms  (avg 8.69 ms/text, dim=384)
[sqlite-vec]  registered, version =       v0.1.9
[sqlite-vec]  inserted 8 rows in         433.12 µs
[sqlite-vec]  vec0 query (8-row corpus)  ~130 µs / query
[footprint]   release binary size                ~21 MB
[footprint]   model cache (BGE-small-EN-v1.5-Q)  ~33 MB
[total wall-clock]                          11.23 s (dominated by first-time fetch)

Carina/Firebase regression (brief §1.1 canonical query). Eight planted notes; query was the multi-hop concept “flag do Firebase para o checkout”. Top-3 cosine ranking (lower distance = closer):

  project/checkout-v2                       dist=0.6859
  note/firebase-stack                       dist=0.7440
  note/meeting-carina-2026-04-20            dist=0.7869

All three relevant notes surface in the top-3 with no tuning. This validates the v0.5 thesis: vector retrieval alone bridges the “Remote Config ↔ Firebase” concept gap even before the entity-linking heuristic + NER pipeline adds a graph layer on top.

Reverse query (“Remote Config feature flag eligibility”) was also clean: firebase-stack + meeting-carina + api/firebase-remote-config in the top-3.

Findings that survive the spike: issues for v0.5 production wiring

  1. rusqlite version conflict: AMENDED 2026-06-02 during SYN-81. The spike flagged that sqlite-vec 0.1.9 declares rusqlite ^0.31 while the main crate runs on rusqlite 0.39 (ADR-003). On closer inspection of sqlite-vec’s Cargo.toml, the rusqlite dep is in [dev-dependencies] only: it backs the upstream crate’s own test suite, never its public API. SYN-81’s integration goes through the SQLite C ABI’s sqlite3_auto_extension, which is ABI-stable and version-agnostic. Decision: skip path B (vendor the C source) entirely. We depend on sqlite-vec 0.1 directly as a runtime dep behind feature = "vector" and register it via rusqlite::ffi::sqlite3_auto_extension. Total cost: ~10 lines in src/index/vector.rs and one OnceLock. The original path A (downgrade) was never viable; path B was real engineering work the spike’s surface read mis-attributed.

    • unsafe exception. Wiring sqlite3_auto_extension requires a single transmute of a C function pointer. The crate’s #![forbid(unsafe_code)] is relaxed to #![deny(unsafe_code)] so that src/index/vector.rs can scope a documented #![allow(unsafe_code)]. Every other file in the crate still fails the build on unsafe. This is the only unsafe site introduced by v0.5 and the only one allowed by the project’s architectural rules going forward.

  2. First-run model download is 11 s + ~33 MB for BGE-small (~570 MB for the production-default BGE-M3 and an additional ~150 MB for GLiNER-multi). This download cannot land on the hot path of syn init: UX-blocking. Decision: lazy initialise the EmbeddingBackend and NerBackend on the first write call, not on workspace open. Print an explicit “downloading model…” banner on the first fetch (SYN-83 + SYN-164 acceptance).

    Correction (SYN-214, 2026-06-07): the size estimates above were wrong. Measured from Hugging Face: the production-default BGE-M3 the code wires (fastembed::EmbeddingModel::BGEM3 → BAAI/bge-m3) is fp32, ~2.2 GB (the “~570 MB” was the int8 gpahal/bge-m3-onnx-int8, offered as a choice in SYN-237), and GLiNER-multi quantised is ~350 MB, not ~150 MB. Real default footprint: ~2.5 GB. The English opt-downs are ~65 MB (BGE-small) + ~260 MB (GLiNER-base) ≈ ~325 MB. Canonical figures live in docs/MODELS.md.

  3. sqlite-vec is flat-scan only in v0.1.9: no HNSW index yet. For Synapse’s ≤ 10k entities v0.5 ceiling this is fine (the spike measured ~130 µs per query in an 8-row table; even O(N) scaled to 10k stays well under the 150 ms search budget). Document the ceiling explicitly in docs/CONFIGURATION.md; revisit in v0.7 if a real user exceeds it.

  4. Fastembed cache discovery: the spike’s du heuristic reported 0 MiB because fastembed-rs 5.x writes under ~/.cache/fastembed_cache/ (lowercase, no leading dot) rather than the ~/.fastembed_cache/ the spike probed. Cosmetic for the spike, but core::embeddings (SYN-83) must read FASTEMBED_CACHE_PATH first, then fall back to the platform-correct path (XDG on Linux, ~/Library/Caches/fastembed_cache on macOS, %LOCALAPPDATA% on Windows), and Synapse should surface the chosen path in syn embed --check so the user can sweep it manually.

    • AMENDED 2026-06-03 during SYN-83. Synapse owns the cache directory and passes it to fastembed via InitOptions::with_cache_dir. Precedence: [enrichment] model_cache_dir → SYNAPSE_MODEL_CACHE → platform default (~/Library/Caches/synapse/models/fastembed on macOS, XDG on Linux, %LOCALAPPDATA% on Windows). The fastembed-owned FASTEMBED_CACHE_PATH env var is not read; one knob, one owner. syn embed --check will print the resolved path so users can du -sh it themselves.

  5. GLiNER via gline-rs blocked by ort RC pinning, NEW 2026-06-03 during SYN-164. The v0.5 brief asserted “fastembed-rs ships GLiNER via ORT: no separate ORT dep”. Verified false at wiring time: fastembed-rs 5.x carries embedding + reranker catalogues only, no NER. The canonical NER crate gline-rs 1.0.1 pins ort = "=2.0.0-rc.9" exactly while fastembed 5.15 pins ort = "=2.0.0-rc.12" exactly: Cargo cannot resolve two pinned RCs of the same crate. Three paths were considered:

    • A. Vendor / fork gline-rs ourselves and bump its ORT to rc.12. Owner of the fork; ORT RC API drift between rc.9 and rc.12 is non-trivial.
    • B. Use kreuzberg-gliner-rs 0.2.0: Kreuzberg’s Apache-2.0 fork of gline-rs whose published crate re-pins ort to =2.0.0-rc.12 and ships an aligned kreuzberg-orp 0.1.1. Public API surface (GLiNER<SpanMode>, Parameters, TextInput, SpanOutput) is identical to upstream, so the consuming code ports back the moment upstream re-aligns.
    • C. Ship v0.5 with heuristic-only NER via the existing RegexNer until upstream gline-rs re-aligns ort.

    Initial reading (2026-06-03): path C, on the assumption that path B would land a second ORT copy in the binary (+50 MB).

    Amended 2026-06-03: path B chosen. A build probe (/tmp/probe-kg, cargo build --release) with kreuzberg-gliner-rs = "0.2" + fastembed = "5.15" resolved to a single ort 2.0.0-rc.12 in Cargo.lock. Both backends share the same ORT copy: no binary bloat. The earlier +50 MB claim came from misreading the fork’s Cargo.toml: the published 0.2.0 carries ort =rc.12, even though the fork’s master branch on GitHub still shows =rc.9. With the real cost at zero MB, the only meaningful trade-off vs. path A is dependency hygiene, the Kreuzberg org’s main project (kreuzberg-dev/kreuzberg, 8.4k stars, 200 releases, last May 2026) is active enough to amortise that risk. SYN-164 lands in v0.5 against kreuzberg-gliner-rs; we revisit when upstream gline-rs re-aligns its ORT pin (issue #156’s exit criterion).

Consequences

  • +1 dep family: fastembed 5.x pulls ort (ONNX Runtime statically linked, ~25 MB compressed), tokenizers, hf-hub, and the BGE / GLiNER ONNX models on first use. Acceptable: the local-first invariant survives and the binary growth lands in the ~50 MB envelope the v0.5 brief §7 advertised.
  • +1 dep: sqlite-vec does not get added to synapse’s Cargo.toml AMENDED 2026-06-02. Per the Findings amendment above, SYN-81 adds sqlite-vec = { version = "0.1", optional = true } as a normal optional runtime dep gated by feature = "vector". No vendoring; the upstream Cargo dep ships the C source the build links in. The dev-deps-only rusqlite ^0.31 constraint inside sqlite-vec is irrelevant to consumers.
  • Per-workspace language config: workspaces are now meaningfully different from each other in what model they load. A user with three pt-BR repos pays the ~720 MB cache once (shared); a mixed install pays twice. Documented in docs/IDENTITY.md and docs/CONFIGURATION.md.
  • Migration story: opening a v0.4 workspace with a v0.5 binary triggers syn embed --all automatically on the first write (or runs it explicitly when the user types syn embed --all); the vector migration ships as migrations_vec/0001__entities_vec.sql per SYN-80.
  • Upgrade path documented in advance (added 2026-06-02 after the owner-prompted critical review at docs/researchs/2026-06-02-vector-storage-options-vs-sqlite-vec.md): if a workspace ever exceeds ~50k indexed entities and the flat-scan latency becomes visible, switch to vectorlite (HNSW-based SQLite loadable extension by 1yefuwang1, ~10× faster searches than sqlite-vec’s flat scan, same load_extension shape, same SQL surface). The swap costs ~30 lines of FFI changes and zero business-logic change. Pre-staged ticket: SYN-203 in v0.7+. LanceDB / SurrealDB / DuckDB+VSS were evaluated in the same review and rejected as v0.5 migration targets: they would force rewriting Workspace::tx, FTS5 callers, the pending queue and the audit-trail SQL (estimated ~2,400 lines + ~100 tests), which is a v1.0-shaped commitment, not a v0.5 spike.
  • Reverse compatibility: the schema additions are append-only; removing the vector feature flag at build time omits the entities_vec table cleanly. v0.4 binaries opening a v0.5 workspace ignore entities_vec (FTS5 path still works). Documented in SYN-80.
  • MSRV stays at 1.88 (ADR-009); no fastembed-imposed bump in 5.x.

Alternatives considered (recorded for completeness)

The “why rejected” table above covers Ollama, candle, OpenAI/Voyage, lance, hand-rolled flat scan, and full GraphRAG. Two more decisions deserve explicit recording so the next reader does not have to relitigate them:

  • Embed model = nomic-embed-text-v1.5 instead of BGE-M3: Nomic is English-strong, 768-dim, ~280 MB cached, MTEB EN 62.39. The brief rejected it as default because Synapse is per-workspace multilingual per BR-13 §1.3; Nomic would force every pt-BR workspace to opt-up to a different model. BGE-M3 covers both languages competently at one model.
  • NER backend = Relik or spaCy over an inference sidecar: both outperform GLiNER on top-tier benchmarks but require either a Python sidecar (Relik) or a separate runtime (spaCy). GLiNER is the Rust-native ONNX option. The 2026 Neo4j blog ([Entity Linking and Relationship Extraction With Relik]) explicitly recommends GLiNER for the “ingestion-time, no Python” branch of the decision tree.

Amendment 2026-06-04 (SYN-168): opt-out, not opt-in

The original decision shipped the semantic layer behind --features vector (off by default), deferring the default flip to “v0.6 if adoption pulls it.” That was reversed inside v0.5: the layer is the product, not an experiment, so it now ships in the default binary (default = ["mcp", "v05_backends"]). Two constraints shaped how:

  • Binary cost is small; model cost is not. Compiling fastembed + GLiNER adds ~25 MB of statically-linked ONNX Runtime: acceptable as a default. The ~920 MB of model weights are a different category, so they are never pulled implicitly. A per-model readiness sentinel (core::embeddings::readiness, written the moment a backend loads) is the single source of truth; run_on_write, hybrid search, and syn embed --check all gate on it and degrade gracefully (skip embedding / fall back to FTS+graph) instead of triggering a download. The only consent paths that fetch are the syn init onboarding prompt and syn embed. Without the models the tool is a fully functional keyword + graph brain; the models add meaning-search and NER discovery on top.
  • network_backends and test_backend stay opt-in. Hosted providers send content to a third party (privacy: local-first is the default per BR-13). The deterministic stub returns fake embeddings and must never reach a production binary; the release pipeline therefore builds --features network_backends (additive to the default set), not --all-features, which previously leaked test_backend into shipped binaries.

References