ADR-042: Retrieval quality: chunked embeddings + cross-encoder reranking

Date: 2026-07-05 Status: Accepted (2026-07-06) Author: Rômulo Febasi

Measured outcome (owner-run persona-bench --features real, M3 Pro). Track A (chunking) shipped default-on: full-haystack (6 personas, 463 entities) recall@1 0.75 → 0.85, no recall@10 regression (a clean 1.0 across the whole curve); average recall@1 across the six tiers 0.783 → 0.842. Track B (cross-encoder reranker) shipped opt-in, off by default: on this corpus chunking already captured the recall@1 gain, so the reranker was neutral at scale (0.85 → 0.85 at the headline; average 0.842 → 0.850, its only edge being the small-N tier and a clean recall@10). It is viable, tested, and available via [enrichment] rerank, but did not clear the “measured win” bar as a default. Honest per §3 of this ADR: a change that does not move the number does not ship as a default.

Reranker input: the snippet, not the full body (tested). A follow-up measured feeding the cross-encoder each candidate’s full entity body instead of its query-focused snippet. It was worse: recall@1 at the small tier dropped to 0.75 (vs 0.85 with the snippet) (the long body dilutes the relevance signal with unrelated text) and it pushed a single search to ~45 s (reranking the top-k over ~512-token inputs on CPU). So the shipped reranker uses the snippet: faster and better. This is the counter-intuitive but measured result; the pool size (top-k) remains a latency knob for anyone enabling it on large workloads.

Scopes SYN-301 (v1.2 “Retrieval quality”). Extends ADR-008 (embedding stack) and composes with ADR-029 (trust) and the SYN-250/255 rankers. Motivated by the agent-in-the-loop benchmark: persona-bench measured recall@1 ~0.78 (docs/BENCHMARKS_EXPLAINED.md), the clearest measured weakness in the product.

Context

Retrieval today is a three-leg RRF fusion (k = 60, src/core/search/hybrid.rs:49): FTS5/BM25 + vector cosine over entities_vec + one-hop graph neighbourhood, with the ADR-029 trust score folded in as a bounded [0.5, 1.0] multiplier (SYN-250) and an optional --near graph-distance reranker (SYN-255, anchor_rerank). Two structural limits cap its precision:

1. One vector per whole entity. src/core/embed.rs embeds each entity as a single entities_vec row (entity_type, entity_id, embedding). A long note or runbook is averaged into one point, so a specific fact buried in a large document dilutes against the surrounding text. The needle loses rank.

2. No re-scoring of the top candidates. RRF is a rank-fusion heuristic; it never compares the query against a candidate jointly. The bi-encoder (BGE-M3) scores query and document independently, which is fast but coarse. The community-standard fix for exactly this: a strong recall@10, weaker recall@1: is a cross-encoder reranker over the shortlist.

persona-bench confirms the symptom: recall@10 holds at 1.0 while recall@1 sits at ~0.78 (docs/BENCHMARKS.md §3). The agent masks this end-to-end by issuing multiple searches, but the raw retrieval quality is the honest lever, and recall@1 is the number that matters when a single ranked answer is consumed (digests, search --limit 1, a small-context agent).

The user also asked which classic NLP techniques (TF-IDF, TextRank, YAKE, RAKE, KeyBERT, LSA, K-Means, TextTiling) apply. Most are superseded by the current stack (BM25 already is global TF-IDF; dense BGE-M3 subsumes LSA) or violate the deterministic/explainable ethos (K-Means topic clustering is opaque ML). The two that map to a real, measured gap are chunking (the structure-aware cousin of TextTiling) and, outside the list, cross-encoder reranking.

Decision

Adopt two independent, benchmark-gated retrieval upgrades, both confined to the disposable index (Markdown truth and the governance gate are untouched):

1. Chunked embeddings (structure-aware, deterministic)

Embed entities at passage granularity, not whole-entity. A deterministic, dependency-free Markdown chunker splits an entity by structure (headings → paragraphs → token-bounded windows with a small fixed overlap); each chunk gets its own vector in a chunk-keyed vec0 table with a parent link (entity_type, entity_id, chunk_ord). Short entities produce a single degenerate chunk, so their behaviour is unchanged. At search time, chunk hits map back to their parent entity (best-chunk pooling into the existing RRF), and the snippet is taken from the winning chunk. This is a reindex/re-embed migration on the disposable index; no source file changes.

The chunker is structure-aware and deterministic on purpose: same bytes → same chunks, no ML segmentation model. TextTiling and embedding-based semantic chunking are considered and rejected below.

2. Cross-encoder reranker over the RRF shortlist

After RRF (and the SYN-250 trust fold) produce the top-k candidates, a cross-encoder rescoring pass reorders that shortlist before the results are returned. The model is bge-reranker-v2-m3 (the reranker paired with BGE-M3, multilingual), run through fastembed 5.15’s text-reranking API on the same ort copy already shared by BGE-M3 and GLiNER (ADR-008): no new backend, no new ADR-008 amendment, no new heavy dependency. Reranking touches only the final ordering of the shortlist: it never changes what is stored, never feeds the trust score, and never bypasses or informs the write gate. It is a ranking aid, not a governance signal, so ADR-029’s “no opaque ML in governance” line holds.

The reranker is model-gated and opt-in like semantic search: available only when its model is present (guided download, like BGE-M3), enabled via [search] rerank = true in config, defaulting on when the model is present and a silent no-op otherwise. --no-rerank forces it off for a query.

3. Ship each only on a measured win

Both are benchmark-gated: each is merged only if persona-bench (--features real) shows recall@1 up with no recall@10 regression and no p95 latency budget break (ADR-040 methodology). If a change does not move the number, it does not ship: the benchmark that exposed the gap is the same one that approves the fix.

Consequences

• Improves: directly attacks the one measured retrieval weakness (recall@1 ~0.78). Chunking sharpens recall on long documents; the reranker sharpens the top of the list for every query. Both reuse the existing ort/fastembed/sqlite-vec stack: zero new charting/runtime deps.
• Worsens / limits: chunking multiplies entities_vec rows (more embed time at reindex, more index size, both linear and one-time); the reranker adds a per-query cross-encoder pass over the shortlist (bounded, O(k), top-k only, measured against the p95 budget). The reranker model is another ~hundreds-of-MB download on top of BGE-M3, so it is opt-in, not forced.
• Boundary held: neither feature crosses into governance. The index stays disposable and rebuildable; a bad chunker or reranker is fixed by rm .synapse/index.db && syn reindex, never by touching a source file.
• Open / revisit: chunk size + overlap are tunables to sweep empirically; per-type chunking policy (a runbook chunks differently from a person) is a follow-up; a distilled/quantised reranker for lower latency is a follow-up if the p95 pass is tight. Query expansion / HyDE is a separate future lever, not in this ADR.

Alternatives considered

• TF-IDF / LSA (from the technique list). Rejected as redundant: FTS5/BM25 is already a global TF-IDF ranking, and dense BGE-M3 subsumes LSA’s latent-topic role. Neither adds signal over the current stack.
• TextTiling / embedding-based semantic chunking. Rejected as the default: both are non-deterministic or model-dependent, which fights the “same bytes → same index” property and adds a segmentation model to maintain. Structure-aware Markdown chunking captures most of the benefit deterministically. Revisit if a measured recall gap on long prose survives structural chunking.
• KeyBERT / YAKE / RAKE / TextRank keyphrase extraction. Not a reranking lever; deferred to a separate “auto-tags + facets” line of work (they improve metadata and the digest, not recall@1 directly). Out of scope here.
• K-Means topic clustering. Rejected for governance-facing use: opaque ML that tensions ADR-029’s explainability. Acceptable only as an advisory navigation view later, never in ranking or the gate.
• An external reranker API (Cohere/Voyage rerank). Rejected: breaks local-first and the offline guarantee (ADR-008 already rejected hosted embeddings for the same reason). The reranker must run on device.
• A learned/weighted RRF (train fusion weights). Rejected for now: introduces a trained artifact and a training pipeline; the cross-encoder gets more precision for less machinery and stays a drop-in over the existing fusion.
• Bump HNSW first (SYN-225). Orthogonal: HNSW is an ANN speed index for large N, not a quality lever; it is correctly gated on a corpus-size trigger (ADR / plan 015) and does nothing for recall@1 at current scale.