Benchmarks

Synapse is benchmarked along two axes that matter to a real user: does it scale to my project and machine? and does the governed-memory story actually hold when a real AI agent drives it? The first is deterministic and offline; the second drives a live agent (Claude Code) over MCP and grades the resulting state, not the transcript.

• Methodology contract (scaling): ADR-040
  • the scaling research note.
• Agent-in-the-loop design + driver decision: ADR-041, plan 022, and the findings notes (methodology · first results).
• All numbers are owner-run, not CI (agent runs cost tokens + are non-deterministic; ADR-041). The harness lives in synapse-bench/.

1. Scaling: latency & index size vs corpus size

Will it handle your project, and where does it break? The honest answer is a curve over corpus size, not a single number.

The shape is the point: FTS-only (BM25) stays nearly flat: it tracks the result set, not the corpus: while vector-flat climbs with N (a brute-force cosine scan over every embedding, O(N·d); the line that eventually breaks), and hybrid (RRF) tracks both. The index grows linearly on disk:

(These two are the stub backend at large N, the latency/size mechanics are real; the vectors aren’t semantic, so these charts are about scaling shape, not quality.)

2. Resources: will it fit on my machine?

With the real backend (BGE-M3 on-device), the answer is floor + slope: the model is the ~1.8 GB floor, your data adds to it gently.

The binding constraint is the model, not your notes, a 16 GB machine fits far more than a single user will ever write. Embedding runs at ~13 docs/s on this CPU (a one-time ingest cost); reindex of the index structure stays in the tens of ms.

3. Retrieval correctness at scale

“Fast” is worthless if the right fact stops ranking. The honest correctness signal comes from realistic data: 6 fictional professionals’ full quarters (persona fixtures, ~460 entities, 320 cross-referenced notes) + a 120-question needle bank (persona-bench, SYN-296). We fix one professional’s 20 facts and watch recall as the other five careers pile into the same brain:

The raw observations contain six measured points. At 77 entities, 19 of 20 target facts appear in the first ten results; at the five larger measured sizes, all 20 do. First-result recall ranges from 75 to 90 percent and ends at 85 percent for 463 entities. This internal synthetic fixture does not establish quality, capacity or robustness for other workloads. The original run’s source commit is unknown.

The historical illustration in brand/images/retrieval-quality.png now derives from this same canonical chart. Its previous hand-positioned curve did not match the recorded values. No new benchmark or matched competitor comparison is claimed here.

recall@1 improved from 0.75 to 0.85 (SYN-301 / ADR-042). The measured weakness at the top of the list was that an entity was embedded as a single whole-entity vector, so a fact buried in a long note was averaged away. Embedding at passage granularity (a deterministic Markdown chunker, one vector per chunk pooled back to its parent by best distance) lifted full-haystack recall@1 from 0.75 to 0.85 with no recall@10 regression (recall@10 is 0.95 at the first measured point and 1.0 at the other five). An opt-in cross-encoder reranker ([enrichment] rerank, ADR-042 Track B) is also available; on this corpus chunking already captured the recall@1 gain and the reranker was neutral at scale, so it ships off by default: useful to try on longer-document workloads, not a headline win here. Raw data: persona-recall · stub · real.

Reproduce (the model is cached & reused; clean-models reclaims the GBs):

cd synapse-bench
cargo run --release -- sweep --tiers 500,2000,8000 --reps 5 --out ../docs/assets/bench
cargo run --release --no-default-features --features real -- sweep --tiers 200,800,2000 --out /tmp/r
cargo run --release --no-default-features --features real -- persona-bench --out /tmp/p
cargo run --release -- clean-models

4. Agent-in-the-loop: does governed memory hold under a real agent?

The distinctive benchmark: drive a real agent (Claude Code, headless) through the full governed-memory loop over MCP (insert / correct / contradict / retrieve) and grade the resulting memory state via syn verify’s canonical dump (the τ-bench get_data_hash() analog), not the transcript. The agent is isolated to the MCP tools (a system prompt + --disallowedTools block file/shell/web access) so the benchmark measures MCP-driven memory, not the agent reading project files. Because the CLI gives no seed control, every task runs N times and reports pass@1 (capability) with a 95% Wilson CI, pass^k (reliability = every run passes its assertions), and cost.

Run it (owner-run; needs claude + credentials; see the harness README):

cd synapse-bench
cargo run --release -- drive --task fixtures/tasks/insert-01.json --runs 15 --syn-bin "$(command -v syn)" --out /tmp/insert.json
cargo run --release -- gate-probe --seed seeds/aurora --payload fixtures/gate-probes/aurora-archived.json --out /tmp/gate.json
cargo run --release -- qa --needles fixtures/needles/ana-yamamoto.json --persona fixtures/personas/ana-yamamoto --limit 10 --judge-model claude-sonnet-4-6 --syn-bin "$(command -v syn)" --out /tmp/qa.json

Results (Claude Code 2.1.197; judge = Sonnet over an Opus agent)

Two findings that shape the positioning:

1. Contradiction defence is two independent layers. A capable agent surfaces a contradiction in its own reasoning and asks the human before ever filing the write, so the agent-in-the-loop path shows non-corruption, and the gate (SYN-242) is the non-bypassable floor for when the agent doesn’t notice (a weaker/autonomous agent, a subtle conflict, too much data). gate-probe measures that floor directly: it detects and gates a conflicting update_status, 100%, with a hard-negative control.

2. Scenario D separates retrieval from generation, and the agent beats single-shot retrieval. Each needle is answered three ways over a persona haystack: retrieval (gold in syn search top-k), oracle (gold injected, the generation ceiling), and end-to-end (agent via MCP), with a reference-guided judge on a different model. On a 10-needle run: recall@10 = 80% but e2e = 100% (oracle ceiling 100%). The gap is the finding: the two needles the top-10 retrieval missed were still answered correctly end-to-end, because the agent issues multiple MCP searches and reasons across related notes. So the MCP-driven loop recovers retrieval gaps a single ranked query would drop; the oracle−e2e gap (here zero) isolates genuine read failures when they occur.

Under noise, retrieval holds. Piling synthetic distractors into the haystack (qa --distractors N; the agent’s token cost is ~flat in N because it searches via MCP, so only the one-time embed scales) leaves the answer intact:

¹ the lone e2e dip is a 5-needle-sample claude -p blip, not degradation. Recall@10 and the oracle ceiling stay at 100% through 8 000 distractors, the retrieval cliff the flat-vector scaling curve predicts lives at far larger N (tens of thousands+), not the low thousands a real second brain holds. The 100 000 tier is a ~2 h one-time embed (documented, not run).

Reliability, not just capability, is the headline for a “can’t silently corrupt” tool: the recurring ~1/N miss in these runs is a transient claude -p crash (now retried), not Synapse: the governance behaviours (update-on-existing, abstention, non-corruption, gate detect-and-gate) are effectively clean.

Honest limits. Judge and agent are both Claude (a weaker preference-leakage guard than a cross-family judge): audit judge–human agreement before trusting the e2e number; the raw verdicts are kept for that. The distractor-scale sweep (1k/10k/100k) and the bitemporal --as-of correction assertion are the next slices (plan 022).