Benchmark redesign: scaling, resources, and correctness-at-scale

Executive summary

The current README benchmark: median 7.97 ms / p95 10.21 / p99 11.53 on a 300-doc stub corpus: proves the renderer works, not that Synapse scales. It has no scale axis, no resources (RAM/CPU/disk), no correctness-under-load, no workload profiles, no breaking point, and no honest comparison. Every credible benchmark in the field (ClickBench, DuckDB, ann-benchmarks, Qdrant, Meilisearch, TigerBeetle, uv) replaced bars with curves over a scaling variable + a floor-vs-slope resource model + percentiles + machine stamp + raw data + one-command reproduce.

The redesign: benchmark across 5–6 scale tiers (T1 daily ~500 → T4/T5 stress 50k–1M), measure per-subsystem latency, RAM (floor vs slope), disk, reindex, throughput, and recall-at-scale, render curves with confidence bands and a breaking-point band, and compare Synapse only to itself (FTS / vector-flat / hybrid; fp32 / int8; cold / warm; flat / future-HNSW) plus ripgrep as a labelled naive floor: never to a hosted cluster, never at unequal recall.

Two real code findings surfaced en route (each → its own ticket): the FTS5 table is self-content despite a comment claiming external-content (~doubles the text footprint, ~3 KB/entity waste); and the “flat vector correct below ~100k” rationale (SYN-225 deferral) is conservative by ~5–10×: flat is fine to ~500k–1M at 1024-dim on a laptop; the real wall is page-cache, not 100k.

1. Why the current benchmark is useless (the critique)

1. No scale axis. A single N=300 number. The reader’s question is latency × N: a curve from 1k → 10k → 100k. Without it, “will it handle my vault?” is unanswerable.
2. No resources. Nothing on RAM (the BGE-M3 model floor alone is ~2.5 GB!), CPU, or .synapse/index.db size. For a local tool, “does it fit on my machine?” is the question.
3. stub hides the real path. Stub latency skips the semantic path (embeddings + sqlite-vec). And vector search is flat brute-force O(N·d): it degrades linearly; that is exactly where it breaks and the chart never shows it.
4. No correctness under load. “Fast” is worthless if retrieval stops surfacing the right fact as the graph grows. Recall/precision × N is the proof of usable.
5. No workload profiles, no breaking point. No “daily (medium)” vs “extreme (stress)”, no cliff.
6. No comparison. 8 ms vs what?

2. The cost model (verified against the source)

Let N = entities (= FTS rows = vectors), E = edges, W = federated workspaces, d = 1024 (vector dim).

The two that bite, in order: flat vector KNN (linear in N), then federation (multiplies it by W). Everything else is sub-linear or free until N is enormous.

Crossover estimates (1024-dim fp32, laptop): to be confirmed by measurement

Anchored on the sqlite-vec author’s own M1 benchmarks (GIST 500k×960 = 41 ms; 100k×1536 = 105 ms):

• < 10 ms (snappy): ~50–100k vectors.
• < 100 ms (interactive): ~500k–1M.
• < 1 s (usable): 1M+.
• Breaking point: ~1M vectors or whenever the fp32 vector blob (4 KB × N) outgrows page cache → every query becomes disk-bandwidth-bound and jumps a tier. This validates that the “100k flat ceiling” is conservative; the bench must locate the real page-cache cliff.

3. Workload tiers (grounded in real vault data)

Anchors: Obsidian forum self-reports (median ~5,500 notes; large 18k–20k; 60k+ “approaching indexing failure”); Luhmann’s Zettelkasten ~90k notes / 40 yrs (~2,250/yr), the lifetime single-user ceiling.

Realistic single-user upper bound: ~50k–100k entities. T1–T3 are must-pass; T4/T5 are the “does flat still hold or do we need HNSW (SYN-225)” probe.

The corpus must look like a real graph, not uniform random

• Scale-free (Barabási–Albert, γ≈2.2–2.5), mean degree ~4–10, with a few hub projects/people carrying 100–1,000+ edges. Uniform-random graphs lie: they have no near-miss distractors, so recall@k looks artificially perfect.
• Log-normal note lengths (median ~150–400 words, p95 ~2,000+, a long tail).
• Zipfian tags; 20–50% AI-authored (exercise trust-ranking + audit under load); supersession chains (exercise --as-of + “stale shouldn’t outrank current”).

4. Index / disk growth (.synapse/index.db)

Per-entity (≈500 words ≈ 3 KB text), under current defaults (fp32, self-content FTS):

Vectors + self-content FTS are 75% of the DB. Two cheap wins: true external-content FTS (−28%) and int8 vectors (~−28%) ≈ roughly halve the index. And disk size is the latency story, the slowdown happens when that blob outgrows page cache.

5. Resource measurement methodology (RAM/CPU/disk)

The honesty trap: (a) don’t report peak RSS as “required RAM” (page cache is opportunistic/evictable: Qdrant: “10 GB RSS ≠ needs 10 GB”); (b) don’t use dhat for the headline number: it sees only the Rust global allocator and is blind to the 2.5 GB ONNX model (off by ~100×).

Floor vs slope (the only honest method):

RSS(N) ≈ baseline + Model_floor(~constant) + SQLite_working_set(N) + Vector_set(N)

1. Baseline: RSS of a no-op (syn --version).
2. Floor: RSS with the model loaded on an empty workspace → report as “baseline RSS with model loaded (fp32 / int8)”. This is the dominant, ~constant cost.
3. Slope: Δ RSS per 10k entities (vectors = +4 KB/entity fp32, +1 KB int8). Subtract the floor: never attribute the 2.5 GB to “100k entities”.

Measurement stack: /usr/bin/time -v (Linux) / -l (macOS) for peak RSS envelope; in-process VmHWM / getrusage phase checkpoints (after model-load, after index-N, after search); dhat only for the Rust working-set slope (labelled, not as total); heaptrack (Linux) for one-off full-process attribution incl. ORT.

Landmines (verified): ru_maxrss is KiB on Linux, bytes on macOS (1024× error if unbranched: sanity-assert peak > model floor); measure index.db only after OPTIMIZE + VACUUM (fragmentation over-reports); don’t gate wall-time in CI (~50% variance): gate memory-peak regressions (stable) and run timing on a pinned box, reporting CPU-seconds + thread/core count, not bare wall time.

CPU/throughput: warmup-then-median; cold (model load + first embed) vs warm (steady state) labelled separately; embedding throughput in entities/sec + tokens/sec with thread count stamped; reindex in CPU-s/entity (machine-comparable) + wall-s (felt).

The “will it fit?” headline this produces: the model is the floor (~2.5 GB fp32 / sub-GB int8); each 10k entities adds ~41 MB (fp32) of vectors, so on a 16 GB machine you fit millions of entities; the binding constraint is the model, not your data.

6. The chart catalogue (curves, not bars)

x-axis = N (log scale) on the scaling charts. Every chart: median line + p5–p95 confidence band (≥5 reps), breaking-point band (green/amber/red threshold rules), dark/light via CSS, and a caption with the machine stamp + reproduce command + commit + date.

Charts 1, 2, 3, 7 are the hero set. SVG (crisp, diffable, themeable), confidence bands, annotated knee (“≈42k entities: p95 crosses 200 ms”), committed results.json beside the SVGs, --quick/--full reproduce tiers.

7. Correctness-at-scale (the premium differentiator)

The claim to prove: the right fact still ranks top-k when 50k entities compete. Measure rank-aware IR metrics as a curve over N, not a single number:

• recall@{1,5,10,20}, MRR, nDCG@10, precision@k, computed per tier for FTS / vector / RRF separately (so degradation is attributable).
• Tier-stable planted-needle query bank (≥100 queries/tier): synthesize the graph, so truth is known; re-run the same queries as the corpus grows → pure haystack-growth degradation (the LongMemEval “same question, bigger haystack” design).
• Hard negatives: 2–5 near-miss entities per needle sharing the hub/tags/words.
• Abstention queries (invalidated/never-existed): test it declines vs hallucinates.
• Watch RRF k=60 × corpus-size interaction: verify RRF doesn’t win at T1 but lose at T3 purely from a small-scale-tuned k.

Agent metric: Recall@k_budget(N) under a declared context budget (e.g. top-10 ≈ 3k tokens), reported separately from end-to-end task success (decouple “was it retrieved” from “did the LLM use it”) + distractor load = noise fraction of the window. Field anchor: LongMemEval shows a 30–60% accuracy drop without targeted retrieval: exactly what this catches.

8. Honest comparison (and what to refuse)

Ship: Synapse vs itself: FTS / vector-flat / hybrid-RRF; fp32 vs int8; cold vs warm; flat vs future-HNSW (flat = the recall=1.0 ceiling once SYN-225 lands); ripgrep as a labelled naive floor (ripgrep should win raw latency and lose relevance: state the asymmetry).

Refuse: comparing at unequal recall (Qdrant’s “fixed precision threshold” rule); warm-only with cold hidden; cherry-picked N as a bar; tuned-vs-untuned asymmetry; hosted-cluster comparisons (apples-to-oranges for a single-user embedded tool); mean-only latency.

9. Two code findings (→ tickets)

• FIND-1: FTS5 is self-content, not external-content. migrations/0001__initial.sql declares fts5(…) with no content= clause, but the comment claims external-content. The table stores its own copy of title/body/tags → ~doubles the text footprint (~3 KB/entity, ~28% of the DB at 100k). Fix → expand-contract migration to a contentless/external-content FTS5 table. A genuine, measurable storage win.
• FIND-2: the “flat correct below ~100k” rationale is conservative ~5–10×. Per the sqlite-vec numbers, flat 1024-dim search stays interactive (<100 ms) to ~500k–1M on a laptop; the real wall is the page-cache cliff. The HNSW deferral (SYN-225) is more justified than stated, but the specific “100k” figure in CLAUDE.md / the ADR should be re-validated by chart #1 before HNSW is ever built.

10. Recommended implementation phases (→ plan 021 / a benchmark-v2 ticket)

This supersedes the thin SYN-290 chart (which proved the renderer, not the product).

• Phase A: scaling latency + DB-size curves (highest value, ships first). Scale-free corpus generator (BA, hubs, log-normal lengths); per-subsystem timing (FTS / vector / hybrid / graph) at T1–T4; median+p5–p95 bands; chart #1, #2, #4; committed raw JSON.
• Phase B: resources. Floor-vs-slope RSS, disk (post-VACUUM), reindex, throughput; /usr/bin/time + VmHWM checkpoints; chart #3, #5, #6; fp32/int8 series.
• Phase C: correctness-at-scale. Planted-needle tier-stable query bank + hard negatives; recall@k / nDCG / MRR per tier per ranker; chart #7; the agent Recall@k_budget metric.
• Phase D: premium polish + CI. Confidence bands, breaking-point bands, dark/light, captions; --quick/--full; pinned-runner release workflow + artifact; README embeds the hero trio.
• Parallel: FIND-1 (external-content FTS) + FIND-2 (re-validate the flat ceiling) as their own tickets.

Discipline: real, reproducible, honest: curves not bars, cold+warm, percentiles, floor-vs-slope, equal-recall comparisons, raw data + one-command reproduce, machine stamp on every figure. Quality over a pretty single number.

Sources (selected)

sqlite-vec benchmarks https://alexgarcia.xyz/blog/2024/sqlite-vec-stable-release/index.html · SQLite FTS5 https://sqlite.org/fts5.html · ClickBench https://benchmark.clickhouse.com/ · DuckDB db-benchmark https://duckdblabs.github.io/db-benchmark/ · ann-benchmarks https://ann-benchmarks.com/ · Qdrant benchmarks https://qdrant.tech/benchmarks/ · getrusage(2) https://man7.org/linux/man-pages/man2/getrusage.2.html · dhat https://docs.rs/dhat/latest/dhat/ · BEIR https://arxiv.org/pdf/2104.08663 · LongMemEval https://arxiv.org/abs/2410.10813 · Obsidian vault sizes https://forum.obsidian.md/t/how-many-notes-do-you-have/36987 · Luhmann Zettelkasten https://www.ernestchiang.com/en/posts/2025/niklas-luhmann-original-zettelkasten-method/ · pythonspeed CI benchmarking https://pythonspeed.com/articles/consistent-benchmarking-in-ci/.