The Synapse benchmark, explained for non-experts

This is the plain-language companion to BENCHMARKS.md. It explains what the metrics mean, how the recorded tests ran, which results they support and where their evidence is limited. The observations are owner-run engineering experiments, not a human field study or a promise about every workload, agent or released executable.

The question it tries to answer

Imagine an assistant maintaining a team’s notebook: API owners, decisions, constraints and open tasks. A useful benchmark needs to check whether the assistant preserves the expected recorded state, whether supported conflicting changes receive the intended review and whether later questions retrieve the right evidence.

The question is: under the recorded scenarios, what changed, what remained intact and what did the checks actually cover? Storage integrity, supported structured conflicts and factual truth are different questions. A result about one does not establish the others.

There are two test families:

1. Scaling, with deterministic offline measurements of latency, size and resources.
2. Agent in the loop, where a real agent uses MCP and the harness grades the resulting state or answer. These runs require the selected provider and are not offline tests.

The current source overview adds a plot of the stored persona retrieval observations. Its 20-question fixture is separate from the 10-question agent Q&A scenario below. Do not merge their sample sizes or present one as a rerun of the other.

Glossary, in human terms

Distractors. Irrelevant records deliberately added to make retrieval harder. Finding the same target facts among 2,000 or 8,000 additional records tests those particular noise fixtures. It does not reproduce every kind of accumulated organizational knowledge.

Recall@10. The fraction of target facts found within the first ten ranked results. recall@1 asks whether the fact ranks first. A hit measures retrieval against the recorded reference; it does not certify that the reference or generated answer is true.

Noise chart. A plot of retrieval results as distractors increase. In the recorded agent Q&A noise probes, top-ten recall was 80% without distractors and 100% at the two measured noise levels, +2,000 and +8,000. That is a small set of observations, not a flat curve proven at every corpus size. The separate persona fixture starts at 95% top-ten recall and reaches 100% at its larger measured corpus sizes.

Saturation. A metric reaches 100% on its tested sample. This can indicate a successful scenario or a test too small/easy to discriminate. “10 of 10 runs” makes the sample visible; it is not a general guarantee.

Gate. The deterministic proposal guard for supported structured changes, such as certain status or relationship conflicts and identity collisions. It does not require a model to notice those supported cases. General disagreements in prose are outside that detector’s scope, and unsupported proposal classes must not be counted as semantically consistent. See ADR-062.

e2e (end-to-end). The agent receives a question, uses only the allowed MCP tools and produces an answer. This checks the complete tested agent/tool loop rather than one search. It remains an engineered scenario with a selected model, fixture and grading method.

The three attribution legs. Each question is examined in three ways:

• Retrieval: is the reference fact present in the raw top-k results?
• Oracle: can the model answer when given the reference evidence directly?
• e2e: can the agent find and use evidence through MCP without that direct injection?

Differences help investigate failures. A high oracle score with a lower e2e score suggests problems in the retrieval/reasoning/tool loop. It does not identify one cause conclusively without inspecting the run.

pass@1 vs pass^k. The harness reports single-attempt success frequency as pass@1. Its pass^k field records whether every graded run passed its assertions. In this harness it is an all-runs result, not a separately estimated population success probability. Neither a lucky success nor a small all-pass sample proves future reliability.

95% Wilson confidence interval (CI). With 12 successes in 15 insertion attempts, the observed rate is 80%, with a Wilson interval of about 55% to 93%. The width reflects sampling uncertainty under the experiment’s assumptions. It is not a forecast that production performance will fall within that range on a different workload.

How the tests were actually run

The agent-in-the-loop harness limits Claude Code to Synapse MCP tools. A --disallowedTools block removes file, shell and web tools from the allowed interface. This keeps the scenario focused on MCP-driven memory instead of direct repository reads. It is a harness configuration, not a sandbox against a separately privileged host process.

After each task, the harness grades the canonical recorded state and explicit assertions. A verification report checks its named invariants; signing depends on the available key and selected policy. A canonical dump or signature does not establish the truth of all prose. This state-oriented grading is analogous to the τ-bench get_data_hash() approach, rather than treating a convincing transcript as successful storage.

The agent is nondeterministic, so the harness repeats each task. A different-model judge (Sonnet judging an Opus agent) scores Q&A against references. Both models are from the Claude family, so that is not independent human adjudication or a cross-family assessment.

The recorded scenarios are insertion (A), correction (B), a supported conflicting change (C, with agent behavior and the isolated gate measured separately), and retrieval Q&A (D). The harness in synapse-bench/ in the source repository and methodology retain reproduction details.

What the recorded runs show

The agent Q&A noise probes are distinct recorded samples:

The methodology attributes the +2,000 e2e dip to a transient claude -p failure on a five-question sample and records a retry. The observed dip remains part of the evidence; a suspected infrastructure cause does not prove the absence of retrieval or answer-quality problems in other runs.

In the baseline Q&A sample, two reference facts were missing from the first raw top-ten search, yet the agent produced correct answers after additional searches. This shows recovery of those two gaps in that run. It does not imply that agents recover every retrieval miss.

The conflict scenario separates two mechanisms. The agent voluntarily notices a conflict; Synapse separately holds a supported structured conflict when the isolated guard is tested. The first depends on agent behavior. The second checks its documented rule classes and configured write path. Neither result establishes arbitrary natural-language contradiction detection, universal non-corruption or a privileged-process security boundary.

Where the benchmark is weak (honest self-critique)

1. Small samples. The agent scenarios use 8 to 15 runs. The intervals are wide, and repeatedly running a similar fixture does not create broad field coverage.
2. Related judge and agent. Sonnet and Opus share a model family. Style preferences or shared blind spots can affect grading. Cross-family and human review would add evidence; neither is implied by the current scores.
3. Saturated metrics. Several samples reach 100%. More diverse cases, harder negatives and repeated seeds are needed before drawing broader conclusions.
4. Execution failures affect results. The five-question noise probe includes an infrastructure-associated answer-score dip. Report retries and failures instead of presenting only the successful rerun.
5. The 100,000-distractor tier was not run. Its approximately two-hour embedding cost is an estimate in the methodology. It supplies no measured quality result at that size.
6. Stub scaling is mechanical evidence. The large scaling curves use nonsemantic vectors. Their latency/size measurements do not establish semantic retrieval quality.
7. The persona fixture has incomplete provenance. Its stored JSON records 20 questions, macOS arm64 and six corpus sizes, but the original run’s source commit is unknown. It cannot establish performance of every later development commit or packaged release.

Weaknesses of Synapse itself that the benchmark exposed

1. Insertion needs better reliability. Three of fifteen attempts failed their complete insertion assertions. Saving the wrong or incomplete context is a practical weak point, even if a later storage-integrity check passes.
2. First-result retrieval still misses targets. Early persona notes reported around 0.78 recall@1; the dedicated chunking comparison reported a full-haystack improvement from 0.75 to 0.85 (SYN-301, ADR-042). In the stored 463-entity endpoint, 17 of 20 targets rank first and all 20 rank in the top ten. Those are separate recorded comparisons, not a new measurement of every current release. An opt-in reranker did not improve that corpus’s endpoint further.
3. Agent caution is not a Synapse guarantee. The 10/10 preserved-fact sample depends on the agent asking first. The isolated structured gate has its own evidence and scope; it cannot replace reasoning about every possible conflicting sentence.
4. Agent-driven temporal correctness remains a separate coverage question. The original agent benchmark did not assert complete --as-of correctness. The bitemporal-probe checks the supported write path; later native tests and current source contracts must be inspected independently. Do not infer a completed agent-loop temporal proof.
5. Semantic models have an ingestion and memory cost. The documented multilingual default download is about 2.5 GB. The measured embedding throughput, around 13 documents per second on its recorded CPU, makes large initial imports expensive. That is not a universal throughput estimate or a guarantee that later queries are unaffected by resource contention.

Use the raw persona observations, complete benchmark methodology and verification scope together when assessing these results.