Agent-in-the-loop benchmark for governed memory

Thesis (three load-bearing claims)

1. Grade the final STATE of the memory, not the transcript. Every serious agent benchmark (τ-bench hashes the mutated DB; SWE-bench runs tests against the end state; Terminal-Bench verifies the container) converges here. Synapse is uniquely ready: syn verify already emits a canonical logical dump, a drop-in analog of τ-bench’s get_data_hash().

2. The contradiction-surfacing metric is un-benchmarked, and it’s our positioning. Across HaluMem / MemoryAgentBench / Mem0 / Letta, every system either silently auto-resolves conflicts or grades only “did the latest value win.” Nobody rewards “detect the conflict and route it to a human gate”: exactly Synapse’s contradicts signal + BR-12.3. We get to define the metric. Do NOT grade contradiction as “latest wins” (that’s a correction); grade the surfacing.

3. Decouple retrieval from generation or the numbers are uninterpretable. A wrong answer has three distinct causes: the fact was never written, was written but not retrieved, or was retrieved but misused. A single end-to-end number can’t tell them apart. Our loop has a third stage the chat benchmarks lack, the write:

   WRITE (gate) → RETRIEVE (search) → READ (answer)
   right fact +    surfaced among      used correctly
   right op?       N distractors?      (+ abstains when absent)?
   

   Report a number per stage + an oracle-context ceiling (feed the gold fact directly → the generation bound).

Reusable designs (what to steal)

• τ-bench / τ²-bench (2406.12045, 2506.07982): canonical state hashing; ground-truth by action-replay (stays in sync with schema); a composable reward basis (DB-state ∧ invariant ∧ communicated-string ∧ tool-trace). Introduced pass^k.
• SWE-bench Verified (2310.06770): FAIL_TO_PASS (must flip true, the fix landed) + PASS_TO_PASS (must stay true, nothing corrupted). Maps perfectly: assertions that flip true after the write plus syn verify INV-1/2/3/5 that must stay green (the non-corruption story). And the “Verified” lesson: hand-audit the task set: broken graders are the norm.
• LongMemEval (2410.10813): 5 abilities (extraction, multi-session reasoning, knowledge-update, temporal, abstention); staged indexing→retrieval→reading with recall@k reported separately from QA; _oracle split = generation ceiling. The template for scenarios B and D.
• HaluMem (2511.03506): operation-level metrics, extraction (Recall / False-Memory Resistance / F1), updating (Update Accuracy / Hallucination Rate / Omission Rate). Exactly how to grade “did the RIGHT facts land.”
• MemoryAgentBench (2507.05257): the only one with a Conflict Resolution competency: counterfactual-edit pairs (MQuAKE) ordered later-supersedes-earlier. Steal the construction; note it grades resolution-given-“prefer-later”, not autonomous detection: our gate is the novel part.
• LoCoMo (2402.17753): persona + event-graph generation recipe (causally-coherent histories). Do NOT copy its grading: an audit found ~6.4% of its answer key wrong and a judge accepting ~63% of intentionally-wrong answers. The cautionary tale.
• RULER / NoLiMa (2404.06654, 2502.05167): non-parametric needles (coined names/uuids the model can’t answer from pretraining) with low lexical overlap to the question (force associative, not keyword, retrieval) + hard in-distribution distractors.

The four scenarios + how to grade each

Ground rules for all: fixed seeded corpus, pinned agent model + pinned judge snapshot, N≥5 trials/task, grade final state via syn verify dump + direct index queries. Task = {seed_state, user_messages, target_mutations(replay), assertions, gold_answer+evidence}.

• A: Insertion. User message with K plantable facts + M non-facts (chit-chat, hypotheticals) that must NOT be written. Grade: extraction precision/recall (query the result), False-Memory Resistance (the M non-facts don’t appear), operation correctness (capture_fact vs link_entity vs update_status: from the tool trace), and state∧INV after accept.
• B: Correction/update. Plant F1 early; later a user corrects it to F2 (counterfactual pair, F2 after F1). Grade: chose update/supersession (not a duplicate); F1 invalidated-not-deleted (x-valid-until / x-supersedes); “what is X now?” → F2, and --as-of <before> → F1 (bitemporal correctness: nobody else tests this).
• C: Contradiction (NOT correction). Plant F1; later user asserts F1′ that conflicts, framed as a fresh claim (no “actually/I was wrong”). Grade the governance, not which value won: contradicts fires ∧ routed to the gate (no auto-apply even under auto_apply=safe) ∧ no silent overwrite AND no silent duplicate. New metric: Contradiction Surfacing Rate + Contradiction Precision on a hard-negative control (compatible facts that look conflicting but aren’t: without it, a flag-everything system scores 100% and the metric is vacuous).
• D: Retrieval Q&A under noise. Question whose answer is one planted needle among N distractors (N sweep: 1k/10k/100k). Grade: retrieval leg (recall@k/MRR, isolated) + oracle leg (ceiling) + end-to-end (LLM-judged) → attribute the failure. Include abstention (a fraction whose answer was never planted → correct = “I don’t know”).

Grading mechanics

• Deterministic wherever possible (A/B/C): outcomes are checkable by querying DB state
  • hashing the syn verify canonical dump (τ-bench style) + INV invariants. No judge variance. Lean on this.
• LLM-as-judge only for open answers (D), with discipline (Zheng 2023 2306.05685): reference-guided (supply gold + evidence); judge with a DIFFERENT model family than the agent (preference-leakage, 2502.01534); pin the judge snapshot; audit judge-human agreement and report it before trusting (if <~80%, the judge is noise).
• Trajectory grading (τ² ACTION basis): assert the MCP tool-call sequence contains the expected op-kinds.

Non-determinism + cost protocol (honest reporting)

Agents are stochastic + expensive; single-run numbers are dishonest (Kapoor, 2407.01502).

• pass^k (reliability) is the headline for a tool whose pitch is “can’t silently corrupt”: it requires the identical correct mutation across all k trials, surfacing nondeterministic corruption pass@1 hides. Report pass@1 (capability) + pass^k (reliability) on write/contradiction scenarios.
• N≥5 trials/task; 95% CIs, cluster SEs on the randomization unit (Anthropic 2411.00640: naive SEs run ~3× too small).
• Pin the substrate: exact agent model snapshot in the run name, fixed seeded corpus, pinned judge, disclosed temperature/step caps.
• Cost as a first-class axis: a cost-vs-accuracy Pareto (HAL/Kapoor), tokens + USD per run (Claude Code reports total_cost_usd per headless run). Bound with step caps.
• Not CI-able (credentials + cost + variance): owner-run, like the real-backend bench.

Driving Claude Code headlessly (verified + corrected against claude v2.1.197)

Getting write tools to fire headless means clearing three stacked gates: a fact the PoC learned the hard way (see below). --permission-mode bypassPermissions clears none of them (it governs the permission prompt, not MCP approval or tool-search):

1. Approval / trust. A project .mcp.json (or inline --mcp-config) server sits at ⏸ Pending approval in headless and its tools never load; only its resources do. Since v2.1.196 enableAllProjectMcpServers in project settings is ignored in an untrusted folder: it must live in ~/.claude/settings.json. Fix: register the server local-scoped: claude mcp add --transport stdio --scope local synapse -- <abs>/syn mcp (auto-approved; --scope user has open bugs #32939/#54803).
2. Schema validation. Claude Code ≥2.1.x validates each tool’s outputSchema and drops the entire tool list if any fails. Synapse tripped this (see PoC / SYN-297). Fix landed in src/mcp/output_schema.rs.
3. Tool-search deferral. ≥2.1.7 defers all MCP tools behind ToolSearch. Fix: ENABLE_TOOL_SEARCH=false + --allowedTools "mcp__synapse__*" (eager-load + pre-approve).

Working recipe (proven E2E on this machine: capture_fact enqueued a real pending proposal):

claude mcp add --transport stdio --scope local synapse -- "$(command -v syn)" mcp   # once, per workspace
ENABLE_TOOL_SEARCH=false claude -p "<user turn>" \
  --allowedTools "mcp__synapse__*" \
  --output-format stream-json --verbose --model <pinned>

Other confirmed flags: --bare (skip CLAUDE.md/settings → clean context), --strict-mcp-config, --output-format (json/stream-json), --model/--effort/--no-session. Each -p is a fresh session. stream-json events carry type: tool_use (name+input) → trajectory; the result event carries final text + total_cost_usd + token usage. No temperature/seed control → average over N runs. Robust alternative for the harness: the Claude Agent SDK (Python/TS) takes mcp_servers + allowed_tools in code and asserts init.mcp_servers[].status == "connected" before the first call: bypassing the CLI trust / tool-search machinery entirely. Recommended when the driver must be fully non-interactive.

Proof-of-concept findings (run on this machine, 2026-07-03 → root cause 2026-07-04)

A live PoC (claude v2.1.197 + the synapse MCP over a fresh workspace) validated the harness mechanics and caught a real, shipping bug: on ourselves, exactly as intended.

• ✅ The harness works. We drove claude -p against the synapse MCP, captured the full stream-json trajectory (tool calls + inputs + final answer + cost) and can inspect the resulting workspace state. This is the eval loop.
• ✅ Read resources connect even when tools don’t (they follow a separate gate), synapse://workspace/server-card returned live data, which is what first proved the server was not “off”.
• ❌→✅ Write loop was BLOCKED, now fixed. First read as “Claude Code 2.x defers all MCP tools behind ToolSearch” (the agent called ToolSearch 16× and mcp__synapse__propose_batch returned “No such tool available”). Deferral is real (gate 3), but it was masking the true root cause. Once the approval gate was cleared with --scope local, the status became Connected · tools fetch failed with Invalid input (at tools.2.outputSchema.properties.result): Synapse’s SYN-177 read envelope emitted outputSchema with result: true (the boolean schema serde_json::Value derives). Draft 2020-12 permits true; Claude Code’s strict (Zod) validator rejects it and drops the whole 24-tool list → the agent sees zero Synapse tools. This breaks any current-Claude-Code user, not just the bench. Fixed in src/mcp/output_schema.rs (a permissive AnyValue → {}, never true; regression test); filed as SYN-297. After the fix + the working recipe above, a headless claude -p called mcp__synapse__capture_fact and enqueued a real pending proposal: end-to-end.

This is exactly what the benchmark is for. The eval’s first act was to find a governed-write regression the unit tests missed (they assert schema shape, not strict-client acceptance). Phase 0 of the harness is that fix (done) plus a syn verify-style strict-schema conformance test so it can’t regress.

Corpus design (the haystack)

Extend the deterministic generator (synapse-bench/src/corpus.rs) + the persona fixtures (SYN-296): non-parametric planted needles (coined companies/uuids: no pretraining leak), low question↔needle lexical overlap (associative retrieval), hard in-distribution distractors, causally-coherent correction/contradiction pairs (persona + event graph), each with a recorded intent label (correction vs contradiction) so the gold governance behavior is known. Keep it in-repo as the example workspace (synapse-bench/fixtures/example-workspace/): deterministic + versioned; a separate repo only if it balloons past ~50 MB.

Top pitfalls (the critique)

1. One end-to-end number (hides which stage failed). 2. Single-run scores. 3. Grading the transcript, not the state. 4. Same model generates corpus + answers + judges (preference leakage). 5. Trusting the answer key (LoCoMo’s ~6.4%-wrong key). 6. Conflating contradiction with correction: grading “latest wins” measures the auto-overwrite the gate exists to prevent. 7. No hard-negative control for the gate/abstention (flag-everything scores 100%). 8. No cost axis. 9. Ignoring the bitemporal --as-of correctness (our strongest, least-tested story).

Sequencing → plan 022 / SYN-298 (Phase 0 fix = SYN-297)

Phase 0: make the headless write loop work, the invalid-outputSchema fix is done (SYN-297); harden it with a strict-schema conformance test + document the working recipe. Phase 1: synapse-bench loop mode: task spec + a driver that runs claude -p per user turn, captures the trajectory, and grades final state via syn verify dump + index queries. Phase 2: the four scenarios + the deterministic graders; ship Contradiction Surfacing Rate with a hard-negative control (the novel, on-brand number). Phase 3: retrieval/oracle/end-to-end decoupling for Q&A + an LLM judge (different model, audited). Phase 4: pass^k + cost-Pareto reporting with CIs over N≥5; a committed example workspace with planted needles + scripted corrections/contradictions.

Sources (selected)

τ-bench https://arxiv.org/abs/2406.12045 · τ²-bench https://arxiv.org/abs/2506.07982 · SWE-bench https://arxiv.org/abs/2310.06770 · LongMemEval https://arxiv.org/abs/2410.10813 · HaluMem https://arxiv.org/abs/2511.03506 · MemoryAgentBench https://arxiv.org/abs/2507.05257 · LoCoMo https://arxiv.org/abs/2402.17753 + audit https://dev.to/penfieldlabs/we-audited-locomo-64-of-the-answer-key-is-wrong-and-the-judge-accepts-up-to-63-of-intentionally-33lg · RULER https://arxiv.org/abs/2404.06654 · NoLiMa https://arxiv.org/abs/2502.05167 · LLM-as-judge https://arxiv.org/abs/2306.05685 · AI Agents That Matter https://arxiv.org/abs/2407.01502 · statistical evals https://arxiv.org/abs/2411.00640 · Claude Code headless https://code.claude.com/docs/en/headless.md.