Agent-in-the-loop benchmark: first empirical findings

Headline numbers

¹ contradict cost is the sum of the two N=10 passes (bald + directive turn). pass@1 shown is for the reframed non-corruption grade (see below); the first assertion set mis-measured.

Finding 1: capability (80%) ≠ reliability (pass^k = no)

insert-01 held pass@1 = 80% at both N=5 and N=15: a stable rate, not sampling noise. The three failures decompose into three distinct causes, which is the whole point of grading state per-run:

• 1/15 infra flake (run 7): the agent surfaced zero Synapse MCP tools (agent invoked no Synapse MCP tool). This is residual SYN-297-class flakiness: even with the fix + the local-scope + ENABLE_TOOL_SEARCH=false recipe, headless tool-loading fails intermittently (~7%). Actionable: the driver should detect-and-retry a zero-tool turn.
• 1/15 agent no-op (run 3): the agent explored (search_knowledge, list_by_type) and wrote nothing. Genuine agent unreliability: sometimes it reasons and gives up.
• 1/15 brittle assertion (run 1): the person was created (via capture_fact), but the body didn’t contain the literal “data engineer”: the agent authored in pt-BR on some runs. This measured phrasing, not extraction. Fixed: dropped the substring check; grade the structural fact (entity exists). By that measure capability is 13/15 (87%).

pass^k = no and 12 distinct state-hashes across 12 passing runs: every successful capture produced a unique syn verify dump (timestamps + agent-authored prose + tags all vary). This validated redefining pass^k on the graded assertions, not on a bit-identical hash: the τ-bench hash-equality trick doesn’t transfer to Markdown-with-timestamps + freeform prose.

Finding 2: the contradiction is caught by the AGENT, not (only) the gate

The marquee, on-brand result, and it inverts the naïve expectation. Told “record that Rafael is the lead of the Prefetcher work stream” over a workspace where Marina is the seeded lead, the agent wrote nothing in 10/10 runs (bald framing and directive framing). Not a failure: reading its final answer shows why:

“Antes de registrar, preciso te alertar sobre um conflito direto… A entidade da Marina diz explicitamente… ‘Marina is the lead of the Prefetcher work stream.’… Não quero sobrescrever isso silenciosamente… Como toda escrita no Synapse passa pela fila de aprovação (pending), nada é aplicado até você confirmar.”

The agent searched, found the conflict itself, refused to overwrite, laid out the options (leadership transition / co-lead / mistake), and deferred to the human. So:

Governed memory has two independent layers of contradiction defence:

1. The agent’s own judgement: a capable agent surfaces the conflict and asks. This is the first line and, with Claude 2.1.197, it fired every time.
2. Synapse’s gate (SYN-242 + BR-12.3): the second line, for when the agent does not notice: a less-capable/more-autonomous agent, a subtler conflict, or too much data to spot it. This is precisely where the deterministic gate earns its keep.

Implication for positioning: don’t pitch the gate as the thing that stops a careless write in the common case: a good agent already hesitates. Pitch it as the non-bypassable floor that holds when the agent’s judgement doesn’t. And the strongest, cleanly-true claim the runs support today is non-corruption: across every insert and contradict run, no existing fact was ever silently overwritten and no unsanctioned write was ever applied (accept_pending=false kept any proposal gated; INV-1/2/3/5 held after a rebuild).

Implication for the benchmark: to measure SYN-242 in isolation we must bypass the agent’s judgement: a direct-MCP probe that files a conflicting update_status/link_entity with no agent reasoning, then asserts the contradicts signal fires and the write is gated. That is the next plan-022 increment (a gate-probe mode distinct from the agent-in-the-loop drive).

Harness corrections this shook out (all landed)

• pass^k keyed on assertions, not bit-identical state (timestamps + prose vary).
• file-contains case-insensitive; dropped a brittle substring assertion: grade structural facts, not LLM prose (multilingual output makes literal matching noise).
• Reindex after the gate before grading: accept’s incremental index can diverge from a full rebuild (INV-1) though the Markdown is correct, the index is disposable, so rebuild.
• contradict-01 reframed to grade non-corruption (which holds), not “a proposal was queued” (which assumed a write the agent chose not to attempt).

Cost

~$0.24/run (insert), ~$0.18/run (contradict, read-heavy). $3.65 for N=15. A full four-scenario suite at N=15 with a judge on scenario D will run tens of dollars per sweep: consistent with “owner-run, not CI” (ADR-041).

Round 2: all four scenarios + gate isolation + MCP-only isolation

After the round-1 fixes, added: MCP-only isolation (a per-turn system prompt + a --disallowedTools block for Read/Grep/Bash/Web* → the agent cannot read the host repo’s files/docs; it must use Synapse’s MCP tools, the benchmark measures MCP-driven memory, not the agent reading project docs), a zero-tool + crashed-CLI retry, 95% Wilson CIs + cost-per-pass, and two new scenarios + the gate-isolation probe.

The recurring failure is claude -p crashing (exit 1), not Synapse: one crash in each N=8 run (started around a host suspend/resume). The driver now retries a crashed turn, so future pass@1 reflects Synapse, not CLI flakiness. With that, the governance behaviours (update-on-existing, abstention, non-corruption, gate detect+gate) are effectively clean.

Scenario B works reliably: the agent consistently searched → get_entity → update_status on the existing Aurora entity, never spawning a duplicate project. Scenario D abstention works: given a never-planted fact the agent said (in pt-BR) it had no record and wrote nothing: false-memory resistance held. Gate isolation is the clean, deterministic Contradiction-Surfacing number the agent-in-the-loop path can’t produce (because a capable agent hesitates before ever filing the write).

Round 3: bitemporal correction: a real gap the benchmark surfaced

Building the scenario-B --as-of bitemporal assertion (“correct a fact, and --as-of <before> still returns the old value”) exposed that it cannot be driven by the agent’s natural correction, and why:

• update_status overwrites the field in place; it is NOT bitemporal. After an agent correction (Aurora active → archived), the entity row is valid_from=<created>, valid_until=NULL, supersedes=NULL and search --as-of <a date before the correction> returns the entity with the current status (archived), not the old one. The old value survives only in the git/audit trail, not the queryable valid-time axis. So the “correct it → --as-of before → the old value” story holds for fact supersession (x-supersedes + x-valid-from), not for field-level update_status corrections. Filed as a gap (SYN-301).
• Supersession closure runs on the insertion path, not bulk reindex. A hand-authored F1 + F2(x-supersedes: F1) reindexed shows the --as-of filter working before F2 (F1-only ✓), but after F2 both facts return: F1’s valid_until was never closed, because the “new-after-old closes F1” rule (ADR-031) fires when a fact is inserted (propose/accept), not when pre-existing Markdown is bulk-reindexed. A deterministic bitemporal-probe must therefore go through the write path (like gate-probe), not a reindex of fixtures.

Consequence for the benchmark: the bitemporal-correctness signal is best measured by a deterministic bitemporal-probe on the write path (propose F1, propose F2 with x-supersedes, assert --as-of before→F1 / after→F2 and F1 invalidated-not-deleted), the agent-driven scenario B can only assert the durable corrected state, not the --as-of reconstruction. And the marketing pillar should be scoped honestly: bitemporal --as-of is a fact-supersession property, not something a field update_status gives you.

Round 4: bitemporal-probe + noise-scale sweep (both landed)

• bitemporal-probe (deterministic, write-path): propose F1, then F2 with x-supersedes
  • a later x-valid-from, accept both (the accept’s resolve_supersession closes F1), rebuild, and assert F1 closed-not-deleted, F2 open, --as-of before→F1 / after→F2, INV-5 holds. PASS: the on-brand bitemporal-correctness signal the agent scenario can’t produce (SYN-301). This is how the “temporal story nobody tests” is actually tested.
• Scenario D under noise (qa --distractors N): recall@10 and the oracle ceiling stay at 100% through +8 000 distractors (e2e ~100%; the one dip is a 5-needle claude -p blip). The agent’s token cost is ~flat in N (it searches via MCP), so only the embed scales, the 100 000 tier is a ~2 h one-time embed, documented not run. Retrieval does not degrade at the low-thousands a real brain holds.

Next (plan 022)

1. ✅ gate-probe: SYN-242 detect+gate in isolation. Done.
2. ✅ Driver zero-tool + crash retry: absorbs the headless flakes. Done.
3. ✅ Scenario B / D: abstention + three-leg judge + --distractors noise sweep + the deterministic bitemporal-probe (the --as-of signal the agent path can’t drive).
4. ✅ 95% Wilson CIs + cost-per-pass. Done.
5. Remaining: the 100 000-distractor tier (a ~2 h embed) on demand; a judge–human agreement audit before trusting scenario-D e2e as a headline; a possible product fix so update_status corrections become valid-time-reconstructable (SYN-301).