Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

ADR-041: Agent-in-the-loop benchmark driver

Date: 2026-07-04 Status: Accepted Author: Rômulo Febasi

Scopes SYN-298 (#286), plan 022. Records how the agent-in-the-loop benchmark drives a real AI agent through the governed-memory loop over MCP. Companion to ADR-040 (the retrieval/scaling methodology contract) and depends on SYN-297 (#285), which made Synapse’s MCP tools loadable by strict clients at all.

Context

The agent-in-the-loop benchmark (plan 022) must drive a real agent, not a mock: through insert / correct / contradict / retrieve, then grade the resulting memory state (via syn verify’s canonical dump). “Real agent” concretely means Claude Code: it speaks MCP, ships headless, and is the client we position against. The state grader (synapse-bench loop) already exists and is deterministic. The open question is the driver: the component that, per user turn, spins a fresh zero-context session, lets the agent choose and call Synapse MCP tools, and returns the trajectory + end state.

Two viable driving mechanisms, both verified to work after SYN-297:

  • Raw claude -p (headless CLI). A fresh session per invocation (no cross-turn bias), --output-format stream-json yields the tool trajectory + total_cost_usd. Needs the three-gate recipe: register the server local-scoped (claude mcp add --scope local), ENABLE_TOOL_SEARCH=false, --allowedTools "mcp__synapse__*". Zero new dependencies: the harness shells out to a binary the developer already has. Proven end-to-end on this machine (a headless run called capture_fact and enqueued a real pending proposal).
  • Claude Agent SDK (Python/TS). Takes mcp_servers + allowed_tools in code, bypasses the CLI trust/tool-search machinery, and lets the harness assert init.mcp_servers[].status == "connected" before the first call: the robust path for a fully unattended fleet. But it adds a non-Rust sidecar (a Python or Node process + package) to a project whose discipline is “no dep without an ADR” and whose harness is otherwise pure-Rust.

The benchmark is owner-run, not CI (credentials + token cost + agent non-determinism), so operational robustness matters less than it would for an unattended service; simplicity and zero-toolchain-drift matter more right now.

Decision

Drive the benchmark with raw claude -p (headless CLI) for now; adopt the Claude Agent SDK later, when the harness needs a fully-unattended fleet or programmatic PreToolUse/connection-status control.

Concretely:

  1. The synapse-bench driver shells out to claude -p … --output-format stream-json per user turn, using the SYN-297 three-gate recipe, and parses the stream-json stream for tool_use events (trajectory) and the final result event (answer + total_cost_usd + tokens). No new dependency.
  2. The driver is isolated behind a small internal interface (a Driver trait / one module) so swapping in an Agent-SDK-backed driver later is a localized change, not a rewrite of the harness.
  3. Every run records the exact claude version and model snapshot in the results, because the CLI’s behavior (and its gates) are version-sensitive: SYN-297 is the standing proof of that.

Switching to the Agent SDK is a future ADR amendment (it adds a sidecar dep), triggered by any of: needing assertable connect-before-call in an unattended fleet, needing PreToolUse hooks / programmatic permission control, or the CLI’s approval/tool-search gates shifting again in a way the recipe can’t absorb.

Consequences

  • Improves: zero new dependencies; the harness stays pure-Rust + a binary the developer already runs; a fresh -p session per turn gives clean zero-context isolation for free; total_cost_usd comes straight from the result event, feeding the cost-vs-accuracy Pareto (plan 022 Phase 4).
  • Worsens / limits: the driver is coupled to CLI surface that is version-sensitive and lightly documented (the SYN-297 saga): mitigated by stamping the claude version per run and by the strict-schema conformance test. No programmatic connect-status assertion (the CLI just fails the turn); the driver must detect “0 tools available” defensively rather than pre-flight it. No temperature/seed control → reliability must come from N runs, not a single pass (see below).
  • Open / revisit: the Agent-SDK amendment above. Also: if a future Claude Code release removes ENABLE_TOOL_SEARCH or changes local-scope auto-approval, the recipe breaks and the SDK path likely becomes mandatory sooner.

Reliability protocol (binding, since there is no seed control): every task is run N ≥ 5 times; report pass@1 (capability) and pass^k (reliability, identical correct end state across all k trials, the headline for a “can’t-silently-corrupt” tool), with the syn verify canonical dump digest as the equality key. Single-run numbers are not reported.

Alternatives considered

  • Claude Agent SDK now. The robust unattended path, but it drags a non-Rust sidecar into a pure-Rust, ADR-disciplined harness for robustness the owner-run benchmark does not yet need. Deferred to a future amendment with explicit triggers, not rejected.
  • A mock/scripted “agent”. Cheap and deterministic, but it measures the harness, not Synapse-under-a-real-agent: it cannot surface a real tool-selection or contradiction-handling failure (SYN-297 is exactly the class of bug a mock would hide). Rejected as the primary driver; a canned-trajectory replay stays useful for grader unit tests.
  • Another agent framework / a different model client. Out of scope now: Claude Code is the positioning target and the client that exposed the real MCP-conformance bug. Cross-client grading is a later extension.