Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

ADR-040: Benchmark methodology + chart pipeline

Date: 2026-06-21 Status: Accepted Author: Rômulo Febasi

Scopes SYN-290 (#278). Extends the synapse-bench harness (SYN-162/163). Records the methodology contract that makes the numbers honest and the decision on how charts are produced for the README and synapse-releases.

Context

synapse-bench already generates a deterministic corpus, scores retrieval (P@k / MRR / nDCG), and measures search latency (median / p95) with PASS/FAIL gates. It is manual (owner decision: no CI) and produces a results.json + a .md report, but it has no charts, no machine/version stamp, no corpus checksum, and only p50/p95 (no tail). The README needs credible, reproducible performance visuals; a “premium chart” with a bare mean and no provenance is marketing, not evidence.

Market research (2026) on credible benchmarking converged on: fixed, checksummed corpora; report distributions (median + p95 + p99 + error context), never bare means; stamp every result with the machine + versions; keep cold/warm and stub/real separate; publish the raw data + a one-command reproduce; and never present shared-CI wall-clock as a headline. The gold standard for a Rust CLI is Astral’s uv, whose charts come from a pure-Rust pipeline (hyperfine JSON → poloto SVG → resvg).

Decision

Adopt the methodology contract below as binding, and render charts with a small in-crate SVG writer: no new charting dependency.

Methodology contract (binding on synapse-bench):

  1. Checksummed corpus. The corpus is seeded/deterministic; every results file carries its SHA-256 (chart::corpus_sha256) so a number is tied to exact bytes.
  2. Distributions, not means. Report median + p95 + p99 (added here). Never a bare mean.
  3. Machine/version stamp. Every results file carries a SpecStamp (os, arch, cpus, backend, synapse-bench version, git commit): printed in the chart footnote.
  4. stub vs real stays separate. stub = mechanics + latency (real timings, non-semantic vectors); real = the only configuration whose quality numbers (P@k/MRR/nDCG) are meaningful. Charts committed to the repo render latency only (honest under stub); quality charts require a real run.
  5. Publish raw + reproduce. The results.json is committed next to the SVG, with the exact command to regenerate it.
  6. No shared-runner headline wall-clock. CI numbers are relative/illustrative unless produced on a pinned/dedicated runner.

Charts: hand-rolled SVG (synapse-bench/src/chart.rs), no charting dep. A render subcommand turns a results.json into a deterministic, premium-styled SVG (Synapse palette, value labels, machine-stamp footnote). Rationale:

  • Determinism + zero new toolchain. Same JSON → byte-identical SVG; no Python, no Node, no headless browser; honors the house rule “no dependency without an ADR.” The only new dep is sha2 = "0.10" (already the parent crate’s pin) for the corpus checksum: justified by contract item #1.
  • SVG (not PNG) so the README stays crisp and the chart diffs in git.

Consequences

  • Improves: the README/releases get reproducible, stamped, distribution-aware charts with published raw data; the harness gains p99 + a corpus checksum + a machine stamp; no new toolchain or charting dependency.
  • Worsens / limits: the hand-rolled SVG is intentionally simple (horizontal bars); richer chart types would want a real library. Committed charts show latency only (quality needs a real run on a dedicated machine).
  • Deferred (follow-ups, not blockers): warmup + N-sample-per-query for tighter percentiles; explicit cold-vs-warm series; a real-backend quality chart from a dedicated runner; PR-time regression gating (bencher.dev / CodSpeed, relative-only). If the hand-rolled SVG outgrows its remit, revisit poloto (uv’s choice) or vl-convert (Vega-Lite → SVG, Rust-native), each a new dep needing its own ADR amendment.

Alternatives considered

  • poloto + resvg (uv’s stack). Strong and proven, but adds 2+ deps for a styling upgrade we don’t yet need; the hand-rolled SVG covers the current charts deterministically with zero charting deps. Revisit when chart types grow.
  • hyperfine for the latency track. Excellent for CLI wall-clock, but it is an external binary the harness would shell out to; the in-harness sampler (already present, now with p99) keeps the measurement self-contained and pure-Rust. hyperfine stays a documented “also run this” option.
  • matplotlib / Vega via Node. Rejected: drags a non-Rust toolchain into CI, against the project’s dependency discipline.