Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

ADR-032: Identity fields: deterministic dedup, detect-don’t-merge

Date: 2026-06-15 Status: Accepted (owner-delegated review, 2026-06-15) Author: Rômulo Febasi

Extends ADR-028 (open types) and ADR-022 (write gate). Companion to ADR-031/SYN-242 (contradiction → gate).

Context

mem0’s #1 unsolved bug is duplicate facts: “My name is X” then “My name is Y” stores both (exact-hash dedup only). In Synapse a re-capture of the same real-world thing under a different title produces a different slug (the id = slug = filename) and so a different entity: a silent duplicate the human gate can’t easily spot. SYN-242 catches structured contradictions (same subject+predicate, different object) but not “same thing, captured twice.” We need a deterministic way to recognise that two captures denote the same identity: without an opaque entity-resolution model and without breaking the pillars.

Decision

A custom type may declare identity fields in types.toml: the fields whose combined values denote the same real-world thing:

[types.book]
indexed  = ["isbn", "author"]
identity = ["isbn"]          # two books with the same isbn are the same book

At propose-time, a capture_fact whose identity-field values match an existing entity of the same kind (with a different id) is flagged as a duplicate-identity collision and routed to the human gate: exactly like a contradiction (SYN-242), surfaced with a contradicts/collision signal so the agent can self-correct (re-capture as a x-supersedes update, or drop the duplicate). Three hard constraints:

  1. Never auto-merge. Detect, don’t resolve: the human decides (consistent with core::contradiction; auto-merge is the <26%-correct stage HaluMem measures).
  2. Never derive the id/slug from the identity hash. The id stays the slug = filename = human-editable, git-diffable, stable. The identity is a comparison fingerprint, not the key. This protects Markdown-as-truth + git-native.
  3. identity ⊆ indexed. Every identity field is already a bound query key (ADR-023 invariant): the match is a parameterised lookup over the core columns + entity_fields, never interpolation.

Detection is a deterministic set-intersection of candidate ids, one identity field at a time (bound values), bailing the moment the intersection is empty. Opt-in per type (empty identity = no dedup). Fuzzy/MinHash/LSH matching is deferred: single-user corpora don’t have the volume that makes blocking pay, and fuzzy matching introduces false-merge risk.

Consequences

  • Improves: the deterministic first tier of entity resolution that the field’s best systems use; sharpens SYN-242 (a duplicate identity is a kind of contradiction); keeps the gate’s incoming quality high (the gate’s value is the scarce reviewer attention). Zero new deps; reuses entity_fields.
  • Worsens: one more types.toml knob; the O(n) candidate scan per propose (fine at single-user scale; revisit only under a SYN-227 budget breach). Identity fields must be indexed, slightly constraining the declaration.
  • Open: cross-kind identity; the merge action (today it’s detect→gate→manual supersession): deferred until real demand. The merge-safe multi-author angle is a Commercial/team concern, not single-user.
  • Update (SYN-254 expansion, 2026-07-06): identity matching now folds case, accents (Unicode NFD + combining-mark strip), and whitespace before comparison (model::slug::normalize_for_match), so a re-capture differing only in those is detected: while punctuation is preserved (C++ ≠ C#) to avoid over-merge. This is safe here precisely because the design is detect-don’t-merge: widening detection only routes more candidates to the human gate, never auto-merges. Fuzzy/similarity (MinHash/LSH/edit-distance) stays rejected: it is over-engineering at single-user scale and would trade the deterministic guarantee for a similarity threshold to tune.

Alternatives considered

  • Derive the entity id from hash(identity fields). Rejected: it makes the id unstable under any identity-field edit, fighting git blame, syn at <ref>, human editability, and diffability. The hash is a comparison fingerprint, never the key.
  • Fuzzy/MinHash/LSH blocking. Over-engineering at single-user scale and a false-merge hazard; a flat exact-match scan over a few thousand candidates is correct and cheap.
  • Auto-merge on collision. Violates the moat (the human gate) and is the exact stage the field gets wrong (<26%). Detect-don’t-merge.
  • A full entity-resolution model. Opaque, non-deterministic, off-thesis.

Research grounding

mem0 #4896 (duplicate facts); HaluMem auto-update correctness <26%; Zep/Graphiti + mem0 LLM reconciliation as the hard tier; classic deterministic-first / blocking entity resolution. See docs/researchs/2026-06-15-v0.11-issue-direction-study.md §3 (SYN-254).