Typed decisions, small NLI and trust: an evaluation protocol

Date: 2026-10-12 · Status: research/design; no model benchmark executed Author: Rômulo Febasi · Baseline: 3ceddd0

Question

Can a narrow decision model reduce review cost for prose changes and contradictions while preserving deterministic acceptance authority and explicit integrity boundaries? User reference: Jev by TypeSafe AI, not JEPA.

Findings from primary sources

TypeSafe documents Jev as a hosted System One model returning Choice, Score and Noul rather than generated prose. It describes focused questions, typed answers and confidence-aware routing. This is an appropriate architectural reference for triage and source relevance; it does not establish a small local artifact or proven Synapse performance. Introduction.

The current model page identifies versioned jev-1.13.0 and moving aliases. Pin resolved versions when evaluating thresholds; do not adopt vendor latency/cost comparisons as product results. No deployable local weights or parameter count were established in this research. Model specification.

Known limitations include literal/negation interpretation, arithmetic/date handling, irrelevant context, hostile input and option-order effects. Keep exact calculations in code and test rubric/label perturbations. Typed output can still be a wrong decision. Official limitations, confidence.

cross-encoder/nli-deberta-v3-small is an Apache-2.0 English NLI reference, trained for contradiction/entailment/neutral. Its task-specific scores are not business-domain calibrated probabilities. Evaluate compatibility with the existing ONNX runtime before use. Model card.

MoritzLaurer/mDeBERTa-v3-base-mnli-xnli is a MIT multilingual NLI comparison candidate. Multilingual metadata is insufficient to assert Portuguese architectural/API quality; measure PT fixtures explicitly. Model card.

Two recent preprints investigate typed decision robustness and paired System One evaluation. They motivate tests rather than a conclusion about Synapse or a proven replacement engine. These papers are emerging evidence, not independently reproduced here: type/label robustness, paired evaluation.

Recommended experiment

1. Define a finite relation vocabulary and allowed actions. Model suggests review; code controls permission/acceptance.
2. Produce a human-reviewed EN/PT corpus of architecture and shared API facts, with source/environment/version/date scopes.
3. Split by source/project/claim families to prevent near-duplicate leakage. Freeze test data before threshold tuning.
4. Compare exact rules + retrieval, local English NLI, multilingual NLI and optional consented Jev under identical candidate sets.
5. Measure candidate retrieval recall separately from pair-classifier recall. Test joint pipeline as well as oracle candidate pairs.
6. Evaluate both pair directions when entailment matters; distinguish genuine conflict from later valid change or separate environments.
7. Freeze model/rubric/schema/input digests and hardware stamp; retain sanitized artifacts and repo commit.
8. Choose thresholds on validation only; review low-confidence/unsupported cases, then report held-out outcomes and intervals.

Cases: negation, allowed exceptions, multiple clauses, v1/v2 coexistence, prod/staging, dates/counts, source wording drift, implicit architectural consequences, missing evidence, unrelated text, adversarial instructions, renamed/permuted labels and source access loss.

Metrics: alert precision/recall, candidate recall, macro-F1 by language/relation, false-positive review minutes, abstention/coverage, Brier score or calibration curves where probabilities are available, p50/p95, cold-start/RAM, query/model input cost and timeout rate. Report estimates separately from official measured usage. Treat all thresholds as domain-specific.

Proposed pilot gate: precision >=0.90 for conflict-review alerts and scoped conflict recall >=0.80, with sample size and bootstrap confidence intervals reported. These targets are not claims of achieved reliability; choose final workload budgets and review budget before benchmarking. A failed gate means keep the adapter experimental or do not ship it.

Action contract

• Same fact: suggest deduplication/relation; never silently merge identities.
• New scoped fact: propose creation with source revision.
• Later valid fact: propose temporal supersession/update; preserve previous validity.
• Potential conflict: show both evidence spans and require review.
• Missing/ambiguous context: abstain and request context or reviewer input.
• Suggested patch: a generative model or deterministic template drafts it separately; Jev does not write prose. Apply only with upstream revision precondition and explicit write capability.

A successful classifier does not add a universal semantic Pass to syn verify. Integrity proof evaluates exact records; a classifier estimates relations between claims.

Deployment posture

Local rules remain available without models. Local model downloads require explicit configuration, artifact digest/license/runtime checks and defined memory/latency budget. A local model swap changes disposable model-derived data, not accepted truth.

Jev is an optional edge adapter, not a default dependency. Read source ACL/egress policy before sending content, minimize data, keep credentials outside knowledge, set deadlines/rate/budget, pin returned model and fail to abstention/review. No paid API calls, downloads, training or local benchmark were performed for this document.

Integrity, identity and blockchain analysis

Useful blockchain-related ideas are authenticated append-only records, hash links, content addressing, signatures and independently witnessed heads. Existing Synapse already has some of them. The gap is coverage of actual mutations and preserved receipt/trust material, not the absence of a distributed consensus network.

OIDC login, device key possession, membership authorization and independent witnessing solve distinct problems. A blockchain address does not map itself to an employee, and a checkpoint cannot prove that the stored fact was true. Begin with sealed transitions and independent checkpoints; benchmark/log-volume evidence may later justify Merkle proofs. No on-chain content/identity; optional private/public anchor adapters require privacy and operational evaluation.

Standards consulted: OIDC Core, OAuth native apps, OAuth BCP, RFC-9162, RFC-8785, Sigstore bundles. Their mechanisms are inspiration/compatibility targets; Synapse does not claim to implement them merely by referencing them.

Decision and revisit trigger

Accept the provider abstraction and the evaluation protocol. Defer choosing a default model, training a new architecture, adding a second native runtime or operating a blockchain until the experiment demonstrates a benefit or a customer provides a concrete trust requirement. See ADR-050 through ADR-053 and the linked implementation issues.