Scoped EN/PT decision evaluation: 2026-10-13
Decision: ship a conservative read-only provider seam and experimental adapters; enable no new model default. Neither evaluated model reached the proposed 90% precision / 80% scoped-conflict recall gate. Model probabilities are diagnostic evidence, never acceptance or integrity proofs.
Measured evidence
The original synthetic corpus has 96 cases, 24 topic families, 32 training cases and 64 heldout cases (32 EN, 32 PT). Entire family variants remain in one split. The corpus hash/split were fixed before threshold selection. Labels were reviewed by the coding agent against the narrow engineering contract, not independently human-labeled or obtained from a Febasi pilot. Negation, exceptions, environments, versions, dates, changed option names, distractors, injected instructions and window overflow are represented.
| Provider | Heldout alert precision | Conflict recall | Abstention | Warm model-only p95 | Peak process RAM | Cold model initialization |
|---|---|---|---|---|---|---|
| Exact-text + scoped filters | undefined (no alerts) | 0% | 87.5% | not applicable | 25 MiB | not applicable |
| English DeBERTa small | undefined (disabled by training threshold) | 0% | 87.5% | 47.79 ms | 932 MiB | 4.18 s |
| Multilingual MiniLM six layers | 100% (9 alerts) | 25% | 50.0% | 10.90 ms | 1269 MiB | 4.58 s |
| Jev | not measured | not measured | not measured | not measured | hosted | not run: no credentials |
Multilingual recall: EN 22.2% (4/18), PT 27.8% (5/18). Whole-family bootstrap 95% pooled recall interval is 7.1%–46.4%; language intervals are EN 0%–50%, PT 7.1%–50%. All emitted alerts in this small sample are positive, so the empirical bootstrap precision interval degenerates to 100%–100%; this is not a population guarantee. English training selected 1.01 (abstain-all), not a deployable threshold. The conservative baseline uses the same exact-text rule as production; its non-alert results principally come from explicit context filters.
Candidate recall was 100% among the explicitly supplied synthetic conflict pairs. This says nothing about finding unknown conflicts across a whole company. Review load for MiniLM was 65.6% EN / 62.5% PT. Raw uncalibrated reliability ECE was 0.161 for MiniLM and 0.210 for DeBERTa; no calibrated confidence is exported. Resource numbers include framework/model initialization and hash verification. The final reproducible run overlapped other local verification workloads; timings characterize this development host rather than an isolated performance lab. Warm pair latency excludes CLI startup, cold model loading and the second judgment direction. The native adapter cold-loads for each invocation and has a bounded total operation deadline.
Artifact and runtime checks
Pinned MiniLM revision 0a71e92a985b6e1ad1828cf67ce9c459639c1dca is MIT; pinned DeBERTa revision fa2804872c3b4bd748f38c0185cc85775361e735 is Apache-2.0. Public file hashes and sizes are retained in the fixture manifests. Safetensors and reviewed local-only code were used; no custom remote model code ran. The multilingual model card’s published XNLI table does not include PT, so Portuguese quality is measured here rather than inferred from its name.
Both pinned ONNX artifacts loaded in Python ONNX Runtime 1.23.2, arm64 CPU. MiniLM’s two bounded probes agreed with PyTorch and differed by at most 3.5e-6 logits. DeBERTa ARM64 int8 agreed on the English probe but changed the Portuguese top class, with a 4.25 logit difference. Successful loading does not establish behavioral compatibility. Native Rust ONNX execution was not validated and no native default is shipped.
The actual Synapse binary ran the local pinned MiniLM on an accepted source fixture through CLI and MCP; both returned the same revision-bound advisory result with null calibrated confidence. Source Markdown hashes remained unchanged, and unaccepted source drift failed closed. Repository CI runs the default baseline/process contract without downloading models; the opt-in experiment receipt is separate.
Reproduction and remaining research
Raw per-case measurements, grouped uncertainty and provenance are sanitized and contain no credentials or source secrets. Reproduction commands and host configuration describe fixed corpus/model hashes, runtime versions, source/egress boundaries and budgets. Jev has official-contract tests for model identity, closed option names, distribution/confidence validation and consent fallback; these are not live quality measurements.
Before recommending a default: obtain independent domain/human labels, expand families and realistic retrieval misses, evaluate calibrators without heldout leakage, measure full operation latency and test the actual native runtime artifact. The current evidence supports experimentation and explicit review, not a general contradiction verifier or measured field ROI.
Primary sources: MiniLM model, DeBERTa model, TypeSafe Choice API, confidence contract.