Scoped decision providers and evaluation

SYN-323 adds a read-only port, not a replacement for INV-13, source authority or governed review. Source evidence is untrusted input. Typed classifications and signatures on their provenance cannot prove a classification correct.

Optional host configuration

Set SYNAPSE_DECISION_CONFIG to a bounded, owner-private JSON file outside the knowledge workspace. CLI/MCP requests select only an alias. No executable, endpoint, secret or configuration path is accepted from Markdown or tool arguments.

{
  "schema": "synapse.decision-providers/1",
  "providers": {
    "experimental-local": {
      "kind": "local_nli",
      "python": "/absolute/reviewed/venv/bin/python",
      "python_sha256": "SHA256_OF_INTERPRETER_BYTES",
      "model_dir": "/absolute/reviewed/model",
      "manifest_sha256": "SHA256_OF_MANIFEST_BYTES",
      "threshold": 0.9,
      "timeout_ms": 15000,
      "max_calls": 8
    },
    "experimental-jev": {
      "kind": "jev",
      "model": "jev-1.13.0",
      "key_env": "SYNAPSE_JEV_KEY",
      "egress_scopes": ["EXACT_GOVERNED_WORKSPACE_GUID"],
      "acknowledge_source_egress": true,
      "timeout_ms": 5000,
      "max_calls": 2
    }
  }
}

The Python adapter runs the repository’s fixed worker with isolated interpreter mode, local-only safetensors, trust_remote_code=false, artifact hashes and no inherited key environment. This is a trusted host runtime, not an OS sandbox: review the interpreter, installed packages and manifests. Optional setup uses a separate virtual environment; pinned experiment dependencies are torch 2.9.1, transformers 4.57.3, onnxruntime 1.23.2 and sentencepiece 0.2.1. manifest.json has schema synapse.local-decision-model/1, repository, exact revision, license and files mapping relative names to {sha256,bytes}. Model artifacts remain outside this repository.

Each process cold-loads its model; benchmark warm pair latency is not end-to-end CLI latency. Input is bounded to 8 KiB per source and 384 tokenizer tokens per pair; overflow abstains rather than hiding truncation. Up to eight calls share a 25-second operation budget, with a 15-second maximum per call. Errors and timeouts abstain. Hash mismatch or a different observed model/rubric rejects the response. Interpreter and model configuration are reviewed host capabilities, not remote permissions.

Hosted Jev requires the network_backends feature, a pinned model version, explicit consent for both source scopes and a separately configured environment key. It uses the official Choice HTTP contract, disables redirects and limits request/response bytes and calls. No credentials are persisted in source. Provider confidence is explicitly provider_claimed; local softmax exports confidence:null. Jev was not run in this experiment because credentials were unavailable; API contract tests are not measured quality, latency or calibration.

Reproduction

The corpus is original sanitized synthetic evidence, reviewed by the coding agent for the narrow policy contract. It is not human field labeling. Its immutable hash and whole-family split were fixed before threshold selection. EN/PT and adversarial variants from a family never cross train/heldout. No external model or prompt instruction owns approval authority.

python3 tools/decision-bench/run.py --provider rules --output /tmp/rules.json
HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 /reviewed/venv/bin/python tools/decision-bench/run.py --provider multilingual --model-dir /reviewed/minilm --output /tmp/minilm.json
HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 /reviewed/venv/bin/python tools/decision-bench/run.py --provider english --model-dir /reviewed/deberta --output /tmp/deberta.json
/reviewed/venv/bin/python tools/decision-bench/onnx_check.py --model-dir /reviewed/minilm --onnx onnx/model.onnx --output /tmp/onnx.json
SYNAPSE_DECISION_CONFIG=/reviewed/config.json SYNAPSE_TEST_LOCAL_NLI=1 python3 tests/decisions_smoke.py target/debug/syn

Thresholds use training cases only. Threshold 1.01 in a report means disable all model predictions; it is not a deployable production threshold. Candidate recall concerns the explicitly scoped synthetic subset, not corpus-wide search completeness. Bootstrap intervals resample whole heldout families (2,000 samples, seed 731); sparse synthetic families limit external validity. Raw reliability bins/ECE characterize uncalibrated scores; they are not fitted or field-calibrated confidence.

See measured artifacts and limitations. Before any local default, require heldout gates, domain/human-reviewed validation, permissive artifacts and the native ONNX/runtime and repository gates. The optional Python ONNX compatibility probe is recorded separately and does not claim native Rust ONNX validation.

Primary contracts: TypeSafe API, TypeSafe confidence, MiniLM card, DeBERTa card.