Your app swapped AI models. Nobody checked whether the behavior survived.

unswayed is a statistical parity gate: freeze a suite, sample every provider, and fail CI only on proof with an honest "undecided" when your sample count can't certify anything.

Swapping models became a one-line change. Verifying the swap didn't.

2026 made model-swapping trivial: Apple's Foundation Models ships provider-swappable sessions (Apple on-device, Private Cloud Compute, Claude, Gemini one line), every serious app keeps a cloud fallback,and "the new model is cheaper, let's switch" happens weekly. Here's what nobody's toolchain answers: did your app's behavior survive? The JSON your parser needs,the one-word reply your UI slots in, the benign question that must not get refused did all of that hold? You can't answer it by running your suite once and eyeballing the output, because models arenondeterministic. One good run proves nothing. Five good runs prove less than you think and that intuition gap is exactly where silent regressions ship. The demo that convinced me: a model failing parity with itself unswayed's three moves: 1. Freeze a suite prompts plus deterministic checks (valid JSON, required keys, one word, regex,length), versioned and SHA-256-hashed. 2. Capture each provider N samples per prompt, refusals recorded as first-class outcomes. 3. Compare with real statistics and fail CI only on certified regressions.

For the first real run I didn't even need two models. Same Mac, same Apple on-device model greedy decoding as baseline, temperature 1.0 as candidate:

Sampling alone broke format compliance "reply with exactly one word" drops from 100% to 40%, and the gate certifies it: the entire 95% confidence interval sits below zero. Not a vibe. A verdict. The honest part: 19 cells came back "undecided" Five samples at 100% vs five samples at 100% is not evidence of equivalence the difference interval spans ±43 points. Most tools would stamp that "pass." unswayed refuses to: it reports undecided and tells you the fix (more samples). How many samples does a verdict actually take at 95% confidence?

1. Catch a total break (100% → 0%): 5 per side
2. Catch a big regression (100% → 25%): ~20 per side
3. Certify equivalence within ±10 points: ~60 per side
4. Deterministic providers (greedy on-device): N identical runs prove determinism, then comparisons are exact

That last line is a real asymmetry nobody talks about: on-device greedy baselines are cheap to gate(determinism is measured, then everything is exact); cloud providers are expensive to certify. unswayed is the tool that makes that visible it uses Wilson score intervals for pass rates and Newcombe difference intervals for the comparison, closed-form and deterministic, so the same captures always produce the same verdicts.

**Built paranoid, like its siblings

** A baseline capture that is 100% provider errors (stale cassette, dead API key) is refused a healthy candidate would "improve" on garbage and turn the gate green. Found by an adversarial audit probe;fixed and locked as a regression test. Malformed suites, captures, and cassettes throw instead of shrinking coverage. Cross-suite comparisons are refused (name + version + SHA-256 must match). Cassette record/replay makes CI fully offline no API keys, no flakes, and the replay throws rather than inventing data. Refusals are first-class: a provider swap that turns answers into refusals is its own finding class, ranked with the regressions.

What it is not Checks are deterministic assertions, not quality judgments pair it with a cassetted LLM-judge for those.And Apple's new Evaluations framework (WWDC26) judges quality within Apple platforms; unswayed certifies parity across providers and runs on today's stable OS. Complementary, not competing. One capture format, two ecosystems unswayed ships as a Swift package and an npm package same name, same versioned capture JSON. Capture on a Mac or iPhone in Swift, judge in TypeScript in CI (or the reverse); a cross-ecosystem test locks both implementations to the identical verdict, down to the confidence-interval values. The npm build is 16 kB with zero runtime dependencies, and its Vercel AI SDK adapter means the same gate reaches React Native's on-device models (react-native-ai, llama.rn, MLC) with no extra glue.

The family unswayed is the sixth piece of the testing & safety layer I've been building for on-device AI: golden_lens — visual regressions an AI agent can act on llm_replay_eval — deterministic record/replay + evals for on-device LLMs redact — on-device PII redaction around every LLM call vouch — freeze an eval baseline, fail CI on silent model-swap regressions underfoot — the public record of what OS-bundled models silently do unswayed — the statistical parity gate for swapping providers GitHub (MIT, Swift): https://github.com/Umer9538/unswayed npm (MIT, TypeScript):https://www.npmjs.com/package/unswayed