A/B Test Significance Calculator

What this calculator tells you

Run an A/B test, plug in how many visitors saw each variant and how many converted, and this calculator does the statistics for you — no spreadsheet formulas, no stats background required. Instead of just handing back a p-value and leaving you to interpret it, it gives you a direct, plain-English verdict: whether one variant beat the other by a margin unlikely to be random chance, or whether the difference you're seeing still falls within normal statistical noise and the test needs more data before you can trust it.

It uses a standard two-proportion z-test — the same underlying method used by most professional A/B testing and CRO (conversion rate optimization) tools — comparing conversion rates rather than raw counts, since a variant with more visitors will naturally have more conversions even with no real difference in performance.

Reading the result

The headline result tells you plainly whether your test reached significance at the confidence level you chose, and which variant came out ahead. Underneath, four supporting numbers back that verdict up: each variant's conversion rate, the relative uplift between them, the p-value (the probability the observed difference could have happened by chance alone if there were actually no real difference), and statistical power (roughly, how capable your current sample size is of reliably detecting a real difference of this size, if one exists). A significant result with low power is worth treating cautiously — it means your sample size is thin enough that the result, while technically significant, could still be less stable than a test run with more data.

One-sided vs. two-sided, and why it matters

A two-sided test checks whether the variants differ in either direction — B could beat A, or A could beat B — and is the safer, more conservative default for most real-world tests, since you usually can't rule out a new design performing worse ahead of time. A one-sided test only checks one specific direction (typically, whether B beats A) and requires a somewhat smaller effect to reach the same confidence level, but it comes with a real trade-off: if the actual result runs the opposite way, a one-sided test simply reports "not significant" rather than telling you A won by a wide margin. Use one-sided only when you have a strong, pre-existing reason to believe the change can only help, never hurt — otherwise two-sided is the more honest choice.

Confidence level and what it actually promises

A 95% confidence level — the most common default in A/B testing — means that if there were truly no difference between your variants, you'd only expect to see a difference this large purely by chance about 5% of the time. It's a statement about how trustworthy the result is, not a promise that the winning variant will out-convert the loser by that exact margin forever. 90% confidence is more lenient and reaches a verdict faster but accepts a higher chance of a false positive; 99% is stricter and more conservative, appropriate when a wrong call is expensive to walk back — a pricing page change, for instance, rather than a button color.

Common mistakes that produce misleading results

Peeking at results early and stopping the test the moment it first crosses significance is the single most common way real A/B tests go wrong — significance can flicker in and out of "reached" as data accumulates, especially early on with a small sample, and stopping at the first favorable-looking peek meaningfully inflates the odds of a false positive. Deciding on a sample size or a minimum test duration in advance, and sticking to it, is the standard fix. It's also worth remembering that statistical significance answers a narrower question than it might seem to — it tells you the difference probably isn't random noise, not that the difference is large enough to matter for the business, or that it will hold up identically once the change ships to 100% of traffic.