Free tool
A/B test significance calculator
This A/B test significance calculator tells you whether the gap in lead conversion rate between two landing page variants is real or noise. Enter visitors and leads for each: it returns the uplift, a 95% confidence interval, the p-value of a two-proportion z-test and the Bayesian probability that B beats A.
Variant A (original)
Variant B
| A | B | |
|---|---|---|
| Conversion rate | — | — |
Relative uplift
—
Absolute difference
—
p-value (two-sided)
—
Probability B beats A
—
Expected loss if you choose B
—
How to read the A/B test significance calculator
Each number answers a different question about your landing page conversion, and most bad calls come from reading one as if it answered another. Here's what each one tells you, and what it doesn't.
| Output | What it tells you | What it doesn't |
|---|---|---|
| Conversion rate | Leads ÷ visitors for each landing page. | Whether the gap would survive more traffic. |
| Relative uplift | B's lift as a share of A's rate. | How sure you can be of it. |
| Absolute difference and 95% CI | The gap in percentage points, and the range of gaps compatible with the data. | Much, when the range runs from negative to positive. |
| p-value | How surprising a gap this large would be if both landing pages converted at the same rate. | The probability that B is better. |
| Probability B beats A | How likely it is, given the data, that B's true rate is above A's. | By how much. |
| Expected loss of choosing B | The conversion rate you'd give up on average by switching to B, counting the cases where A is really better. | Whether each lead was attributed to the right variant. |
p-value vs probability to beat: which one answers your question?
The p-value comes from the frequentist test. It assumes the two landing pages convert at the same rate and asks how often chance alone would produce a gap at least this big. Below 0.05, the result is significant at 95%. It doesn't mean B has a 95% chance of being better, although that's how most dashboards get read.
The probability to beat is Bayesian. Each variant's conversion rate gets a Beta distribution, Beta(1 + leads, 1 + visitors − leads), which is a flat prior updated with your data. The calculator works out how likely it is that B's rate sits above A's: the question you actually ask before switching.
With a decent sample the two point the same way, and the probability to beat lands close to 1 − p/2. So significant at 95% (p below 0.05) corresponds to roughly a 97.5% probability to beat, and a 95% probability to beat to a p-value near 0.10. With only a few leads per variant they drift apart: the p-value leans on a normal approximation, the Bayesian number doesn't.
How the calculator computes significance
- z-test. z = (pB − pA) ÷ √[ppool(1 − ppool)(1/nA + 1/nB)], where ppool is the conversion rate of both variants combined; the two-sided p-value comes from the normal distribution. This is the standard two-proportion z-test, equivalent to a chi-square test without continuity correction.
- Confidence interval. (pB − pA) ± 1.96 × √[pA(1 − pA)/nA + pB(1 − pB)/nB], the Wald interval with unpooled variance. The test pools and the interval doesn't, so right at p = 0.05 they can disagree.
- Probability B beats A. Computed exactly with Evan Miller's closed-form formula for two Beta distributions, not by simulation, so it gives the same number on every reload. Above a million leads per variant it switches to a normal approximation, which agrees to six decimal places.
- Expected loss of choosing B. E[max(pA − pB, 0)] under the same posteriors, also in closed form. It's the quantity behind Chris Stucchio's Bayesian decision rule: stop when the expected loss of the leader falls below your threshold of caring, the smallest difference you'd bother acting on.
Worked example
Example: landing page A gets 1,200 visitors and 60 leads (5.00%); landing page B gets 1,180 visitors and 78 leads (6.61%).
- Relative uplift +32.2%; absolute difference +1.61 pp, with a 95% CI from −0.27 to +3.49 pp.
- Pooled rate 138 ÷ 2,380 = 5.80%, z = 1.68, two-sided p = 0.0929: not significant at 95%.
- Probability B beats A: 95.3%. Expected loss of choosing B: 0.019 pp, about 0.4% of A's conversion rate.
Same data, two readings. The z-test says the evidence can't rule out chance. The Bayesian view says B is probably better, and that if it isn't, switching costs very little. The interval is the sobering part: with 138 leads in total, B could be slightly worse or much better.
Why Thompson Sampling uses the Bayesian view
Thompson Sampling, the multi-armed bandit rule behind DropCPL's automatic A/B testing, keeps exactly these Beta posteriors. For every visitor it draws one random value from each variant's distribution and serves the landing page with the highest draw. With two variants, the chance that the next visitor goes to B is the probability to beat shown above: 95.3% in the example.
That's why a bandit has no significance threshold to cross. Allocation follows the evidence continuously: a variant at 50% gets about half the visitors, one at 99% gets almost all of them, and a losing landing page stops costing leads long before a fixed-sample test would end. The A/B testing glossary entry compares the two approaches side by side.
Reading these numbers by hand works for one test; DropCPL reads them on every visitor. Its automatic A/B testing allocates traffic between landing page variants with Thompson Sampling and shows each variant's win probability. Start free →
Sample size and the peeking problem
A p-value is only valid for the sample you planned. Decide the number of visitors before the test with the A/B test sample size calculator, then read this calculator once, at the end. Checking every day and stopping at the first p below 0.05 is peeking, and it lets noise through as a winner far more often than 5% of the time.
The Bayesian numbers don't carry the same fixed-sample assumption, but stopping the moment the probability first touches 95% still rewards lucky streaks, especially early, when a handful of leads can swing it. Pair it with a minimum sample and with the expected loss.
When should you stop an A/B test?
| Approach | Stop when |
|---|---|
| Fixed-sample test | Both variants have reached the planned sample and at least seven full days have passed. Read the verdict once. |
| Bayesian reading | At least a full week has passed and the expected loss of choosing the leader is below your threshold of caring, for example 0.1 pp. |
| Neither in sight | The sample size calculator asks for months of traffic. The difference you're chasing is too small for your volume: stop, and test a bigger change. |
FAQ
What does "statistically significant at 95%" mean in an A/B test?+
The p-value is below 0.05: if both landing pages really converted at the same rate, a gap at least as large as yours would show up less than 5% of the time. It doesn't mean there's a 95% chance B is better, and it says nothing about whether the lift is big enough to matter.
Is a p-value of 0.05 a 95% chance that B is better?+
No. The p-value measures how surprising the data would be if there were no difference. The probability that B is better is a Bayesian quantity; with a flat prior and a decent sample it lands close to 1 − p/2, so p = 0.05 corresponds to roughly 97.5%, not 95%.
How is the probability that B beats A calculated?+
Each variant's conversion rate is modelled as Beta(1 + leads, 1 + visitors − leads): a flat prior updated with your data. The calculator computes the probability that B's rate is higher with an exact closed-form sum rather than a simulation, so the result doesn't change between reloads.
How many leads do I need before the result means anything?+
As many as the sample size you planned before starting. Below that, intervals stay wide: in the worked example, 138 leads in total still leave the difference anywhere between −0.27 and +3.49 percentage points. A verdict on a few dozen leads is mostly a verdict on noise.
Can I use this calculator for more than two variants?+
Compare each variant with the original one pair at a time, and be stricter than 95%: with three variants there are two comparisons, so require p below 0.025 to keep the overall false-positive rate near 5%. That's the Bonferroni correction the sample size calculator applies when you plan the test.
Why can the p-value and the confidence interval disagree?+
The z-test uses the pooled conversion rate of both variants, the textbook test of no difference; the interval uses each variant's own rate. Right at the 0.05 boundary one can say significant and the other not. Either way it's a borderline result: treat it as not yet.