A/B testing compares two or more variants — of a landing page, an ad, an offer — by sending traffic to each and comparing outcomes.
The requirements for a valid test
- Simultaneity. Variants must run at the same time. Comparing this week against last week compares populations, not variants.
- One change. Test one difference at a time, or you cannot attribute the effect.
- A decision rule fixed in advance. Either a sample size computed before the test (fixed-sample), or an allocation rule that does not depend on peeking (bandit).
- A metric that matters. Conversion rate is a proxy; the outcome is CPQL or revenue.
A fifth requirement is usually missed: the test needs an identifier that ties a lead back to the variant that produced it. Without a click ID written server-side, a lead arrives with no variant attached and the test measures noise.
Two families
| Fixed-sample | Bandit (Thompson Sampling) | |
|---|---|---|
| Traffic split | equal, until the end | adapts continuously |
| Stopping rule | computed in advance | none needed |
| Cost of a loser | paid for the whole test | shrinks as evidence arrives |
| Read mid-test | invalid | meaningful |
The mistake that ruins most tests
Stopping when the numbers look convincing. With a fixed-sample test, checking repeatedly and stopping at the first significant result inflates the false positive rate — you will “find” winners that are noise. A bandit does not have this problem because it is not testing a hypothesis; it is allocating traffic.
Statistical significance
Statistical significance measures how unlikely an observed result would be if there were no real difference between the variants.
In a fixed-sample test the usual threshold is a p-value below 0.05: if the two variants were truly identical, there would be less than a 5% chance of seeing a difference this large.
It answers one question: “how surprising is this data, assuming there is no difference?” It does not answer three others:
- “How likely is B better than A?” That is a different, Bayesian question, and the one a bandit answers directly with a win probability.
- “Is the difference big enough to matter?” A significant 0.1% lift on 40,000 visitors is real and irrelevant.
- “Should I stop the test now?” Significance is computed for one look at the data. Looking repeatedly and stopping at the first significant result inflates false positives.
The practical consequence: significance is the right tool for a decision that must be defended — a change with a large blast radius. For continuous traffic optimisation, where being wrong for a few days costs little, an adaptive allocation rule fits better than a threshold.
Thompson Sampling
Thompson Sampling is a Bayesian multi-armed bandit algorithm that allocates traffic between variants while learning which is best.
Each variant holds a Beta distribution over its conversion rate, starting from Beta(1, 1). For every visitor:
- Draw one sample from each variant’s distribution.
- Serve the variant with the highest sample.
- On a lead, increment the variant’s first parameter; on no lead, the second.
After s leads and f non-leads, the variant holds Beta(s + 1, f + 1), whose mean is about s / (s + f) and whose spread narrows with evidence.
Why it is used:
- No stopping rule. You read a win probability at any time, instead of waiting for a sample size.
- Cheap exploration. Traffic concentrates on the leader automatically, so the cost of a losing variant falls as the gap grows.
- It never fully abandons a variant, which lets it recover if the audience changes.
What it is not:
- Not a significance test. There is no p-value; the output is a probability that a variant is the best of the set, not a proof.
- Not immune to bad data. If leads are attributed wrongly, the allocation optimises the wrong thing, faster.
Which one to use
| The decision | Use | Why |
|---|---|---|
| A page change with a large blast radius, one look at the data | fixed-sample + significance | the result must be defensible in a meeting |
| Continuous optimisation of creatives and landing pages | bandit | the cost of being wrong for three days is small, and traffic is finite |
| Traffic too small to reach a sample size | bandit | it produces a useful allocation without ever “concluding” |
| Leads attributed by hand or by UTM only | neither, fix attribution first | both methods optimise whatever the data says, including a wrong join |
Related terms
- Multi-armed bandit — the problem class behind Thompson Sampling
- Tracking identifiers — how a lead is joined to a variant
- Attribution window — how long a click can claim a lead
- A/B testing landing pages — the practical guide