A landing page test has two costs: the traffic it takes, and the traffic it wastes. Classic A/B testing optimises the first and ignores the second.
The fixed-sample problem
A classic A/B test works like this:
- Decide the minimum effect you care about detecting.
- Compute the sample size needed to detect it with acceptable error rates.
- Split traffic 50/50 until you reach that sample size.
- Declare a winner, or declare no difference.
Step 3 is where the waste is. For the whole duration of the test, half your traffic goes to the variant that is (in most cases) losing. If the test needs 4,000 visitors per variant and one page converts at half the rate of the other, you have deliberately bought thousands of clicks on the worse page.
And you cannot stop early when the numbers look good: peeking and stopping inflates the false-positive rate. That rule exists for good reasons, and it makes the waste unavoidable within this method.
Bandit allocation
A bandit does not split traffic evenly. It allocates continuously, based on what it currently believes:
- Each variant keeps a distribution over its conversion rate.
- On each visitor, sample from each distribution and serve the variant with the highest sample.
- Update the served variant’s distribution with the outcome.
Early on, the distributions overlap and traffic is roughly even — exploration. As evidence accumulates, the better variant gets more traffic automatically. The test never “ends”; allocation just concentrates.
Thompson Sampling is the specific rule DropCPL uses. The important property for a marketer: the cost of testing falls as the gap between variants grows, which is exactly when a fixed-sample test would be burning the most money.
| Fixed-sample A/B | Bandit (Thompson Sampling) | |
|---|---|---|
| Traffic split | 50/50 until the end | continuous, adapts |
| Cost of a clear loser | full test duration | shrinks quickly |
| Stopping rule | pre-computed sample size | none needed |
| Reading mid-test | invalid (peeking) | meaningful (win probability) |
| Best for | a decision you must defend statistically | ongoing optimisation |
The mistakes that invalidate the result
Both methods share the same failure modes, and they are not statistical:
Changing more than one thing. A new headline and a shorter form, tested together, cannot tell you which mattered.
Comparing periods instead of variants. Traffic on Monday and traffic on Friday are different populations. Test variants simultaneously, not sequentially.
Judging on a tiny sample. With 12 leads, a “winner” is noise. Either the difference is large enough to see in your volume, or you cannot see it yet.
Testing something that cannot move the number. Button colour on a form nobody scrolls to is a test that will produce a confident, useless answer.
Forgetting what the page is for. A page that converts at 30% with unqualified leads is worse than a page that converts at 8% with qualified ones. Conversion rate alone is not the objective — see cost per qualified lead.
What is worth testing on a lead-gen page
In rough order of typical impact:
- The offer — what the visitor actually gets, and how it is framed.
- The form — number of fields, order, what is asked first.
- The first screen — does the visitor understand what this is in five seconds.
- Proof — numbers, logos, testimonials, with specificity.
- The CTA — verb, placement, number of competing actions.
Where to test in a paid social context
The page is not the only variable — the creative × page combination is. The same page can win for one creative and lose for another, which is what the attribution matrix exposes. Testing at the combination level, with bandit allocation, is what makes continuous testing affordable enough to actually keep doing it.
Related
- Thompson Sampling explained — the algorithm
- How to read the attribution matrix — where the results show up
- Automatic A/B testing — the product implementation