Meta Pixel, Conversions API and attribution

How is incrementality testing different from an A/B test?

An A/B test says which version converts better. An incrementality test says whether the ads caused the conversions at all. Two questions, two designs.

Published

An A/B test compares two versions of an ad or a landing page and says which one performs better. An incrementality test compares advertising with no advertising and says how many conversions the ads actually caused. The first decides which variant to keep; the second decides whether the budget should exist. They answer different questions, and one cannot substitute for the other however good the tracking is.

Two questions that sound similar

Attribution assigns credit: this lead came from that ad, so that ad owns the cost. Incrementality asks a harder question: would the lead have arrived anyway, with no ad running at all? Everything else follows from that split.

Attribution and A/B testsIncrementality
What it answerswhich ad should get the credit, or which version winsdid the conversion need an ad at all
How the answer is produceda rule or a model applied to the clicks and views you recordeda controlled comparison with a group that sees no ads
People who never saw an adinvisible to itthey are the group that makes the measurement possible
What comes outconversions per ad, cost per resultextra conversions, lift, cost per extra conversion
What it costs youtracking worktraffic or spend held back while the test runs

Attribution cannot answer the incrementality question, however good the tracking is, because it only ever looks at people who saw or clicked an ad. Someone who was going to fill in the form anyway, and happened to see a view-through impression, lands in the same column as someone the ad persuaded.

Incrementality has the short definition, and A/B testing covers what makes the other kind of test valid.

How a lift test works

Meta describes its Conversion Lift studies as a randomised test group of accounts that can see the ads and a control group that cannot, with conversions compared between the two to determine the increase the campaign generated. The arithmetic is simple:

  1. Divide the audience at random, nine parts test and one part control.
  2. Run the campaign so that only the test group can be served.
  3. Multiply the control group’s leads by nine, to put them on the same footing as the test group.
  4. Subtract that figure from the test group’s leads.

An example makes the scale visible. The test group produces 270 leads and the control group 20. Scaled up nine times, the control stands for 180 leads that would have happened anyway, so the ads caused 90 and the lift is 50%. On €3,000 of spend, that is €11.11 per lead and €33.33 per incremental lead. Same campaign, a number three times higher.

That gap is the whole point of the exercise. A campaign that looks profitable on cost per lead can be paying for leads that were already coming.

How a landing page A/B test works

A fixed sample test does something different, and it does it on purpose:

  1. Decide the minimum effect worth detecting.
  2. Compute the sample size that detects it with acceptable error rates.
  3. Split traffic evenly until that sample size is reached.
  4. Declare a winner, or declare no difference.

Both variants receive advertising, so nothing in the result says advertising was necessary. What it says is that version B converts better than version A, which is a decision about the page. The cost of the design is the traffic sent to the loser for the whole duration, and the rule against stopping early exists because peeking inflates the false positive rate.

A bandit allocation, such as Thompson Sampling explained, changes the cost rather than the question. Traffic concentrates on the leader as evidence arrives, so a clear loser stops being expensive after a few days. It still compares two versions of the same thing.

Why a small account usually cannot run a lift test

The limit is the number of conversions in the control group. With a 10% holdout, the control sees a tenth of the audience, and a tenth of a small number is a very small number.

Take the earlier campaign with 270 leads in the test group. If the ads did nothing, that would come with about 30 leads in the control group, because the control is scaled up tenfold. Here is what two other outcomes look like against that baseline:

Example, 270 leads in the test groupLeads expected in the controlGap from “ads did nothing”Random noise, 95% rangeReadable?
Half the leads caused by the ads1515plus or minus 8yes
One lead in ten caused by the ads273plus or minus 10no

An account of this size can see that its ads double its leads. It cannot tell a lift of around 10% from zero, and a 10% lift is usually the answer people run the test for. Volume, more than tooling, decides whether a lift test is worth paying for.

Access is the second constraint. Meta’s developer documentation opens with a warning that Conversion Lift Measurement is limited and sends you to a Meta representative, and the Ads Manager version of the test shows spend minimums or recommended spend amounts for some tests. Check what your own account is offered before planning a quarter around one.

What to do when the volume is not there

Three options, in order of usefulness for a lead generation account.

First, decide on click-based numbers you own. Count the leads your server received, split by ad and by landing page, and treat view-through and modelled credit as claims you have not tested. That is a smaller claim than incrementality, and it is one your own data supports.

Second, if you test anyway, test at the level that has volume. Pause an entire campaign in one region for a few weeks and compare it with similar regions. It is cruder and it is exposed to seasonality, but it asks the right question.

Third, keep the A/B tests running while you wait. They are cheap at this volume, they compound, and they are how a small account improves before it is big enough for a lift test. A/B testing landing pages covers the practical side, and Meta Ads creative testing framework puts the creative side in order.

The mistakes that ruin both

Both methods fail in the same non-statistical ways. Changing more than one thing at a time, so nothing can be attributed. Comparing this week with last week instead of variants running at the same time. Judging on twelve leads, where a winner is noise. Testing something that cannot move the number. And optimising the conversion rate while forgetting what the page is for: a landing page that converts at 30% with unqualified leads is worse than one that converts at 8% with leads that answer the phone.

The last one is why the metric for both designs should be a qualified lead, not a submission. Facebook lead quality explains how to measure it, and the counting side, from the Pixel to the Conversions API, is in Meta Pixel, Conversions API and attribution.

Questions on this topic

All questions on this topic →

FAQ

Can I measure incrementality without a holdout?+

Not really. The defining feature of an incrementality test is a group that could not see the ads, and without it you are reading attribution with extra steps. A regional pause is a crude holdout, not a substitute for a randomised one, but it asks the right question.

Is the incremental attribution setting in Ads Manager an incrementality test?+

No. Meta measures it against its standard attribution model, so it is a model Meta runs over your campaign data. It does not create a control group, and it does not tell you how many conversions would have happened anyway.

Which one should a lead generation account run first?+

The A/B test, almost always. It needs less volume, it improves the funnel while it runs, and it does not require holdout traffic. Run the lift test when the account is big enough that the control group produces a readable number of leads.

Does an A/B test ever show incrementality?+

No. It compares two versions served to the same audience, so both sides received advertising. It can tell you that version B is better than A, never that either of them was needed.

Ready to find out what
converts, for real?

Free to start.

Start free →