A campaign comes back with a high conversion rate, and the conclusion seems obvious: if it worked with a few hundred customers, just send it to more people. Before multiplying your sends, ask a question that almost never shows up in the report: how many of these customers would have come back without the message?
The real effect of a campaign is the difference between the return rate of the people who received the message and that of a randomly drawn group that did not. In marketing, this is called incrementality.
- The conversion rate counts who bought after receiving the message. It does not separate out who bought because of it.
- A large sample makes the number precise, but it does not prove the campaign caused the sale.
- To find the real effect, randomly pick a group that does not get the message and compare the two.
Why campaign conversion is misleading
In a recent conversation with a client, a campaign came up with conversion far above what you would expect from the channel used. The audience had one detail: they were people who had bought just a few weeks earlier, right in the period when that store’s customers usually come back. When the message goes to people who were already on their way to the store, part of the return happens anyway, and the report counts that return as the campaign’s doing.
The tool measures what it was asked to measure: who received the message and bought again. The flaw is in the question. If the store sends a R$ 20 coupon and the customer comes back without using it, they still count toward conversion.
An example with numbers
Example Store had 6,000 customers who bought between two and three weeks ago. Before sending the campaign, it randomly picked half to receive the message and left the other half with nothing. After 15 days, it compared the two groups.
- 16bought again within 15 days. This is what the report shows.
- 14would have come back with no message at all, as the group left out showed.
- 2came back because of the message.
The full math, with the 3,000 customers in each group:
| With message | Without message | |
|---|---|---|
| Customers in the group | 3,000 | 3,000 |
| Bought again within 15 days | 480 | 420 |
| Return rate | 16% | 14% |
| Campaign effect (16% − 14%) | 2 points | |
Of the 480 customers who came back after the message, about 420 would have come back on their own. The campaign caused 60 extra sales out of 3,000 sends. At the same sending cost, each sale it caused came out eight times more expensive than the report suggested (480 ÷ 60). And if the store triples its sends to other audiences, sales will not triple: the 16% belonged to the chosen audience, not to the channel.
Does a large sample prove the campaign worked?
It is common to think that once you pass a few hundred sends, the number is reliable. It becomes precise: with 3,000 sends, the true rate falls between 15% and 17%. But precision is one thing and cause is another. If the audience was going to come back anyway, sending to 30,000 people with the same profile will show the same 16%, with even more certainty. Without the group that was left out, the question about the message’s effect remains unanswered.
The hidden cost of discounts
When the campaign includes a discount, the problem gets bigger. The customer who would have come back anyway uses the coupon because it is there. In the example, if half of the roughly 420 customers who would have come back on their own used a R$ 20 coupon, that would be R$ 4,200 in margin handed to people who needed no incentive at all.
How to measure the real effect with a control group
It is the same principle as the incrementality tests run by ad platforms. Google, for example, describes Conversion Lift (opens in a new tab) as a controlled experiment: the audience is split between people who see the ad and a control group that does not, and the difference in conversions between the two is the lift caused by the ad. In a WhatsApp campaign, the logic is the same.
- Define the metric before you sendChoose what the campaign should move and over what period, for example the rate of customers who come back within 15 days. Choosing the number afterwards, based on what looks good, leaves room for coincidence.
- Set aside a control groupFrom the chosen audience, randomly pick who goes without the message: that is the control group. With a large audience, 10% to 20% is enough; with a small one, set aside up to half. A simple way to draw them is by the last digit of the phone number: numbers ending in 0 to 4 receive it, 5 to 9 are left out.
- Compare over the same windowThe campaign’s effect is the difference between the return rates of the two groups. At Example Store, 16% versus 14%: 2 points.
- Only then decide whether to scaleIf the difference is clear, it is worth sending to more people with the same profile. If it is small, sending more raises the cost, and sales barely change.
Large differences, of 8 or 10 points, show up with a few hundred customers in each group. Small differences, like Example Store’s 2 points, need thousands, which is why the example uses 3,000 on each side.
Without a control group, the bare minimum is to compare with the historical repeat purchase rate for the same window: how many customers usually come back, with no campaign, in the same period after a purchase. It is a weaker comparison, because the month, the season and the customer profile change, but it already keeps you from mistaking natural returns for results.
Not every action needs to convert
A billboard may not change the month’s sales and still strengthen the brand, and that has value. But if an action is presented as a conversion tool, it needs to move a metric, whether sales or repeat purchases, on any channel: SMS, email or WhatsApp. To understand who tends to come back before planning a campaign, read “Quem já comprou, compra de novo”. MeContaí has worked with post-sale follow-up and customer reactivation over WhatsApp since 2019; learn more about who is behind it on the about us page.
Frequently asked questions
Does a high conversion rate mean the campaign worked?
Not necessarily. The conversion rate counts every customer who bought after receiving the message, including those who would have come back anyway. If the chosen audience was already close to its next purchase, conversion comes out high even if the message changed very little. Only a comparison with a group that did not receive it shows the effect.
What is a control group in a campaign?
It is a part of the audience, picked at random, that does not receive the message. Since both groups have the same profile, the return rate of the people left out shows how many customers would come back on their own. The difference between the two groups is what the campaign actually caused.
How big should the control group be?
It depends on the size of the audience and on the difference you want to detect. With large audiences, setting aside 10% to 20% is usually enough. With small audiences, it is worth setting aside up to half, or repeating the test over several sends. Small differences, of one or two points, need thousands of customers in each group.
Can you measure it without a control group?
You can estimate it, with less certainty. The way to do it is to compare the campaign's return with the historical repeat purchase rate for the same window: how many customers usually come back, with no campaign, in the same period after a purchase. Since the month, the season and the customer base change, the number works as a reference and needs to be read with care.