Customer retention · · 5 min read

Did your campaign convert, or were those customers coming back anyway?

A high conversion rate may be counting customers who would have come back on their own. Before you scale up your sends, measure how much the campaign actually caused.

A campaign comes back with a high conversion rate, and the conclusion seems obvious: if it worked with a few hundred customers, just send it to more people. Before multiplying your sends, ask a question that almost never shows up in the report: how many of these customers would have come back without the message?

The real effect of a campaign is the difference between the return rate of the people who received the message and that of a randomly drawn group that did not. In marketing, this is called incrementality.

In short
  • The conversion rate counts who bought after receiving the message. It does not separate out who bought because of it.
  • A large sample makes the number precise, but it does not prove the campaign caused the sale.
  • To find the real effect, randomly pick a group that does not get the message and compare the two.

Why campaign conversion is misleading

In a recent conversation with a client, a campaign came up with conversion far above what you would expect from the channel used. The audience had one detail: they were people who had bought just a few weeks earlier, right in the period when that store’s customers usually come back. When the message goes to people who were already on their way to the store, part of the return happens anyway, and the report counts that return as the campaign’s doing.

The tool measures what it was asked to measure: who received the message and bought again. The flaw is in the question. If the store sends a R$ 20 coupon and the customer comes back without using it, they still count toward conversion.

An example with numbers

Example Store had 6,000 customers who bought between two and three weeks ago. Before sending the campaign, it randomly picked half to receive the message and left the other half with nothing. After 15 days, it compared the two groups.

Out of every 100 customers who received the message
  1. 16bought again within 15 days. This is what the report shows.
  2. 14would have come back with no message at all, as the group left out showed.
  3. 2came back because of the message.

The full math, with the 3,000 customers in each group:

Fictional example: 15-day return of the group that received the message and the group that did not, at Example Store
With messageWithout message
Customers in the group3,0003,000
Bought again within 15 days480420
Return rate16%14%
Campaign effect (16% − 14%)2 points
The campaign report would show only the first column: 16%. Fictional example, just to show the math.

Of the 480 customers who came back after the message, about 420 would have come back on their own. The campaign caused 60 extra sales out of 3,000 sends. At the same sending cost, each sale it caused came out eight times more expensive than the report suggested (480 ÷ 60). And if the store triples its sends to other audiences, sales will not triple: the 16% belonged to the chosen audience, not to the channel.

Does a large sample prove the campaign worked?

It is common to think that once you pass a few hundred sends, the number is reliable. It becomes precise: with 3,000 sends, the true rate falls between 15% and 17%. But precision is one thing and cause is another. If the audience was going to come back anyway, sending to 30,000 people with the same profile will show the same 16%, with even more certainty. Without the group that was left out, the question about the message’s effect remains unanswered.

The hidden cost of discounts

When the campaign includes a discount, the problem gets bigger. The customer who would have come back anyway uses the coupon because it is there. In the example, if half of the roughly 420 customers who would have come back on their own used a R$ 20 coupon, that would be R$ 4,200 in margin handed to people who needed no incentive at all.

How to measure the real effect with a control group

It is the same principle as the incrementality tests run by ad platforms. Google, for example, describes Conversion Lift (opens in a new tab) as a controlled experiment: the audience is split between people who see the ad and a control group that does not, and the difference in conversions between the two is the lift caused by the ad. In a WhatsApp campaign, the logic is the same.

  1. Define the metric before you sendChoose what the campaign should move and over what period, for example the rate of customers who come back within 15 days. Choosing the number afterwards, based on what looks good, leaves room for coincidence.
  2. Set aside a control groupFrom the chosen audience, randomly pick who goes without the message: that is the control group. With a large audience, 10% to 20% is enough; with a small one, set aside up to half. A simple way to draw them is by the last digit of the phone number: numbers ending in 0 to 4 receive it, 5 to 9 are left out.
  3. Compare over the same windowThe campaign’s effect is the difference between the return rates of the two groups. At Example Store, 16% versus 14%: 2 points.
  4. Only then decide whether to scaleIf the difference is clear, it is worth sending to more people with the same profile. If it is small, sending more raises the cost, and sales barely change.

Large differences, of 8 or 10 points, show up with a few hundred customers in each group. Small differences, like Example Store’s 2 points, need thousands, which is why the example uses 3,000 on each side.

Without a control group, the bare minimum is to compare with the historical repeat purchase rate for the same window: how many customers usually come back, with no campaign, in the same period after a purchase. It is a weaker comparison, because the month, the season and the customer profile change, but it already keeps you from mistaking natural returns for results.

Not every action needs to convert

A billboard may not change the month’s sales and still strengthen the brand, and that has value. But if an action is presented as a conversion tool, it needs to move a metric, whether sales or repeat purchases, on any channel: SMS, email or WhatsApp. To understand who tends to come back before planning a campaign, read “Quem já comprou, compra de novo”. MeContaí has worked with post-sale follow-up and customer reactivation over WhatsApp since 2019; learn more about who is behind it on the about us page.

Frequently asked questions

Does a high conversion rate mean the campaign worked?

Not necessarily. The conversion rate counts every customer who bought after receiving the message, including those who would have come back anyway. If the chosen audience was already close to its next purchase, conversion comes out high even if the message changed very little. Only a comparison with a group that did not receive it shows the effect.

What is a control group in a campaign?

It is a part of the audience, picked at random, that does not receive the message. Since both groups have the same profile, the return rate of the people left out shows how many customers would come back on their own. The difference between the two groups is what the campaign actually caused.

How big should the control group be?

It depends on the size of the audience and on the difference you want to detect. With large audiences, setting aside 10% to 20% is usually enough. With small audiences, it is worth setting aside up to half, or repeating the test over several sends. Small differences, of one or two points, need thousands of customers in each group.

Can you measure it without a control group?

You can estimate it, with less certainty. The way to do it is to compare the campaign's return with the historical repeat purchase rate for the same window: how many customers usually come back, with no campaign, in the same period after a purchase. Since the month, the season and the customer base change, the number works as a reference and needs to be read with care.

How much of your repeat purchase happens without a campaign?

The Recoverable Revenue Diagnosis is a 30-minute conversation with André, based on your store’s data. You come away knowing how many customers bought only once, when to contact each one and where to start. Bring a list of sales from the last few months, in a spreadsheet or exported from your system.

The numbers in this article are fictional and are only meant to show the math. They are not a promise of results.

Newsletter · every Tuesday

Get articles like this one in your inbox

One issue a week on everyday retail topics. Each topic comes with a summary and a link to the full article.