Lesson 5 of 5 · WhatsApp Offers
What the message actually added
"Ten percent came back" sounds like a good result. It only is if those customers wouldn't have come back anyway, and if their return put money in the till. In this lesson you learn to measure both.
To measure what a message actually added, the MeContaí course uses two calculations, in this order. Extra returns (in marketing, the incremental effect): the return rate of those who got the message minus the rate of a control group drawn at random before sending. Extra revenue: the difference in revenue per customer between the two groups, times the number of customers who got the message.
Updated on .
Before you start: remember the traps?
Without looking at lesson 4, on the offer ladder, name the three traps in reading the ladder. Then open the box and check. One of them is today's topic.
Check the three traps
- The group gets harder: the customers who were easy to bring back already came back in week 1. Compare the same week across two groups.
- Time brings people back too: some of the people who ordered would have ordered anyway. Randomly pick about 10% of the group to get nothing.
- The customer learns to wait: if the ladder always goes up, regular customers notice that waiting pays more.
The SMS case
An SMS campaign had a conversion rate of more than 10%. But the customers were in the period when many of them start buying again on their own. So: did the SMS bring these people back, or would they have come anyway? (case reported to the MeContaí team)
Without a group that got nothing, there's no way to answer. That group, drawn at random beforehand and sent no message at all, is the control group (lesson 4, the ladder traps). The 10% mixes two things: people who came back because of the message and people who were going to come back anyway. In the MeContaí team's experience, few people in Brazil read SMS: on many phones it sits out of sight. If nobody read it, the 10% is almost entirely people who would have come on their own.
The two calculations for measuring a campaign
| Question | Calculation |
|---|---|
| 1 · Extra returns How many came back beyond those who would have come back on their own? | return rate of those who got the message − return rate of the control group |
| 2 · Extra revenue Did it put money in the till? | (revenue per customer of those who got the message − revenue per customer of the control group) × customers who got the message |
| Estimated profit Was there money left after costs? | extra revenue × margin − cost of the messages |
The difference between two rates is counted in percentage points: going from 10% to 11% is 1 point more.
Why the second question matters so much: the offer also gives a discount to people who would have come anyway. That customer pays less for an order they would have placed regardless. The return rate goes up, and revenue may even go down.
And why measure by the revenue of the whole group, not just the orders that used the coupon: in 70 tests of email offers, about 90% of the extra spending came from purchases made without the coupon. (Sahni, Zou & Chintagunta, 2017) Deep discounts for long-standing customers also reduced their purchases in the following months. (Anderson & Simester, 2004)
How to set up the control group
- Draw it before sending. From the week's group, set aside about 10% at random, with at least 30 to 50 customers (a rule of thumb used in this course). Don't hand-pick: a random draw is what makes the two groups alike. (Kohavi, Tang & Xu, 2020)
- Same window for both. Count 15 days from the send date, for those who got the message and for the control group (a rule of thumb used in this course).
- Count every Goomer order from these customers within the window, with or without a coupon, and add up the amount paid.
Try it: an example similar to the SMS case
The numbers below are just an example, similar to the case: 11% of those who got the message came back, and so did 10% of the control group. Change the values and watch what happens to the profit and to the chance warning; "Back to the example" undoes your changes. The margin is the part of the amount paid that the business keeps, after subtracting costs and the discount.
Notice: the 110 who came back amount to only about 10 extra customers. And with 200 in the control group, a 1-point difference could still be chance. For 1 point to be solid, you would need tens of thousands of customers. A smaller business doesn't need to get there: with the same groups, a larger difference is already solid, and you can repeat the send and add the results together. Try entering 150 who came back among those who got the message (15%): with a 5-point difference, the warning changes. The profit doesn't change, because the revenue of those who got the message is still R$ 9,900: as far as the meter knows, the extra customers paid nothing. Enter a higher revenue as well and watch the profit change.
What if there was no control group?
That was the case with the SMS. You can make a weaker comparison (MeContaí guidance): take similar customers (same number of orders, same time without ordering) in a period with no send, and see how many came back within 15 days. Use that rate in place of the control group. It's an estimate: the week, the weather and holidays may be different. For the next campaign, draw the control group beforehand.
Quiz
1. 12% of those who got the message came back. The control group had 11%. How much did the message bring in?
Extra returns are the difference: 12% − 11% = 1 point. The rest would have come anyway.
2. Extra returns came to 3 points, but revenue per customer was the same as in the control group. What happened?
More people came back, but paying less, including those who would have come on their own. Returns without extra revenue don't pay for the campaign.
3. How do you choose who goes into the control group?
Hand-picking makes the groups different, and then the difference no longer shows the effect of the message.
4. Why count every Goomer order, not just the ones with a coupon?
In Sahni et al.'s 70 tests, about 90% of the extra spending came from purchases without the coupon: the message reminds people of the business.
5. The meter warns that the result "may be chance". What should you do?
With small groups, 1 or 2 points fit within the normal ups and downs. Measuring more customers gives a solid answer.
Mixing in the earlier lessons
6. On the ladder, week 1 brought a 6% return. What should you compare it with?
The control group from the same group and week is the only one where everything is equal except the message (lesson 4, how to read the ladder).
7. A customer who orders every week came back with R$ 30 off. What does that tell you?
Loyal and recent calls for a light offer (lesson 2, on how much to offer). A big discount for them means less revenue.
Exercise: prepare your next campaign
All we know about the SMS case is what was mentioned: "more than 10%". Without the numbers, there's no way to know whether it worked, and that is exactly the lesson. That's why the exercise is to prepare your business's next campaign so the same doubt doesn't come up again. Before sending, write down:
- How many customers will get the message and how many will stay in the random control group.
- The send date and the end of the 15-day window.
- Your business's margin and the cost of each message.
At the end of the window, enter the numbers in the meter below. Until then, test it with your guess.
To wrap up: send the MeContaí team on WhatsApp your plan for the next campaign (group sizes and dates). After the window closes, send three numbers: the extra returns, the extra revenue and the estimated profit, and whether the meter warned that it "may be chance".
Tomorrow
Without looking, write down the two measurement questions, in order, and the calculation for each one. Then check them in the measurement cheat sheet.
You've finished WhatsApp Offers
The five lessons, one line each:
- The five parts of an offer: the five parts of a repeat-purchase message, in order.
- How much to offer each customer: the more likely a customer is to come back, the lighter the offer.
- The action that turns into an order: one step, starting with a verb, leading to where the order happens.
- The offer ladder: start small with one group and go up one step a week, only for those who did not buy.
- What the message actually added: how much the message brought in beyond what would have come anyway, in customers and in money.
To review without retaking the lessons, use the cheat sheets: offer cheat sheet · offer-strength scale · action and coupon · offer ladder · measuring the impact.
Training