MeContaíTraining

Cheat sheet · Oferta no WhatsApp

Measuring the impact

The return rate alone is misleading. What counts is how much extra the message brought in beyond what would have come anyway, in customers and in money.

Two questions, in this order

QuestionFormula
1 · Extra returnrate among those who got it − control rate
2 · Extra revenue(revenue per customer among those who got it − control's) × customers who got it
Profitextra revenue × margin − cost of the messages

Example

With messageControl
Customers1,000200
Came back within 15 days110 (11%)20 (10%)
Revenue per customerR$ 9.90R$ 8.50

Extra return: 1 point, about 10 customers, not 110. Extra revenue: R$ 1.40 × 1,000 = R$ 1,400. At these sizes, 1 point could still be chance.

Rules

  1. Draw the control group at random before sending: ~10%, at least 30 to 50 customers.
  2. Same window: 15 days from sending, for both groups.
  3. Count every Goomer order in the window, with or without the coupon, at the amount paid.
  4. Small difference with a small group: repeat and add up the results before drawing conclusions.
  5. No control group? Compare with similar customers over a period with no sends. That's an estimate.

Sources: Sahni, Zou & Chintagunta, 2017 (≈90% of the effect outside the coupon); Anderson & Simester, 2004; Kohavi, Tang & Xu, 2020. Lesson: 5 · What the message added. Tool: the calculator in lesson 5.