Businesses often find that it takes too long to see the results of their costly marketing campaigns. By the time they measure exactly whether an advertising plan or discount promotion works, the market can change and opportunities are missed, including how to target further efforts.
As a remedy, MIT researchers experimented with “statistical surrogacy” to estimate outcomes and make adjustments before the year or so they may otherwise have to wait for results. They studied churn management at The Boston Globe to see if discounts for digital subscriptions maximized long-term revenue. At an IDE seminar this fall, MIT Sloan Associate Professor, Dean Eckles, explained that in order for the news organization to increase retention and profits over the long term, the researchers developed and applied methods for learning surrogate outcomes. “Over three years, our approach had a net-positive revenue impact in the range of $4 million to $5 million compared to The Boston Globe’s current policies,” according to Eckles, a co-author of the research paper, Targeting for Long-Term Outcomes. The paper won this year’s Best Paper Award at the INFORMS Annual Conference in the eBusiness category. It was co-authored by MIT IDE Director Sinan Aral, Sloan PhD candidate Jeremy Yang, and former Sloan Postdoc, Paramveer Dhillon.
MIT IDE Content Manager, Paula Klein, asked Eckles and Yang to explain the significance of the study and the implications for other marketing strategies.
Q: From the macro perspective, what was the impetus for this study; how widespread are the problems of long-term outcomes and targeting for business marketing campaigns?
Eckles: Many decision-makers have this problem: They want to learn from early trials, but can only find out what happened to customers, patients, etc., if they wait. In a clinical trial, we might care about mortality from any cause after five years, but basing all decisions on this criteria is problematic. Patients may have multiple mortality risks, and a lot of the variation in mortality could have nothing to do with the treatment. In business settings, we often care about long-run profits associated with a customer — this is especially true when thinking about costly efforts to acquire or retain customers.
Yang: I’d also say this type of problem appears as long as you care more about the consequences of your actions tomorrow (long-term) than today (short-term), and you want to decide right away. Our approach has many applications in marketing or business in general, but it can also be applied much more broadly.
Q: Can you explain the concepts of treatment effects and target interventions? Are they commonly used by businesses and are they more accurate now with AI algorithms? How are digital technologies changing the game?
Yang: The concepts of treatment effects and target interventions are quite intuitive. Suppose a firm has some customer interventions (actions) it can take such as sending promotional discounts to them. The question is, ‘To whom should the firm send these discounts in order to maximize a long-term outcome (revenue)?’ This is what it means to target intervention (i.e. discounts) to the appropriate customer. In our case, a treatment effect is the difference in revenue the firm earns from a customer when it sends her discount A versus discount B. Although these definitions aren’t directly linked to AI, or more specifically machine learning, AI can help to learn what the treatment effects are and how to target interventions from the available data.
Digital technology makes a huge difference in product and service measurement and delivery. To use the discount example again, the firm can learn and target much better if it has more data on its customers. In turn, more information about these customers can be measured, stored, and then analyzed with digital technology. Also, the firm can deliver these discounts digitally to customers’ smartphones, rather than using snail mail for printed coupons.
Q: What specific changes did The Boston Globe seek to learn about its customers and how did you approach the problem? What were the results on churn and revenue, and what were the biggest challenges?
Yang: One thing the Globe did was to lower its introductory rate to acquire more new subscribers, but once that rate expires, subscribers will be paying the full price. These new subscribers could be at higher risk of canceling their subscription, creating churn. Therefore, churn management is a pressing problem–which is exactly what our approach was applied to solve.
We ran two rounds of experiments, the first in 2018, and the second a year later. Based on our estimation, our approach reduces churn and increases the three-year revenue per subscriber by $40, on average. If you multiply that by the total number of subscribers in the experiments, it would amount to a substantial $4 million to $5 million. Our approach has a few key components including experimental design, learning surrogate outcomes, optimal targeting using learned surrogate outcomes, and validation. It took us a while to build the pipeline. We were also deeply involved in the implementation.
Q: In the current economic environment and pandemic what takeaways can you offer executives for campaigns underway or those about to start? Are there totally new considerations to weigh or can these lessons still apply?
Eckles: One of the things that the pandemic has highlighted is that the future isn’t always like the past — sometimes things are just a bit different, like maybe we start attracting a different type of customer, and sometimes things can abruptly change. So this means we should always keep experimenting. We may have learned in the past that some intervention works, or that this surrogate outcome is predictive of long-run outcomes. But this can change, so we should use that knowledge but also keep experimenting to keep up with changes.