By clicking “Accept All Cookies,” you agree to the storing of cookies on your device to enhance site navigation and analyze site usage.

Skip to main content

CoDE Unites the Digital Experimentation Community

November 12, 2021

By Paula Klein

Online surveys about your vaccination status, incentives to use a product or make a lifestyle change, and nudges to change your ride-sharing habits are likely the results of digital experimentation. And it’s on the rise.

The convergence of online experimentation and behavioral science was the focus of this year’s MIT Conference on Digital Experimentation (CoDE). The annual two-day convocation — hosted virtually by the MIT Initiative on the Digital Economy (IDE) — featured eight plenary speakers and 100 presentations. Many offered methods that can scale to hundreds of thousands of people and be used by policy makers to address problems from COVID vaccination reluctance to online marketing preferences. Discussions also pointed out limitations in some of the work.

The rapid increase of online tests and trials was evident in the technical papers and studies presented to CoDE’s 500 attendees. Also evident was the collaboration among researchers who are building on each other’s work and refining results. “This year we showcased a number of impressive experiments in public health, where our speakers brought state-of-the-art experimentation to increasing vaccination and preventative measures against COVID-19,” said Assistant Professor, Dean Eckles, who leads the Social Networks & Digital Experimentation Research Group at the MIT IDE.

Mixed Messages

For instance, Eckles said that one highlight of the event was a discussion between UCLA Anderson Professor Hengchen Dai and Associate Professor, Jake Bowers, from the University of Illinois at Urbana-Champaign, who each studied the impact of text message reminders for vaccination — with different results. “Dai’s experiment showed substantial positive effects, while Bowers’ yielded precise null results. This led to an enlightening conversation about changing populations, limits to generalizations, and the importance of rapidly deploying nudges,” according to Eckles.

In real-world field settings there are many variables that can’t be determined in experimental trials; no one size fits all. In the case of COVID nudges, “reminders can accelerate vaccination among those who are likely to eventually get vaccinated — and this is quite valuable and likely to save many lives,” Eckles said. Yet, the same reminders applied at a later time may not have much effect on those who remain unvaccinated, he said. That’s why digital experiments continue to examine the best methods to use in different settings.

Prompts and Nudges

The use of prompts and nudges was also discussed by University of Pennsylvania Wharton School Professor, Katherine Milkman, who studied incentives for flu vaccines with Walmart pre-COVID-19. Her findings, based on large population trials, are now being revised to apply to COVID vaccination incentives and were cited by Dai in her research.

Another research project spurred by the pandemic, studied mask-wearing incentives in South Asia. Ahmed Mushfiq Mobarak, Professor of Economics, Yale University, described his research on successful and failed policy responses, including the failure to ensure widespread COVID vaccine coverage in low- and middle-income countries (LMICs). Mobarak said that these countries very much want vaccines, but are not getting them from wealthier nations like the U.S.

“In the absence of vaccine coverage, non-pharmaceutical interventions like masking remain important,” he said. Mobarak discussed the results of a randomized controlled trial on efforts to distribute masks and vaccines to more than 100 million people concluding that personal, in-person education in local communities is very effective in many cases.

Eckles noted that there was a huge interest in some of the most technical material at the conference. Among the parallel sessions, attendees flocked to the most technical, methods-oriented tracks, he said. “This reflects continued and growing participation by researchers in the tech industry and involvement from quantitative researchers from statistics, economics, and computer science.”

For example, several talks addressed “interference” and “spillovers” in experimentation, such as in two-sided marketplaces like ride sharing or short-term apartment rentals. “This community has made substantial recent advances in making these methods both more practical and more theoretically rigorous.”