By Paula Klein
Digital experimentation –long thriving in academia—is now permeating corporate environments, as well, yielding new insights into online interactions. At the same time, issues of online privacy and selection bias raise concerns daily. At the CODE 2017 event hosted by the MIT IDE in October, these and other topics were explored in-depth by a diverse set of experts from the fields of social science, computer science, economics, analytics, marketing, and statistics.
Many focused on the practical value of digital experimentation as it expands into new applications. For instance, conducting real-time digital experiments has the potential to monitor and reduce security breaches, better understand how social media affects behavior, and even to enhance the quality of online services—from ridesharing to health care– according to speakers.
During two days of presentations and discussions experts evaluated advances in the field. Organizers and MIT IDE leaders Sinan Aral, Erik Brynjolfsson, and Alex (Sandy) Pentland, stated that: “The newly emerging capability to rapidly deploy and iterate micro-level, in-vivo, randomized experiments in complex social and economic settings, at population scale, is one of the most significant innovations in modern social science.”
It’s also changing rapidly. “As more and more social interactions, behaviors, decisions, opinions and transactions are digitized and mediated by online platforms, our ability to quickly answer nuanced causal questions about the role of social behavior in population-level outcomes such as health, voting, political mobilization, consumer demand, information sharing, product rating and opinion aggregation, is becoming unprecedented.”
A Sea-change Underway
“This new toolkit portends a sea-change in our scientific understanding of human behavior and dramatic improvements in social and business policy as a result. When appropriately theorized and rigorously applied, randomized experiments are the gold standard of causal inference and a cornerstone of effective policy.”
Organizers and participants also acknowledged the fine line that has to be walked when issues of user privacy and selection bias can color responses and results noting. The hosts noted that “the scale and complexity of these experiments create scientific and statistical challenges for design and inference.”
In his talk on Experimenting from the Inside-Out: The SBO Project, Carnegie Mellon University (CMU) Professor, Alessandro Acquisti, said that early versions of Internet testing—such as CMU’s HomeNet in 1995 –were the precursors to today’s deeper analytics. The Security Behavior Observatory (SBO) is an infrastructure for long-term monitoring of client machines that began in 2014– 20 years after HomeNet.
Emily Falk, Associate Professor at the University of Pennsylvania, spoke about Neuroscience Approaches to Understanding How Ideas and Behaviors Spread in social media campaigns, and what motivates behavior and outcomes.
From Healthcare to Uber Riders
Also addressing the behavioral-science implications of digital experiments, MIT Sloan Professor, Renée Richardson Gosline, talked specifically about The Outsourced Mind: Using Behavioral Science for Better Patient UX. She discussed how to use algorithms to help manage diseases, raise awareness, and ultimately to improve patient care.
In another session, Stanford Professor, Susan Athey, explained her current research on Service Quality in the Gig Economy: Empirical Evidence from Uber. Using ridesharing as her marketplace example, Athey examines –and attempts to dispel–the assumption that low-priced market entrants weaken safety and quality of services.
During a panel discussion [photo above] with Athey (center) and Gosline (right), IDE leader Aral (left) pointed out that as corporations do their own external experimentation, transparency may be biases or reduced, while collaboration-funding can become limited. In all, it’s a complex and rapidly-changing field of interest.
For more: See the full agenda and speaker bios here.
And watch the full videos of all the Plenary Session presentations on YouTube here.