Please join us for breakfast and networking pre-conference. Take the elevators to the 7th floor where you will check in and find your name badge and visit our sponsor tables in Salon East (7th floor).

• CODE@MIT Organizing Team

• Hema Yoganarasimhan – Professor, University of Washington “A Bias Correction Approach for Interference in Ranking Experiments”

Online marketplaces use ranking algorithms to determine the rank-ordering of items sold on their websites. The standard practice is to determine the optimal algorithm using A/B tests. We present a theoretical framework to characterize the Total Average Treatment Effect (TATE) of a ranking algorithm in an A/B test and show that naive TATE estimates can be biased due to interference. We propose a bias-correction approach that can recover the TATE of a ranking algorithm based on past A/B tests, even if those tests suffer from a combination of interference issues. Our solution leverages data across multiple experiments and identifies observations in partial equilibrium in each experiment, i.e., items close to their positions under the true counterfactual equilibrium of interest. We apply our framework to data from a travel website and present comprehensive evidence for interference bias in this setting. Next, we use our solution concept to build a customized deep learning model to predict the true TATE of the main algorithm of interest in our data. Counterfactual estimates from our model show that naive TATE estimates of click and booking rates can be biased by as much as 15% and 29%, respectively.

• Susan A Murphy – Professor, Harvard University “Did My Reinforcement Learning Algorithm Personalize?”

Reinforcement Learning provides an attractive suite of online learning methods for personalizing interventions in Digital Health.   However after an reinforcement learning algorithm has been run in an experiment, how do we assess whether personalization occurred?  We might find users for whom it appears that the algorithm has indeed learned in which contexts the user is more responsive to a particular intervention.  But could this have happened completely by chance?   We discuss some first approaches to addressing these questions. 

Enjoy a coffee break with light refreshments.

Session A: The Practice of Experimentation

Session B: Long Term Outcomes and Surrogates

Session C: Online Lab Experiments

Grab some food and find a seat to watch the rapid-fire poster presentations while enjoying lunch.

Note to presenters: Day 1 poster presenters will participate during this slam, day 2 poster presenters will participate on Saturday also during lunch in the same format.

• Martin Tingley – Netflix, “A Tale of Complexity:  Report from the Causal Inference and Digital Experimentation Roundtable (CIDER)”

In April 2023, Netflix hosted the Causal Inference and Digital Experimentation Roundtable (CIDER), which  brought together practitioners from industry and applied researchers from academia for candid discussions on the current state and next opportunities for experimentation and causal inference at scale. This talk provides an opinionated summary of the discussions, reinforced with specific examples drawn from Netflix. I will first discuss the agricultural roots of digital experimentation, and then propose that a handful of root complexities –scale, time dependence, constraints on treatment delivery, human decision makers– challenge the commodification of experimentation beyond the simple agricultural model.

• Jean Pouget-Abadie – Google Research, “Designing Experiments for Marketplaces and other Bipartite Graphs”

When the treatment assignment of one unit affects the outcome of another, we say there is interference. Interference is especially prevalent in marketplaces, where buyer and seller interactions lead to complex dependence structures. As a violation of the stable unit treatment value assumption, the presence of interference can lead to bias of standard estimators under naive randomized designs. In this talk, we will cover a set of design and estimation paradigms used at Google to conduct causal inference research in a bipartite graph setting, inspired from, but not limited to, marketplace experiments, with specific attention paid to clustered randomized designs.

Enjoy a coffee break with light refreshments.

Session D: Platforms and Markets

Session E: Continuous Experimentation

Session F: Bipartite Experiments

Session G: Networks and Teams

Enjoy a coffee break with light refreshments.

Session H: The AI-Human Interface

Session I: Bandits and Adaptive Experimentation

Session J: Personalization and Experimentation

Enjoy a coffee break with light refreshments.

• Widad Machmouchi – Principal DS Manager, Microsoft
• James McQueen – Principal Applied Scientist, Amazon
• Tushar Shanker – Director of Data, Grow Therapy
• Wenjing Zheng – Tech Lead Ecosystem Data Science, Roblox

A discussion of the challenges of conducting experiments in industry contexts featuring data scientists and researchers from a number of prominent firms.

Enjoy networking with conference attendees and speakers while viewing poster presenter work and have the chance to chat with the author(s).

Thank you for coming, see you on day 2!

Please join us for breakfast and networking pre-conference. Visit our sponsor tables in Salon East (7th floor).

• Annie Franco – Research Scientist, Meta, “Assessing the Impact of Social Media Affordances on Political Attitudes and Behavior during the US 2020 Election”

Among the most normatively important and heavily debated questions about social media are whether and to what extent they affect elections and democracy. In this talk, I present new evidence and insights on this topic from three recently published experimental studies conducted as part of most comprehensive industry-academia research collaboration to date examining the political impact of Facebook and Instagram in the context of the US 2020 Election. These studies focus on key features of social media hypothesized to affect political attitudes and behaviors: algorithmic ranking algorithms; reshares and virality; and exposure to like-minded sources. I will first provide an overview of the overall project, the design of each of the experimental interventions, and their shared methodologies. I will then discuss the findings with respect to participant’s platform experiences and four core substantive outcomes identified at the outset of the project: political polarization, political knowledge, political participation, and democratic attitudes.

• Jorge Guzman – Gantcher Associate Professor of Business, Columbia University & NBER, “Climate change framing and innovator attention”

Drawing the attention of innovators to climate change is important for green innovation. We report an email field experiment with MIT using messages about the impact of climate change to invite innovators (SBIR grantees) to apply to a technology competition. We vary our messages on the time frame and scale of the human cost of climate change across scientifically valid scenarios. Innovator attention (clicks) is sensitive to climate change messaging. These changes in clicks also predict higher application rates. The response varies by individual characteristics such as location-based exposure to climate change risks and whether innovators have climate-related innovations. Finally, using a structural model of innovator attention, we provide estimates of the implied discount rate of time and the elasticity of attention to lives at stake.

Enjoy a coffee break with light refreshments.

Session K: Health & Education

Session L: Experimentation in Markets

Session M: Misinformation and Polarization

Session N: Privacy Preserving Experimentation and Partial Identification

Enjoy a coffee break with light refreshments.

• Payel Das – Principal Research Staff Member and Manager, Trusted AI, IBM Research
• Xiao Ma – Software Engineer, Google
• Daniel Rock – Assistant Professor, Wharton School of the University of Pennsylvania
• Brandon Stewart – Associate Professor, Princeton University

*Please note: This session will be live-streamed and made available to the public free of charge. Please join the livestream HERE at 11:45am ET on Saturday, Nov. 11 and you may go back to watch this session on-demand any time post-event.

Grab some food and find a seat to watch the second part of rapid-fire poster presentations while enjoying lunch.

Session O: Experimentation Under Interference

Session P: Beyond Average Treatment Effects

Session Q: Generative AI and Large Language Models

Enjoy a short coffee break and make your way back to the plenary room.

• Peng Ding – Associate Professor, UC Berkeley “Causal Inference in Network Experiments: Regression-Based Analysis and Design-Based Properties”

Network experiments have been widely used in investigating interference among units. Under the “approximate neighborhood interference” framework introduced by {Leung2022}, treatments assigned to individuals farther from the focal individual result in a diminished effect on the focal individual’s response, while the effect remains potentially nonzero {Leung2022} establishes the consistency and asymptotic normality of the inverse-probability weighting estimator for estimating causal effects in the presence of interference. We extend these asymptotic results to the Hajek estimator which is numerically identical to the coefficient from the weighted-least-squares fit based on the inverse probability of the exposure mapping. The numerically equivalent regression-based approach offers two notable advantages: it can provide standard error estimators through the same weighted-least-squares fit, and it allows for the integration of covariates into the analysis. Furthermore, we introduce the regerssion-based network-robust variance estimator, adopting the form of the Heteroskedasticity and Autocorrelation Consistent estimator, and analyze its asymptotic bias. Recognizing that the variance estimator can be anti-conservative, we propose an adjusted variance estimator to improve empirical coverage. Although we focus on regression-based point and variance estimators, our theory holds under the design-based framework, which assumes that the randomness comes solely from the design of network experiments and allows for arbitrary misspecification of the regression models.

• Hannah Li – Assistant Professor, Columbia Business School, “When Traditional Power Analysis Falls Short – A Public Health Case Study”

A concern about service interventions, commonly found in domains like public health, education, and vocational training, is that promising interventions at the randomized controlled trial (RCT) stage may not perform well at scale. In this work we highlight and isolate the effects of one such factor: capacity constraints and the ensuing spillover effects. We consider a case study of a mobile health platform designed to improve patients’ adherence to treatment protocols. Due to capacity constraints on the system, the effect observed in an RCT may diminish when servicing a larger population. We show that this has counterintuitive implications for the typical power analysis in experiment planning and that a joint power and capacity analysis should be implemented.

• CODE@MIT Organizing Team

Enjoy networking with conference attendees and speakers while viewing poster presenter work and have the chance to chat with the author(s).

Thank you for coming, see you in 2024!

Dr. Payel Das is a Research Staff Member and manager in the AI Science Department of IBM Thomas J Watson Research Center in Yorktown Heights, NY. She is also an adjunct associate professor at the department of Applied Physics and Applied Mathematics (APAM), Columbia University.  She received her B.Sc. degree from Presidency College, Kolkata and M.Sc. degree from Indian Institute of Technology, Chennai in India. In 2002, Das came to USA to pursue a Ph.D. degree in theoretical physical chemistry with Prof. Cecilia Clementi at Rice University, Houston. Her Ph.D. thesis work was at the intersection of statistical physics and machine learning. During her Ph.D., she developed coarse-grained energy function and novel sampling techniques to directly explore low-dimensional manifold of a very complex, dynamic problem, such as protein folding.

She has been a visiting fellow at the Institute of Pure and Applied Mathematics (IPAM) at UCLA, where she worked on extracting efficetive dimensionality of high-dimensional systems. She has also been a visting student in Princeton University, where she worked with Prof. Yannis Kevrekidis on developing a multi-scale simulation methodology, based on data mining tools for inferring low-dimensional reduction coordinates. During her years in India, she was a summer intern at National Chemical Laboratory, Pune, with Prof. Sourav Pal, where she worked on theoretical investigation of hard-soft acid base relation. Her research interest is in understanding deep neural networks from a statistical mechanics perspective as well as in developing novel interpretable deep learning algorthms.

In her current role, she technically leads and manages research projects related to artificial intelligence (AI) for creativity and  discovery, with inspirations from and applications in material science, chemistry, physics, biology, and neuroscience. Many of these projects lie at the intersection of data-driven and physics-based  modeling. A major focus is to develop novel deep generative models for heterogenous data, which is abundant in real world applications. Prior to her current role, Dr. Das was a post-doctoral researcher at the Computational Biology Center at IBM TJ Watson Research, where she worked on free energy perturbation theory to study disease mechanisms.

Das has co-authored over 40 peer-reviewed publications and several patent disclosures, given dozens of invited talks at several university colloquiums, department seminars, top rated conferences, and workshops. She serves in the editorial advisory board member of the ACS Central Science journal. Das is the recipient of IBM Outstanding Technical Achievement Award (the highest technical award at IBM), two IBM Research Division Awards, one IBM Eminence and Excellence Award, and two IBM Invention Achievement Awards.

Peng Ding is an Associate Professor in the Department of Statistics, UC Berkeley. Peng obtained his Ph.D. from the Department of Statistics, Harvard University in May 2015, and worked as a postdoctoral researcher in the Department of Epidemiology, Harvard T. H. Chan School of Public Health until December 2015. Previously, Peng received his B.S. (Mathematics), B.A. (Economics), and M.S. (Statistics) from Peking University.

Dean Eckles is an Associate Professor of Marketing at MIT Sloan. He is affiliate faculty at the Institute for Data, Systems & Society in the Schwarzman College of Computing. Dean is also the ‘New Data Analytics’ Research Group Lead at the MIT Initiative on the Digital Economy.

His substantive research examines people’s interactions with and through communication technologies, especially the ways these technologies mediate, amplify, and direct social influence. This work sometimes requires or benefits from new analytical methods, so Eckles also works on applied statistics, design of field experiments, and causal inference.

Prior to joining MIT, he was a scientist at Facebook, where he worked on many product areas and analytical methods, including News Feed, messaging, advertising, tools for randomized experiments, and survey methods. Eckles previously worked in research at Nokia and Yahoo.

Eckles received his BA in philosophy, a BS and MS in cognitive science, an MS in statistics, and a PhD in communication, all from Stanford University.

Annie Franco is a senior research scientist at Meta, where she combines computational methods and causal inference both to build products and algorithms, and to understand their impact on people and society. She currently co-leads the US 2020 Facebook & Instagram Election Study, a collaboration with academic researchers to study the role and impact of Meta products in the context of the US 2020 election. In prior work at Meta, she led technical and cross-functional teams working in areas related to political polarization, misinformation, algorithmic amplification, fairness, and social cohesion.

Professor Jorge Guzman is the Gantcher Associate Professor of Business in Columbia University and a Faculty Research Fellow at the National Bureau of Economic Research (NBER). He studies the way startups relate to their geography, and the strategic decisions that underly startup choices. Prof Guzman received a PhD and an MBA from the Sloan School of Management at MIT, and was previously a postdoc at the NBER.

David Holtz is an assistant professor in the Management of Organizations (MORS) and Entrepreneurship and Innovation groups at the Haas School of Business. He earned his PhD at the MIT Sloan School of Management, in the Information Technology (IT) group. He also holds an MA in Physics and Astronomy from Johns Hopkins University, and a BA in Physics from Princeton University.

Hannah Li is an Assistant Professor in the Decision, Risk, and Operations division at Columbia Business School. Her research focuses on developing data science methods for social systems–marketplaces, education systems, and online platforms. Her research combines techniques from operations research, statistics, and economics to develop theoretical insights for practically motivated problems. She informs her work with industry experience, collaborating closely with large online platforms. Her work has received several awards, including Best Student Paper award at ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization (EAAMO) and 2nd place in the INFORMS Revenue Management and Pricing Student Paper Award.

Prior to joining Columbia, Hannah received her BA in mathematics from Pomona College, received her Ph.D. in operations research from Stanford University, and worked as a postdoctoral associate at MIT.

Xiao Ma – Software Engineer, Google

Widad Machmouchi is a Principal Data Science Manager in the AI Platform at Microsoft. She leads a team of data scientists in the Experimentation Platform group focusing on trustworthy metric design and LLM experimentation. Widad enables products like Office, Teams and Bing to make data-driven decisions through AB testing and experimentation. She is passionate about developing intelligent products that give users agency over their data while maximizing the utility they receive. Widad holds a PhD in Theoretical Computer Science from the University of Washington, Seattle and is a co-founder of a technology hardware start-up.

James McQueen is a Principal Applied Scientist working at Amazon on their primary A/B testing platform, Weblab. James works on core statistical methodology applied at scale, particularly covariate adjustment and Bayesian methods.

Susan A. Murphy, Mallinckrodt Professor of Statistics and of Computer Science, Associate Faculty, Kempner Institute,
Harvard University. Susan A. Murphy’s research focuses on improving sequential decision making in health, in particular the development of online, real-time reinforcement learning algorithms for use in personalized digital health. She is a member of the US National Academy of Sciences and of the US National Academy of Medicine. In 2013 she was awarded a MacArthur Fellowship for her work on experimental designs to inform sequential decision making. She is a Fellow of the College on Problems in Drug Dependence, Past-President of Institute of Mathematical Statistics, and a former editor of the Annals of Statistics.

Jean (non-French speakers can call him “John”) is a research scientist at Google Research New York on the Algorithms & Optimization team, led by Vahab Mirrokni. He completed his PhD in Computer Science at Harvard University, advised by Edoardo Airoldi and Salil Vadhan, where he received the Siebel scholarship. Prior to that, he was an undergraduate at Ecole Polytechnique in Paris. His recent research interests focus on causal inference and experimental design, particularly when network interference is present. In the past, he looked into using neural networks to generate distributions.

Daniel Rock is an Assistant Professor of Operations, Information, and Decisions at the Wharton School of the University of Pennsylvania. His research is on the economic effects of digital technologies, with a particular emphasis on the economics of artificial intelligence. He has recently worked on studies addressing the types of occupations that are most exposed to machine learning, measuring the value of AI skillsets to employer firms, and adjusting productivity measurement to include investments in intangible assets. His research has been published in various academic journals and featured in outlets such as The New York Times, Wall Street Journal, Bloomberg, Harvard Business Review, and Sloan Management Review. Daniel received his B.S. from the Wharton School of the University of Pennsylvania, and his M.S. and Ph.D. from the Massachusetts Institute of Technology.

Before transitioning to the corporate world, Tushar received rigorous training in growth strategies for Platforms and Multi-sided Markets under the guidance of Professor Marshall Van Alstyne at Boston University. With over a decade of experience in data science and analytics at world-leading multi-sided platforms like Airbnb, LinkedIn and Uber, his teams have focused on scalable data products that drive consumer satisfaction and business growth.

Tushar’s expertise spans machine learning, statistical modeling, experimentation, and causal inference, and he has successfully guided his teams in solving complex strategic problems in areas such as user acquisition, retention, engagement, and monetization.

Currently serving as the Director of Data at Grow Therapy, Tushar leads a team of data engineers, scientists and analysts to deliver innovative data solutions that support the company’s mission of making affordable mental health care accessible to all. His experience and leadership make him a valuable contributor to the data science community.

Brandon Stewart is Associate Professor of Sociology at Princeton University where he is also affiliated with the Politics Department, the Office of Population Research, the Princeton Institute for Computational Science and Engineering, The Center for Information Technology Policy and the Center for the Digital Humanities. He develops new quantitative statistical methods for applications across computational social science. Along with Justin Grimmer and Molly Roberts, he is the author of the 2022 book Text as Data: A New Framework for Machine Learning and the Social Sciences.

Martin Tingley leads a multidisciplinary team on the Netflix Experimentation Platform, focussed on developing and scaling both statistical methodology and software solutions to improve decision-making across the company. Prior to joining Netflix in 2017, Martin spent several years modeling and pricing catastrophe risk at Insurance Australia Group. In an earlier academic career, Martin was an Assistant Professor in Statistics and Meteorology at Penn State University. Martin holds a Ph.D. in Earth and Planetary Sciences and an M.A. in Statistics from Harvard University, and a B.Sc. in Physics from the University of Toronto.

Dr. Hema Yoganarasimhan is a Professor of Marketing at the Foster School of Business, University of Washington. She also holds affiliate appointments in Computer Science and Engineering, Department of Economics, and Center for Statistics in the Social Sciences. She is recognized as one of the leading experts in quantitative marketing. Hema’s research brings together large-scale marketing data, economic theory, and econometric and machine learning tools to help firms optimize and automate their marketing decisions. Her recent work focuses on combining machine learning tools and statistical econometric methods to address important problems in the domain of digital marketing and online platforms.

I lead core data science research for growing the Roblox Ecosystem. She is responsible for developing the vision and strategy and driving the execution of a system of core data science models that aim to understand and quantify complex causal relationships across the Roblox Ecosystem, and to inform our business growth. She mentors and collaborates with data scientists across the product domains on the execution of these strategies. Wenjing works closely with senior leadership to align strategic roadmaps and priorities, ensuring our objectives and continuing progress resonate with overarching business goals and deliver impactful results.