Please take the elevators to the 7th floor to check in and get your name badge. Enjoy breakfast and networking with other conference attendees and speakers.

• David Holtz – Assistant Professor, UC Berkeley Haas School of Business and CODE@MIT Organizer

• Johan Ugander – Associate Professor, Stanford School of Engineering
  • Counterfactual Evaluation of Peer-Review Assignment Policies

Peer review assignment algorithms aim to match research papers to suitable expert reviewers, working to maximize the quality of the resulting reviews. A key challenge in designing effective assignment policies is evaluating how changes to the assignment algorithm map to changes in review quality. In this work, we leverage recently proposed policies that introduce randomness in peer-review assignment–in order to mitigate fraud–as a valuable opportunity to evaluate counterfactual assignment policies. To address challenges in applying standard off-policy evaluation methods, such as violations of positivity, we introduce novel methods for partial identification based on monotonicity and smoothness assumptions. We apply our methods to peer-review data from two computer science venues, including a major conference with over 8000 submissions and over 3000 reviewers.

• Sida Peng – Sr. Principal Economist, MSFT
  • The Productivity Effects of Generative AI – Evidence from Three Field Experiments with Software Developers

This study evaluates the impact of generative AI on software developer produc- tivity by analyzing data from three randomized controlled trials conducted at Mi- crosoft, Accenture, and an anonymous Fortune 100 electronics manufacturing com- pany. These field experiments, which were run by the companies as part of their ordinary course of business, provided a randomly selected subset of developers with access to GitHub Copilot, an AI-based coding assistant that suggests intelligent code completions. Though each separate experiment is noisy, combined across all three experiments and 4,867 software developers, our analysis reveals a 26.08% increase (SE: 10.3%) in the number of completed tasks among developers using the AI tool. Notably, less experienced developers showed higher adoption rates and greater pro- ductivity gains.

Enjoy a 25-minute coffee and networking break.

Session A: Marketing and Recommendations          

1. Consumer Inferences from Product Rankings: The Role of Beliefs in Search Behavior
   • Jessica Fong (Michigan), Olivia Natan (Berkeley), Ranmit Pantle (Northwestern)
2. Product Visual Similarity in Recommendation: A Randomized Field Experiment on Dynamic Retargeting
   • Dimitrios Tsekouras (Erasmus University), Ting Li (Erasmus University), Zherui Yang (IE University)
3. Privacy-enhanced retargeting? Field experimental evidence of privacy-enhanced and traditional ad effectiveness
   • Shunto J. Kobayashi (BU Questrom), Garrett A. Johnson (BU Questrom), Zhengrong Gu (BU Questrom)
4. Fair Active Learning for Targeted Marketing Campaigns
   • Aurélie Lemmens (RSM, Erasmus University Rotterdam), Zhuoyu Shi (RSM, Erasmus University Rotterdam)

Session B: Generative AI and LLMs I        

1. Generative AI Can Harm Learning
   • Hamsa Bastani (Wharton) Osbert Bastani (Upenn SEAS) Alp Sungu (Wharton) Haosen Ge (Wharton) Ozge Kabakci (Budapest International High School) Rei Marimann
2. As Generative Models Improve, People Adapt Their Prompts
   • Hong-Yi TuYe (MIT), Eaman Jahani (UoM), Ben Manning (MIT), Joe Zhang (Stanford),Mohammed Alsobay (MIT), Christos Nicolaides (University of Cyprus), Siddharth Suri (MSR), and David Holtz (UC Berkeley)
3. The Synthetic Laboratory: Language Models as Auction Participants
   • Anand Shah (MIT), Kehang Zhu (Harvard), Yanchen Jiang (Harvard), Karem Dayi (Harvard), Jeffrey Wong (Harvard), John J. Horton (MIT), David C. Parkes (Harvard)
4. LLM-based Recommendation in Content Promotion: Empirical Evidence from an ABEMA Field Experiment
   • Shota Yasui(CyberAgent), Shingo Uto(AbemaTV), Shuhei Nakano(AbemaTV), Shunya Suga(AbemaTV), Takuya Iijima(AbemaTV)

Session C: Online Platform Governance

1. Field Experiments about the Relative Effectiveness of Suspension Durations on a Popular Social Platform
   • Jeffrey Gleason (Roblox; Northeastern University), Alex Leavitt (Roblox), Bridget Daly (Roblox)
2. Transforming Vigilante Justice: Restorative Appeals Outperform Retribution on a Simulated Social Media Platform
   • Pengfei Zhao (Cornell), Natalie N. Bazarova (Cornell), Inhwan Bae (Cornell), René F. Kizilcec (Cornell), Winice Hui (Cornell), Drew Margolin (Cornell)
3. Governance Shocks and User Engagement: Lessons from the 2023 Reddit ‘Go Dark’ Protests
   • Mimansa Bairathi (UCL), Shrabastee Banerjee (Tilburg), Qinglai He (UW Madison), Justin Huang (University of Michigan)
4. Perceived legitimacy of layperson and expert content moderators
   • Cameron Martel (MIT), Adam J. Berinsky (MIT), David G. Rand (MIT), Amy X. Zhang (University of Washington), Paul Resnick (University of Michigan)

1. Optimizing Returns from Experimentation Programs
   • Timothy Sudijono (Stanford), Simon Ejdemyr (Netflix), Apoorva Lal (Netflix), Martin Tingley (Netflix)
2. Multi Armed Bandits with Network Interference
   • Abhineet Agarwal (UC Berkeley), Anish Agarwal (Columbia University), Lorenzo Masoero (Amazon Weblab), Justin Whitehouse (Carnegie Mellon University)
3. Generalized Randomization Tests in Detecting Network Spillovers
   • Zhen Zhong (HKU), Shan Huang (HKU), Lu Deng (Tencent)
4. C-Learner: Constrained Learning for Causal Inference
   • Tiffany Cai (Columbia), Yuri Fonseca (Columbia), Kaiwen Hou (Columbia), Hongseok Namkoong (Columbia)
5. Estimating the Returns from an Experimentation Program
   • Simon Ejdemyr (Netflix), Martin Tingley (Netflix), Yian Shang (Netflix), Travis Brooks (Netflix)
6. Unbiased Estimation for Total Treatment Effect Under Interference Using Aggregated Dyadic Data
   • Lu Deng (Tencent), Yilin Li (Tencent), Jingjing Zhang (Tencent), Yong Wang (Tencent), Chuan Chen (Tencent)
7. Semiparametrically Efficient Prediction-Powered Inference
   • Wenlong Ji (Stanford), Tijana Zrnic (Stanford), Lihua Lei (Stanford)
8. Better Design Uncertainty Quantification for Tighter Experimental Confidence Intervals
   • Jennifer Brennan (Google), Sébastien Lahaie (Google), Nick Doudchenko (Uber), Adel Javanmard (Google, USC), Jean Pouget-Abadie (Google)
9. Large Scale Prognostic Score Matching
   • Xudong Bai (Tencent), Shichao Han (Tencent), Yilin Li (Tencent), Jingjing Zhang (Tencent), Chuan Chen (Tencent), Yong Wang (Tencent)
10. Optimization of Experimental Designs: A Unified Context-Aware Approach
    • Ruoxuan Xiong (Emory), Mohsen Bayati (Stanford)
11. Leave-two-out Randomization Inference for Synthetic Controls in Small Samples
    • Lihua Lei (Stanford), Timothy Sudijono (Stanford)
12. Your MMM is Broken: Identification of Nonlinear and Time-varying Effects in Marketing Mix Models
    • Ryan Dew (Wharton), Nicolas Padilla (LBS), Anya Shchetkina (Wharton)
13. Performance of Synthetic Diff-in-Diff models for Geo-Randomized Experiments
    • Paula Meloni (UC Berkeley), Stefan Hut (Amazon.com), Mahnaz Islam (Amazon.com)
14. Experimentation for Different Scheduling Policies on Queues: Mixed Differences-in-Q Estimators Based on Little’s Law
    • Nanshan Jia (UCB), Ramesh Johari (Stanford), Nian Si (HKUST), Zeyu Zheng (UCB)
15. Variance Reduction using Pre-experiment and In-experiment Data
    • Zhexiao Lin (UC Berkeley), Pablo Crespo (Etsy)
16. Counterfactual Evaluation of Ads Ranking Models through Domain Adaptation
    • Mohamed A. Radwan (Meta), Himaghna (Meta), Bhattacharjee, Quinn Lanners (Duke University), Jiasheng Zhang (Meta), Serkan Karakulak (Meta), Houssam Nassif (Meta), Murat Ali Bayir (Meta)
17. The Shared Control Solution: Boosting Experimentation Throughput and Efficiency
    • Antonio Davoli (Meta), Albert Cardenete (Meta), Shicong Yan (Meta), Shivesh Ranjan (Meta), Ovunc Bozcan (Meta), Gheorghe Postelnicu (Meta), Sonam Saxena (Meta), and Kenneth Hung (Meta)
18. Large Scale Longitudinal Experiments: Estimation And Inference
    • Apoorva Lal (Netflix), Alexander Fischer (Trivago), Matthew Wardrop (Netflix)

• Susan Athey – Professor, Stanford
  • Interventions to Support Labor Market Transitions

Despite the growth of jobs in the IT sector in many parts of the world, disadvantaged workers struggle to transition into these jobs.  This talk will present large scale evaluations of two different interventions designed to facilitate such transitions. In the first, we created a digital program where Polish and Ukrainian women created portfolios to demonstrate skills; this program substantially increased the share of women transitioning into IT.  The second intervention facilitated the sharing of Coursera certificates on the LinkedIn platform.  Disadvantaged workers, those without a college degree and those from developing countries, were more likely to transition to a new job in the area covered by their credentials.

• Daniel Björkegren – Assistant Professor, Columbia University
  • Experiments in Public and Private Transit in Lagos

Many of the world’s fastest growing cities rely on private transit, commonly informal minibuses. As these cities become wealthier they have been investing in centralized public transit. But if private incumbents respond, public investment will indirectly affect even those who do not use the new system. This paper studies how providing public transit affects informal providers in Lagos, Africa’s largest city. We collect new panel data on the informal network to measure how it responds to the rollout of 14 new formal bus routes in the city’s Bus Reform Initiative. Following government entry, informal departures fall by around 26%, and prices fall on routes where the new option has relatively lower prices. To measure the impacts on consumer welfare, we develop a custom app to measure commuter valuations for price and wait time in an RCT. We combine these estimates to measure the welfare impact of expanding the formal system that arises from the changes induced in the informal system.

Enjoy a 20-minute coffee and networking break.

Session D: Heterogeneous Methods Presentations I

1. Producer-Side Experiments Based on Counterfactual Interleaving Designs for Online Recommender Systems
   • Yan Wang (LinkedIn), Shan Ba (LinkedIn)
2. Estimating the Global Average Treatment Effect under Structured Interference
   • Kevin Han (Meta), Shuangning Li (Chicago Booth), Johan Ugander (Stanford)
3. Graph Neural Networks for Causal Inference Under Network Confounding
   • Pantelis Loupos (UCD), Michael Leung (UCSC)
4. Experiments in Two-Sided Platforms Using Recommender System
   • Mahyar Habibi (Bocconi), Zahra Khanalizadeh (University of Washington), Negar Ziaeian (Warwick)

Session E: Advertising and Auctions

1. Ad Clustered User Randomized Trials
   • Edvard Bakhitov (Meta), Ivan Sherstobitov (Meta), Brett Daley (University of Alberta), Houssam Nassif (Meta), Daniel Ting (Meta), Congshan Zhang (Meta), and Sergei Leonenkov (Meta)
2. Interference Among First-Price Pacing Equilibria: A Bias and Variance Analysis
   • Luofeng Liao (Columbia), Christian Kroer (Columbia), Sergei Leonenkov (Meta), Okke Schrijvers (Meta), Liang Shi (Meta), Nicolas Stier-Moses (Meta), Congshan Zhang (Meta)
3. How do Advertisers React to Changes in Ad Auction Pricing? Evidence from a Clustered Multiple Randomization Experiment
   • Lorenzo Masoero (Amazon), Stefan Hut (Amazon), Jiaqi Li (Amazon), Suhas Vijaykumar (Amazon), Zhenning Wang (Amazon), Guido Imbens (Stanford)
4. Behavioral Study of Dashboard Mechanisms
   • Paula Kayongo (Northwestern), Jessica Hullman (Northwestern), Jason Hartline (Northwestern)

Session F: The Practice of Experimentation

1. Overcoming the Winner’s Curse: Leveraging Bayesian Inference to Improve Estimates of the Impact of Features Launched via A/B tests
   • Ryan Kessler (Amazon)
2. Estimating the True Effect Size Distribution with SIMEX
   • Zhaoqi Li (Meta), Daniel Ting (Meta), Ilya Gorbachev (Meta), Ehsan Emamjomeh-Zadeh (Meta), Houssam Nassif (Meta)
3. An exploration of levers impacting returns from experimentation programs
   • Martin Tingley (Netflix), Simon Ejdemyr (Netflix), Travis Brooks (Netflix)
4. Making Better Decisions with Continuous Experimentation
   • Gabe Bessler, Nick Doudchenko, Roman Garnett, Hossein Safavi, Howard Zhang

Enjoy a 15-minute coffee and networking break.

Session G: Two-sided and Bipartite Experiments

1. Design-based causal inference in bipartite experiments
   • Yue Fang (CUHK); Sizhu Lu (UC Berkeley); Lei Shi (UC Berkeley); Wenxin Zhang (UC Berkeley)
2. Experimentation on Endogenous Bipartite Graphs
   • Dominic Coey (Meta), Edvard Bakhitov (Meta), Wenshuo Wang (Meta), Okke Schrijvers (Meta)
3. Debiasing Seller-Side Experiments via Multinomial Logit Models in Two-Sided Platforms
   • Chenyu Zhang (BNU), Yuhang Wu (UC, Berkeley), Zeyu Zheng (UC, Berkeley), Nian Si (HKUST)
4. Regression Adjustments for Experimental Designs in Two-Sided Marketplaces
   • Timothy Sudijono (Stanford), Lihua Lei (Stanford), Lorenzo Masoero (Amazon), Suhas Vijaykumar (Amazon), Guido Imbens (Stanford), James McQueen (Amazon)

Session H: eCommerce          

1. The Revenue Impact of Review Solicitation: Evidence from a Large-Scale Experiment
   • Hana Choi (Rochester); Minkyung Kim (CMU); Jinsoul Seo (Rochester)
2. Eliciting Advice Instead of Feedback Improves Developmental Input
   • Hayley Blunden (American), Ariella Kristal (Columbia), Ashley Whillans (Harvard), Michael Yeomans (Imperial)
3. Social Influence in Online Reviews: Evidence from the Steam Store
   • Adam Di Lizia (Warwick)
4. Business Policy Experiments using Fractional Factorial Designs: Consumer Retention on DoorDash
   • Yixin Tang (DoorDash),Yicong Lin (DoorDash),Navdeep S Sahni (Stanford)

Session I: Adaptive Experimentation and Heterogeneous Effects

1. Mathematical Programming for Adaptive Experiments
   • Ethan Che (Columbia), Daniel Jiang (Meta), Hongseok Namkoong (Columbia), Jimmy Wang (Columbia)
2. Using Explore-Exploit to Reduce Experiment Cost for Launch Decisions
   • Yan Wang (Airbnb), Jiayun Dong (Airbnb), Yicheng Li (Airbnb)
3. Exploration Sizing via Model-Predictive Control
   • Ethan Che (Columbia Business School), Hakan Ceylan (Netflix), James McInerney (Netflix), Nathan Kallus (Netflix and Cornell University)
4. Scalable Heterogeneity Detection in Online Experiments
   • Hammaad Adam (MIT), Merlin Heidemanns (Amazon), Doug Hains (Amazon), James McQueen (Amazon)

Enjoy a 15-minute coffee and networking break.

• Session Moderator:
  • David Holtz – Assistant Professor, UC Berkeley Haas School of Business and CODE@MIT Organizer
• Panelists:
  • Rina Chang – Staff Product Designer, Netflix
  • Antara Dave – Product Designer, Microsoft
  • Hang Le – Head of Design, Eppo
  • Joni Rustulka – Product Design, LaunchDarkly

This panel discussion will feature UX designers who work on experimentation-related products.

• David Holtz – Assistant Professor, UC Berkeley Haas School of Business and CODE@MIT Organizer

Enjoy a networking reception featuring the work of our poster presenters on display and poster presenter authors available for questions.

See you tomorrow!

Please take the elevators to the 7th floor.

If you are just arriving to the conference, please check in and get your name badge. Enjoy breakfast and networking with other conference attendees and speakers.

• Dean Eckles – Professor, MIT Sloan and CODE@MIT Organizer

• Avinash (Avi) Collis – Assistant Professor, Carnegie Mellon University
  • The Consumer Welfare Effects of Online Ads: Evidence from a 9-Year Experiment

Research on the causal effects of online advertising on consumer welfare effects is limited due to challenges in running large-scale field experiments and tracking the effects over extended periods. In this study, we analyze a long-running field experiment of online advertising, launched in 2013, in which a random 0.5% of all users are assigned to a group that does not see ads (the “no-ads hold-out group”). We recruit a representative sample of Facebook users in the ads and no-ads groups and estimated their welfare gains from using Facebook using a series of incentive-compatible choice experiments. We find no significant differences in welfare gains from Facebook or time spent on Facebook between the two groups. Our estimates are relatively precisely estimated reflecting our large sample size (53,166 participants). Specifically, the minimum detectable difference in median valuations at standard thresholds is $3.18/month compared to a baseline valuation of $31.95/month for giving up access to Facebook. That is, we can reject the hypothesis that the median disutility from advertising exceeds 10% of the median baseline valuation. Our findings suggest that either the disutility of ads for consumers is relatively small, or that there are offsetting benefits, such as helping consumers find products and services of interest.

• Jessica Hullman – Ginni Rometty Professor of Computer Science, Northwestern University
  • The Rational Agent Benchmark for Data Visualization

Understanding how helpful a visualization is for decision-making is difficult because the observed performance in an evaluation is confounded with aspects of the study design, such as how useful the information that is visualized is for the task. I will discuss how decision-theoretic frameworks that conceive of the performance of a Bayesian rational agent can transform how we design and evaluate visualizations and other decision-support interfaces, such as explanations of model predictions.

Enjoy a 15-minute coffee and networking break.

Session J: Surrogate Models    

1. Using Short-term Outcomes and Past Experiments with Different Treatments to Estimate Long-term Causal Effects
   • Yi Zhang (Harvard), Winston Chou (Netflix), Kosuke Imai (Harvard)
2. Long-Term Causal Inference from Short-Term Experiments in Semiparametric Markov Decision Processes
   • Lars van der Laan (University of Washington); Aurelien Bibaut (Netflix Research)
3. Evaluating and Utilizing Surrogate Outcomes in Covariate-Adjusted Response-Adaptive Designs
   • Wenxin Zhang (Berkeley), Aaron Hudson (Fred Hutch), Maya Petersen (Berkeley), Mark van der Laan (Berkeley)
4. Beyond Correlation: A Bayesian Multilevel Model for Effective Proxy Metric Use in A/B tests
   • Miha Gazvoda (Booking.com)

Session K: Management and AI 

1. Vertical Integration and Consumer Choice: Evidence from a Field Experiment
   • Chiara Farronato (Harvard Business School), Andrey Fradkin (Boston University), Alexander MacKay (University of Virginia)
2. AI-enabled Technology and Gig Workforce: The Role of Experience, Skill Level, and Task Complexity
   • Benjamin Knight (Instacart), Dmitry Mitrofanov (Boston College), Serguei Netessine (The Wharton School)
3. Communicating Uncertainty Can Increase AI Adoption
   • Rozhina Ghanavi (MIT), Mohsen Foroughifar (University of Toronto), Avi Goldfarb (University of Toronto), Ryan Webb (University of Toronto)
4. Facilitating Meetings with LLMs: An Experimental Study of Group Decision Making
   • Mohammed Alsobay (MIT), David M. Rothschild (Microsoft), Jake M. Hofman (Microsoft), Daniel G. Goldstein (Microsoft)

Session L: Heterogeneous Methods Presentations II  

1. Sequential Testing on Repeated Observations: Method and Field Implementation
   • Zhimei Ren (Wharton), Ruohan Zhan (HKUST), Lu Deng (Tencent), Shichao Han (Tencent), Penglei Zhao (Tencent), Yong Wang (Tencent)
2. Enhancing External Validity in Experiments with Ongoing Sampling
   • Chen Wang (HKU), Shichao Han (Tencent), Shan Huang (HKU)
3. Data-Driven Recruitment for Generalizable Experiments
   • Hammaad Adam (MIT), Lars Hulstaert (Johnson & Johnson), Yu Mao (Johnson & Johnson), Hans Verstraete (Johnson & Johnson), Marzyeh Ghassemi (MIT)
4. Transformed-Outcome Regression that Works: Flexible and Precise Treatment Effect Heterogeneity Estimation with Cross-Fitting
   • Duy Pham (UCLA), Adam C Sales (WPI), Kirk Vanacore (UPenn), Johann Gagnon-Bartsch (University of Michigan)

Session M: Media and Politics             

1. Political Polarization and AI: The Effect of Politically Opinionated Conversational Agents on Users’ Political Views
   • Naama Ilany-Tzur (CMU), Matthew Mingus (CMU), and Edison Dao (CMU)
2. Estimating The Effect of Online Voter Registration Advertisements At the Household Level Using Ghost Bidding
   • Katherine Haenschen (Northeastern), Anil Onal (Voter Participation Center), Yi Wu (Voter Participation Center)
3. Investigating the Impact of Media Bias on News Readers
   • Amir Tohidi (University of Pennsylvania), Samar Haider (University of Pennsylvania), Duncan Watts (University of Pennsylvania)
4. Politicized Scientists: Credibility Cost of Political Expression on Twitter
   • Eleonora Alabrese (University of Bath), Francesco Capozza (WZB Berlin), Prashant Garg (Imperial College London)

Enjoy a 15-minute coffee and networking break.

• Session Moderator:
  • Dean Eckles – Professor, MIT Sloan and CODE@MIT Organizer
• Panelists:
  • Malika Korganbekova – Assistant Professor, The University of Chicago Booth School of Business
  • Seth Neel – Assistant Professor, Harvard
  • Eric Benjamin Seufert – Owner, Mobile Dev Memo; General Partner, Heracles Capital

Please note this session is live-streamed and open to the public HERE free of charge. You may follow the link to pre-register or simply join live at the time of the event. The video will be available on-demand post event via the same link.

19. Anytime-Valid Continuous-Time Inference for Inhomogeneous Poisson Point Processes

• Michael Lindon (Netflix), Nathan Kallus (Netflix, Cornell)

20. Unifying DiD Estimators and TWFE Regression via Network Flows for Arbitrary Treatment Patterns and Heterogeneous Effects

• Yudong Chen (University of Wisconsin-Madison), Calvin Tolbert (Cornell), Xumei Xi (Cornell), Christina Lee Yu (Cornell)

21. Bayesian updating and information cascades experiments using ChatGPT instances: a discussion of language models as subjects.

• Fabrice Le Guel (University of Paris-Saclay, France, RITM Research Center)

22. A Simple Solution to Improving Human Supervision of Algorithms: Evidence from Smart Vending

• • Minda Zhao (Gatech), Brian Han (UIUC), Xin Chen (Gatech), Tao Zhu (Fengyi Technology Co., Ltd.)

23. Outlier Impact: Detection by Consequences

• • Daniel Ting (Meta), Ilya Gorbachev (Meta), Sammy Shen (Meta)

24. When Do Adaptive Experiments Help Participants *and* Accelerate Discovery? Insights From Winning XPrize for Experimentation

• • Joseph Jay Williams (University of Toronto), Nina Deliu (Sapienza Università di Roma), Sofia Villar (University of Cambridge), Laura Hawkins (University of Toronto), Jay Olson (University of Toronto), Mohi Reza (University of Toronto), Ilya Musabirov (University of Toronto), Tong Li (University of Toronto)

25. Difference-in-Differences Subset Scan

• • Will Stamey (University of Notre Dame), Sriram Somanchi (University of Notre Dame), Edward McFowland III (Harvard)

26. Combining Observational and Experimental Causal Estimators to Boost Precision

• • Yanping Pei (Worcester Polytechnic Institute), Adam Sales (Worcester Polytechnic Institute)

27. Meta-Encouragement: Estimating incremental value of newly launched business verticals by condensing 100s of business-as-usual encouragement design experiments

• • Nadia Tomova (Booking.com), Nils Skotara (Booking.com), Anirudh Tomer (Booking.com), Wally Toh (Booking.com)

28. When Carryover Meets Spillover: Incrementality Measurement in Multi-channel Marketing

• • Zhengrong Gu (Boston University), Garrett Johnson (Boston University), Ziwei Liao (Wayfair), Jerry Chen (Wayfair)

29. Developer Productivity and Cloud Emissions

• • Alibek Korganbekov (Boston University), Malika Korganbekova (The University of Chicago)

30. Causal Feature Selection Method for Contextual Multi-Armed Bandits

• • Zhenyu Zhao (Roblox), Yexi Jiang (Roblox)

31. Measuring Experimentation Platform Improvements with Relative Efficiency

• • Daniel Ting (Meta), Kenneth Hung (Meta), Artem Vorozhtsov (Meta)

32. AI-Enhanced Task Performance: An Experimental Study on Collaborative Work

• • Tamilla Triantoro (Quinnipiac University), Aleksandra Przegalinska (Harvard, Kozminski University), Leon Ciechanowski (MIT, Kozminski University), Anna Kovbasiuk (Kozminski University), Richard B. Freeman (Harvard University), Konrad Sowa (Kozminski University)

33. Observing the Unobservable: Leveraging Proxy Potential Outcomes from A/B Tests for More Precise Causal Inference

• • Michael Valancius (University of North Carolina), Vinod Bakthavachalam (Netflix), Yinghong Lan (Netflix)

34. WAPTS: A Weighted Allocation Probability Adjusted Thompson Sampling Algorithm for High-Dimensional and Sparse Experiment Settings

• • Haochen Song (University of Toronto); Ilya Musabirov (University of Toronto); Ananya Bhattacharjee (University of Toronto); Audrey Durand (Laval University); Meredith Franklin (University of Toronto); Joseph Jay Williams (University of Toronto)

Session N: Violations of SUTVA II

1. False Positives in A/A Experiments with Correlated Outcomes
   • Ramesh Johari (Stanford), Hannah Li (Columbia), Anushka Murthy (Stanford), Gabriel Weintraub (Stanford)
2. Detecting interference effects with dyadic data
   • Yilin Li (Tencent), Lu Deng (Tencent), Yong Wang (Tencent), Chuan Chen (Tencent)
3. Clustered Switchback Experiments
   • Su Jia (Cornell), Peter Frazier (Cornell), Nathan Kallus  (Cornell), Christina Yu  (Cornell)
4. Price Experimentation and Interference
   • Wassim Dhaouadi (Stanford), Ramesh Johari (Stanford), Orrie B. Page (Stanford), Gabriel Y. Weintraub (Stanford)
5. Off-policy causal estimation under arbitrary network interference
   • Sahil Loomba (MIT), Elchanan Mossel (MIT), Dean Eckles (MIT)

Session O: Personalization and Targeting   

1. Incrementality Representation Learning: Synergizing Past Experiments for Intervention Personalization
   • Ta-Wei Huang (HBS), Eva Ascarza (HBS), Ayelet Israeli (HBS)
2. Deep Learning for Policy Targeting with Continuous Treatment
   • Zhiqi Zhang(WashU),Zhiyu Zeng(WashU),Ruohan Zhan(HKUST),Dennis Zhang(WashU)
3. Improving Targeting Policies Using Transfer Learning
   • Marat Ibragimov (Emory University), Duncan Simester (MIT), Artem Timoshenko (Northwestern University)
4. Policy-Aware Experimentation: Sampling Smart to Improve Targeting Policy
   • Yi-Wen Chen (Columbia Business School), Eva Ascarza (Harvard Business School), Oded Netzer (Columbia Business School)
5. Data-Adaptive Experimentation to Find Contexts with the Most and Least Discrimination
   • Jennah Gosciak (Cornell), Daniel Molitor (Cornell), Ian Lundberg (UCLA)

Session P: Evolving and Unknown Networks

1. Experimentation Under Unknown Network Interference
   • Sadegh Shirani (Stanford Graduate School of Business), Mohsen Bayati (Stanford Graduate School of Business)
2. Treatment Effect Estimation Amidst Dynamic Network Interference in Online Gaming Experiments
   • Yu Zhu (UC Santa Cruz), Zehang Richard Li (UC Santa Cruz), Yang Su (Tencent), Zhenyu Zhao (Tencent)
3. Interference and Network Dynamics: Assessing the Bias in Average Treatment Effect Estimates When Networks Evolve
   • Eaman Jahani (University of Maryland, Smith School of Business), Xiaoyue Niu (Penn State University, Department of Statistics), Peng Ding (University of California, Berkeley, Department of Statistics)
4. A/B testing under Interference with Surrogate Network Information
   • Shiv Shankar (UMass), Ritwik Sinha (Adobe), Madalina Fiterau (UMass)
5. Causal Inference in Social Platforms Under Approximate Interference Network
   • Yiming Jiang (Gatech), He Wang (Gatech), Yong Wang (Tencent), Lu Deng (Tencent)

Session Q: Generative AI and LLMs II

1. Words that work: Using language to generate hypotheses
   • Rafael M. Batista (UChicago-Booth; CAAI), James Ross (UChicago-Booth; CAAI)
2. Does reading the literature improve the ability to predict the effects of interventions? Language models as tools for research synthesis and evaluation
   • Robin Na (MIT), Abdullah Almaatouq (MIT)
3. Generative AI and Distributed Work: Evidence from Open Source Software
   • Manuel Hoffmann (Harvard Business School), Sam Boysel (Harvard Business School), Frank Nagle (Harvard Business School), Sida Peng (Microsoft Corporation), Kevin Xu (GitHub Inc.)
4. Deriving Value from AI Characters in Online Social Community: Evidence Through a Field Experiment
   • Frank Li (UBC), Shiyang Gong (Beijing Normal University), Li Guo (University of International Business and Economics), Yuetao Gao (Xiamen University), Yinliang (Ricky) Tan (China Europe International Business School), Erik Brynjolfsson (Stanford)
5. On In-Silico A/B Testing with LLMs
   • Shiv Shankar (UMass), Ritwik Sinha (Adobe), Madalina Fiterau (UMass)

• Tesary Lin –  Assistant Professor, Boston University
  • Revealing Privacy Preferences: Experimental Insights on Consumer Data Sharing Decisions

Consumer privacy preference has long been an elusive topic for researchers and policymakers alike. A well-documented phenomenon is the “privacy paradox”, referring to the disconnect between stated attitudes and revealed preferences (i.e., actual behaviors) for privacy; additionally, the context-dependent nature of privacy preferences further complicates their measurement. In this talk, I present three experiment studies that tackle these challenges via various experiment design techniques. I demonstrate how to elicit revealed preference for privacy to ensure the managerial and policy relevance of the results, the role of theory in guiding experimental variation design, and how to implement user-side experiments in the field when the nature of the research question precludes industry-facilitated testing. I then discuss the insights these studies reveal about the nature of consumers’ privacy preferences.

• David Choi – Associate Professor, CMU
  • Agnostic Characterization of Interference in Randomized Experiments

Mechanisms for interference–such as transmission of disease, sharing of information, peer influence, or economic competition–often play fundamental roles in our understanding of social behaviors, and their empirical characterization–such as their nature, prevalence, or strength–may be of scientific interest. In this work, we give a general approach for characterizing the prevalence of interference. This approach is applicable to experiments with binary-valued outcomes. It can be used to to explore questions such as Q1. How many units are affected by any treatment (including their own)? Q2. How many units are affected by the treatment of others? of distant others? Q3. How many units are affected by the treatment of others, provided that their own treatment satisfies some condition? For each of these questions, our approach gives conservative point estimates and one-sided confidence intervals, both of which lower bound the true value. Our point estimates are asymptotically equal to inverse propensity weighted (IPW) contrasts, such as comparisons of treated versus control, or comparisons of different levels of indirect exposure. Under reasonable experiment designs, our point estimates will converge to a lower bound on the true value, while the one-sided interval will have coverage probability converging to at least the desired level. These consistency and coverage properties will require no assumptions or restrictions on the nature of the interference, relying only on the known experiment design. As a result, our estimates will remain valid even if they use an observed social network that is only a crude proxy for the actual social mechanisms. We find empirically that the efficiency of our interval estimates (i.e., the interval widths) is competitive with, and often better than, that of the EATE, an assumption-lean treatment effect.

• Dean Eckles – Professor, MIT Sloan and CODE@MIT Organizer

Enjoy a networking reception featuring the work of our poster presenters on display and poster presenter authors available for questions.

Thank you for attending – see you in 2025!

Professor Athey is the Economics of Technology Professor at Stanford Graduate School of Business. She received her bachelor’s degree from Duke University and her PhD from Stanford, and she holds an honorary doctorate from Duke University. She previously taught at the economics departments at MIT, Stanford, and Harvard. She is an elected member of the National Academy of Science and is the recipient of the John Bates Clark Medal, awarded by the American Economics Association to the economist under 40 who has made the greatest contributions to thought and knowledge. Her current research focuses on the economics of digitization, marketplace design, and the intersection of econometrics and machine learning. She has worked on several application areas, including timber auctions, internet search, online advertising, the news media, and the application of digital technology to social impact. As one of the first “tech economists,” she served as consulting chief economist for Microsoft Corporation for six years, and has served on the boards of multiple private and public technology firms. She also served as a long-term advisor to the British Columbia Ministry of Forests, helping architect and implement their auction-based pricing system. She was a founding associate director of the Stanford Institute for Human-Centered Artificial Intelligence, and she is the founding director of the Golub Capital Social Impact Labopen in new window at Stanford GSB. From 2022 to 2024, she took leave from Stanford to serve as Chief Economist at the U.S. Department of Justice Antitrust Division. Professor Athey was the 2023 President of the American Economics Association, where she previously served as Vice President and Elected Member of the Executive Committee.

Daniel Björkegren is an Assistant Professor at Columbia University School of International and Public Affairs (SIPA). He works on digital transformation and applied machine learning, with a focus on developing economies. He is an affiliate of the Bureau for Research and Economic Analysis of Development, and MIT’s Jameel Poverty Action Lab. He holds a Ph.D. in Economics and a Master’s in Public Policy from Harvard University, and a Bachelor degree in Physics from the University of Washington.

Rina Chang leads the product design and user experience strategy for the entire suite of Experimentation Platform products at Netflix, which enable experiment-driven decision making across the company. She is passionate about making the experience as easy and clear as possible to a wide range of users, while balancing statistical rigor and technical feasibility.

David Choi is an Associate Professor in Heinz College, Carneigie Mellon University, with courtesy appointment in the Department of Statistics and Data Science. He obtained his PhD from the Department of Electrical Engineering from Stanford University, and has previously worked as a staff scientist at MIT Lincoln Laboratory, and as a postdoctoral researcher in the School of Engineering at Harvard University, and in the Department of Statistics at UC Berkeley.

Avinash (Avi) Collis is an Assistant Professor at the Heinz College of Information Systems and Public Policy at Carnegie Mellon University. He holds a Ph.D. in Management Science from MIT Sloan School of Management. His research interests include the economics of digitization, focusing on measuring the welfare gains from digital goods. His research has been covered in major media outlets and policy reports, including the New York Times, Wall Street Journal, Washington Post, the Economist, CNN, BBC, Financial Times, Bloomberg, and NPR, and reports by the US White House, Federal Reserve, Senate, and UK treasury. He is a member of the Federal Economic Statistics Advisory Committee (FESAC), which advises the Department of Commerce, BEA, BLS, and U.S. Census.

Antara Dave is a Product Designer with a unique blend of skills, combining her foundation in Data Analytics with her passion for Product design. With over 9 years of total experience, she has made significant contributions in both multi-national corporations like Microsoft and The Home Depot.

Antara’s interdisciplinary background in user experience and data analytics allows her to bridge the gap between Design and Data. She strongly advocates for making design decisions that are data-driven and visually simple.

At Microsoft, Antara plays a pivotal role in shaping one of the world’s largest experimentation platforms – Experimentation Platform. Trusted by premier Microsoft teams such as Bing, and Office, this platform impacts millions of users globally. Antara’s responsibilities span from ideation to creating fully curated interactive designs, ensuring that the product is not only visually appealing but also highly functional.

Dean Eckles is an Associate Professor of Marketing at MIT Sloan. He is affiliate faculty at the Institute for Data, Systems & Society in the Schwarzman College of Computing. Dean is also the ‘Data and Analytics’ Research Group Lead at the MIT Initiative on the Digital Economy.

His substantive research examines people’s interactions with and through communication technologies, especially the ways these technologies mediate, amplify, and direct social influence. This work sometimes requires or benefits from new analytical methods, so Eckles also works on applied statistics, design of field experiments, and causal inference.

Prior to joining MIT, he was a scientist at Facebook, where he worked on many product areas and analytical methods, including News Feed, messaging, advertising, tools for randomized experiments, and survey methods. Eckles previously worked in research at Nokia and Yahoo.

Eckles received his BA in philosophy, a BS and MS in cognitive science, an MS in statistics, and a PhD in communication, all from Stanford University.

David Holtz is an assistant professor in the Management of Organizations (MORS) and Entrepreneurship and Innovation groups at the Haas School of Business. He earned his PhD at the MIT Sloan School of Management, in the Information Technology (IT) group. He also holds an MA in Physics and Astronomy from Johns Hopkins University, and a BA in Physics from Princeton University.

Jessica Hullman is Ginni Rometty Professor of Computer Science and a Fellow at the Institute for Policy Research at Northwestern University. Her work develops interface tools and theoretical frameworks for helping people combine their knowledge with statistical models. Jessica’s work has been awarded best paper awards at top visualization and HCI venues, a Microsoft Faculty award and NSF CAREER, Medium, and Small awards as PI, among others.

Malika Korganbekova is an Assistant Professor at the University of Chicago Booth School of Business. She got her PhD in Quantitative Marketing from Kellogg School of Management, Northwestern University. Before graduate school, Malika worked at the Advertising division at Google.

Malika Korganbekova’s research applies tools from causal inference, empirical industrial organization, and machine learning to study the impact of technology on societal issues, with the focus on privacy, data regulation, and sustainability.

Her dissertation work has received MSI A.G. Clayton Best Dissertation Proposal, ISMS Sheth Foundation Dissertation award, and Shankar-Spiegel Best Dissertation Proposal awards.

Malika holds Master’s degrees in Economics from Penn State and Central European University, and an undergraduate degree in Economics from Lomonosov Moscow State University.

Hang is Head of Design at Eppo, shaping tools that enables impactful experiments at scale. She previously led teams at Dropbox, Earnin, and SAP. She cares about simplifying people’s work lives so that they can spend more time at home, and fostering diverse perspectives to design discourse.

Tesary Lin is the Isabel Anderson Career Development Assistant Professor of Marketing at Boston University. She is also affiliated with the Hariri Institute for Computing and the Technology & Policy Research Initiative. She received her PhD in quantitative marketing from the University of Chicago and worked at Microsoft Research and JD Silicon Valley Lab before joining Boston University.

Tesary’s research focuses on how information and data influence the relationship between firms and consumers. Specific topics include consumer privacy preferences and their impact on firm analytics, the development of analytics tools that adapt to a privacy-first data landscape, and how companies employ choice architecture to influence consumers’ data-sharing decisions. She has received multiple awards and grants for her research, including the John D.C. Little Best Paper Award, the Alessandro di Fiore Best Paper Award, the Sheth Foundation ISMS Doctoral Dissertation Award, the MSI Alden G. Clayton Doctoral Dissertation Award, and the Internet Society Foundation Grant.

Seth Neel is an Assistant Professor in the Department of Technology and Operations Management at Harvard Business School, and Faculty Affiliate in Computer Science at the Harvard School of Engineering and Applied Sciences, and in the Kempner Institute for the Study of A.I. He is Co-PI of the Secure and Fair AI Lab, which studies issues of privacy, reliability, and fairness in generative models, often from a data-centric perspective. Outside of academia, he is a co-founder of Welligence, a global energy data startup with offices in the US, Europe, and Asia.

Sida Peng is a sr. principal economist in the Office of Chief Economist at Microsoft. His research interests include econometrics, industrial organization, machine learning and artificial intelligence. His work has been published in economics, statics and CS journals and conferences, including Biometrika, Marketing Science, Journal of Health Economics, and AISTAT. He received his Ph.D. in Economics from Cornell University in 2017. And received his M.S. in Statistics, B.S. in Mathematics and B.A. in Economics from University of Virginia in 2011.

Joni Rustulka leads the product design team focused on Experimentation, Metrics, and Risk Mitigation at LaunchDarkly. Her work on LaunchDarkly’s Experimentation product has focused on designing a tool for users who need to make decisions about the software they release with feature flags. Joni’s focus when building this product has required a delicate balance: ensuring the platform’s user-friendliness while embedding the sophisticated knowledge needed to accurately interpret uncertainty. Before joining LaunchDarkly in 2021, Joni worked on designing and building developer tools that bring the Internet to life, including Apache Cordova and NodeJS.

Eric Benjamin Seufert is a media strategist, quantitative marketer, author, and investor who spent his career working for transformative consumer technology and media companies — including Skype, Wooga (acquired by Playtika) and Rovio, the developer of Angry Birds — before launching Heracles Capital, an early-stage venture capital fund focused on the mobile ecosystem. Eric is also the author of the book Freemium Economics, published by Elsevier in 2014, and he founded Agamemnon, a start-up that built a mobile marketing analytics platform, which was acquired in 2017. Eric also developed Theseus, an open-source Python library for marketing cohort analysis.

Eric runs Mobile Dev Memo, a mobile advertising and freemium monetization trade blog, and QuantMar, a knowledge-sharing platform for quantitative marketers. Eric’s writing has been featured in The Information, Business Insider, and the Harvard Business Review, and he has been quoted in The Wall Street Journal, The New York Times, The New Yorker, The Sunday Times, and The Financial Times, among others.

Johan Ugander is an Associate Professor at Stanford University in the Department of Management Science & Engineering, within the School of Engineering. His research develops algorithmic and statistical frameworks for analyzing social networks, social systems, and other large-scale social and behavioral data. Prior to joining the Stanford faculty he was a post-doctoral researcher at Microsoft Research Redmond 2014-2015 and held an affiliation with the Facebook Data Science team 2010-2014. He obtained his Ph.D. in Applied Mathematics from Cornell University in 2014. His awards include a NSF CAREER Award, a Young Investigator Award from the Army Research Office (ARO), three Best Paper Awards (2012 ACM WebSci Best Paper, 2013 ACM WSDM Best Student Paper, 2020 AAAI ICWSM Best Paper), and the 2016 Eugene L. Grant Undergraduate Teaching Award from the Department of Management Science & Engineering.