Nick Jones, Sam Barrows: Uber's Synthetic Control | PyData Amsterdam 2019

Full Title: Uber's Synthetic Control Experimentation Framework When A/B Tests are Not Possible
When doing experimentation at Uber, we sometimes have to expose an entire market to treatment, because network effects between riders and drivers or legal/regulatory concerns make an A/B test infeasible. In these cases we can construct a synthetic control out of a series of potential control cities to still do causal inference, using a Python package developed at Uber. In our presentation, we discuss the motivation and use cases for this approach in our marketplace and product teams, the theory behind this approach, its implementation in Python at a high level, and a comparison of this method with other alternatives when an A/B test is not possible, such as diff-in-diff or Google's CausalImpact - synthetic control using Bayesian Structural Time Series. Many organizations face similar causal inference challenges, for example, in estimating the effect of media campaigns that have to be rolled out at the city or regional level. A correct application of synthetic control can allow for rigorous decisions based on testing, even in these situations.
www.pydata.org
PyData is an educational program of NumFOCUS, a 501(c)3 non-profit organization in the United States. PyData provides a forum for the international community of users and developers of data analysis tools to share ideas and learn from each other. The global PyData network promotes discussion of best practices, new approaches, and emerging technologies for data management, processing, analytics, and visualization. PyData communities approach data science using many languages, including (but not limited to) Python, Julia, and R.
PyData conferences aim to be accessible and community-driven, with novice to advanced level presentations. PyData tutorials and talks bring attendees the latest project features along with cutting-edge use cases.
00:00 Welcome!
0:05 - Introduction
0:36 - What to do when A/B testing is not possible
2:00 - Motivating example: Cash trips at Uber
3:47 - Experiment: Tell drivers if a trip is cash
5:07 - Why an A/B Test doesn't work here: Spillover effects
6:53 - What to do instead of A/B: Look at before and after
7:40 - What to do instead of A/B: Switchbacks
8:46 - What to do instead of A/B: Difference-in-Differences
10:21 - A/B Alternative: Synthetic Control
11:47 - How do we form our synthetics?
13:42 - How do we estimate treatment effect?
14:57 - Synthetic control performance
15:32 - Synthetic control implementation at Uber scale
16:25 - Other use cases for Synthetic Control at Uber: Uber POOL
18:17 - Other use cases for Synthetic Control at Uber: Surge pricing
19:04 - Limitations of Synthetic Control
22:06 - Key takeaways
S/o to github.com/fsammarc for the video timestamps!
Want to help add timestamps to our CZcams videos to help with discoverability? Find out more here: github.com/numfocus/CZcamsVi...

[@romeoanzueto6014]

Op

[@romeoanzueto6014]

06p