AB Testing Case Study | Google and Spotify Data Scientists | Data Science Interview
Vložit
- čas přidán 28. 06. 2024
- 🚀 Land your dream data job using datainterview.com/.
====== ✅ Details ======
🤔 "How would you design an AB test to decide whether to change the color of the 'Buy Now' button from orange to green on Amazon.com?"
Dan (Ex-Google/PayPal Data Scientist) and Felicia (Spotify Data Scientist) partnered up to create a discussion video and technical design doc on how to approach an AB test problem based on Amazon's shopping experience.
The video is a preview of the entire discussion covering:
1. Problem Statement
2. Background (User Journey, User Target)
3. Hypothesis Statement (Success Metric, Guardrail Metrics)
4. Experiment Design (MDE, Power, Alpha, Sample Size)
5. Running an Experiment (Ramp Up, Validation Checks)
6. Launch Decision (Decision Tree, Post-Launch)
If you want to see the entire discussion and get access to AB testing courses, case course, product SQL course, Slack group and much more, make sure to check out datainterview.com/
👍 Feel free to subscribe, like and share!
====== ⏱️ Timestamps ======
0:00 Intro
06:11 Problem Statement
08:18 Background
14:15 Hypothesis Statement
====== 📚 Other Useful Contents ======
1. Principles and Frameworks of Product Metrics | CZcams Case Study
/ principles-and-framewo...
2. How to Crack the Data Scientist Case Interview
/ principles-and-framewo...
3. How to Crack the Amazon Data Scientist Interview
/ crack-the-data-scienti...
====== Connect ======
📗 LinkedIn (Felicia) - / feliciarutberg
📗 LinkedIn (Dan) - / danleedata
📘 Medium (DataInterview) - / datainterview - Věda a technologie
Hi, Please share the remaining part
4. Experiment Design (MDE, Power, Alpha, Sample Size)
5. Running an Experiment (Ramp Up, Validation Checks)
6. Launch Decision (Decision Tree, Post-Launch)
Playback Speed 1.5x recommended
Not really sure why we need the "first session only" constraint? If you just sum/average per user you can also get results "promptly", right?
I agree, the first session only would also be impacted by the novelty effect.
Hey Dan, I really love your courses on CZcams and purchase the monthly subscription a week ago. Now I could not login due to Google's prevention: "Attackers tried to steal your information"
Incredible! Professional yappers, very useful for practicing corporate English✍️😁
Question on metric tested in the hypothesis test - avg. # of clicks per user. Why wouldn't it be avg revenue per user per day? I believe the user could click the "buy now" button and still exit, or they may buy but spend less money with the new button,etc....
You mentioned that CTR is not a good metric which is understandable, but how about CTP? In CTP, we are looking at unique page visits and unique clicks, therefore, if a user has multiple sessions and multiple visits, they will all be counted as one.
Use delta method to account for the repeated measures
What's the purpose of having a guest speaker if you don't let her talk?
Well they made the presentation together and she had quite a bit of engagement imo. He asked her to talk about the hypothesis statement and they had some solid dialogue... I guess the idea was that he would guide the presentation passively, and she'd give great insights/alternate. I learned quite a bit
thats your whole take away from the video ?
Gold
On 24:30 about not using CTR due to violation of iid, I think you’re missing some important info, would love your feedback:
1. If you’re aggregating CTR to per user across session, your unit are users and still iid, assuming users are independent
2. Even if measuring at session level, you can still use delta method to estimate variance where otherwise variance would be underestimated due to sessions are not iid
A user would be way more likely to click a 'buy now' button for a cheap, insignificant item. I think this definitely needs to be controlled for in the experiment.
*God bless me, please. I wish I could get my dream job.*
So wordy….. get to the points
some people love the sound of their own voice
Sorry, I think you don't know statistics. T-test is a parametric test.