Netflix Product Designer | Navin Iyengar | Design Like a Scientist
Vložit
- čas přidán 27. 08. 2024
- Product Designer at Netflix, Navin Iyengar reveals how Netflix applies experimental thinking to design using A/B testing and tells us how to design like a scientist at Awwwards Conference San Francisco.
Subscribe now for the latest talks and interviews from digital design industry leaders: www.awwwards.c...
#usertesting #abtesting #ux #design
Experience our conferences! Get out from behind your screen, learn from industry leaders and meet cool designers from all over the world - get your ticket to awwwards New York bit.ly/awwwards...
For the latest web design trends - follow us:
Blog: awwwards.com/
Twitter: / awwwards
Instagram: / awwwards
Facebook: / awwwards
They found that 46% of people would like to see the catalog before signing up. Their hypothesis that showing the catalog would lead to more sign-ups is quite biased. Based on this finding, it should be something like "showing the catalog upfront will lead people to making a more informed decision of whether to sign-up or not". But of course Netflix is more biased towards formulating hypothesis which tend to result in measuring experiment success by looking only at business metrics and ignoring user needs.
They should have then run surveys for each variant, to find out which variant helped people to make a better decision whether or not to sign up for Netflix, and tried to correlate that with behavioral data. But apparently they ignored this and declared the winning variant solely on what benefits the business goals (more sign-ups). This is not what UX Design is about - you should always work hard towards finding a balance between solving user needs AND business goals. Also, not every experiment should have a clear winner. A/B testing is more about learning than it is a tool to iterate on the product.
I'm also curious to see what the retention rate for each variant was. It's one thing to increase sign-ups, but if people are signing up just to see the catalog and then churning because they don't find value in the content, did you really even help the business goals? From what was presented, it's not clear what other metrics were considered for the success of the experiment.
100% agree
Albert Einstein.....Is that you?
You have valid points. However, the guy said showing them catalog might be time-wasting for users since they are not gonna get access to the content unless they sign up anyway. Imagine scrolling through catalogs for couple of minutes to find what movie suits your mood, then end up discovering you must sign up and make a payment before you watch that movie? Their catalogs then was a catastrophe, but I think it's better now in the sense that a new user will be informed of a paid subscription before moving forward.
I really like your point here! I had to go back and forward for a couple times during his talking about this point because I couldn’t make sense out of what he said. Like… why people don’t sign up? Because they have gone through all the content so this is equal to failure. For business, yes but if you were one of the users, No one would come to you and say you failed the experiment because you didn’t sign up for it. UX is fun!
I get your point.. but whatever they are doing is going to affect their business decision and thus financials.. that's why I don't think they would have incentive to botch the process or be biased towards one thing or other.. at least knowingly
Wow! Simple and mega-informative!
Really enjoyed the message : Act based on tested results, not assumptions
Definitely bookmarked for sharing and future viewing.
Users had indicated that they'd like to know what content they can get before they decide whether or not to get the subscription. So the measure of success here should be that it leads to better decisions, not more subscriptions.
So the users were not wrong to mention that in the survey.
It is important to differentiate that "Improving user experience in decision making" is different from "maximizing subscription rate" are 2 different hypothesis.
Sathishkumar Jagadeesan 🤔but we would also like to know the correlation between the two. Does improving user experience in decision making ultimately lead to more subs?
But Taking the account of user behaviour with a/b testing, the metrics for the business need to change which is why the result is measured in Business terms.
good point also he said " people choose to pay us" as if to suggest that netflix is a free platform & people "Choose" to pay like supporting a youtube streamer or something but you have to pay to acess the content
@@pnutdraws I bet if Netflix sent a reminder every month asking if you'd like to renew they'd see a major drop-off rate. People don't "choose to pay" monthly, they don't notice how they pay and at the same time that's why subscriptions are convenient.
Great talk! 💯 I'm sure has an amazing data science team, but in a lot of companies, I think A/B testing causes teams to focus on what's easy to measure, and that doesn't necessarily correlates with what the person using the product is trying to achieve
That's a great point and thanks for raising awareness. I agree with this talk and your comment is a key point to keep in mind too when developing products.
HOT DAMN! this is one of the best UX - Product Design videos out there, including paid content platforms. Thanks for this internet, Navin, and awwwards
I believe there are some natural explanations to why not showing specific content worked better.
There are two kinds of streaming users: 1) those who are looking for something entertaining but don't have any specific in mind, and 2) those who know which specific movie/series they want to watch.
1) For those who visit Netflix without a specific movie/series in mind and just want to explore if there's anything worth watching there, it's obvious that forcing them to sign up in order to do that, will lead to more sign ups.
2) Human beings judge things by their subjective perception. So if they're allowed to browse all the specific content and few of the titles are interesting to them, they will deem the value of the service as lower (-> decreasing their probability to sign up). On the other hand, by keeping it abstract then each person can imagine that there will be more content to their liking. (in addition to the one specific series/movie they visited Netflix for in the first place).
This was super simple to understand and a great example to inspire UX Designer like me! thank you! for this content!
Wow really understood alot about how designers think. By going through so many iterations
Brilliant talk. I heard before AB testing divides the UX design world, I'm new and so far, from books and my studies, I gathered quanL and quanT research unbiased analysis (as much as is humanly possible ) are key to any successful product design specification. But this talk has gave me insight into how research heavily supports defining a hypothesis for subsequent testing. Thanks for sharing
wow.. this is kinda my first semster in Psychology plus the really basics of UX design ..
This is completely misleading.
Nerflix' business model is getting as much money as possible and to assure the longevity of the income stream. This is how their app is designed.
They want you to keep watching any content they can keep you watching.
Example : you talk with your significant other about watching a movie for date night. The conversation prior to using the app is about seeing "something we haven't seen before" and then you both start thinking about what you have seen recently mixed with a discussion about recent popular movies that "should be on Netflix by now".
You get to Netflix and you can not answer those questions using the app.
It does not allow you to find recently released movies sorted/filtered by popularity in the box office and movies you already have watched on their platform.
The user experience categories, favorites type lists or searches you have the option of searching don't let you explore your desires according to your values.
Instead you get results that frustrate you to the point you pick something one of you probably already watched.
Netflix just got you to effectively pay them for another two hours this month. Combine that with the rest of the time you use their platform this month, and you have a reason to keep the service because you associate all that time with value, never really being delighted by being able to find what you want when you want it, but instead see a lot of "good enough" content.
Does Walmart show the latest box office hits mixed in with everything else alphabetically?
THAT'S why they need so much experimentation WITH a substandard UI/UX - it's about quantifiable cashflow forecasting.
I could relate to your statement as I was going through it. Fantastically written!
Don't use it then if it's so terrible. I actually find Netflix after four years a pretty enjoyable service to use. We have also Amazon prime TV and in comparison the user interface is clunky and feel most times fighting to get the thing doing what I need it to do. Like continue watching never works and starts any series from the beginning when I'm already half way through a season. I honestly don't understand how they can get away with such disregard for the user experience. Guess Amazon is now too big to give a damn. Only reason we still have our Amazon membership is because my partner always forgets to cancel the subscription.
@@VirgifusI agree that, as consumers, we all have the option to NOT use a product we don't feel are worth the money we spend on it.
As a software engineer, and business person, I find that getting honest feedback from users is a much better means for improving the user experience.
There's a sales phrase that comes to mind, about pleasing or pissing off customers. When you make one happy, you've kept a user, and when you piss one off, you loose ten users.
It is almost impossible to infer anything meaningful from user attrition, except that something is not right.
Your suggestion for just going away doesn't help Netflix improve anything, and certainly won't improve existing user's experiences.
like steve jobs said, people dont know what they really want
Henry Ford said that! (Sort of) he said, “ if I gave people what they wanted, I’ve given them faster, horses”. Steve Jobs reprised this as part of his strategy and direction at Apple, of course
please stop autoplaying videos with sound dear lord have mercy
amen to that
Wait for the iteration of their scientific approach.
Bryan Guenther you can turn it off now
@@desharrsaddler5305 Really? How?
Chile, this had me laughing .
Amazing talk.This is inspiring and informative, Thank you!
Galileo never did this expperiment, because observations would have been insufficient and dangerous. What he did is to throw the same mass through molasses, water and air to extrapolate how it'll behave in a theoretical empty space. Then he made a thought experiment that concluded masses fall at the same velocity, otherwise a string attached to a heavy and a light mass would cause a parachute effect while the system would fall faster as it's more massive than splut masses, therefore it'd be nonsensical to consider masses fall differently.
Awesome, great easy to follow advice.
So much testing and thinking while I never know the movie library - it seems random, the search works in misterious ways, often My List doesn't save what I want to save, Recommendations are just insane - I never watch Bollywood movies yet I get them like I'm a fan (?), the movies I watched are constantly in front everywhere instead of being at the last of any List - seriously why do I want to keep rewatching the same movie over and over.... and the list goes on. But really helpful the stuff about Galilei, really smart and relevant!
Oh can you please Auto mute the previews/Trailers on atv. Its so annoying!!! I think this is a Good example where u actually have to Listen to your Users and Not ab Test.
thank you all very much
This is a great video to understand their practical research methodologies and how to synthesize and use the results to understand users. Also very easy to understand the concept of A/B testing. Still lots of designers misuse what a/b testing means.
I don't have Netflix, but excellent talk! Keep rocking! 🙂👍
Excellent talk
thank you for sharing your knowledge, Navin!!! :)
Amazing! Thank you!
Thanks for Sharing. Learnt some new approaches to my product.
Very WELL explained buddy. It is brilliant presentation ❤
Yes I watch Netflix on my Apple TV most of the time and the experience is horrible since day one
So many things I hate about its UI/UX on tv
The best
love this, so informative.
This is a very basic introduction to what product designers at most companies do every day. I hope they teach this to new designers, as what's shared in this video is useful to know -- but NOT at all a new concept. It's been quite a standard approach for 2 decades now
So basically it's more about getting feedback to improve your product.
Netflix doesn’t run on my s9 plus. I tried all the possible solution. Still doesn’t work. And I love it.. ✌
Wow, this piece of information is really handy. Thank you :-)
Awesome
This was really helpful! Thank you.
dope!!
Lets designers be designer and scientists be scientists. I wouldn't want either to work like anyone else.
Doing A/B test doesn't make a designer scientist.
Amazing
Is it by design that movies don’t have trailers while TV shows do?
How about science like a designer?
Awesome!
how can i find subtitle
Please
Turkey and Russian language
Navin's null hypothesis was "A/B testing is the best testing for experimentation". He should have designed a between subject comparison using T test to prove his null hypothesis. After he skipped the evidence by just making this statement, now I don't feel like watching the rest of the video.
This guy's understanding of p values would make good stats teachers upset.
Dude doesn't it bother you that large number of your users never find the titles they are looking for...? I wouldn't try to sell that as an achievement if I were you :-) Just today I tried to watch that movie Netflix ui promoted the day before, I never managed to find it on obvious reasons. You guys really make me laugh!!! (...And curse sometimes)
Wow what a revelation. Great video. Can someone tell me which font's used in that slideshow? It looked like Helvetica Now, but I may be wrong
DM sans and Circular are similar looking.
Sorry, this isn't science. And it isn't design. This is user experience.
Scientific user design methodology
@@jimihendrixx11 say what?
good one
Lol his summary of philosophy at the beginning is ludicrous.
What I don't get is how all these amazing designers thought that Auto preview was a good idea? did you guys tested it? Please, just do an a/b test with and without the auto preview and let's see which one is better.
Why isn't Auto preview a good idea? Just very curious.
@@dk8770 it's an invasive method, it's playing something I didn't ask for, if I'm interested in a movie i tap on it to see more detail, it should give me the option to play the trailer not shove it up my face
@@alexandrustefanica I think what happens here is that Netflix as a company, has some experience decisions that look like Apple's. We know what's best for you. But Apple can get away with for ex. not having an iPad calculator till this day or a weather app.
Netflix, on the other hand, does not have enough breathing room to execute that (they don't produce as much features as Apple does).
The idea behind autoplay mode is to show you the content as soon as they possibly can. (auto-next episode timer is like 5 seconds, not really enough to decide whether I want to binge watch another episode or leave - they definitely do not want you to leave). Another thing is maybe that they miscalculated input methods, it's not the same if you hover over something on pc and it auto-plays the trailer, or you are watching on a tv where hover option behaves differently.
This reminds me of Evolution
Anyone that knows what A/B testing is can skip this one.
This sounds great on the surface. But, optimizing for subscription? Just make it hard to unsubscribe! Hello, UX dark patterns.
receipt of information assurance: left at4:00
Can we end this “edit out the static” editing?
I can't stand the user experience of the Netflix app. So should I even listen?
luc, its your therapy.
Thanks for posting this! Just to note: the close up camera shot moves too much, making me dizzy.
nice shoulder bro
*s
Interested to know how do they evaluate big experiments like high level hypothesis on Personalization, It is difficult to analyse why that experiment is better.
Netflix recos are useless. I constantly see exact same movies recommended. Thrre is very less customization. And what about search.. Netflix searching sucks. Prime video is no better. Wonder why such a critical feature has been constantly overlooked.
audio could be better
It's like I always say: Never believe in your users. They're lying to you.
You know what, make content people love. Your Hindu-phoic content is the worst. You might have some number of customers who appreciate Hinduphobic content. The truth is it doesn't scale and you will never reach the hearts of India by making anti-India content. This is the main reason I never bought Netflix subscription even though I can easily pay. I can't pay for something that harms me and my people.
"Go to India and find out their unmet needs"=> Their aspiration to self-loathe. So lets make stories where cultural Hindus are presented as biggest villains on the planet. Even when Laskar Taiba attacks India, show that Hindus masterminded attack on themselves. Netflix really read our people well.
EDIT: Added context
Was his audience in this keynote 5 year olds?
13:03 i don't get it, why would you have this hypothesis?
Somebody design Netflix?
Bill Nye isn't even a scientist lol
starts off interesting and about customer experience, then ends on a bitter note on how to better sell a product people might actually not even want in the first place. "if we didn't show what we got, more people sign up" is less about customer experience but more about sales tactics..
kindof a bad example for an otherwise interesting talk.
Scratching ass haters comments down there.
His fundamental understanding of science is questionable. There is a difference between physical science and social science. You can't compare experiments with stones and people in the same breath.
His insta Id@
Hate people that talk in a corporate way
This isn’t product design it’s web design. Why is ui now labelled as product design....
It’s not a ui bro
@@tekiero yes it is
he literally talks about the science of the user interface?
He is product designer
Nothing new
Omg worst talk waste of time, no he really works with netflix ,such a team
People dont tell the truth? Branding gurus have been saying this for ages. Cool guy - not very valuable information.
OK sir
Uuughhhh what a COMPLETE WASTE of time. I had to skip many parts and basically watch 1minute.
some wrong information but ok :)
Please don't just say "some wrong information"; would be really great if you could explain your statement here. Would really love to know some new things always.
Netflix sucks. Hate the recommendations. Super irritating content
Netflix promotes sexuality, so I guess what's the benefits it is giving too our society
sex sells. It's a business which literally exists to make money so yes
Sex is not bad. You exist because of it. I think it's your religious brainwashing thats making you think like this.
@@nandanachar5597 nope, I think you don't have knowledge about my religion.
Sex is something which be private, if you don't think so then animals are there who follows your logic
@@mdrahbarahmedkhan no one cares about your religion or any of the 5000 active religions in the world when designing a product. They only care about the product , not some shitty ideas
@@techindia3602 no one remember people like you who are so naive to distinguish the right from wrong.
No one cares your bullshit
I like taking advise for people wearing business attire as oppose to some sweater or whatever that cliche atire you see in shows like silicon valley
So cliche !! I'm done listening ..!!