How NETFLIX onboards new content: Video Processing at scale 🎥
Vložit
- čas přidán 31. 05. 2024
- Everyday, #Netflix handles billions of requests regarding movies, trailers and other video content. Delivering at such a large scale needs an #engineering marvel. This #video will talk about how Netflix is able to onboard new video content onto their platform. We go from video chunking to collating 4 second shots into scenes.
Amazon S3 is used to store the video chunks. Netflix also provides open connect servers to internet service providers, which acts like a cache of movies. Most requests to Netflix can be served by this cache, and the remaining are sent over the network. This reduces the bandwidth and time required for Netflix to operate at scale. Synergy at it's finest.
00:00 Problem Description
00:32 Video formats and resolutions
03:18 Chunk processing
05:52 Storage
06:19 OpenConnect for video caching
10:13 Summary
Looking to ace your next interview? Try this System Design video course! 🔥
interviewready.io
With video lectures, architecture diagrams, capacity planning, API contracts and evaluation tests. It's a complete package.
References:
/ high-quality-video-enc...
/ optimized-shot-based-e...
/ video
/ stream
/ keystone-real-time-str...
media.netflix.com/en/company-...
System Design Playlist: • System Design for Begi...
You can follow me on:
LinkedIn: / gaurav-sen-56b6a941
Instagram: / applepie404
Quora: www.quora.com/profile/Gaurav-...
Twitter: / gkcs_
External content used:
Fast and Furious 7 (Car chase scene)
Boone Loves Video: • 🌎Create 3D PATH ANIMAT...
Sacred Games (Available on Netflix)
Johnny English Reborn (What's it called?)
Netflix Logo
I believe there is a variation to Gaurav's understanding on open-connect caching mechanism. A new uploaded video DOES NOT goes directly to the caching server, instead they are resolved to a 'a small Netflix DB replica" in that region, which in-term populates the caching layer for new videos or most frequently accessed videos. The distinction is important because otherwise the design is assumed to directly populate OC CDNs, which is not the case. So for example, if a viewer in Australia wants to stream a non-cached US content:
Instead of directing all the Netflix traffic through under-sea cables, Netflix copies the video file (transcoded) from US-based storage to a storage location in Australia once during off-peak hours. Once the video has reached the continent, it’s copied to all the Open Connect servers present in the ISP networks.
This is not to point out a flaw in the video, just a clarification.
Thank you for this amazing clarification 😁
@@gkcs I am glad. Keep making fun content😄
Content apart , the new animations that you've added are really nice. Extra efforts are appreciated!
Keep em coming !
Wow, excellent video. Love your unique style of breaking down the topics in chunks and explaining them neatly. Keep them coming.
Man, you know how to teach with fun! I must say presentation and video editing is very well done in your videos.
Blown by the simplicity you explained such a complex topic!
holy shit! Just fall for these videos; Thank you from bottom of my heart! Please make more video on this topic!
What a beast! Loved your reference to Amazon S3 out of nowhere! 😎✔️
The material you have in this playlist is gold! I did not expect to learn so much from it. Thanks a lot!
You're welcome!
Very helpful and simple for anyone who is preparing for software design interview. Your way of explaining is interesting and funny 😊
Bro - Thanks for the great video! Learned a lot. Just subscribed. Keep the great content coming!
Keywords = CDN , predictive content delivery , content pre-fetch and so on...
Yes Content Delivery Network plays a really important role if the geographic location of physical server is far and also I think what these OTTs would be doing is what we are getting in recommendation might be based on geo location rather than your watch history , this way if one guy from ur location has seen some movie it gets cached to a CDN and then when u request for the same content , since it comes in ur recommendation its extremely fast
Wow this is sweet! Super smart solution to the issues of video streaming, will be interesting to see where it is improved upon in the future too!
Awesome man! Simple and to the point. The best kind of online content.
1:23 AWESOME GAURAV, perfect example of what you are explaining, Huge respect bro
Whoa!!! Thank you so much for making such unique and great content videos ;)
Hi Gaurav, thanks for the video. Impeccable. Keep doing the great work!!
You have the most unique content on CZcams, man. No bs, pure knowledge. Keep going.
Thanks!
Another informative video from you. Nice explanation. Thanks. Just wanted to add something on the caching part. Many websites take help of CDN providers like Akamai to do the caching on behalf of them. The CDN providers have the required infrastructure across the globe wherein they have placed their caching servers in most of the countries.
That's interesting, thanks!
Showing Gupt in local popular movie would make Booby Deol really Proud. Savage AF Gaurav! :D
I'm very impressed! Great explanation! I just subscribed! Keep going!
As i have some experience in this area, i would like to say that you delivered crisp content on the same. Good job!
Thank you!
Moral : Steal that Open Connect Box
Have been working in this arena for a while now, you got everything correct man, other stuff you post usually goes over my head because I haven't dabbled in a lot of those things but for once, felt nice to already know what you were gonna say. Haha.
Thanks! Your replies on the comments are excellent btw :D
It's amazing to see how Guarav improves the quality of his content, and I can tell you, guys, as a newbie tech CZcamsr, it's a big deal. Keep it up, bro! ;-)
Thanks Victor!
Gaurav, I can't tell how interesting this video was ! Thanks a lot ! I totally enjoyed learning this ! Keep making more such videos ! :)
Thanks!
Love the energy. Your videos are great.
To be honest learned something useful today, and willing to learn from you more like this kinda of topic!
Thank You for sharing. Very nice details and very good description
Thanks for this wonderful insight on the engineering side of Netflix, looking forward to more system design videos.
Thanks!
Please do some lectures on docker internals like Cgroups, namespaces, union capable file systems
Champion presentation. Just wanted to see a graphical representation of the adaptive bit rate ladder through Netflix's eyes.
I just started watching your videos. Your content is Amazing.
Hey Gaurav,
I was just wondering how the Open Connect or CZcams Red boxes are any different from the long existing concept of CDN (Content Delivery Network). The only change is, being big companies they have the money and resources to create their own personal CDNs and brand them which is kind of building your own house instead of renting one. Is there something I am missing here ?
They are very similar, yes.
I was having the same question as how these Oppen connect boxes are different from CDNs.
Awesome bro ..!! Simple and very informative.. loved it .. keep it up 👍
You’re awesome. Thanks 😁😍 this was so helpful
great efforts thanks a lot ! i remember my childhood days after seeing the GUPT poster! :-)
That was a nice one !
Thanks for the video.
Can you explain some of the advanced concepts of video streaming too? For eg., Bandersnatch? I wonder how Netflix is providing conditional streaming of video chunks based on user selection.
that's pretty interesting concept... very well explained 👏👏.....
ISP cache is really interesting, wondering how they have handled security on top of it across multiple geography??
Don't be surprised if you hit 500K subscribers by the end of 2019. Your content quality is skyrocketing.
Thank you!
@@talesara74 It's system design and it's not for everyone to understand, what did you expect? People mainly use this channel for technical system design interviews.
You said that from your perspective. But the sad reality is People really don't like good contents😅😂. They can watch a 3hrs movie, but couldn't watch this type videos for 10mins😑
Gaurav I love your content and soon I guess will be joining your InterviewReady platform...Please offer me some special discount bhai 😍
It's almost 2023 and he still has 488k 🥲
Short, simple and full of interesting concepts. Thanks!
Thank you!
The content of this video was very dense. The breather at 5:29 was very much wanted :-). You have very good audience engaging skills. Good Luck !!
Thank you Gaurav for the content . Hope to see many more useful content .
Very detailed. Great work!
Your videos are pure delight to watch !
Thank you!
I was waiting for this video, awesome
Very nice explained and made it interesting. Will be waiting for more good system design question.
That bit at the beginning was hilarious 😂
Thanks for your describtions
Thank you for your reading on this topic...this is enlightening (at least for me)
Excellent video, well explained!
Nice and depth knowledge bro!
must say u made it really Interesting with editing :P and great content!
Thank you!
Great, Nicely explained !!!
Nice explaination, Thank you so much.
This was a very informative video! Awesome job.
Thank you for the explanation
Superb explanation bro
Great video ! Thanks for sharing
real effort here, good job mate
Good job explaining!!
Great presentation skills and content🔥🔥🔥
Great breakdown!
Thanks Gaurav. I am not an engineer but always wanted to get some resource to learn the concepts. I would love to see more of your videos and on advance concepts.
Thanks Ankit!
As always one another brilliant content
Paaji ek dum kamaal ka video hai ...even understoodable for non cs
Fantastic explanation
Brief and precise.. Nice.
Gaurav it would b great if you can go deeper into Netflix design and technology choices done by Netflix team.
Overall a nice video 👍
Great content always!
Happy Teachers Day Sir! U taught me a lot! I really appreciate that!
Thank you!
This is great work Gaurav. Thank you for this!
😁
I just love your system design videos.
Thanks 😁
Great video. Something I would like to point out.
AVI MP4 - these are video containers. H.264 is a codec. Not a big thing to worry about in the grand scheme of a system design overview video like this but just wanted to point out
Thanks!
That's a good point 😁
You're a gem you should have a teaching startup like byjus
So well explained 🤟
That's just awesome
Keep it up
Very informative!
Very well explained, thank you, Gaurav!
Thanks Veronica 😁
Really helpful man!
Video -> Fragmentation -> ABR -> DASH + HLS -> Packaging -> Cache On CDN -> Play
Good explanation!
Bro u are great am a 14+ software engineer really appreciate ur content
Nice work.
Hi Gaurav,
You mentioned that instead of segmenting based on timestamp, Netflix segments based on scenes that are a collection of shots. Do you have any understanding of how they divide each scene? Taking your car scene as an example: how does Netflix know from which point to which point (or to be more abstract: how many shots need to be collated) to create that scene and to download it in a single go?
It's called Shot Transition Detection. Here is a starting point for researching it: en.wikipedia.org/wiki/Shot_transition_detection
Great content.. keep up the good work..
this isp cache thing is just simply amazing,
Awesome Video. Very good explanation. Video content caching is not an easy job. The video is really good.
Software Architecture is the most important thing today. No matter how much you you improve on ML algos or buy hardware etc... It is the archtecture that defines quality and reliability. How you connect components together using a good deisgn tactic and make each component work flawlessly by the deisgn pattern matters the most. It is archtecture everywhere. The fact that you make videos on it is really a good thing. Developers today must understand the imporatnce of it. It is not just writing code but many other things too.You are doing a great job by making such videos.
Bhai saab.. Creative content...
OMG!!!! the opening😂😂
Thanks for the great info
I wonder why I like Netflix more than any other OTT
Thank you!
Great Sir,
Thanks!!
Awesome!!
Gupt yrr, haha Very wonderfully explained Thanks for that
Thats a really great explanation. Thanks
Thank you!
Not gonna lie, you got me in first second, sorting in O(n) lol
Loved this video please create one on Google sheets also.