System Design: Live Streaming Events like ESPN and Hotstar
Vložit
- čas přidán 2. 07. 2024
- From the system design course: get.interviewready.io/?source...
Use the "HELLOWORLD" coupon to get 20% off.
This lesson talks about workflows that can handle video uploading and transformation in real-time. Such systems are necessary for live streaming systems like Disney Hotstar and ESPN.
You can watch the remaining videos from this design on the InterviewReady website.
You can follow me on:
Github: github.com/InterviewReady/sys...
LinkedIn: / interview-ready
Instagram: / interviewready_
Twitter: / gkcs_
#SystemDesign #InterviewReady #LiveStreaming
can't thank you enough for your videos and the amount of information the contains.
recently found your content. very well versed! keep up the good work!
Gaurav mentioned backpressure without actually mentioning backpressure. Noice!😎
you are so charming while explaining smtg. im not on the system design stuffs but watching your videos and started to learn smtg about. thanks a lot:))))
Thank you 😁
This video is above me, but I enjoy these technical videos just to get an idea on what's going on via the backend.
Hey Gaurav, great video again! Thank you. Do you think you'll ever do a video on System Integration? How do you get System A to talk to System B? Or how would you migrate a database from one system into another? These are common occurrences in mergers and acquisitions. I'd love to hear your thoughts on it. Thank you for the consideration.
Love your content! Keep it up. If possible make videos in LLD
Thank you!
for each stream or match, do we have a different queue ? I guess by adding queue we are introducing the delay .
We had stream outage in ipl finals 2022! Well explained! 👏 processing at transformer service will make delay isn't? So we see a delay as opposed to satellite ?
Gaurav has some ideas and maybe good for practice for like interviews but in real world there are a lot of complexities to deliver such service in a scalable matter.
also Guarav rushed to publish this I think, he could have added some points on CDN, error handling, different codecs, latency, and others issues related to live streaming, anyway go take a look at people working in streaming company tech presentations and you learn a lot and also try to implement it yourselve.
This is an old video but I must say that it's a terrible idea to include any sort of file storage or queues in the live publishing path. What Gaurav's done here is taken a simplistic "design youtube video upload" architecture and labeled it as live streaming. Generally, with live video, you'd want to minimize end to end latency, and you'd want to re-stream transcoded frames to multiple replicas. File storage, queues, and async transcoder tasks are all antithetical to minimizing latency. As for protocol, UDP would be a great choice streaming the video to users (no need to maintain thousands of TCP connections). You can deal with out of order packets and packet drop in a live setting. When learning system design from these videos, we should always be skeptical of what's being said. No one person is an expert in all domains, or has ideas that are 100% good.
I need help with streaming...can we connect
greatly explained
Great Content
How to scale queue and configure partition in this case?
We cannot predetermine how much live streams will happen on platform and accordingly make partitions.
Can you highlight this?
You are the best ❤️
Hi, can you explain what is purpose of DFS here? Thank you
Hi! Can you make a video on live streaming using RTMP vs HLS?
I need a suggestion from you. Do u know how can i get the subscription package for streaming for website?
Hey, I want that load balancer hoodie. where do i get it ? BTW big fan of yours
YOU THE BEST!
Hi Gaurav, what would best suite to be the transformation service in an actual setup? I think a serverless computing service like AWS Lambda would work well right?
Great content++
You are the best
Thanks for the vid!
Thank you!
Hey Gaurav, just a suggestion- please try to avoid zoom in, zoom out in videos, gets a bit frustrating at times. Very good content though, thanks a lot!!
Question is how will subscribers like tablet , mobiles read from the queue ? kafka wont be running on these or any other messaging queues , correct ? will it be a websocket call ?webhooks or httpstreaming ?
What is the frequency of tasks done by workers now !
👏👏👏
Thank you for all the great contents Gaurav. I've learnt a lot from your videos. I have a question, instead of using a queue, can't the stream of video be ingested in some streaming platform like AWS Firehose transformation done there and the transformed data be pushed to say S3 bucket to be ingested by the intended app.
Yes, you can do that too.
Any mock interview series wich match the maang level.
If we are storing every video fragment in a DFS, then after it's been converted the first time, do we need to send a message to the second queue informing that the worker completed its job?
Ie:
1. We convert a video fragment at ts 1m20s - 1m30s using a worker.
2. That fragment is stored in the DFS.
3. The worker pushes an event to the queue saying it's finished its work.
4. The subscriber grabs that message and then the client is given the video fragment.
5. A second user steams those exact same 10 seconds. We don't need that fragment reprocessed. We can just grab it from the DFS.
So the current architecture makes us reprocess every fragment repeatedly every time. So instead of having an event sent that the processing job is done, shouldn't we first check to see if that fragment even exists in the DFS, and use it if it does, rather than repeatedly reprocessing it?
Hey Gaurav I want to buy out your course but I don't see any where over portal also coverage of code and practical Implementation . Can you or your team will help me on that part?
Could you send us the query at contact@interviewready.io, or on the chat of the website? We'll take it forward from there 👍
Hi Gaurav , Great video. I emailed you a question can you please look into and respond when possible
would love to see some practical example at some point, instead of all these whiteboard drawings. nonetheless, great explanation.
Not clear
Are you in the southern hemisphere?
Nope. India 😁
@@gkcs if you feeling too cold, please do a LLD.😁
By Live Streaming, I'm assuming that it is Real time stream to users. How is this system really Real Time as it will be very slow and slugging?
What makes you think it'll be slow and slugging?
The message queue system adds temporary storage by producer and consumers. That’s going to add to ur hand waving latency.
This system design is not suitable for live video but works for VOD. I’d be shocked and sad if anyone is using this design for live steaming.
@@pankajchaudhari2588 Time to get shocked and sad:
czcams.com/video/cd40-l-6EmI/video.html
Watch from minute 42:00 for the relevant bits.
Please don't assume that message queues are slow. Many real-time production systems use them with less than 1 ms of latency.
@@gkcs hahahah…good one. Now I am sad
Hmmmm…I did a quick 1.5x and skip ahead of that video but didn’t find anything showing their video processing pipeline to be what u showed.
Anyhow…if u have anything that shows that this is what Hotstar does, I’d be grateful to check it out. If not, no sweat.
I am simply speaking from my experience of having been an architect for a live transcoding product.
This was oversimplified, I don't think this will help in sys design interview, no CDN, no cache, nothin.
More on how Hotstar works here:
www.scylladb.com/2021/02/11/disney-hotstar-powered-by-scylla-cloud/