System Design Interview Walkthrough: Design Twitter
Vložit
- čas přidán 11. 09. 2023
- *Designing Twitter's System Architecture: A Deep Dive*
Join me as we delve into the intricate system design of Twitter, one of the world's most popular social media platforms. From understanding the core requirements of the platform to the nitty-gritty of microservices, we'll explore how to architect a system that can handle millions of tweets, retweets, likes, and more.
In this video, we'll cover:
- The importance of load balancing and how it's achieved.
- The microservices architecture and how different services interact.
- Data storage solutions, focusing on tweets, replies, and user profiles.
- The unique challenges and solutions of the Twitter timeline service.
- Key security considerations, from authentication to data encryption.
- Monitoring and testing strategies to ensure a robust system.
Key Highlights:
- How "fanout on write" and "fanout on read" strategies optimize the timeline service.
- The role of Elasticsearch in Twitter's search functionality.
- The intricacies of the profile service, including user data storage and follower connections.
- The importance of security, monitoring, and testing in ensuring a robust system.
Disclaimer: This video is a theoretical exploration and does not represent the actual internal workings of Twitter. It's designed to provide a comprehensive understanding of system design concepts.
Want to test your system design skills? Walkthrough a real-time mock interview at [hellointerview.com](www.hellointerview.com) and get instant quality feedback from an AI trained by industry experts.
#systemdesign #twitter #interviewprep #softwareengineering
Youre diagrams are somehow so clear despite having complex lengthy requirements !
The pounding sound while dropping a new diagram object is UNBEARABLE 🔨🔨
Checkout the new videos. Totally different format and much improved. You won’t see it again :)
Great work. Thanks for sharing!
Great! Super common question. Instagram or link shortener next please!
Man, you covered everything. Thank you, please make such videos. Your videos are point-to-point without any distractions.
Wonderful, I've watched all of the videos on your channel and I'm kind finishing up with this one. I guess this was a proof of concept perhaps? Either way, it's great. Love to see that you defined a framework for the other videos. Looking forward to more of those!
Yah this was the first one i made a while ago and honestly consider deleting
@@hello_interview for what it’s worth, I learned a ton from this one.
- You showcase here that it’s worth delineating read/write paths for specific* services, which is not something you folks have done in the other videos (it sounds obvious but it’s not for someone of my level).
- You spend a bunch of time towards the end talking about salient points we could make, whether related to security, monitoring and testing. Again, points you don’t quite bring up in the others.
Now if you’re ever going to make a new Twitter design video, perhaps? I don’t know, you guys are doing a tremendous job leading this anyway. Thought I’d share some user-feedback. Cheers.
I have watched many hours of system design videos. This video is the best one that covers majority of the important topics. Thank you for making this!
Hey this was great! Would love to see more of this.
HelloInterview to the world! Great great content.
Amazing video! One of the best system design videos I've seen. Clean, simple, yet goes over a lot of important topics and beyond. Keep it up!
Excellent👌
Found your content on reddit and this is great. tyty
Excelente video, thanks🎉
May be a noob question, why do you need Load balancer before Api gateway? shouldnt that be after api gateway as gateway itself will be resilient enough to balance the traffic? May be after gateway we decide if request is going to particular service then which server of that particular service it should goto?
You store replies in a separate database, why not store replies and retweets in different databases as well ? their scale is massively different .
By the way, your website is probably one of the best I've seen in terms of learning system design - the way you present information with trade offs, questions and answers is really well done.
You talked about using cache for popular tweets but where this cache is being used? If it's in the timeline service than how does the system knows that one of the tweet in the cache is from one of the user's follower and hence needs to be included in the timeline?
For timeline calculation, instead of messaging queue writing for each tweet to all followers cache, could we use pub/sub? Publisher = tweeter, subscribers = followers
Or could we instead use streaming followed by batching? I guess in this case since timelines are all unique to each user, batch processing is overkill.
Thanks man 👌
This is what API Gateway server special for. It is to handle any non-functional requirements. like rate-limit, caching, security. etc.
But I think you had to mention that to not confuse the beginners
It’s main responsibility is to route api requests to the correct microservice, but it also can handle general responsibilities like auth, rate limiting, ssl termination, etc
Thanks for this video. You have definitely covered a lot of topics in breadth, but isn't the expectation in a design interview to atleast deep dive into one of the components ? Would you give a hire feedback for an E5 candidate who presented exactly the same content as that of this video in their onsite interviews ?
Great question! This solution would be adequate for an E4 candidate, but not for an E5+ candidate, for the reason you mentioned. As candidates get more senior, the expectation is less breadth and more depth. For an E5 candidate, I'd suggest you find 1-2 places to go deeper. For Twitter in particular, the search component could be one of these places.
@@hello_interviewthanks for your response. Do you have some suggestions reagrding the time distribution to finish a design(including deep dive in one component) in a 40 minute Meta interview ?
This is an awesome content. Best so far I have seen when compared to others on this problem .
It would have been great to go into details of some topics like , how are we updating redis cache ? What does data model of followers /followee looks like?
Yah tbh this video is pretty high level, targeting more mid level (ic4) candidates. For much more depth check out the other two videos on the channel!
@@hello_interview will do . thx!
Feedback while watching the video - I'm not sure if it's from the diagramming app or added but the pounding sound effect when adding a new service is annoying and detracts from the viewing experience. The content is super helpful though.
FR it feels like Evan gets mad and hits the desk out of anger 😂😂😂
This is how you do it. Whole twitter in 20 minutes. OP, you are the best. I didn't get though why user profiles need to be stored in a sql db, why mongo would be a poor choice? Analytics can be it's own separate DB. Will a SQL DB sustain analytics and logins with 300+M active users? if you shard the DB, then is it possible to do analytics? Also when a user follows another one, it should be recorded such that timeline takes that into consideration (oh you added it later :) ). What about rate limiters? Is it a separate service such that the throttling data per account is in central place? I liked you touched security, logging and monitoring. Is this something that interviewers are looking for?
"Is this something that interviewers are looking for?" - Depends on the interview. I have had a few where they told me ahead of time they wanted to touch on security, monitoring, and maintainability. This is likely something you can ask the recruiter ahead of time for clarification.
Hey Evan,
Around 15 minute, when you create a message queue for timeline fanout -- in this case, wouldn't it be better if we used something like CDC with Kafka/Kinesis to capture new tweets directly from the database?
I'm just thinking there might be more failure scenarios to handle with pushing directly to message queue from the tweet service -- i.e. how do we make sure we don't miss tweets for fanout if the message queue goes down? How do we handle if database write fails? Only push it to the queue when write is successful?
Also I guess this is a more general question because I've struggled with it interviews -- how do you decide between using a message queue and a stream for a particular problem? Because I've found there is some overlap where both can work
I think in a very recent video, he mentioned that generally CDC should only be used for kicking off anything asynchronous, but should be avoided for synchronous low latency tasks.
Since the task being kicked off here is asynchronous, I think you make a good point of using a CDC.
Hope this helps as a vague guideline.
why sometimes you design the actual API and http methods in detial, other times you draw requests to microservices without mentioning APIs
This is great but just didn't got one thing - why put load balancer before api gateway?
Just an abstraction. Each service (gateway, services) will have a load balancer
Can you please share HLD of something like mixpanel?
Big fan of your explanation since I found you in a Reddit comment! Does this one have a written version? I couldn't find it on the website
Nope, this the only one that doesn’t hit the newsfeed write up should be similar
Awesome content. What is the whiteboarding tool that you use?
Excalidraw
is cdc some endpoint that database provides out of the box, or is it some tool, or a stand-alone microservice?
Most DBs have some form of event capture nowadays. They expose connectors where you can configure a streaming technology where the DB places new events on the stream.
This is a great resourc, but I don't see the interactive AI powered whiteboard in your website. Is that still in development?
Click on mock interviews in the nav then “AI mocks” on the left nav
Does twitter mainly use MySQL?
font and design components are too small. hard to catch those details
You uploaded the video through the API Gateway and service. The bandwidth should be a bottleneck for the the system.
Nit: MongoDB isn’t open source
Thank you for the great content. Really appreciate it!