Video

Thanks

Which of the two types is Amazon SNS? I believe Log based, as it delivers each message in order and to each of the consumer, not Round robin based?

Hey Jordan, slightly confused here. We are writing both count min sketch, as well as windowed top k to TSDB. I probably missed why exactly. Is it to answer queries like fetch top k in last 10 mins (in which case CMS will be used) and fetch top k in last 3 hours (in which case windowed top k will be used)?

I refuse to believe that there's no relationship between Kleenex tissues and lotion

Jordan, how are you learning these concepts? Do you read papers or are you referencing a book?

Is it fair to think that CRDTs using Gossip Protocol will have eventual consistency? In some possible cases might not get the latest updates

I like this idea, Im in a similar boat w my career/interest in system design and would love to learn more about real life applications but don't always have high enough priority to read a full whitepaper. Would definitely tune in to these

How does Flink handle (topicID <-> userID) update? What if there is a new topic? What if a new user subscribes to an existing topic? What if a topic is deleted etc?

The design assumes we have full control on the clients and we can control the logic of the clients to have bidirectional communication, but in reality we often don't have control over the clients. Notification to the clients often requires existing infra (e.g. Apple APNs) which we need to integrate with but cannot modify its protocol.

Storing the idempotency keys on client is a wild idea, and can be easily treated as a red flag. I wouldn't mention this as an option in a real interview.

Hey Jordan, thank you for your video, it's very inspiring. For the scheduler table, could you please clarify the rationale behind using a combination of time_range and random_number as the partition key? At the end of the day, wouldn’t it achieve the same result as using time_range alone?

A CZcamsr who's true to himself, rather than hogging for SEO. Respect++. Hit 100K, already! 🎉

Hi Jordan, I have a question developed around the overall DB choice after following each of your videos. I noticed that you always look at "read heavy/write heavy" + "replication pattern" to make decisions, but almost never look at the overall load. Like in this video for deciding DB choice for comments, since it is obviously read heavy and there is not really challenges against single leader replication, you chose MySQL. I wonder if the number of comments grow very large let's say 100B comments (I think CZcams has more than that scale) and the read QPS is hundreds of Kilos, should we still stay with MySQL? What is the threshold of the overall number of items and QPS load for you to stay with or shun away from MySQL? (My old impression of SQL's QPS limit is just thousands of QPS, but maybe that is for a single Mysql instance, and may be the story before "Mysql Cluster" was introduced?) In between SQL and NoSQL, there is also "NewSQL" DB like Spanner and CockcroahDB. As I am aware of, both of them use LSM + SStable (or variant), which is actually slower in read than BTree index. But in reality, I feel the sentiment is Spanner is favored over MySQL in such large scale use cases. Does the slight edge of "BTree index gets lookup'/range scan queries faster" get offset by other considerations? Thanks a lot!

yours feel relatively complex and speedy tutorials compared to others

Hey Jordan, when the merkel trees mismatch and we found the row that is mismatching, which row of the two dbs do we pick?

Hey Jordan, won't ensuring Quorum reads and writes help in our cart service case? Is it possible to get quorum reads with R+W>N scenario from Riak? Any downsides in this? 2nd question, isn't the inventory data stream flink a spof here? Also since it's all in-memeory, if it goes down how do we plan to get all the inventory data back in memory of the new flink consumer? (I am thinking it'll have to construct the in-memeory data structure by making calls to inventory db and orders db, but that'd be very costly. Another option would be to have a background thread snapshot the data to disk, but that'd miss out on some data - if the consumer processes some order and before the snapshot it crashes?) 3rd qsn I see the flink writing something to products DB, what data is being written? Since this flink node is just capturing inv info.

I'm genuinely so excited for this Jordan! Been following your design 2.0 series which really shaped some fundamental concepts for me - thank you for that. As a fellow SWE who really likes to learn new things outside of the realm of interview preparation, this channel is a godsend. Keep it coming! Just a suggestion, maybe you could also link whatever you're reading so it kinda feels like we're in this together you know? idk learning alone sometimes can become lonely and having a community to discuss things with can crack open the mind like a walnut. lol

Instead of kakfa sharded on chatID (on send path), we can use User-scale realtime pub/sub (FIFO) [Examples I found are Amazon SNS, Azure Event Grid]. All the users will have their topics in which messages will be delivered to. In cases user is offline, the messages will sit in that until they come online and read from it.

8:22 Here in step 5 and 6, why do we need to search all points in the neighbouring boxes? Since we have the longitude and latitude of our position couldn't we calculate the places that fall along or within our circumference and then look up the geo hashes for them in our box and the neighbouring boxes?

Very curious to see the new content and i'm all for it after you taught me alot with the system design 2.0 videos and the system design questions. I'll give it a try and i'm sure i'll find it useful and entertaining with you as always. Keep growing my friend, and thank you! <3

Thanks for the great content again! Question: In your final diagram, the middle flow (new connection service) shows two layers of Kafka (a stateless consumer in between). Why do we need both layers? Can't the "new connection service(s)" directly push to the corresponding Kafka shard and avoid having the Kafka layer and the stateless consumer?

I like it. Now that I've scammed my way into an L5 FAANG position (in part by watching your videos) I need to quickly learn how things actually work beyond a surface level so I can stop feeling like a total fraud lol

Hi Jordan, If we partition by hash of the domain name, could this lead to ‘hot’ partitions? The node where the twitter or Wikipedia domain is mapped could possibly have a lot more work to do than some other nodes in the system. Is this something to be worried about or are we ok with some hotter partitions as a trade off to enable caching on DNS and robot.txt. Thanks again!

Jordan, you have been an inspiration to me and I am sure, you are many hundred thousands out there. I wish you good luck and I am sure you will rock in what ever direction you choose for. Good luck and best wishes. May the God bless you :)

would you also please share the links of the papers and videos you go thru?

Looking forward to learning with you!!!

you are so hot！

Hey Jordan, your videos helped land a Job at Meta IC5. I’ve watched and followed many bloggers and content creators but I couldn’t have cracked with any of the limited content they provide. Thanks for all the work. I’m very grateful to have found your channel. 😊 I’ve recommended your channel to atleast 30 people. I’ll continue to do so.

Thank you Jordan!

Very excited for what’s next! I was able to score two amazing offers recently and your videos were definitely a big factor. Since then I have been trying to continue the learning, so this is just perfect. Thank you for all the effort you put into this, cheers and looking forward to what’s next!

Just broke down screaming and crying in walmart. Where am I supposed to get my corinna kopf / apache flink crossover content now?

Well! I would love for you to go as deep into a paper (maybe also its implementation of the stuff in Rust/C++ or whatever, don't use python please). Would love to learn from you! Cheering you on!

me and my friends as binging on these videos rn and jordan's subtle humour and on point content here is what driving us to finish playlists after playlists.

systems design interview prep content gets silly and redundant because systems design interviews are silly and redundant. the interview process is a cruel joke right now and while i have no faith it will improve i am confident it will change rendering a lot of these scripted design and leetcode prep videos obsolete. real knowledge of systems will always be relevant to our work and this is the right pivot to make, keep it up proud of ya looking foward to it

God speed to you Jordan . You changed my thinking process about systems .

Good luck Jordan for the new adventure!

Hey Jordan, In a partitioned database that uses single leader replication, does each partition have it's own single leader or is there one leader across all partitions?

For pricing server/service, I am still not convinced about the need for Websockets. The user is unlikely to subscribe/unsubscribe to stocks as frequently as they would care about getting the current price of stocks. So bidirectional communication aspect of Websocket is not really useful? For near real time updates, I am wondering if we could get away with SSE considering the cost of maintaining web socket connections and a second/two of extra delay in getting updated prices will have negligible impact on how the client decides to act on it. Thoughts?

RabbitMQ has durable queues, where the mesages are written to disk, they can also be replicated to other nodes. RabbitMQ also has "Streams" which are queue-like in that you can pop and read messages in order, but the messages are not deleted once read, meaning another consumer can start from the beginning (or any point in the stream) and read the messages again.

Hi Jordan, Great video. I did want to know what the relationship between chat and messages is? A chat can have one or more messages and I found it a bit hard understanding that relationship while watching the video Thanks

Man I normally don't laugh at your jokes but this one actually made me laugh, I think it was mainly due to your stare at the end of the intro! 😆

Hey Jordan, I just cleared meta onsites and your videos were quite entertaining and helpful and yes the substack too. So, thank you🎉

That's a great idea dude. Would you be able to share the paper/resource(s) you are targetting to read in next 'x' days? Kind of a follow along concept, I'd like to go through them and follow your videos. It could help motivate me to learn as well.

Good direction, mate. Now that I have a job again, I find I have less interest in software design videos for the sake of job interview and seek to expand my knowledge instead.

I’m sticking around 😉

god

Your deep dives is the reason I'm subscribed to your channel. 😂

Wouldn't indexing on run_timestamp degrade the performance of the DB ? since ideally, we create indexes on columns that don't change often?

You are so underrated brother... this is the best content on the internet regarding System Design interviews 100%... who cares about an interviewer saying 'yes' to every question that the interviewee asks, or an interviewee coming up with the same 10 boxes every single System Design question when you explain stuff like this... keep it up, you are saving lifes!!

There are many system design videos on CZcams that are from people who never built any system so they end up being shallow and have unsolved distributed problems. As much as I liked your previous videos, I’m interested for the next and having more depth. Thanks