Multi Leader Replication - chaos | Systems Design 0 to 1 with Ex-Google SWE
Vložit
- čas přidán 24. 07. 2024
- They call me an all to all replication schema since I'm shooting my DNA out in every direction (this one even grossed me out)
- Věda a technologie
Such a good channel. Thank you for making the concepts simplified. I read DDIA book first and then come to your channel to clarify my doubts. This channel deserves much more subscribers
hey Jordan... love your content... get to learn a lot.
The usual 10 min sessions are amazing. thanks a lot !!
Requesting 1 more conceptual video to be added to your list (non system-design): A* and/or Dijkstra and their applications like you used for Uber/Google maps but a little more detail if possible, ex: how you used it with Contraction Hierarchy.
Sounds good! I'm planning to do much longer videos, and possibly even multi part videos for those when they do eventually come around, so I'll make sure to get into it!
Thank you for the great content!
I have 2 question around the All-to-All topology and ensuring the same write is not applied multiple times to the same database.
1. Why are the other leaders passing around changes made from leaders other than themselves? For example, if there are four leaders, leader 2 should ensure that its unique changes are propagated to 1, 3, & 4. Why would leader 3 pass the changes of leader 2 to other leaders, for instance?
2. This is more of a sub-question from the question above but i'm having trouble understanding how modifying the replication log prevents the same write from being sent to multiple database nodes.
Let's assume there are 3 leaders. If the new data is originally on leader 1 and leader 1 sends a replication log to leaders 2 and 3 then SeenBy = [1].
There is nothing preventing leader 2 from sending the data to leader 3 and vice-versa.
Where is my misunderstanding here?
1) Leader 3 can go down, so this offers some additional protection there
2) You're right that 2 could send to 3 and 3 would send to 2, but after that happens, they'll stop sending (because they'll both know that each other has seen it), at which point this actually terminates, as opposed to being an infinite loop.
Hey Jordan! Can you explain a moment with delay on 12:10, please? Our device sends a request over an NTP, gets a response with 3:58 time from a server with GPS clock but we have 3:59 on our device, why not 3:58? Does our device track a delay in 1 second and sum it with 3:58? If it is then how can our device track a response delay without request delay🤔
No there's a delay due to network lag, you're right that was a dumb drawing - my point was there's 1 second of network lag and as a result we get the update at 3:59 but our phone thinks it's 3:58 but I explained it terribly lol, sorry about that
Now it makes sense, thank you :)
Considering that everyone probably gets their stock tick data from exchanges, surely there must be some agreed upon protocols to avoid write conflicts in quantitative development world?
There is - a single leader
@@jordanhasnolife5163 Reading back my question, i don't know what I was asking. I guess my main thought at the time was we have exchanges that have been running since decades. How often do they undergo architectural changes based on new technologies being developed, like SSDs, faster Rams, faster processors etc ? Or do they postpone changes because of "why change what's not broken" as identifying bottlenecks becomes a hassle?
What a chad
Hey, I think this video is not in the system design 2.0 playlist. I had to come to the channel page and find it. Just a heads up :)
Ty sir
I think you forgot to add this to your system design 2 playlist. Edit: Actually, the issue is it's out of order, it's way further forward in the playlist and makes some of the earlier vids a bit confusing.
Lemme see if I can fix this thanks!
what is meant by all to all replication in your description context 🤨
Tragically I cannot