In regards to second problem you mentioned when the leader goes down, specifically the case where writes 71-80 are lost since those writes were never replicated. I assume the new leader (previously the follower) begins accepting writes starting from write 71. When the initial leader that had gone down comes back online at a later point in time, the initial 71-80 records also come "back into existence". My question here is, in such a case do we really lose the 71-80 data?
1) Do they come back into existence? That's implementation specific. They may be outdated now and nonsensical. 2) The node may never come back up! Maybe the hard drive gets fried by my 360 roundhouse kick
I just pop around to random youtube videos at this point, ideally from actual conferences, since those are a lot more trustworthy than other CZcamsrs lol
Hey Jordan, thanks a lot for this content! It's especially cool that you draw, because it makes so much more sense :) Will you do the same "drawing" series with System Design Questions?
Hello Jordan(Single Leader, alpha male), As always great video I had a small query, as we have established replication can be achieved using Replication log. Just wanted to know how does this actually work between master and follower? Is it a process on master which keeps track of where exactly each follower is and pushes the difference OR each follower polls the master with last item it fetched and master sends back the new data it has.
I imagine there are multiple possible implementations. Perhaps the leader pushes, and then upon receiving a message a follower asks for backfills if it is behind.
Not so easy! What happens when your leader goes down? Like what you're saying, it's not enough to "label" something, every node has to agree on it (or at least a majority).
Initially node participate in an election as (follower or participant )where nodes shares their latest id wether be it a log id or id incremented by commit (depends on what algorithm you are using) if the participating node gets the id from the other participant node which is greater then it’s own it will vote for that node that’s how single node gets the majority and that node becomes the leader as soon as the leader is declared it spreads out the message to all the nodes of the quorum
In the leader goes down scenario 2 where we are loosing the data from 71-80. In this case don't we have a replication log for exactly that purpose. Like the leader wrote into the replication log till 80 and then dies. Since the follower is reading the replication log wouldn't it be able to get that data? Pls correct me if I am wrong!
Replication is done asynchronously. If the leader goes down before it sends its writes to the follower, and it never comes back up, the follower will never see those writes.
Hey Jordan, In a partitioned database that uses single leader replication, does each partition have it's own single leader or is there one leader across all partitions?
Hey man I have been enjoying your system design concepts So far one suggestion I'd like to make is maybe you can talk along with your drawing (the 5 dollars white board you normally used in the end of each videos) I think this can really help the explanation, anyway keep up with the great worj
This entire system design playlist is awesome, so much is packed in these short videos, and your dry humour makes it even more fun to watch.👏
Bro you are awesome. Your sense of humor is too good, making learning system design fun
Thanks man! Glad you're sticking around :)
In regards to second problem you mentioned when the leader goes down, specifically the case where writes 71-80 are lost since those writes were never replicated.
I assume the new leader (previously the follower) begins accepting writes starting from write 71. When the initial leader that had gone down comes back online at a later point in time, the initial 71-80 records also come "back into existence".
My question here is, in such a case do we really lose the 71-80 data?
1) Do they come back into existence? That's implementation specific. They may be outdated now and nonsensical.
2) The node may never come back up! Maybe the hard drive gets fried by my 360 roundhouse kick
Hi Jordan, can you please share the resources for the system design which you follow?
I just pop around to random youtube videos at this point, ideally from actual conferences, since those are a lot more trustworthy than other CZcamsrs lol
Hey Jordan, thanks a lot for this content! It's especially cool that you draw, because it makes so much more sense :)
Will you do the same "drawing" series with System Design Questions?
Yup, it's gonna be a bit but that's the plan
Hello Jordan(Single Leader, alpha male), As always great video
I had a small query, as we have established replication can be achieved using Replication log. Just wanted to know how does this actually work between master and follower? Is it a process on master which keeps track of where exactly each follower is and pushes the difference OR each follower polls the master with last item it fetched and master sends back the new data it has.
I imagine there are multiple possible implementations. Perhaps the leader pushes, and then upon receiving a message a follower asks for backfills if it is behind.
cant we just label a node as a leader / follower, or is this basically just what distributed consensus is?
Not so easy! What happens when your leader goes down? Like what you're saying, it's not enough to "label" something, every node has to agree on it (or at least a majority).
Initially node participate in an election as (follower or participant )where nodes shares their latest id wether be it a log id or id incremented by commit (depends on what algorithm you are using) if the participating node gets the id from the other participant node which is greater then it’s own it will vote for that node that’s how single node gets the majority and that node becomes the leader as soon as the leader is declared it spreads out the message to all the nodes of the quorum
In the leader goes down scenario 2 where we are loosing the data from 71-80. In this case don't we have a replication log for exactly that purpose. Like the leader wrote into the replication log till 80 and then dies. Since the follower is reading the replication log wouldn't it be able to get that data? Pls correct me if I am wrong!
Replication is done asynchronously. If the leader goes down before it sends its writes to the follower, and it never comes back up, the follower will never see those writes.
Thanks a ton!! Waiting for more such awesome/harder ones
Oh things will be getting harder no doubt
Hey Jordan, In a partitioned database that uses single leader replication, does each partition have it's own single leader or is there one leader across all partitions?
First one!
High Tech-Wizard 🌿🧑💻 Thanks for the hard work brotha
Same to you :)
Hey man I have been enjoying your system design concepts So far one suggestion I'd like to make is maybe you can talk along with your drawing (the 5 dollars white board you normally used in the end of each videos)
I think this can really help the explanation, anyway keep up with the great worj
Bit confused here as I'm certainly talking along with my drawings
More problematically - the white board is gone :(
Typo in video description?
It 'reminds' me of myself?
Nit: typo -> yup
Thanks!