Distributed Consensus and Data Replication strategies on the server
VloĆŸit
- Äas pĆidĂĄn 27. 07. 2024
- We talk about the Master Slave replication strategy for reliability and data backups. This database concept is often asked in system design interviews with discussions on consistency and availability tradeoffs.
Very closely tied to the master slave architecture is the concept of distributed consensus. When designing a system, we must make sure that the individual components can agree on a particular value. (Leader election, distributed transactions, etc...) Some popular techniques are 2 phase commit, Multi Version Concurrency Control, SAGAs and Quorum.
Chapters
0:00 Problem Statement
0:53 Replication
1:24 Synchronous replication vs. Asynchronous replication
3:36 Peer to Peer data transfer
4:44 Split brain problem
Also useful when used with Master Slave replication is sharding. You can check out the video mentioned below.
Looking to ace your next interview? Try this System Design video course! đ„
get.interviewready.io
Course chapters:
1) Design an email service like Gmail
2) Design a rate limiter
3) Design an audio search engine
4) Design a calling app like WhatsApp
5) Design and code a payment tracking app like Splitwise
6) Machine coding a cache
7) Low level design of an event bus
The chapters have architectural diagrams and capacity estimates, along with subtitled videos. Use the coupon code of 'earlybird' to get a 20% discount.
References:
Sharding: âą What is DATABASE SHARD...
System Design Playlist: âą System Design Playlist
Designing Data Intensive Applications - amzn.to/2yQIrxH
You can follow me on:
Facebook: / gkcs0
Quora: www.quora.com/profile/Gaurav-...
GitHub: github.com/gkcs
LinkedIn: / gaurav-sen-56b6a941
#SystemDesign #Databases #Replication
Guys, sorry for the last 4 minutes of the video where the camera focus went haywire. I keep running into these technical issues every video. Maybe I should do a course on videography...any suggestions?
Also, if you are preparing for a system design interview, try get.interviewready.io.
All the best!
Where is your video with Tourist? Are you still in the process of editing it? And you might want to make a video about anti-patterns.
@@sreyanchakravarty7694 I thought that would be the next video but my assumption failed
@@mrtourist8066 do you know why he took the first video down? The video with tourist that is
Use small aperture, pre focus and then set focus to manual.
yaa great work bro. Just one advise, disable auto focus. Manually focus it on the board first and then start recording.
Excellent video as always gaurav.
A few points from my side.
1. Taking snapshots at regular intervals of the db is one way to avoid the single point of failure.
2. Log reconstruction is one more way. Before writing to the db first flush to a log file. So if the db crashes the log file can be used to restore the db.
3. I think when you talk about the databases communicating with each other , you mean there is an API that sits in front of them and these API's communicate with each other.
Or is it that there is an API in front of the db which writes to a queue and another process that polls the queue and talks to the second db.
4. One way to deal with the split brain problem is through the paxos algorithm.
Great job and thanks !!! :)
Excellent points Mukund, thank you! đ
Dude thanks very much. Your work is highly appreciated!
Great work brother!!! Awesome as always
Solid content my dude!
Great content (upvoted and subbed). Don't see any other channel with short and precise summaries of fairly complicated concepts.
One more benefit of Master-Slave - Move replicated data closer (geographically) to the end user. For better user experience - lower latency.
Also sharding and replication scheme are tangential and often work in tandem. Generally you will have sharded databases with each shard replicated. Replicated shard could be a slave or a master (consensus based).
I vote for distributed consensus. In fact this would cover the meat of distributed systems and could be broken down into multiple smaller introductory videos
Video 1> Inherent problems with distributed systems (failure detection, n/w partitions, byzantine(non) faults, clock skews).
Video 2> Why do we need consensus ? Protocols to tackle #1 and what they tradeoff to offer consensus (CAP axis). Protocols - 2pc, 3pc, paxos (intro).
Video 3> Real applications and tradeoff they make for better performances. (Why distributed acid applications are hard and to be avoided).
I see lot of interviewees(i'm interviewer in one of the FANG) have surface knowledge of these concepts and often incorrectly adjust to a constraint (been guilty myself). Content like yours can help someone find gaps in their understanding.
Key terms for Data Replication during interview: Atomicity (A of ACID), Master- Slave, Master-Master, 2PC, 3PC, MVCC, SAGA, Split Brain, Distributed Consensus, Regular DB Snapshots, Log reconstruction, Paxos Algorithm
MIT video on 2PC: czcams.com/video/aDp99WDIM_4/video.html
Hey ! It was really helpful video for beginners ! Keep rocking
Awesome explanation Gaurav Sir đ€© I learn a lot from your videos keep the good work up.
Hi Gaurav - thanks for reviewing the differences between ( Master-Slave ) and ( Peer-to-Peer ) architectures : it's helpful to know that Masters have RW operatinos, but slaves have only R operations. Same for synchronous copying versus asynchronous copying : I should remember this in addition to shallow copying versus deep copying for object datatypes :-).
Great explanation Gaurav, You just made my interest in system design alsođ
Thank you!
same here, I started loving this topic.
The point where the video gets blurry, I felt like I got hit by some sort of vision problem đđ Video was very interesting and fun to learn.. Thanks Gaurav Sen
Amazing content brother đ
Hey Gaurav, great videos :), I believe 2PC and MVCC are completely different things, they cannot be compared to each other, MVCC(weak isolation) is a way of handling transaction Isolation on a single node, whereas 2PC/3PC are ways of handling distributed consensus (as you have explained already). Transaction Isolation levels should have no relation with distributed consensus.
Please correct me if I am mistaken here, will help a lot.
Yes that is true. I think I mixed them up because of the close nature of consensus and locks.
I was looking for this comment.
Nice work Gaurav! This is the only channel for which I have enabled notifications. You are helping the whole community. Thanks a lot.
PS: start an instagram page and post all these memes there also.đ
Haha that's a really good idea đ
Great video !
Great humor keep it up ^âą^
Thanks for the video!! I think it would also be interesting to explain how a consensus algorithm like paxos works and what problems it prevents.
Yes, that's something to discuss in a future video (on distributed consensus) đ
Hi Gaurav, good video. Can you make a video over "how large scale Indian Unified Payment Interface works"? Focus on technical/Networking and Database aspects. Thanks !!!
That's actually a great idea and a video I am looking forward to !
You mean, about the Payment Gateways of various financial institutions like credit cards, netbanking, ecommerce & digital wallets ?
Hi, thanks for the great work! I just have a question regarding the syncronization lag between master and slave DB. What is the best practice to solve this issue for a read heavy system ?
Khoob bhalo dada :)
Thanks for the vid!
Thank you!
Good video. Subbed ;)
If there is single database, my application is scaling up in terms of user then the load balancer will not help, all the load will be in single database, for that how to serve fast in terms of user experience(read/write), how to scale or design the database??? i m seeking help for sql server
@Gaurav Sen, Any reason, you didn't talk about Paxos and Raft, for consensus? Also, @13:40 when you said co-ordinator, did you mean something like zookeeper which can do promote to master, or is that anything else ?
I intend to talk on paxos in the Distributed Consensus video. Yes, something like Zookeeper would be a coordinator here đ
Thanks buddy!
Where is Distributed Consensus actually explained ? Did you make a video for that already ? Please share if you did. Thanks for the videos. Keep up the good work.
Hi Gaurav, polling for next video - Reporting engine. In many systems, heavy reportings/analytics is involved. There are various solutions like using sql views/ late update using cron jobs. Using separate database or even servers. Please enlighten us on this.
Sounds interesting, and I will add this to my list :D
Hey Gaurav,
Great timing for the upload btw,
Thanks for a very exhaustive discussion regarding master slave architecture. Much new stuff I came to know. Distributed consensus or "Quorom", as mentioned in your earlier videos, will be highly dependent on many factors as mentioned, but I believe that these concepts dealing with "system vote" in case of potential failure will require a branched study of its own, as it slightly outside the topics in discussion.
Hey, your video was very complete regarding master-slave, can't focus more on that, but it looks like the "quorum" part also contains some of the architectural meat, and can be discussed in dedicated manner at once, for all the possible use cases.
Thanks! That's true, and hence my call out to the "distributed consensus" topic :)
@@gkcs Yes, it is noticeable you are into it right now. Waiting for the resulting vid.
In many Amazon interviews, they usually ask Command execution framework design and play store design. Can i have material for that?
If our system has 'n' replicas, which replica amongst those 'n' stand chance to be master? What is the criteria for selecting one single replica amongst pool of replicas?
For active-active-active data centers, 3 nodes RAC cluster in each, if application has to do DB replication, what could be the best strategy?
Your videos are great and you've done a great job in your explanations. You should do a video on book recommendations that aid people on learning about systems design thinking in this way.
New hair cut! đ
MVCC is used by MySQL InnoDB storage engine as well.
Wow, I had forgotten about that đ
What do you mean by database can crash? Generator to Hoga na!!đ
Amazing content, keep up the good work
Hahaha! đ
awesome.
Amazing vedio,can u give me a brief advise so that I could become a programmer like you.
Hey Gaurav, thanks for making a video on this topic. I had requested you for this. Could you please refer me to some links for the implementation part, as in, how do i implement this syncing of two databases in two different systems?
Master Slave is a very common pattern, so a lot of these systems provide the functionality inbuilt. Like MySQL or Postgres.
You can check out the implementation of these systems đ
Hey man, awesome Videos on important topics! Actually, I haven't come across any other good source on Systems Design.
Can you recommend some literature or a crash course on Systems Design? Or maybe you can make a whole course, going from topic to topic in a structured manner, more like how the topics are structured in a book, so that it can serve as a starter basic overview on the whole field.
Cheers ;)
You can try: get.interviewready.io/courses/system-design-interview-prep
I have the details of the course here: czcams.com/video/beUzxZK6aKw/video.html
dude you just look like Donald Glover in The Martian ahahah great video thx !!
đ
Good work. Just a small concern, what if there are 2 partitioning failures from a to b and c. We're back to square one. I'm guessing this wasn't the part of assumption you made. Can you throw some lights?
HI! i love your competitive programming videos. do u think you can make a serie of fendwick tree (some applications of it in problems, not the theory). i think it is hard to conceptualize a problem and using fendwick tree to solve it, so it would be really helpful.
Thanks!
You can go through this article on Fenwick trees
www.hackerearth.com/practice/notes/binary-indexed-tree-or-fenwick-tree/
Why don't you make a technical meme page. Where we get knowledge with laughter. haha :P
Amazing video and amazing memes in between :P
I have an FB page you can checkout, mentioned in the description :)
Is this considered isolation? Or consistency? I always get those 2 mixed up
how master slave is linked to consistent hashing. Can they both work together or work as an alternatives?
In AWS will multi AZ database work?
Hi Gaurav,
In between you have explained the split brain problem, that if network partition has failed then A & B assumes that they are the master and both updates the data due to split brain problem. But however even if network partition is not failed then also there are couple of problems. What if S1 Send the Write request to DB1 and for same entry in database S3 Send the Write Request to DB2 then also both can updated and database will be in inconsistent state. So We need to make sure before performing write request to any DB Server we would have a lock on the entry in every server. How will we ensure that? Either B needs to talk to A or somehow some other system has to handle this. Because we can't blindly update DB and then tell it's other peer that I have done the update please sync. So without network partition also this needs to be handled. I am interested how this locking things will be handled can you make video on that?
The Batman meme got me dead
heey, great job!
thanks for ur vids
can i ask how do u know this much at this age?
5:25 you missed an opportunity to include the "Look at me. I'm the master now" meme
Hahaha!
Can you make a video on load balancer handling the requests based on Geo-Location along with HTTP authentication tokens?
Hopefully soon đ
Hi gaurav, will you make a video about system design of UPI and Bitcoin please
Please make a video on paxos and raft
06:00 The balance deduction thing, the 2 messages deduction 100 and deduction 50 are different messages right ? so what's wrong with deducting 150 ? I think I am missing a key point.
The problem is that serverA receives the message to deduct 100 and serverB receives the message to deduct 50 from the account, but serverA and serverB have lost their communication. So for serverA the resulting state of the account would be (120-100) = 20 and for serverB the resulting state would be (120-50) = 70 which leads to an inconsistency in the data.
Munish Bhatia thanks man for clearing it out
thanks Gaurav for the wonderful video. Have a question on DB sharding. Do we store whole data in each shards and serve only subset basis shard key. How would shard know that I need to keep whole data but serve only subset of it. Thanks
We don't store the whole data. That would be replication without the benefits.
should the slave and master be in the same subnet?
But won't there be redundancy of the link between A and B as well ? As you said at each step there must not be a single point of failure ,then why spof at the link between A and B?
That's a good point. We can add redundancy to the link.
Can you make a video about proxy
Auto increment, foriegn key having issue in master-master replication. How to get rid off it.
Don't get into such a scenario if possible.
@Gaurav - can 'distributed consensus' help solve distributed concurrency issues? For example, service is incrementing no. of like of a comments. If current count is 10, if two liked at same time. Then can 'distributed consensus' make sure the count is 12 instead of 11. If not, can you cover these concurrency topic in distributed environment.
Yes it can.
Distributed consensus can also be achieved by the Paxos algorithm, which also solves the problems of 3PC.
Even then, Paxos is....slow. In some cases you have to go for eventual consistency to keep the business running.
@@gkcs Netflix and likes rely on eventual consistency right? #Clarifying Nice Content
I know of Raft and Paxos architecture to solve distributed consensus problem. May be Gaurav you can make a video on this sometime later. Or if you already have, then please share the link once.
I will đ
Wat software u use for editing your videos???
Adobe premiere pro
Hi Gaurav, awesome video.
Do you happen to know which system design is probably used in the scenario when you run out of locks during distributed consensus pretty fast. Something like rate limiting for acquiring locks ?
Hello Gaurav. Just want to ask what is your view on data science in industry field? Are you planning to switch to data science field in the near future?
Maybe, I haven't considered it seriously yet.
Bro, please name the video better, leaves me confused most of the time. I've noticed this in other videos as well.
Other than this, your videos are nice, good explanation bro, Keep it up
Hi Gaurav,
Nice video. I have some elementary doubts.
1) how communication between two DB could get broken which results in Split brain problem. My guess is they communicate on particular port. Closing of port is only way to cause this issue.
2) how master master architecture is achieved in DB. Bcz server would be configured to sent data to particular DB say DB1, howcome data could be dynamically sent to DB2. As much I know there is no load balancer behind DB set up.
Thanks lot.
Thanks!
1) There could be tons of issues, hardware or software. Some hardware failure example would be power supply loss, disaster, etc...
2) Master Master would mean you can send the update request to any server and expect a consistent state on the DB. The DB can also have a load balancer. It's a (DB) request response after all.
@@gkcs
Thanks for replying.
1) Hardware failure ex power loss etc it means whole computer is down. It is different from Split brain which means computer is up just communication link is broken.
2) is this like config server at which source code point will list all DB instance. N using load balancer request could could go to any DB.
If it's read request then any DB instance is fine provided Synchronous replication strategy is used. But if write request is also going to any instance then we have to enable this configuration of master slave take form of master master.
Do you know what need to be done to achieve this switch setting.
@@saurabhsingh7020 Those are good points, and I lost context while responding. Let me take some time and get back to you :)
@@gkcs
Benefits of sharing knowledge.
Discussion :-)
can you please make a video on SAGA?
I will later đ
is gossip architecture implements distributed consensus or is it basically just another name for it?
It's a type of eventual consistency mechanism. Have a look at the "service discovery and healthchecks" video for a better understanding :)
So finally, what is the reason about using Master-Slaves architecture but not Master-Masters architecture, after your solution about the third node part (C node). I dont understand :(
Master slave is simpler and doesn't need 3 nodes, so it's cheaper too. The consensus is also faster.
I'm really concerned about your title since CZcams algorithm is sometimes misleading
Oh, yeah that might be an issue...đ
Why don't we just add multiple routers between A and B to avoid split-brain ?
I've Amazon interview in 2 days for SDE-1, any tips?
All the best!
@@gkcs Thanks!
What if I have millions or billions of data, say data of courier service and their mapping with the pincode of origin and pincode of destination in an ecommerce website. The mapping data can be huge. Let's say there are total 30k pincodes and say at least one courier service provides service for all 30k*30k locations/pincodes. I can't have one database for all this data plus can't have slave DB. So i thought to shard it on the basis of origin pincode but then also the data will be huge and many databases will be there again. So to have slave DB for each of them will be expensive. How to handle this situation?? Thanks for the video :)
Thanks for the feedback đ
Are you sure this is a real situation? I can't understand the usecase much đ
@@gkcs it was asked in an interview đ so the question was, if you go to any ecommerce website and select a product and the seller, the application needs to check with the courier service which will collect the product from the seller and drop it to the customer. Now there might be few courier services which don't operate from the seller's location or from the pincode entered by the customer. And the few courier services which do that can serve to other pincodes too. So the use case given to me was that if there are 30k pincodes and say at least one courier service provides it's service to all of them means from each pincode to other 29999 pincodes so the mapping would be 30k x 30k in a table if you store all of them in one table. So now how to efficiently store the data. the columns are courier service id, origin pincode (seller's location) and destination pincode.
I think by pincode, he means like zip code or postal code?
@@jmoo4457 yes. In India, it is called pincode
Hi,
Also there is one more question in master slave architecture we can Ideally keep only one master right if write requests are less? We can avoid 2 master to avoid locking related problems. When master dies then one of slave becomes master now again if our old master is up then this will become slave. So that will solve the locking problem I had mentioned. Ideally master -master design works good when the write requests are high otherwise should the architecture mentioned above is good? Not sure very since I am new to system design there might be problems also feel free to put comment.
First like first comment!
Awesome!
Hi Gaurav, great video on the master-slave architecture. I have been asked a question about this in one of my interviews and wanted your views on the same.
As you mentioned in the master-master architecture when one master fails the requests are directed to the next master. The question to me was that how much time does it take to make this switch to the new master by the load balancer (How does the switching actually happens one layer down) and what happens to the requests which are coming during the interval of the switch? How are these requests getting managed as the earlier master is down and the new master is yet not up?
This has to do with service discovery. Let's see when we can talk about that đ
Hit Thumbs Up button before video even started
đ
don't you sleep in the night?
I like your hairstyle :p
Hahaha, thanks!
Thanks for the video Gaurav.
How do you implement this in real ? Like can you make a video on implementing this through code ?
I'll check it out. You could have a look at MySQL which provides master slave features till then đ
As Always good video ...but please dont include cartoon characters in between ,it breaks the flow of concentration .
is it just me or the audio isn't there..?
It's...there đ
@@gkcs yeah after a couple of refresh it came ..Thanks for the reply
I liked your video, if only I didnât have to hear master/slave words so many times. I like the language Yugabyte uses, leader/follower. đ
DDIA
Second
Close!
hey can you slow down your speed a bit. looks like its 1.3x instead of 1.0x. i know you get excited but if you can tone down your speed people can follow a bit better.
Try running it at 0.75?
0.75*1.3 is close to 1 :P
First
Wow!
Load Balancer looks like Corona Virus depiction.
The terms used in this video are not inclusive, this brings back memories of the past as a person of color.