4. HLD SAGA Pattern | Strangler Pattern | CQRS | Microservices Design Patterns | System Design
Vložit
- čas přidán 17. 07. 2022
- ➡️ Join this channel to get access to member only perks:
/ @conceptandcoding
Hi,
This is the PART-2 video of Microservices Architecture and its Design patterns.
Its a very important Hight level design topic
#saga #systemdesign
In this video we have explained :
- what is Strangle Pattern
- What is SAGA Pattern
- What is CQRS pattern
-
Checkout the others series:
👉Interview Experiences : • Interview Experiences ...
👉Low Level Design: • Low Level Design from ...
👉High Level Design: • High Level Design from...
👉Interview Coding Questions: • Playlist
👉Talk with Employees: • Interview Experiences ...
LIKE | SHARE | SUBSCRIBE 😊
### **Introduction - Part 2**
- This video is a continuation of Part 1, which discussed the "Decompose by Medium Position Pattern".
- The video focuses on three important patterns:
- **Strangler Pattern:** Used for **refactoring** a monolithic service into microservices.
- **Saga Pattern:** Solves the problem of **distributed transactions** across multiple databases.
- **CQRS (Command Query Responsibility Segregation):** Separates command (write) and query (read) operations for improved performance and scalability.
### **Strangler Pattern:**
- **Purpose:** Gradually refactoring a monolithic application into microservices.
- **How it Works:**
- A "controller" is introduced to handle requests.
- Initially, the controller forwards all traffic to the monolithic application.
- Gradually, specific functionalities are extracted into microservices, and the controller routes traffic to them.
- As more functionalities are migrated, the controller forwards less traffic to the monolithic application, eventually strangling it.
- **Advantages:**
- Minimizes disruption to existing services.
- Allows for a gradual transition to microservices.
- **Example:**
- Imagine a monolithic e-commerce website being refactored into microservices.
- The controller initially directs all traffic to the monolithic website.
- Gradually, functionalities like order placement, inventory management, and payment processing are moved to individual microservices.
- The controller gradually routes more traffic to these microservices, eventually reducing the reliance on the monolithic application.
### **Data Management in Microservices:**
- **Two Approaches:**
- **Database for Each Individual Service:** Each microservice has its own dedicated database, promoting autonomy and isolation.
- **Shared Database:** All microservices share a single database, simplifying data access but potentially leading to complexities.
- **Why Database per Service is Preferred:**
- **Scalability:** Allows for independent scaling of individual services without impacting others.
- **Isolation:** Changes in one service's database don't affect others.
- **Technology Flexibility:** Services can choose different databases based on their specific needs.
- **Advantages of Shared Database**
- Join Query
- Transactional Property (ACID)
- **Drawbacks of Shared Database:**
- **Performance Bottlenecks:** Increased contention and performance issues as more services access the same database.
- **Complexity:** Managing dependencies and ensuring consistency across multiple services becomes difficult.
- **Limited Scalability:** Scaling the entire database is necessary, even if only one service needs more resources.
### **Saga Pattern:**
- **Purpose:** Managing distributed transactions across multiple databases, ensuring data consistency even if some operations fail.
- **How it Works:**
- A sequence of local transactions is executed within each participating microservice.
- Each transaction updates the database and publishes an event.
- Subsequent transactions listen to these events and continue the process.
- In case of failure, compensation events are published to undo completed operations and maintain consistency.
- **Types of Sagas:**
- **Choreography:** Each service manages its own transactions and listens to events from other services.
- **Orchestration:** A centralized orchestrator manages the transaction flow and handles compensation logic.
- **Example:**
- An order processing saga involving services for order creation, inventory management, and payment processing.
- If the payment service fails, compensation events are triggered to cancel the order and update inventory.
- **Advantages:**
- Guarantees data consistency in distributed systems.
- Provides a mechanism for handling failures and rollbacks.
- Allows for flexibility in service interactions.
- **Disadvantages:**
- Increased complexity compared to local transactions.
- Requires careful design and implementation to ensure correctness.
- **Interview Question Example:**
- Explain how you would handle a transaction involving transferring money between two users in a microservice architecture.
### **CQRS Pattern:**
- **Purpose:** Separating read (query) operations from write (command) operations for better performance and scalability.
- **How it Works:**
- The system maintains separate models for read and write operations.
- Write operations are performed through commands, updating the write model.
- Read operations access the read model, which can be optimized for fast retrieval.
- **Advantages:**
- **Performance Improvement:** Optimized read models can handle queries more efficiently.
- **Scalability:** Read and write models can be scaled independently based on their specific needs.
- **Flexibility:** Allows for different data structures and query languages for read and write operations.
- **Example:**
- A blog application where write operations are performed on a relational database, while read operations access a denormalized view optimized for fast search.
- **Challenges:**
- Maintaining consistency between the read and write models.
- Ensuring the read model is up-to-date with changes in the write model.
It was a direct question to me for EM's role in Junglee games, how to roll back the transition in Microservices, excellent explanation!
Thanks for making a highly theoretical topic like HLD so simple and interesting….Cheers man!! 🎉
Very well explained...awsome!!!
Amazing content and you made it so easy to understand. Just a friendly advice, consider writing more clearly while explaining.
thanks, sure will take care of it
Your teaching skills are just WOW. Thank you.
Thanks
Explained Very well .Thank You!
It great explanation very simple way. I failed 2 interviews because of system design questions. After that, I watched many videos for system designs but was not able to understand them. After watching your series get confidence and cracked 2 interviews with a good package. Thanks a lot, Concept & Coding ❤
Waiting for the actual implementation of these videos with examples.
Create a Series on depth implementation of each pattern with examples.
Glad to know that, sure
Hi, Great explanation for describing various microservices design principles....
Few queries -
Could you please explain the cycle that would be formed in case Choreography technique while implementing SAGA?
Also, could you please explain in detail how a VIEW will be created that has tables which span across various DB's?
these video series is best as of now . that too in hindi.
Hi @shrayansh, Great Explanation and series . Just an important request , Could you also put a video on Approaching Database design as well for a System Design round in depth?
Many thanks!
Very informative video. Will have to watch it multiple times for sure.
Glad to know that you liked the video
Thanks for providing such high quality content and it is totally free . Amazing
what a lesson eagerly waiting for questions and their solution implementation videos for concepts like SAGA and CQRS in spring/spring boot
sure thanks
Great explanation around microservices !!! Please cover the few HLD design questions as well.
Yes, its in my roadmap
Thank you! Very wonderful explanation.
Thanks
Great video. Keep going there are very less content on in depth microservices design and best practices.
Just want to see in depth implementation of each pattern with examples.
Waiting for more such good videos.
thanks Naman
Great way to explain - keep it up good work... thanks
Thanks
Very nicely explained. Keep up the good work!!
Thank you
Hi, great explanation feels like after a long time I found a place which explains SAGA and CQRS in human understandable way.
just one small doubt isn't CQRS Command Query "Responsibility" Segregation or is it "Request" segregation ?
Hi, I am just in love with your channel content. Following both LLD and HLD series and these are amazing. Complex topics explained in very simple/easy way ❤. Could you share some resources/books from where we can learn more in depth about HLD/LLD concepts.
Thanks.
Roadmap i have given Shalini.
Now internet is the resource :)
For CQRS -> How does the DB triggers/SP helps in maintaining the consistent state as the dbs are different?
Suppose service 1 with db 1 have 2 deletes , 1 updates and 3 create operations, how will it update the respective data using stored procedures for the database table situated in common database(different db instance)?
Thank you so much sir ❤. Please keep uploading more and more videos. Quality is top notch , very well explained 💯❤
Thank you, I will
very well explaied
one more benefit of microservice is that we can choose diff tech stack for diff microservices. If monolithic is implemented using python and it is not necessary to use same for microservice rather we can choose diff langauges as well for microservice implementation.
Hare Krishna Bhaiya!!!
Top Notch Video !!!
Thanks
Very well explained. will not forget it life time. Please make simple Example VIdeo for this SAGA Pattern
Thanks 👍
Hi Shrayansh, Thank you so much for the nice explanation. I have one doubt : Why should not a MS access Db of a different MS ,Whant can be the problems if we do so??
it was great Explanation 🎉 Thanks
Thanks
Thanks!! was great and informative video
thanks RG
Great Explaination Shreyansh, but in CQRS pattern where the view would b created, is it in which database? or view can be independent of database?
Thanks Shreyansh!!
Hi, I have couple of doubts in the explanation
1. In SAGA pattern, let's say, DB1 got updated by an event, but DB2 got failed, so , between the time when DB2 emit failed event and DB1 updates itself accordingly, if someone queried on DB1, then person would get stale/inconsistent data. How do we handle such cases?
2. In CQRS pattern, when you said we'll create a common/shared DB having content of all individual service DBs, doesn't that defeats the purpose of individual service level DBs, because that again will bring issues such as DB scaling, table modifications, etc. Also, as you said, individual service level DBs can be of different type depending upon requirement, like S1 could have SQL, S2 could have NoSQL, S3 could have ElasticSearch, etc. Then how would we aggregate these data of different schema / schema-less in a common DB?
Answer to your 1st query was explained by Shreyansh in a comment to another video. Below is the summary I deduced there.
Yes, In such case there could be a data consistency issue. And this can be resolved by maintaining a status column on database. So, until all commits are done, we should keep that status as “PENDING” and read call should not consider rows which are not marked as “SUCCESSFUL”.
So say when commit was successfully done on micro-service m1, it’s status could be “PENDING”. Let’s assume operation on micro-service m2 gets failed. It would publish an event and m1 will revert the transaction changing the status from “PENDING” to “FAILED”.
When transaction is successful, components will listen to events and change the status to “SUCCESSFUL”.
very very helpful
Thank you for sharing this. Just an idea to add maybe we can use read replicas of the microservices to create the common view?
Awesome video!! Just couple of queries
1. Didn't understand how there will be a cycle in the choreography pattern. Can you explain with an example.
2. In CQRS design, wouldn't the third DB (composition of service DB) face the same issues as faced by common DB architecture like scaling, common records modifications etc.
Thanks in advance!
Ack, will check and get back Sushant
DId you get answers to these questions?
@@ventorz5066 Nope
@@ventorz5066no
@@rechinraj111 same doubt , any one can help me with this ?
content mei dum toh hai..Just be consistent ,you will surely mass famous someday
:) thanks
Bro, is it used in the payment settlement system? If payment fails, then they check certain tables. I worked for the PP settlement team contractually. Trying to compare it. Any insights?
Sir , can you please share this sheet sharable link , that you have drawn and explained.
Awesome explanation
Thank you
Hi Sir, I Need one advice .
I am not able to corelate . .how organization using only graphql is impacted by existing microservices pattern
Thanks its helpful
Thanks
How come in choreographer services maintaining states itself (its kinda same in orchestrator)? Unable to understand cycle dependency part due to this!!
CQRS = Command Query Responsibility Segregation
How triggere and procedure know the changes occur in db1 and db2 if we are not using event
Hello Nice Explanation, One query in Choreography approach do our microservice need to continually pool queue for event? If yes then Interviewer can raise concern on performance as it need to continuously pool queue for events, please clarify.
Mostly the service which keeps listings to queue are demons (means Async Component).
They listen to the queue and if any event arrives they read it and then call the actual component which will act on it.
how many of these videos are in Hindi ? I just joined but need for English ?
Sir can you do one video on ecommerce system design esp related to how apis will be designed, how they communicate and how to handle api failures i.e. making it fault tolerant?
Noted
Hi Shreyansh, if before rollback (bcz of payment fail) in saga pattern, what if someone reads data from balance DB.
Then he will get wrong results , what should we do in that case.
Generally in saga pattern, generally status of txn keeps in pending till the dependent components get succeeded, if any of the dependent components failed, it will publish the fail event, which will be read by the component say balance and update it's status like rollback etc.
but what was the problem where we needed a common db for read in CQRS?
SAGA- Maintains ACID property
CQRS- For JOINS query
Thanks!
Thanks
Hey shreyansh,
if we make an api call in a spring boot jpa transactional block from say order service to inventory service and it fails, wont the local transaction on order service also roll back?
See if you are doing sequentially,
- yes but what if inventory success and order failed (suppose you are inserting in both parallely, order can get rollback but not inventory now)
@@ConceptandCoding right makes sense, thanks a lot !
Have a doubt, in saga pattern if any service say s3 goes down then it can't event publish failure events .In this case how will the other services would know that they have to rollback?
I think, if s3 is down then we can set retry for request. If retry cross certain threshold then update update event bus with KO.
Please provide some high level Notes at the end of Vedio. So that it can help to review while going to interview
Noted
@26:40, you said that as per SAGA pattern, if payment fails, the Payment service creates a fail event which will be consumed by the Balance service to rollback its transaction. But what if Payment service fails to even create a failEvent after failing in transaction, ie, how are we ensuring that an event will always be created corresponding to success or a failure?
Events are published in Queue like Kafka, if there are failure in publishing the event, msg still stays in the queue.
Pls check the Kafka video in HLD playlist itself. It will help you to understand how queue works
@@ConceptandCoding Okay, thank you.
Command Query Responsibility Segregation
CQRS means Command Query Responsibility Segregation and NOT request segregation
Ghotna😀😅🤣....AWSM Content.👍
Ghotna ? Sorry did not get it. But thanks
@@ConceptandCoding You did say ghotna when telling about strangle pattern
@@farazahmed7 ohh acha. Got it now. Strangle means Ghotna :D
Hii can you please guide me how can I apply in attlasian company cause I didn't get any response from there recruiter
Hi Neha, best way is to take out the job id and give it to engineering manager and HRs of the company
do you live near airport? heard a plane landing/taking off at 11:40
:)
Es lecture ke yeh notes bhi upload kar do sir pe please
Yes will do buddy
Hi
CORS = Cross Origin Resource Sharing, which is different from CQRS, you can't spell CQRS as CORS
My bad, thanks for pointing out.
@@ConceptandCoding that's ok, just thought of pointing out, rest of the content is spot on.
I guess great content, but awful english... man, i've never met such strong accent. I'm sure if you are not from India it's impossible to decipher. Do you mix english with hindi in this video? No disrespect man, but it's really really hard to listen (i dropped)
:) no problem buddy. Sorry for the inconvenience
Very nicely explained. Keep up the good work!!
Thank you
Very nicely explained. Keep up the good work!!
Thanks, will do!