What is the Listen to Yourself Pattern? | Designing Event-Driven Microservices
Vložit
- čas přidán 30. 06. 2024
- ► LEARN MORE: cnfl.io/microservices-101-mod...
The Listen to Yourself pattern is a technique to improve latency, while ensuring data consistency in microservices. It is implemented by having a microservice emit an event to a platform such as Apache Kafka, and then consuming its own events to perform internal updates. Since it separates Kafka and database writes into different processes, it can be used to avoid data consistency issues caused by the dual-write problem. However, it also allows microservices to respond quickly to requests by deferring processing to a later time.
To learn more about the dual-write problem, check out the Designing Event-Driven Microservices course on Confluent Developer: cnfl.io/microservices-101-mod...
RELATED RESOURCES
► What is the Dual Write Problem?: • What is the Dual Write...
► What is the Event Sourcing Pattern?: • What is the Event Sour...
► What is the Transactional Outbox Pattern?: • What is the Transactio...
► Microservices course playlist: bit.ly/designing-event-driven...
► Microservices: An Introduction cnfl.io/3ZMt3up
► Event-Driven Microservices Architecture: cnfl.io/48FSYbj
► Migrate from Monoliths to Event-Driven Microservices: cnfl.io/3tsqlhu
► Get Started on Confluent Developer: cnfl.io/48FnKRB
CHAPTERS
00:00 - Intro
01:07 - What is the dual-write problem?
01:30 - What is the listen-to-yourself pattern?
01:51 - How does the listen-to-yourself pattern eliminate dual writes?
02:24 - When are the events processed in the listen-to-yourself pattern?
02:55 - Is the listen-to-yourself pattern eventually consistent?
03:43 - How can we deal with eventual consistency?
04:14 - How do we validate events?
05:02 - Closing
--
ABOUT CONFLUENT
Confluent is pioneering a fundamentally new category of data infrastructure focused on data in motion. Confluent’s cloud-native offering is the foundational platform for data in motion - designed to be the intelligent connective tissue enabling real-time data, from multiple sources, to constantly stream across the organization. With Confluent, organizations can meet the new business imperative of delivering rich, digital front-end customer experiences and transitioning to sophisticated, real-time, software-driven backend operations. To learn more, please visit www.confluent.io.
#microservices #apachekafka #kafka #confluent - Věda a technologie
Wade here. Funny story. When I started building my videos on microservices I considered a section on this pattern and ended up rejecting it. My thinking was that nobody really uses this pattern and so it wasn't really relevant. But, as I was building out the content for other videos I realized that I have used this pattern myself, multiple times. I used it when working on a Real-Estate platform to reduce the response time of certain calls. I used it when working on a Fitness Tracker for similar reasons. It's actually an incredibly valuable pattern and one that I reach for often. I suspect a lot of people use this pattern more than they realize.
It's very similar to the Queue-based Load Levelling pattern.
Lots of domains where the write can be considered a bit more fire-and-mostly-forget or it's just unlikely that a system or user will be immediately fetching it.
@@vanthel Wade here. Absolutely. There are plenty of perfectly valid use cases for this type of pattern. If your domain allows for it, it's a great solution.
@@lkazzzz Wade here. I'm not sure I see the similarities. Perhaps you can elaborate.
Thanks a lot for the video. With the Listen to Yourself pattern, it implies that when our microservice performs a calculation or similar operation, the resulting data should be included in the event payload. This enables the service to rely on the data within the event payload to update its own database when it listens to itself.
In this setup, I feel it's crucial to pay careful attention to the message bus configuration, particularly aspects like order delivery guarantees.
Alternatively, employing event sourcing might be beneficial to provide a mechanism for reconciliation in case of discrepancies. For instance, if the process involves calculating a deposit check, merely having a snapshot of the calculation in the event payload might be insufficient.
Wade here. Depending on the domain, ordering might be critical. If it is, you are definitely going to want to pay attention to how you configure that. In Apache Kafka, you'd need to be careful of your partitions and partition keys to ensure you maintain the necessary ordering.
Event sourcing is a good solution to the dual-write problem on its own. It provides many of the same benefits as the Listen to Yourself pattern. However, it's not likely to be as fast as the Listen to Yourself pattern when it comes to responding to the sender.
The Transactional Outbox pattern (with CDC) always works nicely as a way to avoid the Dual Write problem!
Wade here. Absolutely. The Transactional Outbox is another good way to solve this problem. It just depends on what your needs are. If you want more details you might check out our video on the Transactional Outbox Pattern here: czcams.com/video/5YLpjPmsPCA/video.html or learn more about the Dual Write Problem here: czcams.com/video/FpLXCBr7ucA/video.html
I believe that outbox is just straight up slower and sending an event right away is more perfomant. Do I get it right?
@@Fikusiklol but you may run into dual write problem without oubox
@@MaxingMix Yeah, I get it. I usually just stick to outbox, cause there are no perfomance issues in my domain. Thanks!
@@ConfluentDevXTeam You've thought of everything! 😄 Thanks for sharing and for the great content, Wade.
Thank you so much for sharing this with us on CZcams ❤️🔥 from SeoulNatU
Wade here. You're welcome. I'm glad you enjoyed it. Nice to see people from all over the world enjoying the content.
How is it different than the saga pattern
Wade here. The Saga pattern operates using compensating transactions. If one of the writes fails, then you execute the appropriate compensations on all previous writes to roll back to your initial state. So if your operation was to deposit $10 into a bank account, and the original DB write succeeded, but a later operation (say writing to Kafka) failed, then you would compensate by issuing a correction on that $10 original write. Essentially, Sagas operate using "rollback" mechanisms. Technically they aren't rollbacks because you never actually "undo" the previous state. You just add new data to compensate for the previous changes.
The Listen to Yourself pattern doesn't use compensating actions. Once that original write to Kafka succeeds, you will eventually have to make everything after it succeed as well. This is more of a "roll forward" situation. Once you start the writes, you have to figure out how to keep moving forward with it because you can't really "undo" the initial event.