Apache Kafka Crash Course - Part 1

This comprehensive guide will walk you through Apache Kafka's core concepts, including topics, partitions, producers, consumers, and its role in data streaming. We'll break down how Kafka acts as a high-throughput, distributed messaging system for handling real-time data feeds. You'll learn how producers publish messages to specific topics, while consumers subscribe to those topics to receive the data stream. We'll also explore partitions, which subdivide topics into manageable units, and offsets, which keep track of the position of a consumer within a partition.
By the end of this video, you'll have a solid understanding of the fundamentals of Apache Kafka and be ready to explore more advanced topics.
#apachekafka
LinkedIn: / bytemonk
System Design Interview Basics Playlist:
► • System Design Intervie...
AWS Certification:
►AWS Certified Cloud Practioner: • How to Pass AWS Certif...
►AWS Certified Solution Architect Associate: • How to Pass AWS Certif...
►AWS Certified Solution Architect Professional: • How to Pass AWS Certif...

[@parthsolanki7878]

I like your explanation style. Would be great if you can create a video on Kafka's zero copy. Thanks.

[@basuta-dshrara]

You have the best tutorials, i love this style. Amazing way of explaining topics.

[@satishkumar-vo1kv]

Thanks alot for the eloborated video. Waiting for the next one.

[@sjsh3hee815]

was looking for this, Thanks for the upload

[@indraalapati989]

This session is helpful and looking for next part. I have question about consumer, is the consumer an end client ? Let's take Uber as an example, where the ride requestor wants to know the position of a driver and the position of a driver is streamed to a kafka broker and end user is subscribed to that broker. Can we assume the ride requestor who is the end client as a consumer ?

[@rahul_singh_rajput3292]

waiting for the real world example 🤝

[@pavanbadsha]

Can u create a topic on click or journey tracking for mobile and web applications where there are different journeys

[@mahendragundeti]

Thanks for sharing kafka video. we are using msk from aws and we are migrating data to other mks graviton cluster using mirrormaker2 but we are facing issue with offset translation issue as offset is not translating to new cluster even after using connectors. if you have experience in this can you suggest any solutions.

[@ByteMonk]

This is a common problem, and there can be multiple reasons, I can think of few here, but you should first start with double checking the logs, version compatibility, and configurations.
1. You may first like to examine the MirrorMaker2 logs for any errors related to consumer group offsets or offset translation. These logs might provide clues about the root cause.
2. Ensure you have enabled automated consumer offset sync in MirrorMaker2. Starting with Apache Kafka 2.7, this is achieved by setting sync.group.offsets.enabled and emit.checkpoints.enabled properties to true in your configuration. This allows MM2 to translate and synchronize offsets during data replication.
3. Also verify that MirrorMaker2 is creating the consumer group on the destination cluster. If you've pre-created the group with a different ID, it might lead to offset translation issues
4. If you're using a custom replication policy, double-check its configuration for handling consumer group offsets. Ensure it's designed to translate offsets from the source to the destination cluster.
Good luck!