kafka vs active Mq , IBM MQ ,Rabbit MQ , JMS | Kafka Spark Interview Questions
Vložit
- čas přidán 21. 06. 2018
- As part of our kafka and spark Interview question Series, we want to help you prepare for your kafka and spark interviews. We will discuss various topics about spark and kafka
As part of this video we are covering what is different between Kafka and traditional queue based brokers like active mq , ibm mq,rabbit mq etc . what are advantages of kafka, when to use kafka
Please subscribe to our channel.
Here is link to other spark interview questions
• 2.5 Transformations Vs...
Here is link to other Hadoop interview questions
• 1.1 Why Spark is Faste...
IBM MQ also provide distributed , PubSub oriented, Ordered & guaranteed delivery of messages. It jus that once read message won't be available there in IBM MQ, where as it says in Video Kafka can retain the message for particular amount of time.
Just want to elaborate on the ordering of messages part:
>>Kafka only provides a total order over records within a partition, not between different partitions in a topic.
Imagine you have 2 partitions P0 and P1 which are replicated across two brokers (nodes) B0 and B1.
Now to see why the ordering of the messages between partitions (or in other words ordering of messages for a topic) is not maintained, imagine the following:
A producer produces two messages R0, R1 in this order. Assume that R0 got written to P0 and R1 to P1. Now while reading, the consumer can read from either of the partitions first, since Kafka doesn't specify which partition you are going to read from (each partition is dynamically assigned to one consumer from the consumer group), which can change the ordering of messages.
With that said, there are still ways to ensure the ordering of messages. One way is to use only 1 partition for each topic, so that all the writes/reads go through a single partition which can ensure the ordering. Another is to explicitly add a timestamp to your messages.
Also, I am no expert in Kafka, so please correct me if I am wrong. And great work Harjeet, keep it up! :-)
References: sookocheff.com/post/kafka/kafka-in-a-nutshell/
You are Spot on Diptanshu... I have clarified in one of comments below. Hoever I am not able to update it in video... Completely accepted point
Order of the messages ca be guaranteed , if we can send messages based on the keys , Lets say product number as key then always specific product will go to specific partition so ordering can be still achieved . for ordering is single partition is not good option but using keys is the good option.
Dude I have worked with jms, ur statement that jms deletes the message and Kafka doesn't is absolutely wrong...
Message queues irrespective of the retention policy will not remove the message from the queue untill they receive an acknowledgement message from the consumer. And this applies to both Kafka and active mq. This is how messaging systems guarantee 100% delivery!!
Hi Ajit. you are absolutely right.. Excuse me if my explanation created confusion. I meant to say that Kafka stores message for longer duration. Kafka retains message even after the successful delivery to consumer. So helps Kafka to be used for reconsilation jobs, which can help to recover from any functional discrepancy in data.
Last statement multiple consumers for jms doesn't require duplicate topics.. if you change consumer user id it will treat as a new consumer. Multiple consumers is possible in jms.
Good video with indepth content, fantastic work
Nice and to the point, thanks!
Thanks. Precise content!
Thanks... happy it is useful for u
Very informative video. Keep up the good work!
All your points on first slide can be achieved by IBM MQ, please add more detail to clarify each product separately. Title of video is misleading, Retention/Persistence, Ordering, Multi consumer, PubSub(using IBM MQ Topics) are part of IBM MQ
Thank you for making such wonderful videos. Could you please make some videos about interview questions related to airflow and AWS related to big data?
Sure Paresh... I am restarting the spark interview series, I will cover these topics as part of that
Good job !!
There are many MQ Server implementation available, such as IBM MQ, RabbitMQ, Apache ActiveMQ, and Kafka so on. but wich mq is best to develop the microservice application
answer will be very specific to your usecase. it is difficult to answer this without knowing details
I have another question, when are we going to choose messaging queues over kafka. could you explain this
When your load is very low... Data is consumed as soon as it is generated... You don't need any persistence... Don't want to manage a distributed system to keep system complexity low... Use traditional jms
Hadoop Spark and Big Data Tutorials Thanks Harjeet
rabbit mq also has persistence facility @Data savvy
Can you make videos on Kafka with java code?
I dindt get the uber request to match .....?
I have a Uber design video which shall help you understand this concept more deeply... Please refer that
I found only difference is replay n retention of messages in a queue If you compare to IBM MQ. All banks in the world uses IBM MQ where trillion of transaction happens every day. That to money!!! Secured Assured delivery no duplicate message .IBM MQ Series is in use since from 90s n Kafka started 2011.
IBM MQ also provides message retention however message can be read by application only one time
If we use Topic in ActiveMQ, I think it also send message to multiple consumers.
Hi Jitendra, all features of topic etc are added in activemq in recent years after popularity of Kafka. Activemq was actually a implementation JMS specification... Please share what are your thoughts on this
perfect video to the point.
Thanks mate