Want to learn more Big Data Technology courses. You can get lifetime access to our courses on the Udemy platform. Visit the below link for Discounts and Coupon Code. www.learningjournal.guru/courses/
I have one quick question. In the Partitioner step, you mention that Producer decides partition. But isn't the Kafka / broker which knows the going partition number? How will producer know because the partition config is setup and resides on Kafka side. Great work with the videos... please keep making them.
The partitioner is simply a Java class, and it is part of Producer. So partition number is decided at producer side. If you look at the partitioner class, there is a parameter named "cluster" that can be used to get the available number of partitions from the cluster.
When a producer connects to Kafka, it makes an initial bootstrap connection. This connection can be to any of the servers in the cluster. The producer requests the partition and leadership information about the topic that it wants to publish to
How does a producer know that a transaction happened in the source application? How does it distinguish which are the new transactions and what it has already sent as messages to the broker? Sorry I am new to this...
IF the message is stored in partition buffer and waiting for the partition buffer to be filled does it mean the send method of producer is blocked ? Because the "RecordMetadata" object has to be returned by the send method. Doesn't it leads to problem of synchronous communication ?
We shouldn't set a key serialiser property if we are not sending the key to the broker right? If so, how key serialiser would be a mandatory parameter?
Every Kafka message is a key/value pair. However, the key could be NULL. Serializer is not mandatory if you don't want to send a key. However, key is so fundamental to Kafka messages that it doesn't make any sense to skip the key.
hello sir, your videos are really very helpful. I have one quick question. I want to use kafka in spark streaming using scala. Is Scala api is different from java api?
We don't need to worry about it. It's implemented in the API. Basically, all those issues that have a fault tolerance implementation are recoverable problems because the automatic fault tolerance feature will recover in few milli seconds.
Want to learn more Big Data Technology courses. You can get lifetime access to our courses on the Udemy platform. Visit the below link for Discounts and Coupon Code.
www.learningjournal.guru/courses/
I have look through several videos on Kafka yours have simplest and clearest examples and explanation. Please do more videos on other topics.
I am using this right away at work...best tutorials out there. Right to the point!
Hello Sir, you have explained the concepts in a very simple way . Thank you so much for your work..I look forward for more videos
Thank you for sharing your knowledge, tutorials are crisp and clear
Great tutorial..crisp and clear ..:)
I have one quick question. In the Partitioner step, you mention that Producer decides partition. But isn't the Kafka / broker which knows the going partition number? How will producer know because the partition config is setup and resides on Kafka side.
Great work with the videos... please keep making them.
The partitioner is simply a Java class, and it is part of Producer. So partition number is decided at producer side. If you look at the partitioner class, there is a parameter named "cluster" that can be used to get the available number of partitions from the cluster.
ok now its clear.
When a producer connects to Kafka, it makes an initial bootstrap connection. This connection can be to any of the servers in the cluster. The producer requests the partition and leadership information about the topic that it wants to publish to
Hi, thanks for the nice tutorial, do you have tutorial for Apache Zookeeper also???
loved it!
Well explained.
Sir, you are awesome!!
How does a producer know that a transaction happened in the source application? How does it distinguish which are the new transactions and what it has already sent as messages to the broker? Sorry I am new to this...
IF the message is stored in partition buffer and waiting for the partition buffer to be filled does it mean the send method of producer is blocked ? Because the "RecordMetadata" object has to be returned by the send method. Doesn't it leads to problem of synchronous communication ?
We shouldn't set a key serialiser property if we are not sending the key to the broker right? If so, how key serialiser would be a mandatory parameter?
Every Kafka message is a key/value pair. However, the key could be NULL. Serializer is not mandatory if you don't want to send a key. However, key is so fundamental to Kafka messages that it doesn't make any sense to skip the key.
hello sir, your videos are really very helpful. I have one quick question. I want to use kafka in spark streaming using scala. Is Scala api is different from java api?
The concepts are but API is different. Check this link. goo.gl/pq1qWT
Hi Sir, Thanks for ur video, i have a question..how does producer know if the error it received is recoverable?
We don't need to worry about it. It's implemented in the API. Basically, all those issues that have a fault tolerance implementation are recoverable problems because the automatic fault tolerance feature will recover in few milli seconds.