Spring Boot | Kafka Schema Registry & Avro with Practical Example and Implementation |
Vložit
- čas přidán 20. 01. 2024
- #JavaTechie #springboot #kafka #SchemaRegistry #Avro
📌 In this Video, we will learn about how to use Avro schema to produce and consume messages using Spring Boot. Additionally, we'll explore the concept of Schema Registry.
👉 What does it mean?
👉 what benefits does it provide?
👉 How schema evolution handled in schema registry ?
👉 Internal workflow
👉 Consumer & Producer example using spring boot
🧨 Hurry-up & Register today itself!🧨
Spring boot microservice Premium course lunched with 70% off 🚀 🚀
COURSE LINK : Spring boot microservice course link :
javatechie.ongraphy.com/cours...
PROMO CODE : JAVATECHIE50
Devops for Developers course link :
javatechie.ongraphy.com/cours...
COUPON CODE : NEW24
View All course : javatechie.ongraphy.com/
GitHub:
github.com/Java-Techie-jt/spr...
Blogs:
/ javatechie4u
Facebook:
/ javatechie
Join this channel to get access to perks:
czcams.com/users/javatechiejoin
🔔 Guys, if you like this video, please do subscribe now and press the bell icon to not miss any update from Java Techie.
Disclaimer/Policy:
📄 Note: All uploaded content in this channel is mine and it's not copied from any community, you are free to use source code from the above-mentioned GitHub account. - Věda a technologie
Excellent tutorial on real time scenarios. Avro serialization is one of the real time scenarios used in the projects and also an important topic in the Kafka interviews. Thanks a lot Basant for the efforts put in, much appreciated!!
Thanks Basant. Appreciate your efforts. God Bless you.🙂🙏 Every week waiting for your videos. You are my Guru. 😊 The way of your explanation is awesome.
Thanks for making such a detailed video about schema registry!!
Thanks for this tutorial.. your contribution to the developer community is invaluable.
Thank you Basant bhai.. Lucid explanation, loved it👍
Excellent sir no words to describe your knowledge sharing
Thank you for a wonderful tutorial
Nice video you are putting lot of effort
❤❤ Thank you so much sir 🙏
thanks bro, pls do like this videos, pls keep ur good work
Nice video ❤
Surpringly I had this kind requirement using avro object, many thanks
Glad it helped!
Thanks, looking for this one. Please cover for Kafka connect also😊
Sure buddy
Hello sir,
I already followed you from last two years and you are awesome.
Could you please help me for one interview question on threading.
In you corrent project where you are using thread with example?
Thanks for the detailed information, But how to use different versions at consumer side while deserialising.
Can you also cover Kafka connect and stream
Currently we are using in our application.
Hi Basat ,
what is default compatiblity for avro schema registry .
Hi Sir , I need to convert an input xml file to avro schema dynamically instead of having it under resource folder and then build the Avro object and send it to Kafka topic .can you please help me with this.
"Let's get started" has a separate fan base..
Thank's Basant
which data type we can use for date in schema , Pls suggest ?
Thanks for your useful tutorials.
Short question please: what is the difference between you first change to avro scheme, that when you send Employee information without age/dob was validate successfully , to your second change, when you added the middle name, the validation failed. I didn't notice any change in the process for the two changes...
Check I have explained in control centre ui the difference in schema evolution
everything's working well, but my control centre often shut down itself after some time, need to start it again from docker desktop, why can be the cause?
Can you please make a video on how to make a class immutable in java?
Nice explanation. Just have one doubt, can we use this when we are storing employee info in database like cassandra as database schema is fixed there. And if we want to accomodate new fileds we will have to manually add them.
But in the db schema you need to modify the field.
@@Javatechie yes, I just wanted to know couple of real life use cases of using avro schema
@@letsCherishCoding purpose of avro to avoid failure in schema evolution. Next you can think of any e-commerce or food delivery app who frequently do changes on their payload
@@Javatechie got it. Thanks
can you guide me how to do kafka clustering in spring boot application how to handle
nice tutorial but if u do this concept by creating two differnt projects(producer and consumer) then it wil be more clear and prefect
Yes I agree but to reduce the length of the video i kept both in the same project but what's the challenge here to separate consumers into different projects?
@@Javatechie i think we need to generate the beans class using avro schema in both producer and consumer application and then need to run both app. I thought if we separate producer and consume and if we change schema in producer it should reflect in consumer app without re run but seems not possible we have to re run both app. pls correct me if I'm wrong?
You are correct only need to build your app again that's the things you need to do
hi sir can you please create a video on employee biometric login using Spring boot
I have a doubt what if we are not sure what kind of schema is coming from my database as my app might be handling different types of json data in that case Do i need to define schema for every json data one by one or is there any other way?
Schema is nothing is the contract between producer and consumer buddy. For example to work as IAS you must need to clear the UPSC exam that's the rule or you can say contract set by the government. If you have multiple exams then you need to prepare for different subjects like schema
If consumer in different microsevice, I need to generate avro class for all the consumers right ?. In future, if we need to add new field, first we need to add default as empty, is my understanding correct ?.
Yes that’s correct but if your consumer is different project then you need to only get the updated schema from producer that’s it
Yes exactly because whatever version of avro Object producer is sending consumer is going to consume that object as per version in schema registry when deserializing
@@Javatechieif producer sends new schema, does consumer need to generate the dto files from consumer end before it consumes. Until the new changes deployed to prod, is it uses old schema ? How exactly schema versioning works for consumer without any break if consumer is a separate external service ?
Thanks a lot
Hi Sir,
How can we achieve this kind of version in open-source Kafka ?
Not possible supriya
From 1:00 on you explain that in the plain Kafka world we need to build a new producer and consumer app once a class, whose objects we send to Kafka, changes.
From 41:00 on you generate a new Employee class based on the changed schema and then rebuild producer and consumer as well. From my understanding, using schemas we still need to build a new producer and consumer app; so this is not an advantage of the whole schema overhead.
The only advantage I see is that we do not manually change classes.
Do I overlook something?
i'm stuck in the same, still dont get it..
No your understanding is correct. It means we no need to rely on data like pojo/events even producer modified any payload structure still my consumer will not break
@@Javatechie I used everything same as you but when it came to evolving the schema and dropping some of the fields, I got change in schema error 409. I am wondering because I used same properties as you. Also, another thing is, I have the following dashboard I am not seeing the messages even though the schemas and the topics are there. I am talking about port 9092/clusters/** but the 8081/subjects/** is showing
This are the errors org.apache.kafka.common.errors.InvalidConfigurationException: Schema being registered is incompatible with an earlier schema; error code: 409
2024-04-23T09:29:46.988+03:00 ERROR 12236 --- [apache-schema-registry] [nio-8888-exec-1] o.a.c.c.C.[.[.[/].[dispatcherServlet] : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed: org.springframework.kafka.KafkaException: Send failed] with root cause
org.apache.kafka.common.errors.InvalidConfigurationException: Schema being registered is incompatible with an earlier schema; error code: 409
1. In confluent dashboard we can able to see the sesitive data, how to secure it in one interviewer asked me this question, please suggest, also confluent is free or paid.
2. If consumer is down producer is producing messages continuously and the hardware resources are exhUsted kafka how can handle this situation
Use secure Kafka cluster and yes it’s open source
@@Javatechie Thanks for the reply basant sir, actually content ex: credit card numbers, social security numbers etc. which should display in amsked manner, please suggest
consumers are configured with appropriate heartbeat settings
14:44
hi sir can you please create a video on employee biometric login using Spring boot