Spring Boot | Kafka Schema Registry & Avro with Practical Example and Implementation |
Vložit
- čas přidán 20. 01. 2024
- #JavaTechie #springboot #kafka #SchemaRegistry #Avro
📌 In this Video, we will learn about how to use Avro schema to produce and consume messages using Spring Boot. Additionally, we'll explore the concept of Schema Registry.
👉 What does it mean?
👉 what benefits does it provide?
👉 How schema evolution handled in schema registry ?
👉 Internal workflow
👉 Consumer & Producer example using spring boot
🧨 Hurry-up & Register today itself!🧨
Spring boot microservice Premium course lunched with 70% off 🚀 🚀
COURSE LINK : Spring boot microservice course link :
javatechie.ongraphy.com/cours...
PROMO CODE : JAVATECHIE50
Devops for Developers course link :
javatechie.ongraphy.com/cours...
COUPON CODE : NEW24
View All course : javatechie.ongraphy.com/
GitHub:
github.com/Java-Techie-jt/spr...
Blogs:
/ javatechie4u
Facebook:
/ javatechie
Join this channel to get access to perks:
czcams.com/users/javatechiejoin
🔔 Guys, if you like this video, please do subscribe now and press the bell icon to not miss any update from Java Techie.
Disclaimer/Policy:
📄 Note: All uploaded content in this channel is mine and it's not copied from any community, you are free to use source code from the above-mentioned GitHub account. - Věda a technologie
Excellent tutorial on real time scenarios. Avro serialization is one of the real time scenarios used in the projects and also an important topic in the Kafka interviews. Thanks a lot Basant for the efforts put in, much appreciated!!
Thanks Basant. Appreciate your efforts. God Bless you.🙂🙏 Every week waiting for your videos. You are my Guru. 😊 The way of your explanation is awesome.
Wowww
Awesome
Thank you very much, so helpfull!
Thanks for making such a detailed video about schema registry!!
Thanks for this tutorial.. your contribution to the developer community is invaluable.
Thank you Basant bhai.. Lucid explanation, loved it👍
Nice video you are putting lot of effort
Excellent sir no words to describe your knowledge sharing
thanks bro, pls do like this videos, pls keep ur good work
Thank you for a wonderful tutorial
❤❤ Thank you so much sir 🙏
Surpringly I had this kind requirement using avro object, many thanks
Glad it helped!
Nice video ❤
Hello sir,
I already followed you from last two years and you are awesome.
Could you please help me for one interview question on threading.
In you corrent project where you are using thread with example?
awsome
Could you please discuss more on Forward and Backward compatibility configurations.
Thanks, looking for this one. Please cover for Kafka connect also😊
Sure buddy
Can you also cover Kafka connect and stream
Hi Sir , I need to convert an input xml file to avro schema dynamically instead of having it under resource folder and then build the Avro object and send it to Kafka topic .can you please help me with this.
Thanks for the detailed information, But how to use different versions at consumer side while deserialising.
which data type we can use for date in schema , Pls suggest ?
Thanks for your useful tutorials.
Short question please: what is the difference between you first change to avro scheme, that when you send Employee information without age/dob was validate successfully , to your second change, when you added the middle name, the validation failed. I didn't notice any change in the process for the two changes...
Check I have explained in control centre ui the difference in schema evolution
Hi Basat ,
what is default compatiblity for avro schema registry .
This is awesome content thanks. One question , how do you share the schema across multiple micro services.
It should be store in registry
@@Javatechie Sorry i didn't get which registry you talking about. Could you elaborate further. Can we store it in like a config server? Just thinking..
Thank's Basant
Can you please make a video on how to make a class immutable in java?
can you guide me how to do kafka clustering in spring boot application how to handle
Currently we are using in our application.
everything's working well, but my control centre often shut down itself after some time, need to start it again from docker desktop, why can be the cause?
Nice explanation. Just have one doubt, can we use this when we are storing employee info in database like cassandra as database schema is fixed there. And if we want to accomodate new fileds we will have to manually add them.
But in the db schema you need to modify the field.
@@Javatechie yes, I just wanted to know couple of real life use cases of using avro schema
@@letsCherishCoding purpose of avro to avoid failure in schema evolution. Next you can think of any e-commerce or food delivery app who frequently do changes on their payload
@@Javatechie got it. Thanks
"Let's get started" has a separate fan base..
If consumer in different microsevice, I need to generate avro class for all the consumers right ?. In future, if we need to add new field, first we need to add default as empty, is my understanding correct ?.
Yes that’s correct but if your consumer is different project then you need to only get the updated schema from producer that’s it
Yes exactly because whatever version of avro Object producer is sending consumer is going to consume that object as per version in schema registry when deserializing
@@Javatechieif producer sends new schema, does consumer need to generate the dto files from consumer end before it consumes. Until the new changes deployed to prod, is it uses old schema ? How exactly schema versioning works for consumer without any break if consumer is a separate external service ?
I have a doubt what if we are not sure what kind of schema is coming from my database as my app might be handling different types of json data in that case Do i need to define schema for every json data one by one or is there any other way?
Schema is nothing is the contract between producer and consumer buddy. For example to work as IAS you must need to clear the UPSC exam that's the rule or you can say contract set by the government. If you have multiple exams then you need to prepare for different subjects like schema
hi sir can you please create a video on employee biometric login using Spring boot
Hi Sir,
How can we achieve this kind of version in open-source Kafka ?
Not possible supriya
Thanks a lot
nice tutorial but if u do this concept by creating two differnt projects(producer and consumer) then it wil be more clear and prefect
Yes I agree but to reduce the length of the video i kept both in the same project but what's the challenge here to separate consumers into different projects?
@@Javatechie i think we need to generate the beans class using avro schema in both producer and consumer application and then need to run both app. I thought if we separate producer and consume and if we change schema in producer it should reflect in consumer app without re run but seems not possible we have to re run both app. pls correct me if I'm wrong?
You are correct only need to build your app again that's the things you need to do
1. In confluent dashboard we can able to see the sesitive data, how to secure it in one interviewer asked me this question, please suggest, also confluent is free or paid.
2. If consumer is down producer is producing messages continuously and the hardware resources are exhUsted kafka how can handle this situation
Use secure Kafka cluster and yes it’s open source
@@Javatechie Thanks for the reply basant sir, actually content ex: credit card numbers, social security numbers etc. which should display in amsked manner, please suggest
consumers are configured with appropriate heartbeat settings
14:44
From 1:00 on you explain that in the plain Kafka world we need to build a new producer and consumer app once a class, whose objects we send to Kafka, changes.
From 41:00 on you generate a new Employee class based on the changed schema and then rebuild producer and consumer as well. From my understanding, using schemas we still need to build a new producer and consumer app; so this is not an advantage of the whole schema overhead.
The only advantage I see is that we do not manually change classes.
Do I overlook something?
i'm stuck in the same, still dont get it..
No your understanding is correct. It means we no need to rely on data like pojo/events even producer modified any payload structure still my consumer will not break
@@Javatechie I used everything same as you but when it came to evolving the schema and dropping some of the fields, I got change in schema error 409. I am wondering because I used same properties as you. Also, another thing is, I have the following dashboard I am not seeing the messages even though the schemas and the topics are there. I am talking about port 9092/clusters/** but the 8081/subjects/** is showing
This are the errors org.apache.kafka.common.errors.InvalidConfigurationException: Schema being registered is incompatible with an earlier schema; error code: 409
2024-04-23T09:29:46.988+03:00 ERROR 12236 --- [apache-schema-registry] [nio-8888-exec-1] o.a.c.c.C.[.[.[/].[dispatcherServlet] : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed: org.springframework.kafka.KafkaException: Send failed] with root cause
org.apache.kafka.common.errors.InvalidConfigurationException: Schema being registered is incompatible with an earlier schema; error code: 409
hi sir can you please create a video on employee biometric login using Spring boot