Redis pub-sub vs Kafka? What to use for a chat application?
Vložit
- čas přidán 30. 06. 2024
- This video is about a specific question around System design of a chat application, which generally remains un-addressed unless you actually have a design discussion. The question is : In the publisher subscriber section, why did we use Redis? Why not Kafka?
The short answer is : Redis is the right choice for the use case of real time chat application.
There are 4 reasons
1. SPEED
Redis is in-memory , so it is faster. Where was Kafka has to make disk operations.
2. STORAGE REQUIREMENT
Redis does not store messages , they just pass through to all the consumers
So, you only need to increase the processing and not the storage. Where-as in Kafka you need both compute and storage.
3. COST OF CREATING A TOPIC
Kafka: Has extra cost because the partition needs to be created in all the Redis: no cost of creating a new channel. In fact no channel is created, you just send a message to a channel and you are good.
4. COST OF DELETING TOPIC
Deleting a topic in Kafka is not straight forward. Sometimes, delete topic operation takes long time and then you have to manually delete the topics from server.
COMMANDS
Redis:
You can install Redis on local by following this instruction: redis.io/docs/getting-started...
Start redis server: redis-server
connect to redis server: redis-cli
subscribe channel: SUBSCRIBE channel
publish command: PUBLISH channel1 "hey"
Observation : fast message sending and receive
Kafka
go to kafka folder downloaded from apache Kafka website: kafka.apache.org/downloads
start zookeeper: bin/zookeeper-server-start.sh config/zookeeper.properties
start kafka: bin/kafka-server-start.sh config/server.properties
create a topic: $ bin/kafka-topics.sh --create --topic quickstart-events --bootstrap-server localhost:9092
start consumer: bin/kafka-console-consumer.sh --topic quickstart-events --bootstrap-server localhost:9092
start producer: bin/kafka-console-producer.sh --topic quickstart-events --bootstrap-server localhost:9092
00:00 Intro
01:30 Redis & Kafka Basics
05:45 problem statement
07:08 Traditional solution
10:00 Demo and explanation
25:55 Conclusion
Follow me on Linkedin: / anubhavsri
You can also DM me if you need guidance as a software engineer. - Věda a technologie
Valuable information. Thanks
Keep it up 👍
great video !!
Thank you!
kafka stores data in disc cache page which is in memory and later flushed to actual disk.
Thanks for this really insightful video and very well presented and explained.
I have one question. Does redis pub sub scale horizontally, if we think about scaling this even further for millions of users using this system at the same time.
Yes . Checkout Sharded Pub/Sub section here: redis.io/docs/interact/pubsub/
Excellent Video. For Redis , How both the web socket services differentiate messages from various sessions of users. its the same one topic right?
No, they are not same topic. They are different channel. So, there is separate channel for each session of user.
Hi Anubhav
Thanks for creating this video. I have a question around persistence in Redis. What if Redis host crashes? Won't we loose all in-flight messages? Would you still prefer Redis?
Please correct me if I am wrong Redis can also store data on disk which should be helpful in recovery if the Redis host crashes but not sure about latency in that case. Is there any other significant difference between Redis and Kafka that we should consider before making the decision?
yes it can happens and so, we choose redis only for sending and receiving the data. Storage will done asyc.
we will not redis for storage but for real-time messaging.
main difference between kafaka and redis is that you cannot delete/create topics in kafka as frequently aa redis. you can see the reason in the video.
Good work Anubhav
So i am curious when should I use Kafka or RabbbitMq then?
Thanks Shirsh!
w.r.t chatting application you can also use RabbitMQ as well.
Kafka vs RabbitMQ is another topic I shall cover soon.
What will be the flow to rotate redis data to database mongodb means what will the time to do that should we do for every message as it produce or batches it and store after an interval?
I think you can put the data in Kafka. Thereafter it will be up to you to either batch consume or consume one by one and put in db.
@@anubhav_shrivastava then why don't you explained it into the video !!
one question big brother can i store my data in mongodb by using redis?
yes you can do that ! you will need to write your own code to do that!
@@anubhav_shrivastava thanks for reply
Are you available for custom development?
i did not get what exactly is the ask here. can you ping me on linkedin and we can have a quick chat?
*there is an option in redis as well for disk persistance!!!
Yeah there is an option for that too.
Consider Whatsapp scale and persistence to the messages. Does this scale?
Redis becomes.a bottleneck. U need to shard it, if pub-sub can scale like Kafka then it's good.
yeah persistence is a separate problem not covered here.
@@anubhav_shrivastava thanks for clarifying.
Scaling websockets horizontally is an expensive task thats the fundamental problem redis is solving here
+1, that's the message being conveyed here.
imagine millions of chat messages every second or minute. you think that's healthy for keeping in memory without causing memory overflow?
No, in such cases, we would horizontally scale our backend services. one single machine will not handle all the millions of connections.
Too slow and too long explanation
Thanks for the feedback! Will keep that in mind.
you can also navigate to the section you are interested in using the timeline feature if that helps.
@@anubhav_shrivastava Nah ignore him I think you did a good job of skipping the part that's redundant and tried to keep it concise. Besides, use playback speed/seek if you think it's too slow. However I would like to point out that the comparison between these two tech is a not apples to apples IMO. Redis stream would be much more akin to what Kafka is trying to do. It's probably also the reason why deletion isn't an issue in redis pub/sub because it follows fire and forget mechanism while Kafka is a streaming platform and has to have persistence layer in order to allow consumers to pick their own offset. But, nonetheless great video and great analysis with actual demo than just slides that I see on so many other videos.
Thanks Yojan. I really appreciate your encouragement and watching the full video.
I agree it is not apples to apples comparison. The reason i made this video is to convey that not all pub-sub problems are solved using Kafka and what are the other factors one should keep in mind when they choose a solution.
just watch in 1.75x and appreciate the efforts, i loved everything about this video