System Design: Apache Kafka In 3 Minutes
Vložit
- čas přidán 6. 09. 2023
- Get a Free System Design PDF with 158 pages by subscribing to our weekly newsletter: bytebytego.ck.page/subscribe
Animation tools: Adobe Illustrator and After Effects.
Checkout our bestselling System Design Interview books:
Volume 1: amzn.to/3Ou7gkd
Volume 2: amzn.to/3HqGozy
The digital version of System Design Interview books: bit.ly/3mlDSk9
ABOUT US:
Covering topics and trends in large-scale system design, from the authors of the best-selling System Design Interview series.
Thanks for the video mate now I can add Apache Kafka to my resume
😂
😂😂
😂😂😂
😂😂😂😂
😂😂😂😂😂
Really great overview - precise and succinct!
Great video! Precise and concise
Kafka is one of my favorite pieces of technology. I’ve successfully used it in several projects as a streaming queue and event bus, in a microservices setting, and it’s a joy to work with. Since it tracks what has been consumed with an offset, it greatly simplifies distributed, high-volume writes, and gives you great confidence in data consistency (eventually) ;) Highly recommend!
Do you know where the information "this consumer consumed the latest message SOME_ID from topic" is stored ? On consumer side or broker side ? If there is a temporary connection issue for example, I am trying to understand how the consumer can resume at the exact record id (mostly last record id processed +1)
Could you please help why not Rabbit MQ but KAFKA ?
Save that for your interview bruh
@@jackwinnfield It's not exactly meant to be used "sparingly". You either utilize it or you don't bother w/ the effort. As the video mentions, it's an effective data bus for microservices. If you're not decoupling your microservices properly, you're setting yourself up for a lot of pain as development scales out, as you'll have a lot of "spaghetti services" that can't be deployed independently, cause cascading failures when one link in the chain goes down, and generally loses the ability to be fault-tolerant. If you're making direct calls between your services and it's holding up well, you likely over-engineered to begin with, and don't truly need microservices architecture to begin with.
@@VincentJenks True, but if all you are using Kafka for is "an effective data bus for microservices", unless what you are working on is really big, very distributed, and for a big company, you can use something much more lightweight like RabbitMQ.
Most developers don't work for Uber or Netflix-scale companies, nor anywhere close. I agree with you though, Kafka is a fantastic product, but you need to require its full value proposition in order to justify using it in your system.
Yoy are awesome. thanks for sharing such a deep knowledge
Hello Alex,
I love your style of presenting. could you share which bunch of softwares you use for your lectures ?
0:58- "this sets it apart from simpler messaging systems". What sets it apart from simpler messaging systems? The fault tolerance that you mentioned right before that sentence? In what sense is it fault tolerant? By being distributed and holding messages across multiple nodes?
Great description. What software is used to do these diagrams
Excellent work how you make bits and packets moving and these animated flowgraphs? Which software?
thank you!
I really love your videos. I have subscribed to bytebytego and continue to learn from the content you share. I have one question about your video animation. What do you use to animate the system design animations in this video explanation of Kafka. I have a presentation and I would love to do something like that for my presentation. Thank you.
Powerful tool. I also wanna know
Literally in the video description.
Thanks
@fireship is sweating bullets right now
😂😂😂😂😂
😂😂
why?
Great overview! kudos! If Kafka should not be used for Low Latency then what is the best tech/tool to use for Low Latency Systems or Financial Markets Trading? I would appreciate if you could create a video on Low Latency System Designing?
You can remove the broker with something like UDP broadcasting from the producer side. There are ways to make it more resilient by implementing retries and so forth. However, it is not as resilient as apache Kafka. Like when the client and server crash, there is no easy way to know at what point it crashed and what data the consumer is missing.
Search for LMAX distruptor - it's a well documented example.
@@jackfinkel9434 I've worked with UDP-based protocols, in fact, I removed a UDP protocol with retries and so forth because it performed worse than TCP. There is no free lunch with UDP.
Idk, a lot of stock market feeds seem to use udp multicasting for its low latency: www.nasdaqtrader.com/content/technicalsupport/specifications/dataproducts/moldudp64.pdf
I would like to know how these types of animated videos are created
This was created with Kafka.
You can explain full details of Tomcat Apache service, please
How do you make these animated videos?
Hello, thanks for providing us with these fantastic presentations. I will be very 0:26 if you kindly send me these files. Anyway, it will be appreciated if you let me know the way and method these files are produced.
Celery with Java? or RabbitMQ with Java?
Thanks for the video, great explanation. Not sure if it’s correct to call it real-time streaming though, normally it would refer to websockets while event driven systems can be quite slow.
Nope, Web Sockets are used to communicate between server and browser.
"Real-time" is contextual and it's usage here is fine. It implies processing within a well known tolerance. If you were in a different domain such as microcontrollers, gaming or high frequency trading, then each of those would have a different meaning. I've worked in all those industries and used Kafka.
What's the difference of Kafka vs Any other Queue messaging system?
Hi , What kind of software to write dynamic architecture ?
Don't start this early... learn to walk first.
Anything about alternatives or competitors to kafka?
RedPanda
Google pub/sub
Rabbitmq
Flink
How and why am I subscribed to this channel and I didn't subscribe to here? I've been experiencing this quite a bit on CZcams. I really need to write to CZcams about this because I didn't subscribe and yet I got a notification and when I checked I'm subscribe to here.
Why the name. Is there a connection to the famous writer? I don‘t see any connection to one of his stories and I read quite a few of them. Probably no connection there.
Fireship joked that it's because "it's optimized for lots of writing", not sure if that's true.
please louder from next video sir
Correction: Apache Kafka In 4 Minutes
3:05 "this is also not suitable for ultra low latency applications like high frequency trading, where microseconds matter." can you elaborate this??
Why sound quality is low ?🤨
You can do the same for RabbitMQ.
Not quite. There are many advantages to Kafka, which you can find through searching for comparisons.
Can somebody help me learn Hadoop in simpler language !
What a shame, this video is about nothing. If it was supposed to be for engineers than you should probably talk about how it works under the hood (what is this topic, what it consist of, "replay" feature of kafka -commit log)
It’s not a tutorial, it’s a quick familiarization. Use it to direct your own learning.
What did you expect from a 3 minute long video? The time you wasted on writing your useless comment would have been more useful for you if you just looked for another video
@@24Shredder I do not expect something from video, I expect from people
This video is about nothing! 😡