Reimplementing Apache Kafka with Golang and S3
Vložit
- čas přidán 28. 05. 2024
- This week on Developer Voices we’re talking to Ryan Worl, whose career in big data engineering has taken him from DataDog to Co-Founding WarpStream, an Apache Kafka-compatible streaming system that uses Golang for the brains and S3 for the storage.
Ryan tells us about his time at DataDog, along with the things he learnt from doing large-scale systems migration bit-by-bit, before we discuss how and why he started WarpStream. Why re-implement Kafka? What are the practical challenges and cost benefits of moving all your storage to S3? And would he choose Go a second time around?
-
WarpStream: www.warpstream.com/
DataDog: www.datadoghq.com/
Ryan on Twitter: x.com/ryanworl
Kris on Mastodon: mastodon.social/@krisajenkins
Kris on LinkedIn: / krisjenkins
Kris on Twitter: / krisajenkins
This is a terrific topic - thank you
Terrific is awfully close to terrifying
Your such a refreshing thought provoker. Really appreciate your channel and your guests. Curious on S3 to replace Kafka. Watching now.
My minio is going strong, I don't understand why that wouldn't be cloud native. Why does everything have to be some kind of thinly provisioned resource? The program still runs on a computer, and where exactly it runs is often super important.
Storage as a service, network as a service, cpu as a service, your mom as a service. I'm tired of this.
I feel so sorry for newcomers to programming having to learn this stuff, which noone tells them is completely useless for 99.99% of people, ending up not learning it anyways, because they would need to understand non-cloud in order to even realize why people are doing this.
Seldom ever do I feel this strongly about a topic. Now where's keys to my bare metal postgres cluster...
EDIT: great video, just can't fit more into this comment.
Same man, i blame Python. So many people start there and never progress into more capable languages so to them-frameworks are their way of contributing but there is just too damn much.
Never once has it ever been anything near seamless, i enjoy these videos but hearing the descriptions of how these tools “work” is always an eye roller.
You just know there’s a mountain of bu** sh** you will still have to decipher to make it work for you.
My major takeaway is how crazy expensive inter-AZ traffic is at AWS. 43% of the total cost of running Kafka...
Very useful for me as a dev dealing with Kafka and AWS
AWS MKS is not a good option?
More than a decade i always replicate and backup my audit logs through aws s3, via good old kafka-ish trigger based replication approach, similar to this.☺
I watched a few videos on this channel. This is the best channel about software engineering I've ever listened to. I'm like "wait, he didn't just ask that hard question?" then I'm "wait, the other one gave a great answer?". So much to learn here.
I’m half way through the video. No mention yet about Kinesis yet. Isn’t that an AWS scalable competitor to Kafka?
Класно, коли людина реально оцінює свої сили: "не граю, бо боюсь, що затягне"). Повага.
Россия победит
Classic move to maximize profitability ending up painting yourself into a corner.
have you heard about red panda guys??? just struggling to understand what's the point beyond "just having fun' ??
why? There is Kafka. There is Red Panda. Do you need more?
Why not? Someone wants to learn something, we should celebrate people wanting to learn and try new things. There is plenty of room to explore new approaches to same problems.
@@jc-aguilar and that's fine. I will prefer to use Kafka/Red panda regardless.
@@peppybocan - Build a wheel. Then use wheels bought from the store.
Build the wheel to appreciate and understand the wheel.
This is ~80% cheaper at the cost of some added latency.
Vocal fry
Why not Rust? Why Go?
Because Ryan said that his team knew Go inside out and they could, with some work, get the performance they wanted.
go lang does not have generics.... so not much use other than exe generation that graalvm in java can do too.
is it necessary to reimplement kafka?
Go got generics in 1.18... it's now at version 1.22.3. Keep up to date mate 😉
@@jazzycoder Interesting... Generics in functions???? when it does not have classes.. Go is calling primitive auto widening and narrowing as generics???
It's not a serious language for large scale systems. IT has its place when you want to write C type programs without having to deal with memory handling.
A co-routines without fine grained control of threads?? why would anyone build a server based on this language???
Its use case may be client-side native executable. Its not designed to be anything more. If you want server- the best language is C# and java. C# being better for now until java delivers project valhalla with value objects and universal generics.
Once Java delivers on this next year. It will be very hard to justify anything over Java for high level language.
@@curio78 You do realise Docker and K8s is programmed in Golang? It gets rid of all the bad OOP flaws such as inheritance & favors composition.
@@jazzycoder well that explains the difference isn't it. Java could host server projects for more than 20 years that was perfectly scalable. People deviated from it and now need abstractions and isolations to do exactly what java has been doing for decades.
Its like reinventing a round wheel with a square one and discovering all the problems that has already been solved.
People need to be told this clearly. so that they can finally get past the silliness of Java versbose so its bad jargon that has been going around to be blind to the solutions that has already present and has been reinvented to be solved all over again.
Its really getting silly. Go is good for client applications, but you want to build a proper server you have java. Unless you want to reinvent the wheel in the form of a crooked wheel.