Bringing Pure Python to Apache Kafka (with Tomáš Neubauer)

The “big data infrastructure” world is dominated by Java, but the data-analysis world is dominated by Python. So if you need to analyse and process huge amounts of data, chances are you’re in for a less-than-ideal time. The impedance mismatch will probably make your life hard somehow.
So there are a lot of projects and companies trying to solve that problem. To bridge those two worlds seamlessly, and many of the popular solutions see SQL as the glue. But this week we’re going to look at another solution - ignore Java, treat Kafka as a protocol, and build up all the infrastructure tools you need with a pure Python library. It’s a lot of work, but in theory it would make Python the one language for data storage, analysis and processing, at scale. Tempting, but is it feasible?
Joining me to discuss the pros, cons, and massive scope of that approach is Tomáš Neubauer. He started off doing real time data analysis for the Maclaren’s F1 team, and is now deep in the Python mines effectively rewriting Kafka Streams in Python. But how? How much work is actually involved in porting those ideas to Python-land, and how do you even get started? And perhaps most fundamental of all - even if you succeed, will that be enough to make the job easy, or will you still have to scale the mountain of teaching people how to use the new tools you’ve built? Let's find out.
-
Quix Streams on Github: github.com/quixio/quix-streams
Quix Streams getting started guide: quix.io/get-started-with-quix...
Quix: quix.io/
Tomáš on LinkedIn: / tom%c3%a1%c5%a1-neubau...
Tomáš on Twitter: / tomasneubauer0
Kris on Mastodon: mastodon.social/@krisajenkins
Kris on LinkedIn: / krisjenkins
Kris on Twitter: / krisajenkins
--
#podcast #softwaredevelopment #datascience #apachekafka #streamprocessing

[@JavierBlancoCordero]

great questions as always! great answers too! would love to watch a second part of you two!

[@MichaelRosam-no4zt]

Python as a low barrier, high ceiling stream processing tool for the masses, Gleam as a stream processing tool for Kris!

[@MrHopp24]

Thanks for another great episode!

[@Cha053ngine]

Fantastic thanks!

[@lukashei1870]

Very interesting talk, thanks
Btw, not sure you're already aware but the Spotify version of this episode has somewhat broken sound.

[@DeveloperVoices]

Dang! Thanks for letting me know. Not sure what happened there - it should be exactly the same as the video version, but clearly not. I've just shipped a fix. Thanks again!