ByteWax: Rust's Research Meets Python's Practicalities (with Dan Herrera)

Bytewax is a curious stream processing tool that blends a Python surface with a Rust core to produce something that’s in a similar vein to Kafka Streams or Apache Flink, but with a fundamentally different implementation. This week we’re going to take a look at what it does, how it works in theory, and how the marriage of Python and Rust works in practice…
-
The original Naiad Paper: dl.acm.org/doi/10.1145/251734...
Timely Dataflow: github.com/TimelyDataflow/tim...
Bytewax the Library: github.com/bytewax/bytewax
Bytewax the Service: bytewax.io/
PyO3, for calling Rust from Python: pyo3.rs/v0.21.2/
Kris on Mastodon: mastodon.social/@krisajenkins
Kris on LinkedIn: / krisjenkins
Kris on Twitter: / krisajenkins
--
0:00 Intro
2:06 Podcast
59:56 Outro
#softwaredevelopment #dataengineering #apachekafka #timelydataflow

[@TomPerry83]

Happy birthday Developer Voices! I've been enjoying your videos from (almost) day one. So many thanks for the great interviews, and I'm looking forward to the next 50 !

[@DeveloperVoices]

Thanks!

[@towel9245]

Happy birthday on the podcast! 👏

[@DeveloperVoices]

Thank you! 🥳

[@adicide9070]

would you care to have Jonathan Blow on the show?

[@DeveloperVoices]

Yes, he's definitely on my list. 🤞

[@hubstrangers3450]

"All the values in a given key, end up is same worker" - this is not a novel thought, it's been around prior 2010....

[@kasperjuulhermansen7524]

Hi Kris, I really enjoy your podcasts, but this one was quite hard to follow. Lots of interesting discussion but I'd difficulty picturing how to actually use bytewax or timely for that matter. The discussion jumped right into the nitty gritty and for the most part stayed in the abstract. And I didnt have a foundation to which I could tie the details into
Anyways, I really enjoy your podcast.
Your Data noob,

[@MichaelLehnGermany]

Yeah, I felt the same way. But what, in my opinion, makes this podcast so great and sets it apart from others is that it covers various specialties. And it doesn't just scratch the surface. Unfortunately, you rarely find something similar on CZcams. For this reason, I can cope with not being able to understand everything on some topics. I still enjoyed the episode (with admiration for the expert). Having said that, I am also looking forward for another low-level-C-like-Language topic ;-)

[@kasperjuulhermansen7524]

@@MichaelLehnGermany I agree. I find most of the topics covered on this channel quite approachable honestly. For example I think the Benthos episode did a good job of covering a lot of ground, also the more advanced topics of the tool. I did get exactly what you mention as well, which is a conversation between real engineers, not just enterviewers with surface knowledge.
Again keep up the good work Kris.

[@andrewdunbar828]

apostrophe's are hard

[@DeveloperVoices]

Thank you. Corrected. 🤦‍♂️

[@andrewdunbar828]

@@DeveloperVoices Glad it helped!