Developer Voices
Developer Voices
  • 116
  • 720 036
PyO3: From Python to Rust and Back Again (with David Hewitt)
There’s huge pressure on Python at the moment to get faster, ideally without changing at all. One increasingly-popular way of achieving that impossible task is to push the performance critical code down into C, C++, or Rust. And this week we’re focussing on the Python route, as we take a look at PyO3.
David Hewitt’s the principal committer to PyO3, and he joins us to go through the easy parts, the hard parts, and the works in progress, giving us an insight into how Python and Rust work under the hood, and quite how much work it takes to make them work as one.
-
Become a Supporter on Patreon: patreon.com/DeveloperVoices
Become a Supporter on CZcams: www.youtube.com/@DeveloperVoices/join
PyO3 User Guide: pyo3.rs/v0.22.0/
PyO3 on Github: github.com/PyO3/pyo3
Polars: pola.rs/
Tokio: tokio.rs/
Trio: trio.readthedocs.io/
Robyn: github.com/sparckles/Robyn
Faster CPython: github.com/faster-cpython
Maturin: www.maturin.rs/
David on Mastodon: fosstodon.org/@davidhewitt
David on Twitter: x.com/davidhewittdev
Kris on Mastodon: mastodon.social/@krisajenkins
Kris on LinkedIn: www.linkedin.com/in/krisjenkins/
Kris on Twitter: x.com/krisajenkins
--
0:00 Intro
3:09 How David Got Involved in PyO3
7:01 So Starting With Python To C...
9:20 Taking That To Rust
17:37 Calling Rust from Python
20:54 Understanding PyO3's Place In The Stack
21:52 Memory Safety
30:37 Rust's Lifetimes
35:07 Applying Lifetimes to Python Code
38:24 Let's Get Into Error Handling
55:55 The Performance Cost
1:01:19 Threading
1:16:05 David's Motivations
1:19:22 Supporting Python's Sub-Interpreters
1:24:14 The Limits of Compile-Time Guarantees
1:27:32 Getting Started with PyO3
1:33:01 Outro
zhlédnutí: 4 806

Video

Inter-System Messaging with NATS & Jetstream (with Jeremy Saenz)
zhlédnutí 3,3KPřed dnem
Most message systems have an opinion on the right way to do inter-systems communication. Whether it’s actors, queues, message logs or just plain ol’ request response, nearly every tool has decided on The Right Way to do messaging, and it optimises heavily for that specific approach. But NATS is absolutely running against that trend. In this week’s episode, Jeremey Saenz joins us to talk about N...
Cuis Smalltalk and the History of Computing’s Future (with Juan Vuletich)
zhlédnutí 6KPřed 14 dny
Smalltalk is one of those programming languages that’s lived out of the mainstream, but often referenced as an influence and an important part of programming history. It’s the cornerstone of object-oriented programming, it was into message passing before actors were cool, and it blurs the line between operating system, programming language and personal notebook. But what is it? Joining us to di...
The Inko Programming Language, and Life as a Language Designer (with Yorick Peterse)
zhlédnutí 3,1KPřed 21 dnem
This week we take a close look at the programming language Inko from two perspectives: The language design features that make it special, and the realities of being a language developer. Yorick Peterse joins us to discuss why he’s building Inko, and which design sweetspots he’s looking for. We begin with memory management, aiming for the kind of developer who wants control, but without the comp...
Building the Zed Text Editor (with Nathan Sobo)
zhlédnutí 17KPřed měsícem
I’ve often wondered how you build a text editor. Like many software projects, it’s a simple idea at the core with an almost infinite scope for features. How do you build a solid foundation to expand on? Which features matter for launch? And how do you hope to satisfy the needs of every programmer, working in every language? My guest for this episode is Nathan Sobo. He’s tackled this problem onc...
Reimplementing Apache Kafka with Golang and S3
zhlédnutí 7KPřed měsícem
This week on Developer Voices we’re talking to Ryan Worl, whose career in big data engineering has taken him from DataDog to Co-Founding WarpStream, an Apache Kafka-compatible streaming system that uses Golang for the brains and S3 for the storage. Ryan tells us about his time at DataDog, along with the things he learnt from doing large-scale systems migration bit-by-bit, before we discuss how ...
Extending Postgres for High Performance Analytics (with Philippe Noël)
zhlédnutí 5KPřed měsícem
PostgreSQL is an incredible general-purpose database, but it can’t do everything. Every design decision is a tradeoff, and inevitably some of those tradeoffs get fundamentally baked into the way it’s built. Take storage for instance - Postgres tables are row-oriented; great for row-by-row access, but when it comes to analytics, it can’t compete with a dedicated OLAP database that uses column-or...
Designing Actor-Based Software (with Hugh McKee)
zhlédnutí 8KPřed měsícem
The actor model is a popular approach to building scalable software systems. And isn’t hard to understand when you’re just reading about the beginner’s examples. But how do you architect a complex design using the actor model? Which patterns work well? How do you think through it? Joining me to take us through it is Hugh McKee. Hugh’s a total actor-model fan, and a Developer Advocate for Lightb...
ByteWax: Rust's Research Meets Python's Practicalities (with Dan Herrera)
zhlédnutí 3,7KPřed měsícem
Bytewax is a curious stream processing tool that blends a Python surface with a Rust core to produce something that’s in a similar vein to Kafka Streams or Apache Flink, but with a fundamentally different implementation. This week we’re going to take a look at what it does, how it works in theory, and how the marriage of Python and Rust works in practice… - The original Naiad Paper: dl.acm.org/...
Mojo Lang - Tomorrow's High Performance Python? (with Chris Lattner)
zhlédnutí 41KPřed 2 měsíci
Mojo is the latest language from the creator of Swift and LLVM. It’s an attempt to take some of the best techniques from CPU/GPU-level programming and package them up in a Python-compatible syntax. In this episode we explore why the Mojo programming language was created, and what it offers to Python programmers and non-Python programmers alike. How is it built for performance, and which perform...
Batch Data & Streaming Data in one Atom (with Jove Zhong)
zhlédnutí 1,7KPřed 2 měsíci
Every database has to juggle the need to process new data and to query old data. That task falls to any system that “does stuff and remembers stuff”. But it’s quite hard to really optimise one system for both use cases. There are different constraints on new and old data, and as a system gets larger and larger, those differences multiply to breaking point. That’s something Twitter’s engineers w...
Advanced Memory Management in Vale (with Evan Ovadia)
zhlédnutí 5KPřed 2 měsíci
Rust changed the discussion around memory management - this week's guest hopes to push that discussion even further. This week we're joined by Evan Ovadia, creator of the Vale programming language and collector of memory management techniques from far and wide. He takes us through his most important ones, including linear types, generation references and regions, to see what Evan hopes the futu...
Bringing Pure Python to Apache Kafka (with Tomáš Neubauer)
zhlédnutí 2,8KPřed 3 měsíci
Bringing Pure Python to Apache Kafka (with Tomáš Neubauer)
Developing a Neovim Docker Plugin from Scratch
zhlédnutí 11KPřed 3 měsíci
Developing a Neovim Docker Plugin from Scratch
Taking Erlang to OCaml 5 (with Leandro Ostera)
zhlédnutí 6KPřed 3 měsíci
Taking Erlang to OCaml 5 (with Leandro Ostera)
How Apache Pinot Achieves 200,000 Queries per Second (with Tim Berglund)
zhlédnutí 4,2KPřed 3 měsíci
How Apache Pinot Achieves 200,000 Queries per Second (with Tim Berglund)
Neovim: Creating, Curating and Customising Your Ideal Editor (with TJ DeVries)
zhlédnutí 30KPřed 3 měsíci
Neovim: Creating, Curating and Customising Your Ideal Editor (with TJ DeVries)
The Magic of Property Testing
zhlédnutí 6KPřed 3 měsíci
The Magic of Property Testing
Creating Hackathons that Work (with Jon Gottfried)
zhlédnutí 1KPřed 4 měsíci
Creating Hackathons that Work (with Jon Gottfried)
Automate Your Way to Better Code: Advanced Property Testing (with Oskar Wickström)
zhlédnutí 7KPřed 4 měsíci
Automate Your Way to Better Code: Advanced Property Testing (with Oskar Wickström)
Bridging the Gap Between Languages (with Martin Johansen)
zhlédnutí 2,6KPřed 4 měsíci
Bridging the Gap Between Languages (with Martin Johansen)
How do refactoring tools work? (with Jonathan Schneider)
zhlédnutí 2,5KPřed 4 měsíci
How do refactoring tools work? (with Jonathan Schneider)
Implementing Hardware-Friendly Databases (with DuckDB co-creator, Hannes Mühleisen)
zhlédnutí 10KPřed 5 měsíci
Implementing Hardware-Friendly Databases (with DuckDB co-creator, Hannes Mühleisen)
Exploring Verse, Haskell, Language Design and Teaching (with Simon Peyton Jones)
zhlédnutí 15KPřed 5 měsíci
Exploring Verse, Haskell, Language Design and Teaching (with Simon Peyton Jones)
Shouldn't Data Connections Be Easier? (with Ashley Jeffs)
zhlédnutí 3,9KPřed 5 měsíci
Shouldn't Data Connections Be Easier? (with Ashley Jeffs)
What can game programming teach us about databases? (with Tyler Cloutier)
zhlédnutí 73KPřed 5 měsíci
What can game programming teach us about databases? (with Tyler Cloutier)
Is Odin "Programming done right"? (with Bill Hall)
zhlédnutí 49KPřed 5 měsíci
Is Odin "Programming done right"? (with Bill Hall)
Can Event-Driven Architecture make Software Design Easier? (with Bobby Calderwood)
zhlédnutí 8KPřed 6 měsíci
Can Event-Driven Architecture make Software Design Easier? (with Bobby Calderwood)
How Lisp is designing Nanotechnology (with Prof. Christian Schafmeister)
zhlédnutí 8KPřed 6 měsíci
How Lisp is designing Nanotechnology (with Prof. Christian Schafmeister)
Roc - A Functional Language looking for those Software Sweetspots (with Richard Feldman)
zhlédnutí 12KPřed 6 měsíci
Roc - A Functional Language looking for those Software Sweetspots (with Richard Feldman)

Komentáře

  • @masterchief1520
    @masterchief1520 Před hodinou

    Use the right tool for the job. Most startups have switched to Go for backend. Very small number of companies actually use rust for that. Anybody can easily dive into go and pick up the idioms in sometime. Zig doesnt replace that and it doesnt compare with Rust's maturity.

  • @victoroliari9479
    @victoroliari9479 Před 3 hodinami

    It would be great to see you interviewing Casey Muratori. Casey holds a lot of knowledge about game development and low level code. I suppose a portion of your audience would enjoy lisening to him. I really enjoy the way you interview people. You’ve got real coding experience and you inspire your guests to talk about their technical solutions. It has depth in it. Casey is advertising his new course with cool stuff about low level language and CPU architecture. He keeps going to « young » interviewers that can’t extract the depth like you do.

  • @Incertophile
    @Incertophile Před 6 hodinami

    Sick video Kris

  • @Moousaka
    @Moousaka Před 8 hodinami

    Are there any articles or example code written about these "reduction trees"? Nothing shows up on my searches. Cheers!

  • @linkernick5379
    @linkernick5379 Před dnem

    This project, PyO3, is the life-changing one. I was really surprised, that even async Rust functions work too. Wonderful job, David!

  • @johanngambolputty5351

    pyo3 is awesome :) proc_macros really are a boilerplate killer

  • @SirSomnolent
    @SirSomnolent Před dnem

    Combine python dependency concerns with rust borrow checker and async? Heaven.

  • @pawehenek5948
    @pawehenek5948 Před dnem

    Kris your guests are great! maybe would you consider someone doing software for space industry? I can recommend you Tomasz Czajka from SpaceX or other polish folks from iceye or kplabs space

  • @frankc2119
    @frankc2119 Před dnem

    Could you do an interview with the creator of the C3 language? It is a C alternative that fits in a similar spot to Odin and Zig.

  • @aliasjon8320
    @aliasjon8320 Před 2 dny

    Bookmark 31:44

  • @faldarith
    @faldarith Před 2 dny

    I’ve been a Spotify listener for awhile now; this is an excellent episode of an excellent show.

  • @TomerBenDavid
    @TomerBenDavid Před 3 dny

    Haskell, lisp and alike are good for hobby. But how many real world products are built with them? 0.000001% of the products built with java at best.

  • @TheFreshMakerHD
    @TheFreshMakerHD Před 3 dny

    Fantastic podcast! Joran can talk for days on end about his baby, it must be quite inspiring to be on his engineering team. Developer voices ought to be an order of magnitude more popular, this was great

  • @MultiMojo
    @MultiMojo Před 3 dny

    This channel deserves a lot more subscribers.

  • @vikingthedude
    @vikingthedude Před 3 dny

    Will we be getting a python + zig interop episode next?

    • @DeveloperVoices
      @DeveloperVoices Před 3 dny

      Yes. Kinda. Check back in two weeks for the exact answer. 😀

  • @Alexander_Sannikov
    @Alexander_Sannikov Před 4 dny

    I honestly think at this point "how is it better than C?" question should be forbidden when talking about new languages. It's really not that hard to find something better than a dead horse in 2024. Yes, we know that errno is bad, we know that null terminated strings are bad, we know that C macros are bad. No new language competes against C, it competes against all other languages that compete against C. Tell me how this new language is better than zig, odin and rust.

  • @MrHopp24
    @MrHopp24 Před 4 dny

    Another great episode! Getting Charlie Marsh would be an amazing interview and follow up to this!

  • @driedurchin
    @driedurchin Před 4 dny

    My 2 cents. Another compelling use case for PyO3 is when you have a Rust library implementing some complex logic that you need to share with various levels of your stack. PyO3 allows you to wrap an already complete, full featured, tested Rust library with a Python API with minimal effort.

  • @jamesarthurkimbell
    @jamesarthurkimbell Před 4 dny

    I've always feared that someday, someone would pronounce "numpy" like that

  • @3DArea
    @3DArea Před 4 dny

    Absolutely terrific interview but it makes me thinking... excluding DS and researchers... maybe, just maybe... we should stop asking half of the C, C++ and Rust community to write the heavy lifting code, so the other half can keep using Python?

    • @lycantropos
      @lycantropos Před 4 dny

      Python is a great "glue" language and no-one asks anyone, Python programmers are doing most of it by themselves

  • @lycantropos
    @lycantropos Před 4 dny

    The man, the legend himself, thanks for the great work. PyO3 introduced Rust to me some years ago, and I will be forever grateful for such a smooth experience.

  • @itopaloglu83
    @itopaloglu83 Před 4 dny

    Really love the new detailed chapter markers. 👍

    • @DeveloperVoices
      @DeveloperVoices Před 4 dny

      Glad you like them! I have a new process for gathering them as I edit, so they should be appearing much more regularly from now on. 😅

  • @cj-ip3zh
    @cj-ip3zh Před 4 dny

    PyO3 is exciting, as it Mojo. I've used cpp and python ctypes to remove python in hotspots of programs before, I wouldn't say it was painful but the learning curve was there and works well once it works. Low friction interop between Python and performant compiled languages offers something superior to a Python4. Imagine being able to call, cpp, rust, go, zig, mojo all from python as first class functionality...

  • @SpacemanY2K38
    @SpacemanY2K38 Před 4 dny

    Best programming language of all time. (collide like a super nova)

  • @johnr3936
    @johnr3936 Před 4 dny

    Chris seemed to be genuinely happy to share everything he did, phenomenal interview.

  • @Alexander_Sannikov
    @Alexander_Sannikov Před 5 dny

    What the actual f, when the guy was talking about segment trees/fold maps I did not understand literally anything. Judging by the reaction of the host, I don't think he understood much either. At around 13:20. Lots of hand gestures, trapezoids, ropes, CRDT's -- I guess the guest expects all viewers to be writing high performance text editors on their spare time, but I doubt it's the case..

  • @Onyx-it8gk
    @Onyx-it8gk Před 6 dny

    Omg, I love Gleam! Kris, can we please have a update with Louis!!! It's such a cool language that brought me into the BEAM ecosystem, and it's come so far and the community around it is awesome!

    • @DeveloperVoices
      @DeveloperVoices Před 5 dny

      Yes, I definitely want to get Louis back soon. And with a bit of luck & planning, I'll also have someone else from the lovely Gleam community on soon. 🥳

    • @Onyx-it8gk
      @Onyx-it8gk Před 4 dny

      @@DeveloperVoices Awesome!

    • @aDaily1222
      @aDaily1222 Před dnem

      @@DeveloperVoices cant wait! i wonder who it is...hayleigh? giacomo? idk anybody else lol

  • @gaboralexnagy5609
    @gaboralexnagy5609 Před 6 dny

    First of all, love the channel! I was looking for content like this and thank the celestial rewrite rules that I found it. Second, I have a basic question about functional logic programming to anyone who has the answer. In functional programming, our basic abstraction and means of computation is function application. We know how that can be done as there is plenty of literature on mapping the work of Church and Curry to the world of building functional compilers and interpreters (or just look up how apply+eval works in Lisps :) ). But in the world of functional logic programming, instead of N-ary functions, we have N+1-ary relations, and say I define the relational version of the map function (in a miniKanren-like syntax, modified a bit to decreate "jargon" for legibility): (define-relation map° [f xs ys] (disjunction (conjunction (== xs empty-list) (== ys empty-list)) (conjunction (exist [x xs' y ys'] (cons° x xs' xs) (apply° f [x] y) (cons° y ys' ys) (map° f xs' ys'))))) Looks like a straightforward translation of something like the naive Haskell version of map. But the key thing here is apply°, the relational equivalent of function application. My big problem is that I cannot imagine how to implement apply°. Any implementation I try to come up with, I just cannot find a proper algebra for treating f as the proper logic variable that can be passed around like this (hence the "higher order" phrase I keep using) and would work properly with unification in its environment and would also lead to values evaluated like a regular function would (again, tying back to unification). In addition to this, I don't even know if apply° should be implemented at this level, or at the constraint level (but that doesn't seem constructive on the face of it, as arbitrary function application very much does not seem like a finite domain). In a similar vein, how is an anonymous relation defined and implemented? I wonder how Verse implements this and if anyone can point me to literature where I might find the solution to this. I must admit I feel a bit like a fool, but my computer science background is quite limited in this area.

  • @Alexander_Sannikov
    @Alexander_Sannikov Před 6 dny

    A very good podcast. I like the host, the guest and the subject.

  • @Dgg1234-x8e
    @Dgg1234-x8e Před 8 dny

    As a programmer (not in low level systems), I can’t express how grateful I am for gifted people like Loris doing what they do for the world.

  • @orbatos
    @orbatos Před 8 dny

    This is pretty interesting, but i would never want to work at a company micromanaging to the extent he seems excited for, no way in hell.

  • @100timezcooler
    @100timezcooler Před 8 dny

    Huge convo on rewriting old code and how important it is to figure out the core behaviour of whatever your tryna revive

  • @jonton6981
    @jonton6981 Před 10 dny

    One of my favorite tools. The payload agnosticism and default nearest delivery are my two favorite aspects

  • @joloppo
    @joloppo Před 10 dny

    Awesome interview! Looking at inko, I do think Yorick is missing a big part about syntax which is that approachable syntax does drive engagement and helps the userbase grow.

  • @cho4d
    @cho4d Před 10 dny

    first i have learned about zig. great presentation. i am feeling positive about this project's future! he is a smart dude.

  • @joloppo
    @joloppo Před 10 dny

    This guy can’t answer any questions properly.

    • @nythrox3047
      @nythrox3047 Před 10 dny

      horrible vid, I watched the whole thing and still have no idea how NATS works

  • @nexovec
    @nexovec Před 10 dny

    Chris can you open up a ko-fi account so I don't have to send money to patreon corp?

  • @nexovec
    @nexovec Před 10 dny

    Didn't expect this for sure. I watch Jeremy's casts for longer than yours.

  • @CipovPeter
    @CipovPeter Před 10 dny

    what happens when nats hard down. Whole solution build on it is down too? Some apps just cannot get whole hard down, your SLA will not allow it. so you need to build solution without shared resources like nats and rather building silos. Then you need to diverge to that http stuff, webhooks, grpc. To the other hand I see NATS ok inside silo.

    • @pablovicentico
      @pablovicentico Před 8 dny

      You can build clusters and streams can have replicas so if you have many nodes your applications will switch to other member of the cluster.

  • @JasonCunliffe
    @JasonCunliffe Před 10 dny

    10mins in ... He still hasnt even tried to answer the basic question. Help !! Swiss army penknife ?

  • @abc_cba
    @abc_cba Před 10 dny

    why are there no from scratch tutorial to advanced on duckdb anywhere?

  • @paxdriver
    @paxdriver Před 10 dny

    Congrats to monetizing! Well earned, I'm grateful for the transparency and brevity. Very, very classy method, as const 😘 model CZcamsr, that's how it's done.

  • @albinekb
    @albinekb Před 11 dny

    Interesting topic! Supr as always. Happy to support you on Patreon!

  • @ScottyZA101
    @ScottyZA101 Před 11 dny

    There goes my weekend…

  • @hexomega9445
    @hexomega9445 Před 11 dny

    excellent show, as always!

  • @mrmaniac9905
    @mrmaniac9905 Před 11 dny

    Kris, you are hands down, the best interviewer I’ve seen for programming related topics. Your questions and genuinely thoughtful and I love how much room you give the interviewee to answer the questions while you take it in. Wonderful stuff

    • @DeveloperVoices
      @DeveloperVoices Před 11 dny

      Wow, thank you. That's really kind of you to say. ❤️😊