How To Make Your Python Packages Really Fast With RUST
Vložit
- čas přidán 21. 07. 2024
- Everyone knows that program speed isn't Python's strong point. That's why so many number-crunching data science libraries are written in C.
BUT C can cause headaches. Segfaults are common and memory leaks are something to be feared. What if there was another option? Well, there is. It's Rust. Rust is a blazingly fast, memory-efficient and memory-safe language that's an absolute joy to work with. And I'm going to teach you how to use it to write your Python packages.
Check out the code:
github.com/isaacharrisholt/yo...
Resources:
Maturin: www.maturin.rs/
PyO3: pyo3.rs/
Rustup: rustup.rs/
The Rust Book: doc.rust-lang.org/stable/book/
uuidt: doc.rust-lang.org/stable/book/
Polars: pola.rs
pydantic-core: github.com/pydantic/pydantic-...
__________________________________________
Check out my other socials!
🐦 Twitter ▶ / isaacharrisholt
🖥️ Portfolio ▶ ihh.dev
📝 Blog ▶ isaacharrisholt.com
__________________________________________
Timestamps:
00:00 - Introduction and explanation
00:46 - Prerequisites
00:58 - Script overview
01:19 - Maturin setup
02:09 - Writing the Rust
02:53 - Benchmarking our function
04:08 - Rust limitations
04:43 - Common gotchas
#python #softwareengineer #rust - Věda a technologie
"C has enough foot guns to cripple a centipede..." 🙂
Thank you thank you.
I was WAY too proud of that one.
That made me laugh way too hard
Im a C dev, and my foot hurts
LOL. Nim is also great to interface with Python and very fast.
ERRATA:
4:24 - You can use a u64 on a 32-bit machine! The compiler will do its magic and sort it all out for you.
Was wondering about that since you can use 128-bit integers on 64-bit machines. TIL, thanks :)
There are a lot of videos on this topic but this one earns a like for being focused and to the point. Thanks 🙏
Glad I was able to help!
Why are you so underrated. You have genuinely good content, why are you still don't get any views? Keep going, good content!
That's very kind, thank you!
Damn, after getting more comfortable with the idea of calling opencl code from either python or rust, or c code from python, wrangling c types in all cases, I've been meaning to get to calling rust from python, did not realise it can be this easy (especially for end users too), this might be exactly what I wanted, you've saved me some research :).
I kinda like c, but I do find myself missing generics, traits and maybe sum types and pattern matching, not super fussed about automatically checked safety, you still end up having to think about what needs to be accessed from where (and apart from rust-gpu, we still don't really have this for GPGPU), but hey, automating some of that with the compiler can't be a bad thing. All in all, looking forward to using rust for more stuff, apart from when I would find a repl more convenient (data analysis and plotting). Knowing what functions can and can't fail and where you might have to think about nulls is also pretty cool.
I'm glad you found it useful! You'll still be able to call the Rust functions from your Python REPL, thankfully. You'll just have to import the correct lib :)
Unfortunately Rust doesn't have a REPL, though I'm sure someone's probably made one!
@@IsaacHarrisHolt I think there are a couple, but for just quickly poking around some slices in a matrix, or plots that you want to edit on the fly, good ol' python will do
So informative! thank u!
I can really tell that this vid was made with love
Thank you! I'm glad you enjoyed it :)
Great video! I wonder what packages I'm already using are actually written in Rust.
Hey James! Big fan 😅 and thank you!
I've actually made another video which goes over that, though they're the obvious ones again. Polars, Pydantic, Ruff. There are more and more where authors are choosing to use Rust instead of C/C++ for the critical path. The PyO3 readme has some examples: github.com/PyO3/pyo3
Hi! I liked the presentation u made and was wondering what software u used for making it.
I use Obsidian with the Advanced Slides plugin using the Monokai theme :)
Great stuff, thank you for sharing. Greetings from Morocco.
Awesome! I love Morocco. I want to go back at some point :)
Great video. Thanks
Glad you liked it!
I'm curious by your comment on "u64 might not be supported on every machine". What about u128? I am sure this is more than enough for 99.9% of use-cases.
I was actually incorrect about this! I added a correction in the pinned comment, but essentially you can use u64 or u128 for this too. Of course, there will potentially be memory constraints, especially for larger Fibonacci numbers, but they can be easily overcome using a better algorithm
Hey great tutoriel sadly the command maturin doesnt seem to work. My maschine cant find a cmdlet, function, script or executable Program named Maturin. Maturin is installed with pip just like you did in the tutoriel. Thanks for your time and help.❤
Hey! It's likely that your pip binaries aren't in your PATH. I'd have a quick Google to find out how to do that for your machine and OS.
With code that needs to take data owned by a different process do some modifications to it and give back control
(Like python packages cuda ml kernals etc)
Just us8ng pointers seems so nice and simple
I'm not entirely sure how this works in that regard. I haven't looked into whether PyO3 gets values by reference or by value. I'll have a look!
aaaand ... subscribed !!
Thank you so much!
I tried to create and return a simple list containing 1 million integers in range with the following:
#[pyfunction]
fn create_integer_list(n: usize) -> PyResult {
let mut result = Vec::new();
for i in 0..n {
result.push(i);
}
Ok(result)
}
when I compile it in RustRover it runs in 8 ms. However, when I create it in maturin as per your instructions, it takes roughly 110 ms to run. Any idea what's causing it? I ran maturin develop --release
It's very possibly the interaction with Python. When you say you're compiling it in RustRover, are you simply compiling it as a Rust binary or as a Python library?
@@IsaacHarrisHolt Sorry I was unclear, I meant that I ran the same code as above in RustRover which worked great, as opposed to when I ran that code through maturin. I have narrowed it down to the PyResult return of my function. It seems like returning a vec and "retrieving" a list in python is really slow. Gonna see if there is a way to return a list of values in a better way
That makes sense. The communication with the Python layer will be what takes up most of the time in your case, I would imagine.
Why would you recommend Rust for someone with Python background and not Nim?
Good question! Mostly because I'm not familiar with it, but also Rust and Python is a very popular combination at the moment and I couldn't find many good videos on the topic, so I decided to fill the gap.
What is the name of the font used in vscode?
I use JetBrains Mono!
@@IsaacHarrisHolt Im late haha thanks
Amazing video. You deserve more subscribers than mr beast.
That's very kind, thank you!
nice to see a fellow wsl user
Only for videos and work! I use Ubuntu for most of my personal projects, and then WSL on my work laptop and for recording
What are you talking about? u64 and u128 are supported in 32 bit machines lol, the compiler fixes that for you. Please add this correction as that is highly misleading.
That being said, it's very rare to build for a 32 bit architecture in 2023 if you're not in embedded, and there you have a whole lot of other things to worry about.
I've added a pinned comment with this information. Thanks for calling me out!
I wish there was anything similar for doing it with c++ and pybind
I'd not heard of pybind! It looks like a minimal version of Boost, but it would be nice if there was a build tool for it. Technically maturin and PyO3 are separate tools, so there wouldn't be anything preventing you from creating the equivalent for pybind11.
I try learn Julia but there are only English resources
That's a shame! But it also means there's a gap in the market 😉
When i run "maturin init" it says
error: unrecognized subcommand 'init'
I am using kubuntu. Any idea whats going on?
The command has changed to `maturin new`!
@@IsaacHarrisHolt Still says error: unrecognized subcommand 'new'
Maturin version is 1.3.2
This error is really weird, Also you replied INSTANTLY! What?
Dw if you dont know what this is ill switch to c/c++ or maybe go. Just reply "idk" or smth atleast to tell me.
Maturin help returns:
Build and publish crates with pyo3, rust-cpython and cffi bindings as well as rust binaries as python packages
Usage: maturin
Commands:
build Build the crate into python packages
list-python Search and list the available python installations
develop Install the crate as module in the current virtualenv
sdist Build only a source distribution (sdist) without compiling
help Print this message or the help of the given subcommand(s)
Options:
-h, --help Print help
-V, --version Print version
Ah it's `maturin new` in 1.7.0, which is the latest version. I'd probably recommend not using Go bindings - while I absolutely love Go, the interop with Python is a massive pain in the ass to get set up correctly, and the types don't map that well.
@@IsaacHarrisHolt Oh, I think i know why that happened and it didint the latest version. Thanks man!
Edit: Yeah its fixed, it was caused by a number things actually mainly due to the fact that i use linux and pip is a bit different here.
I installed 2 versions,
1. globally using sudo apt install python3-maturin which was outdated. Couldn't use pip to install it globally because of Linux.
2. The version in a venv environment. Since i could use pip, i did pip install maturin which gave me the latest version.
But running maturin in the terminal still gave me the outdated version.
Thanks!
Ran into another error (i ran into this previously but the following was something i thought was a fix but that gave me the same error.)
During maturin develop:
🔗 Found pyo3 bindings
⚠ Warning: Failed to determine python platform
💥 maturin failed
Caused by: Unsupported Python interpreter for cross-compilation: /home/thearchcoder/Documents/Programming/Python/Trilent/venv/bin/python; supported interpreters are pypy, graalpy, and python (cpython)
Ill just open a github issue for this. I searched on google, No fix.
Caused by the fact its looking in the venv directory and seeing the link, that link is just a shortcut to the actual python interprator
don't know about it. there are good, tested modules like numpy, to make your counts in python, they are fast.
They're really fast! Because they're written in C 😉
C can be fun. It is super simple once you get the hang of it.
True! C is harder to get started with from a tooling perspective though. Which compiler do I use? What formatter? etc. etc.
Rust doesn't give you a choice, and all the tooling is in a single CLI app, which makes it really easy to start. Yes, the language is a bit more difficult, but it's much easier to write safer code in Rust than C
@@IsaacHarrisHoltI think it's better to have a choice, all of rust currently depends on LLVM...
That's not inherently a bad thing. My point is that the barrier for entry to getting started writing Python extensions with Rust is lower than that of C, mostly thanks to the amazing tooling built by the Rust community.
@@IsaacHarrisHoltwhy does this need to be such a difficult choice. Windows you use mingw for multi comparability or msvc for performance, Mac you use clang, and Linux you use gcc. And do people really need intelisense so badly? I’ve personally never used intelisense and just use default vim lol
I think you've just proven my point 😅 for Rust it's cargo on everything!
wow, i did it!!! With poetry!
You got this working with Poetry? Awesome! What was the trick?
@@IsaacHarrisHolt in pyproject toml specify build backend as maturin and something else a littlebit. can i have your contact? I have telegram, whatsapp and email. I could provide a bit more detail because now my laptop is closed and i forgot what i actually did.
My email address should be available on my channel's About page
Let's not fall into syntax similarity trap. Rust is wonderful language, and I'd use it for other reasons than similarity of syntax. In fact popular modern languages such as Rust, Typescript, Scala, and typed Python share syntax with good old Pascal, which would be the first programming language that you would learn in my days (except basic). Personally, I have rewritten PoCs in Python in C++, and I didn't find it any difficult, actually I could achieve most of those rewrites by copy & paste of Python code, and then run some VI macros to add types, compile and all works! I don't think it is that simple with Rust.
You make very good points! I usually only mention the syntax similarity because it slightly reduces people's "fear" of Rust. Lots of folks hear that it's hard and then don't want to try it, so it's just another reason it might not be that scary.
Naturally, it's not the most important thing when considering which language to use to extend Python. Honestly, I think the most important consideration is whether you even SHOULD be using a different language when there are so many things you can do to optimise Python code first.
what ide is this?
It's PyCharm! I'm using the new UI, dark theme and the red-green colour blindness adjustment turned on, if you're curious about the theme :)
Better use something more similar to Pyrhon and easier like cython or for instance D or nim..
I've not tried Cython, but Nim is good, I agree
I just cant wait for Mojo now.
Mojo 👀
Perhaps! I made a video on Mojo, but it seems to be a lot further along now. I'll need to take a closer look at some point
C is fun tbh
I'm not denying it! I just prefer Rust for Python extensions :)
C++ has enough leg bombs to cripple a tank
This is why we Rust 👀
@@IsaacHarrisHolt ohkay, i think i will start with rust then. one question - does it have proper tooling - like the IDE documentations & linting etc?? editors which i have available: vscodium, kate, intellij.
Yes! Rust's tooling is amazing. It has an LSP called rust-analyzer which you'll be able to use in both VSCodium and Kate, and all the JetBrains IDEs have a Rust plugin available (JetBrains are also working on a Rust IDE).
I'd recommend looking at the Rust book and watching a few CZcams videos to get started :)
I'm both a python (on job) and a rust (newbie and hobby) programmer. And before I start my criticism: Yes, pure python code will be slower than rust code most of the time.
My issue is the abundance of oversimplified benchmarks on the internet like this one. Please stop doing those. New programmers will pick up those examples and will think these are mostly okay. Please, use real-world problems whenever possible and solve it with your best effort on both sides. Don't trust the results and don't jump on conclusions. Verify the results as good as you can and be as transparent as possible when publishing the results.
While I do typically agree with your approach, this video is aimed at people who can't really write any Rust, so I needed an example that would easily translate from Python to Rust without causing any confusion.
For that reason, I avoided anything with any data structures as they're topics that require some understanding of the Rust borrow checker.
Ah, that's a good suggestion. I'll keep that in mind for next time, thank you!
@@IsaacHarrisHolt I fully agree with your perspective here. I guess the video was about "how to run rust code in python", not "look how slow python is"...
Yes! Absolutely. Python is slow compared to Rust for certain, but this was just meant to be an example of WHY you should consider rewriting parts in Rust. The idea was to show potential gains, not necessarily to benchmark the two languages.
Wtf you are talking about, the tutorial does what video name says,... What real world problems , what oversimplified benchmark ? Benchmark is good, shows simply comparison of those 2 languages. What is Real World problem ?, in real world, like almost a year, I move data from frontend to backend, to database or file storage, read and open and refactor files,....
This is good for rust developers but not C or C++ for obvious reasons.
anything that enriches python echo system is welcomed by me.
I don't think it's particularly BAD for C/C++ devs. They can continue to write their language - it just gives people another option!
I feel like every language-plugger and C-hater in the world keeps forgetting that their favourite language runs on top of about 5 million lines of well-tested and functioning C code. It's no longer feasible to write any language at all without libbing out to C for most of its basic functionality (like opening files, accessing memory, listening to sockets, connecting to servers etc etc), simply because if you spend 2+ years getting the basics done, your language will get overtaken by something else.
That's probably true! But it doesn't mean that everyone has to write C. My argument in this video is that Rust is easier to get started with for Python extensions. That's what I've found from experience, even as someone who's done more C/C++ than Rust in the past.
zzzZzz
Imagine using pythonzzzzz,. instead of pure Rust
Unfortunately I'm not genius enough to use pure Rust 😓
Yo our pfps are the same
@@TheArchCoder based 🗿⚡⚡
@@TheArchCoder nice pfp btw