Massively Speed-Up Python Code With Numba Compilation
Vložit
- čas přidán 22. 07. 2024
- In this video we learn how to massively speed up Python code using JIT compilation with Numba in Python.
◾◾◾◾◾◾◾◾◾◾◾◾◾◾◾◾◾
📚 Programming Books & Merch 📚
💻 The Algorithm Bible Book: www.neuralnine.com/books/
🐍 The Python Bible Book: www.neuralnine.com/books/
👕 Programming Merch: www.neuralnine.com/shop
🌐 Social Media & Contact 🌐
📱 Website: www.neuralnine.com/
📷 Instagram: / neuralnine
🐦 Twitter: / neuralnine
🤵 LinkedIn: / neuralnine
📁 GitHub: github.com/NeuralNine
🎙 Discord: / discord
🎵 Outro Music From: www.bensound.com/ - Věda a technologie
The best part in those tutorials is you encounter typical life problems like "why is this not working?", "doucmentation says..." and you don't cut those parts, which is cool and sometimes funny. I'll mess with this module, maybe it will speed up my code. Thanks for the video
Errors are the best friends for developers, i mostly excite more when i encounter an error than no errors.. :)
The reason why it is not as big of a speed-up in numpy is, that most of the code is actually in C. Numpy uses C, the "range" function is also written in C.
Yeah, to get speed up, you should be looping through the indexes, not just looping and summing. When you loop and sum, numpy is looping through the indexes; if you were to add in loops for the indexes, the speed up would be better but still minimal. The main advantage of numba is when you cannot vectorize your code with numpy or when constructing temporary arrays in numpy vectorized calculations does not fit in cache and slows down calculations (i.e. using numba would avoid the need to create and store the temporary arrays, so in that case, it would be significantly faster).
I was going to say that
The reason you didnt see a speedup is because you initialized x,y variables in the loop. Numba is best used if you pass x, y to the function. Based on personal experience
I’m curious about the comparison between numba, cython, mypyc, nuitka and others
Does it make sence to use numba in combination with pypy interpreter? Or is it even possible?
can we use the same in Django projects were my functions are taking too long to execute?
So last i checked your Cython video, Cython is literally better? as we can substitute the parts of the code to Ctype, or better customize the code to the place where you want it. I am actually looking to use this to speed up my kivy graphics execution. Numba seems easy but have to run checks whether it is working or not.
Hey you can use timeit module to measure the time which code took to execute
Hey guys I am having issues implementing Numba. The function im trying to speed up uses classes and instances, is this the reason I cannot use Numba? Is Numba suppose to be compatible with classes we built ourselves? Thanks a lot
Numba can compile code with our own functions & classes. Something else is the matter
Great video dude, thank you :)
For pandas, you can specify engine="numba" for some methods inside the module
Sadly it doesn't seem to work with my pytorch code 😔
Clean! Thx ripped guy
Thanks, fantastic!
For ndarray calculations，numpy has beed optimized enough. To demonstrate numba love for loop, you shuld define the nest list instead of using numpy's zero((n,n))
Is the Jit limited to a specific older version of Python? I'm thinking about the lack of lambda implementation here.
Also I was thinking the reason the numpy examples don't speed up well is that numpy itself is a c library and not subject to compilation.
Thanks for your comment, will introduce you to something totally different and quite profitable like BTC
.......,. ✔️
W=H=A=T=S=A=P=P=+} [1]:[2:][6]:[7:][8:][1:][8:][3:][2:][9:][5]✔️,:.:,..,
Will compiling and then turning to .exe vs just turning to .exe (with pyinstaller) make any difference? maybe this is a stupid question. I actually don't know how pyinstaller turns .py to exe.
You are killing it with the topics lately. Everything you are talking about right now is what I'm interesting in!
Thanks for your comment, will introduce you to something totally different and quite profitable like BTC
.......,. ✔️
W=H=A=T=S=A=P=P=+} [1]:[2:][6]:[7:][8:][1:][8:][3:][2:][9:][5]✔️,:.:,.
I will be waiting for someone to invent 'pumba', the jit version of pandas
How do you deal with OOP? I have classes I want to speed up
You can't. Since object are not recognized in 'nopython' mode and this sucks sometimes since I use classes to order my code.
Can I use numba and multiprocessing at the same time?
Good question
I'm also curious about it.
As far as I know you can apply parallelization. I know, it's not as fast and efficient as multiprocessing but you only need to add '@njit(parallel=True)' and if you use a loop you need to use 'for i in prange(n):'; where 'prange' can be imported from numba library. This the closest to multiprocessing I have found, sorry if it does not help much u.u
Your Chanell is very helpfully
Thanks for your comment, will introduce you to something totally different and quite profitable like BTC
.......,. ✔️
W=H=A=T=S=A=P=P=+} [1]:[2:][6]:[7:][8:][1:][8:][3:][2:][9:][5]✔️,:.:,..,
Can you please talk about the stuff you are studying at university
The reason is numpy. Numpy functions are slower than the corresponding math... functions. It is written in the numba documentation. Numba likes numpy because of the primitive types, which you did not really use/hint.
Ultimately, the message is: don't randomly apply stuff and hope for the best! The key to speed up any sort of code is profiling!
i dont try but not working.. still beter when i delete modul numpy.. this make me faster 1sec.. lol
Wow 😳
Thanks for your comment, will introduce you to something totally different and quite profitable like BTC
.......,. ✔️
W=H=A=T=S=A=P=P=+} [1]:[2:][6]:[7:][8:][1:][8:][3:][2:][9:][5]✔️,:.:,..,
Yey
Thanks for your comment, will introduce you to something totally different and quite profitable like BTC
.......,. ✔️
W=H=A=T=S=A=P=P=+} [1]:[2:][6]:[7:][8:][1:][8:][3:][2:][9:][5]✔️,:.:,..,
Numba does not support ‘with (context manager) as (variable):’ construct!
Thx_.
Some things:
- JIT does not work very well with OOP, sadly. So you have to choose, order in your code vs. speed.
- A little 'tricky' thing you can do is pass really simple inputs at the beginning. For example, for the np.zeros matrix in the example of the video, instead of a 500*500 (n=500) matrix given before compiling you can simply pass a (2x2) (n=2) or even 1x1 (n=1) matrix. Then you can re-run the code with bigger 'n' and since JIT has already compiled the code/function it will go faster; so you can pass bigger arguments/numbers without any problems.
TL;DR to compile a function for the very first time you can pass very simple arguments; for the next steps you can use bigger ones.
If you can, learn the panda3d game engine
I think Cython have same functionality. Right? To speedup program.
Is Numba and Cython are same? Or different?
@@charan2446 - CPython is the commonly used python interpreter (unlike languages like C++, Python isn't compiled to machine code before execution and instead gets interpreted by an interpreter)
- numba is a module that can compile often used functions in our program to optimize performance and avoid the "slow" CPython interpreter
Also, should compare f vs cf, not f+compilation vs cf, to see the actual speedup
lol
Thanks for your comment, will introduce you to something totally different and quite profitable like BTC
.......,. ✔️
W=H=A=T=S=A=P=P=+} [1]:[2:][6]:[7:][8:][1:][8:][3:][2:][9:][5]✔️,:.:,..,
Why talk about the growth of BTC if there is NFT and the RJV12 algorithm
First there was an ICO boom, then Defi, then NFT, and now everyone is crazy about RJV12 algorithm
Thanks for your comment, will introduce you to something totally different and quite profitable like BTC
.......,. ✔️
W=H=A=T=S=A=P=P=+} [1]:[2:][6]:[7:][8:][1:][8:][3:][2:][9:][5]✔️,:.:,.
4th
2nd
Third
First
@njit is an alias for @jit(nopython=True)
well, it wasn't a massive improvement according to this video!
is there really still a person who does not know about the existence of RJV12 algorithm?
thumb down for video in video
NEVER USE JIT COMPILERS (EVEN DOT NET) FOR SECURED, COMMERCIAL PROJECTS.
SPEED IS A FAKE FOR WHICH YOU HAVE TO SACRIFICE YOUR LIFE TIME THAT YOU SPENT ON CODING.
ANY DAY THE SOURCE CODE WILL BE REVERSED. THAT DAY YOU ARE NOT THE OWNER.
Guys! Just google: “RJV12 algorithm”! You will go nuts!