Ray: Faster Python through parallel and distributed computing
Vložit
- čas přidán 29. 08. 2024
- Parallel and Distributed computing sounds scary until you try this fantastic Python library. Ray makes it dead simple to run your code on a cluster of computers with minimal changes to the actual code. Check it out!
MY OTHER VIDOES:
○ 5 Common Python Mistakes: • 5 Things You're Doing ...
○ 5 Amazing Python Libraries: • Five Amazing Python Li...
○ Making Python fast: • Can VSCode be a reason...
○ VSCode's Python Interactive Mode: • VSCode's Python Intera...
○ Learning programming language Julia: • How to learn Julia, a ...
Twitter: / safijari
Patreon: / jackofsome
#python #vscode #notebooks
the way you compacted 1 hour worth of content into 10min video is just amazing.
astonishingly efficient pedagogy here. Props man! Thanks!!
Awesome video Jack! Keep on with the good content bro!
Simple and Lucid, thanks a lot. This gives me hope for my current project.
Thank you, I've been trying to do distributed computing using hadoop and spark. And I just found out that there is an easier way for it. Thank you, I will try it next time.
You are great Jack! Learn new things with you. Keep it up!!
What an amazing awesome video! thanks a TON for making this gem!
Great introduction to Ray, thank you!
the best ray intro I have seen
Wish you still made videos.
Thank you! This was an awesome presentation. well planned out and thought through. I'm going to give ray a go.!
Nice work, Jack
Awesome video.
Small nit though… in the second time test… imread comes after start time. Wouldn’t that increase the time taken!?
🙏 🙏 😊 great intro! 😅 show us your tmux, vim setup?
Sure but that's Emacs 👀
@@JackofSome 😅
Interesting stuff, thank you
Excellent.
That is very awesome, will have to try it out
thanks for the info, I am new to setting something like this up. I do have a couple questions. So using the function decorator with .get(), ray will automatically use multiprocessing on each individual machine as well? My current implementation is using python multiprocessing pool with apply_async. Also, is there a tool/easy way to install the same venvs and dependencies for many cloud servers? You also mention to have a good internet connection or LAN but I imagine in most applications it will be a cloud server connection.
Omg best solution for my case
Beautiful!
this is awesome .... thanks for shared
this is very helpful even in 2023
Hi when I try to connect another laptop to head node I get an error can’t connect to GCS address check that it is correct or firewall is turned off can you help us please
Can you please help the client node gets connected to cluster but the worker is just idle all the workload is done by host computer
How would I have it so I just have to start the server and not type the command using ws (websocket)?
Can you use this to parallelize the code for cluster computing?
does it work with windows? I can't seem to get ray installed
I've used it under WSL but it's a suboptimal experience. I'm not sure if it works under windows proper. Are you using anaconda?
Very helpful video. How would this be hooked up to remote AWS servers? Thanks
They have documentation for deploying it to a number of clouds as well as putting it on kubernetes in general. I haven't tried it myself. If I ever do I'll make another video
@@JackofSome Sounds great, thank you. I'll check the docs out.
Great video. Where can I get the source code for this demo?
I have tried run 'ray start --head --redis-port=8888' and an error arisees, which is 'no such option: --redis-port'
The commandline options may have changed. Run it again with --help and see if there's a new option for Redis
Very cool
Hi, I installed ray using pip and I am able to import in python program but when I try using 'ray start --head' then it shows 'command ray not found'. I am using ubuntu. Any suggestions?
Seems to be a path issue. You can always import ray, look at ray.__path__, and then run the binary file near there by giving the full path
@@JackofSome Thanks for the reply but this was resolved. Now there is another problem. I have made a cluster of 4 nodes. All nodes are connected. But when I execute thr python script it is only running on head node only and not using all the nodes. When I checked the logs, I found out something like this: Export to Agent Metrics Failed: Can't connect to all addresses.
Great tutorial, but what to do when neither the virtual machines nor the local machine are on the same network? I have 3 VMs on different networks in Azure and a local machine. How to spin these up? Do I have to have Redis server installed somehow manually? Help please.
A VPN would probably be your best option in that case but you need to careful with this kind of a setup. You'll get charged for network activity and would need to very carefully design your inputs and outputs so that they're very small, otherwise network overheads will dominate your run time
@@JackofSome I reinstalled VMs in a virtual network, so they are regarded as in the same local network. Will test this today. Thanks Jack
@@JackofSome So, the approach of reinstalling VMs and putting them inside a single virtual network succeeded and now ray status shows that nodes are connected. However, port forwarding over ssh for Ray dashboard does not work since dashboard did not start on port 8625. In fact, with netstat I only see port 6734 being occupied.
I cannot see ray dashboard after ray.init() method browser says This site cannot be reached what should I do now?
I have tried in colab kaggle and in my localhost as well
That won't work in colab and kaggle. For your localhost you're trying to access from the same computer where the master node is?
@@JackofSome yes I'm doing everything in same computer
I've installed Ray Ray [default] ray[dashboard] using conda then I've tried ray init() it gives me ip if i copy that ip to browser it didn't show dashboard. everything in same system localhost
That was very helpful ! I am setting up a small office with a couple of servers and will try this for sure. Does anyone know how to set up a VPN to connect from home ? Thanks !!!
On the main server or host, google "my IP address" and it should look something like "123.456.78.90", enter that into your address bar and login to your wifi and port forward. (Look up a video on youtube for more info)
Hello again. I have another problem:
I am using WSL 2 on 2 windows 10 computers to use the Ubuntu console. I have downloaded Anaconda and Ray to be able to make the connection as shown in the video. The server computer works fine and I can see everything on the "Localhost", but the client node that I want to connect cannot do so. The console returns: "RuntimeError: Unable to connect to Redis. If the Redis instance is on a different machine, check that your firewall is configured properly."
I have already disabled the Firewall of both computers from windows, manually (including the full antivirus), and also disable the Firewall from Ubunto with "sudo ufw disable".
Please help. I need it to work for an integrative practical job in a subject at my university.
This is an open wsl2 issue. It needs to be in bridge mode for things to work or proper forwarding rules need to be set up which were to complicated for me so I gave up on it
@@JackofSome What did you do then? did you use Virtual Box with Ubuntu?
Can that work?
VMware but it will work with vbox too. Both can be in bridge mode
@@JackofSome hi again. Im download Ubuntu SO in VMware in 2 PC. The same thing happens to me as with WSL2. The client node, when I put "ray start --address = '***' --redis-password = '***' I get the error" RuntimeError: Unable to connect to Redis. ".
Check your VM settings and make sure it's set in bridge mode. Your VM needs to look like it's another device connected to the same physical network as your host OS
can you please share link of github?
be honest jack, did you expect it to be slower @ 4:10 ? I can almost hear the "what the f-" in your voice lol
Nope. I left it in because I felt it was a great learning exercise
Hi there, how does ray compare to dask?
I'm not too familiar with dask but I think it's different use cases no? Ray is more of a core technology rather than one applied to a specific use case whereas dask is focused on arrays/data frames. I guess dask ML is similar to Ray Tune?
@@JackofSome So I personally work on automated ML pipelines, not AI, meaning that I usually do a lot of models that are easily trained, more so than one big one. I think that Ray is very kewl for if you want to have a shared state of an object, something that is not possible with Dask. The rest I havent figured out yet however :P
how is the ray web link?
(this is my serious account)
ray.io
I'm not sure what you mean exactly. I'm assuming you wanted the website
Help me, reallly important, i can't install ray, i'm on windows
pip install ray
pip3 install ray
"As straight forward as doing pip install ray"... Ahh, the naïveté of x86 people. Ray uses stupid Bazel as a build system and it needs a lot of tinkering around to install from source.
My condolences for having to build things for ARM (I assume that's where you're running into trouble). I've never had a good experience there.
As it gets more prevalent eventually you'll have self contained wheels on pypi, and then my video will be accurate again 👀
@@JackofSome Not ARM but PPC64LE where you can find wheels and Conda channels just for really really popular packages and you're on your own for all the rest. I think it took me two work days to figure out all of Bazel's shenanigans and build Ray.
Damn. Glad it worked out. I'd legit watch an angry rant video about the process if you ever make one 😅