Video

thanks for making this, coiled seems to be what I'm looking for

can someone share dataset link? from where to download 10 csv files of nyc flights dataset?

Thank you

If I run Dask without importing the client, it does not work on many workers ?

The source was one only big parquet file ? Dask set partitions by itself ?

My main issue with dask is the lack of support of the community (very different from pandas!)

Great talk! What would you recommend for ingesting about 100-200GB of geospatial data on premise?

{ "name": "CancelledError", "message": "('mul-floordiv-3770c7fe5e6231d62ed3d68e48276fbd', 0)", "stack": "--------------------------------------------------------------------------- CancelledError Traceback (most recent call last) File <timed eval>:2 File c:\\Users\\mohit.parwani\\.conda\\envs\\parApat\\Lib\\site-packages\\dask_expr\\_collection.py:476, in FrameBase.compute(self, fuse, **kwargs) 474 out = out.repartition(npartitions=1) 475 out = out.optimize(fuse=fuse) --> 476 return DaskMethodsMixin.compute(out, **kwargs) File c:\\Users\\mohit.parwani\\.conda\\envs\\parApat\\Lib\\site-packages\\dask\\base.py:375, in DaskMethodsMixin.compute(self, **kwargs) 351 def compute(self, **kwargs): 352 \"\"\"Compute this dask collection 353 354 This turns a lazy Dask collection into its in-memory equivalent. (...) 373 dask.compute 374 \"\"\" --> 375 (result,) = compute(self, traverse=False, **kwargs) 376 return result File c:\\Users\\mohit.parwani\\.conda\\envs\\parApat\\Lib\\site-packages\\dask\\base.py:661, in compute(traverse, optimize_graph, scheduler, get, *args, **kwargs) 658 postcomputes.append(x.__dask_postcompute__()) 660 with shorten_traceback(): --> 661 results = schedule(dsk, keys, **kwargs) 663 return repack([f(r, *a) for r, (f, a) in zip(results, postcomputes)]) File c:\\Users\\mohit.parwani\\.conda\\envs\\parApat\\Lib\\site-packages\\distributed\\client.py:2235, in Client._gather(self, futures, errors, direct, local_worker) 2233 else: 2234 raise exception.with_traceback(traceback) -> 2235 raise exc 2236 if errors == \"skip\": 2237 bad_keys.add(key) CancelledError: ('mul-floordiv-3770c7fe5e6231d62ed3d68e48276fbd', 0)" } I'm getting this error when i use client can someone please help with any possible solution i definitely need that. please!

So happy to see this. Better late than never. Hopefully Dask gets the popularity it deserves and becomes a serious contender to Spark down the line.

Gemini 1.5 Pro: This video is about an introduction to Dask DataFrames, and it covers when to use them, how to use them, and performance tips. In the video, it is explained that pandas is great for tabular data sets that fit into memory, but Dask is useful for working with data sets that are larger than your machine can handle. Dask can cut up your big data set into smaller bits and execute those smaller parts in parallel. Here are the key points covered in the video: * **When to use Dask DataFrames:** You should use Dask DataFrames if your data doesn't fit into memory and your computations are complex. Pandas might run into a memory error if the data is too large, but Dask can handle those types of large-scale computations comfortably. * **Dask DataFrames vs Pandas DataFrames:** Dask DataFrames are similar to Pandas DataFrames and implement a well-used portion of the Pandas API. This means that a lot of Dask DataFrames code will look and feel pretty familiar to Pandas users. However, there are some key differences. For instance, unlike Pandas DataFrames, Dask DataFrames are lazy, meaning they only create the task graph (a recipe or a root map) to get to the final result but doesn't actually execute it until you specifically tell Dask to do so by calling compute. * **Working with Partitions:** Dask DataFrames are cut up into small bits which are partitions and each partition is actually just a Pandas DataFrame. This means you can perform Pandas operations on these partitions. * **Performance tips:** The video also covers performance tips, such as when to call compute. It is recommended to call compute when you want to combine computations into a single task graph. This is because task graphs for these results have been merged which means that Dask only needs to read the data from the CSV file once instead of twice. The video concludes by mentioning that this is module two of the introduction to Dask tutorial and the next module will cover processing array data with Dask Arrays.

How this setup is solving dependencies for the python code?

The problem is that in fact RUST based tooling actually wins and all the paid promotions just suck . The actual reason why RUST based tooling is sort of suppressed is very simple , hyperscalers (big cloud tech) earn a lot of money and if things are faster there is no huge bills for your spark clusters 😊)) , I was playing with RUST and huge datasets myself without external benchmarks course I don t trust all this market shit .Rust based EDA is maybe witch kraft but this thing runs as beast . try yourself guys with a huge datasets .

When you say "now", from what version are we talking about? To future-proof the video.

It's a great introduction with very cool and easy to follow illustrations. Great job....!!

Clickhouse ftw

Would you please share a link to the github ?

Great work 🙂 Dask will fascilitate to solve some computational data analysis issues of many people

How delayed gets around GIL?

This is such a clear, simple, yet extremely powerful introduction. Alright, you convinced me to try coiled again.

But DuckDB is good if your data fits one single machine. But the benchmarks shows different story when data is distributed. What about that?

Thanks for this tutorial and the other material at Dask and Coiled, will help heaps in a large data project 🙂

Thanks for this tutorial and the other material at Dask and Coiled, will help heaps in a large data project 🙂

What software did you use for generating / displaying your plots? It looked really nice

Great video! Recently switched from Dask to Duckdb on my ~1TB workloads, interesting to see some of the same issues I found brought up here. One gotcha I've found is that it is REALLY easy to blunder your way into making non-performant queries in dask (things that end up shuffling, partitioning, etc. a lot behind the scenes). It was more straightforward for my use case to write performant SQL queries for duckdb since that is much more of a common, solved problem. The scale-out feature of Dask and Spark is interesting too, as we are considering the merits of a natively clustered solution vs just breaking up our queries into chunks that can fit on multiple single instances for duckdb.

Such a good video! So many good insights clearly communicated with proper data. Also love the interfaces you've built, very meaningful, clean and minimalistic. Have you got comparison benchmarks where cloud cost is the only constraint and the number of machines or their size and type (GPU machines with cudf) is not restricted?

you are most welcome (suffered well) worth it for the duck

It's not clear to me if you can use autoscaling with coiled.

I don’t see duckdb and polars kick spark dask ass on 10gb level in my practical usage.😅 we can’t always trust TPC-H benchmarks.

Great talk, thanks

Some ppl were meant to teach and Matt is one of them! One feedback: I know you have covered it elsewhere but it might be helpful to talk about the graphs (like what does a yellow vs red block mean). You have them up on the screen. They must be serving some purpose. Again, brilliant presentation

Thanks, this is amazing! I have tried integrating Optuna hyperparameter search with Dask and it works great, but I have noticed if I increase the number of iterations, at some point my system crashes due to insufficient memory. From what I can see dask keeps a copy of each iteration so it ends up consuming more memory than needed; any way I can release all the memory usages after each iteration?

Does the Task Delayed use GPU as well ?

I had an issue with parallelization and the random sampler for hyperparameter search. When I submit optimize function in parallel, optuna keeps repeating the same hyper-paremeters across all processes. I could not figure out how to reseed the sampler for different processes.

What about Dask Bag and Dask Future ?

Would you please share a link to the notebook ?

Thank you so much. This is very helpful with my research.

The video is very informative and kudos to Richard for making Intuitive. Could you help me with below questions? 1. How can we perform a Time series regression using DASk. I see we are breaking the huge dataset to chunks how are gonna maintain the time continuity between the chunks. 2. You have used coiled clusters and i beleive these are external CPU clusters and how DASK is powerful over Pyspark in this case? 3. So DASK can be only utilised when there is CPU executions and it might be used in case of parallel GPU execution right ? Share your comments on this Thanks in advance

This looks great! I remember trying to use coiled jobs to do something like this a while ago.

What are the coming features that coiled plans to do?

great stuff, Matt !

I have a use case to read data from Oracle table - split this into files and zip it. Move to s3. Would Dask be a benefit or overhead for such a use case? (Cx_Oracle is used. Currently using mutiprocessing on 20 core server)

Update: pandas 2.0 has been released! See www.coiled.io/blog/pyarrow-strings-in-dask-dataframes for the latest on PyArrow strings improvements.

Thank you very much for this usefull information

the one at 15min is really nice...what is the cluster you're running it on?

Большое спасибо вам! Отличная серия видео, продолжайте работать!

Good video! Can you help me? Where can i find notebook from this video?

This is a very good video. I have to ask cause I am in the situation of setting up a DASK cluster that will be querying large weather datasets in AWS S3. I have never done it. Do you have a video on setting up the cluster? Have not explored your channel yet...thx

33:00 surprising that there aren’t existing open source solutions that support “marginal “ arrays, so to speak… has this changed?

Thanks , interesting for oceanographic research .

thanks for this video!! i am not sure how it is benefitial to have dask worker code inside the same process that the user code is called. after all pinging the process that runs the user code, does not need to happen often, and in this way GIL is not blocking for the heartbeat to be communicated to scheduler. am i missing something here? any pointer is appreciated.