Video

Gemini 1.5 Pro: The video mentions that group by operations can fail due to large datasets and unsorted data. Here are the reasons for failure and how to compensate for them: * **Large datasets:** When dealing with large datasets, it is recommended to tune the split-out parameter. This parameter determines the size of the partitions, and a good starting point is to target 100 megabyte partitions. You can estimate the split-out value by considering the number of groups in your data and the size of each group. * **Unsorted data:** Dask performs better when the data is sorted by the group by fields. If your data is not sorted, Dask will shuffle the data to group it, which can be expensive. There are two ways to address this: * Sort your data before performing the group by operation. * Use math partitions. Math partitions can be used when your data is already sorted by an index matching one of your group-by fields. In this case, Dask can perform the group by operation on each partition without shuffling the data. Here are additional tips to improve the performance of group by operations in Dask: * **Optimize memory usage:** * Use pandas string dtype instead of object dtype for strings. * Use categorical data types when applicable. Categoricals are efficient when you have a small number of unique strings and the strings are large. * Drop unnecessary columns before performing the group by operation. * **Repartition your data:** Repartitioning your data ensures that the partitions are uniform in size. This can improve the performance of group by operations by avoiding situations where some partitions are significantly larger than others. * **Prioritize reductions before group by:** Perform any filtering or data reduction operations before the group by operation. This will reduce the amount of data that needs to be shuffled or grouped by.

Gemini 1.5 Pro: This video is about Dask Bag, a library for processing large datasets in parallel. The video starts with a basic introduction to Dask Bag. It explains that Dask Bag is a library that is useful for doing embarrassingly parallel analyses and a lot of pre-processing especially the text JSON or Avro data. Then the video dives into details with an example. The speaker constructs a bag with ten elements separate into four different partitions to demonstrate what a bag is. A bag is like a bunch of lists. Users can perform map, filter and reduce functions on the bag. For instance, the speaker uses map function to square every element in the bag, and filter function to get only the even elements. Next, the video shows how to use Dask Bag on real data. The data used in the example is a bunch of JSON files from a web service called MyBinder. The speaker reads the data using the read text function from Dask Bag. Then the speaker uses map function to convert the JSON encoded text into Python dictionaries. After converting the data into Python dictionaries, the speaker uses frequencies function to count how many times each Github repository shows up. The result shows that ipython is the most common repository that showed up in the data. The video then talks about how to use Dask Bag to pre-process data. The speaker filters out data that does not have "task" in the "spec" field and convert the data back into JSON format. Finally, the speaker writes the data to a text file. The last part of the video talks about the data frame. The speaker mentioned that Dask Bag may not be the right choice for complex analyses. Dask Dataframe might be a better option for such cases. The speaker also mentioned that Dask Bag can be converted to a Dask Dataframe using the to_dataframe function.

The resolution is very bad.

Excellent video, I wish all tech videos were this good.

Very interesting. Thank you for this view on the new dask_databricks functionalities.

Dask on Databricks is really cool. There's so many times you're on Databricks doing Python data science and don't want to use Spark.

Question regarding Array Expressions: how do they play together with the Dask (high-level) graph? A concrete xarray example: a problem with very large arrays is that even just their computational graph is too large to be materialized. A strategy is to read them without Dask (chunks=None), slice, and then again turn them into a dask-backed array by chunking. Would Array Expression simplify this, pushing the slicing before the graph materialization, or are those operating at different levels?

Show its use with xarray

Obrigado por ter legendas em Português .

Where can I get Paul Hobson's source code ?

Awesome video Trevor. Do you have any idea about the resources that I can use to learn more about the Zarr and its inbuilt configurations? I have seen the documentation, but it seems little overwhelming to me.

Nice video. Is there a detailed review how your colleagues are analyze billions of records? you've mentioned it here: czcams.com/video/8aQ3xcX8e9Y/video.htmlsi=0FRQOT9TEnDz9FUs&t=1621

@martinfleis can we access your notebook?

Had some issues with Ray, but Dask worked out of the Box! Congratulations to the Developers!

What is the name of this enviromnet where you are running this commands?

Great intro. Also, how do I show those additional panes on the right shows an 2:05 to display memory usage and progress etc. That is pretty awesome. Thanks so much

Great work you guys

1:08:00

Could I use async/await with dask?

55:15

Is there an official Dask community channel?

Hi Matt. Amazing stuff as always. Do you know if there is something similar for VScode? Thank you!

pin me

Kept checking my Slack because I didn't realize it was coming from the video...

is the notebook for the local gpu availablr

Thank you!

Dask is the bomb.

Hiya, you mentioned Xarray in passing. Is there a multi-demensional equivalent to cudf?

Please correct if I am wrong, but maybe it is better to open file for writing at 4:48 with 'a' mode or every worker will override the data inside and you will have only the result of the last firing worker.

where can we download the CSV files?

Highest resolution available is 360p. It’s hard to read the code

These videos are fantastic but sometimes difficult to hear (even with my volume set to max)

Hi, since this video was posted, the dask-report.html page has an extra tab called "Summary" - is there a doc where I can read what the various stats in that summary mean?

cant even create dataframe from python list. need to create a pandas dataframe first. which kinda defeats the whole purpose.

Thank you for this recorded Dask Demo Day! Are these Jupyter NB available for users?

Hi! Thank you for this! Regards

Does the Dask have some kind of linter?

Really a great talk!

Thank you for the explanation. Now it clears up my confusion on compute() vs persist()

Where can I get access to the notebooks used here?

The quality of this video makes it impossible to read the code

This lib is awesome!!! Thanks a lot 😍😍

what a boring speaker, such a disgusting english!

Thanks for the great explanation!

This is without doubt the best short guide on dask futures. Been reading lots of documentation but this video makes it so simple yet so powerful. Thanks a lot!

Thank you Dask Team, will explore this and join the community

That's a quality video, well done

Hello. The notbooks are available somewhere ?

Dask and all the python magic aside, Matt should hold master classes in delivering public lecture ♥ Also +100 on the "mature deployment" issue.

💔 🄿🅁🄾🄼🄾🅂🄼