Question regarding Array Expressions: how do they play together with the Dask (high-level) graph? A concrete xarray example: a problem with very large arrays is that even just their computational graph is too large to be materialized. A strategy is to read them without Dask (chunks=None), slice, and then again turn them into a dask-backed array by chunking. Would Array Expression simplify this, pushing the slicing before the graph materialization, or are those operating at different levels?
Expressions will eventually replace high-level graphs. They generate low-level task graphs directly. Slicing is definitely pushed through before graph generation, which will likely help reduce overall graph generation overhead. It's still possible to create large graphs though, just less likely. We're also shipping the expressions directly to the scheduler, so there will be less pain to large graphs (they won't have to travel over a wire).
![Data Modeling for Power BI [Full Course] 📊](http://i.ytimg.com/vi/MrLnibFTtbA/mqdefault.jpg)
Question regarding Array Expressions: how do they play together with the Dask (high-level) graph?
A concrete xarray example: a problem with very large arrays is that even just their computational graph is too large to be materialized. A strategy is to read them without Dask (chunks=None), slice, and then again turn them into a dask-backed array by chunking.
Would Array Expression simplify this, pushing the slicing before the graph materialization, or are those operating at different levels?
Expressions will eventually replace high-level graphs. They generate low-level task graphs directly. Slicing is definitely pushed through before graph generation, which will likely help reduce overall graph generation overhead. It's still possible to create large graphs though, just less likely.
We're also shipping the expressions directly to the scheduler, so there will be less pain to large graphs (they won't have to travel over a wire).
@@Coiled Thanks for the answer! That actually sounds great, would help our workflows quite a bit.