Working with Skewed Data: The Iterative Broadcast - Rob Keevil & Fokko Driesprong

"Skewed data is the enemy when joining tables using Spark. It shuffles a large proportion of the data onto a few overloaded nodes, bottlenecking Spark's parallelism and resulting in out of memory errors. The go-to answer is to use broadcast joins; leaving the large, skewed dataset in place and transmitting a smaller table to every machine in the cluster for joining. But what happens when your second table is too large to broadcast, and does not fit into memory? Or even worse, when a single key is bigger than the total size of your executor? Firstly, we will give an introduction into the problem. Secondly, the current ways of fighting the problem will be explained, including why these solutions are limited. Finally, we will demonstrate a new technique - the iterative broadcast join - developed while processing ING Bank's global transaction data. This technique, implemented on top of the Spark SQL API, allows multiple large and highly skewed datasets to be joined successfully, while retaining a high level of parallelism. This is something that is not possible with existing Spark join types.
Session hashtag: #EUde11"
About: Databricks provides a unified data analytics platform, powered by Apache Spark™, that accelerates innovation by unifying data science, engineering and business.
Read more here: databricks.com/product/unifie...
Connect with us:
Website: databricks.com
Facebook: / databricksinc
Twitter: / databricks
LinkedIn: / databricks
Instagram: / databricksinc Databricks is proud to announce that Gartner has named us a Leader in both the 2021 Magic Quadrant for Cloud Database Management Systems and the 2021 Magic Quadrant for Data Science and Machine Learning Platforms. Download the reports here. databricks.com/databricks-nam...

[@raviiit6415]

great talk both of you.

[@LuisFelipe-qe2pj]

Very nice presentation!! 👏👏👏

[@rishigc]

@22:13 - where can i find an example of implementation with the SQL API ?

[@bikashpatra119]

Can you please provide the link to benchmark in githug

[@JimRohn-u8c]

Go to 23:25 in the video, he shows the GitHub URL in that part of the video.

[@vishakhrameshan9932]

Hi, I am facing skewed data issue in my spark application. Here I have 2 tables both are of same size (in the sense same rows but different column size) and am checking table A not in table B. This Spark SQL is taking lot of time.
I have given 100 executers in production env and also tried writing the both tables to a file to avoid in memory processing for such huge data and tried reading it to do the sql operation.
My application contains a lot of spark sql operation and this sql comes in some what in between the entire operation. When i run my application, it runs till this sql and then takes more than 6hrs to run 2M records
How can I achieve faster result with repartitioning, or iterative broadcast. Please help.

[@arpangrwl]

Hi VIshakh did you found the solution for the problem you mentioned ?

[@shankarravi749]

@@arpangrwl May i know the Solution What was needs to be done??

[@JoHeN1990]

Try bucketing the table before writing, it might take longer during write. But will be faster during joins

[@TechWithViresh]

check this : czcams.com/video/HIlfO1pGo0w/video.html