22 Optimize Joins in Spark & Understand Bucketing for Faster joins

Video explains - How to Optimize joins in Spark ? What is SortMerge Join? What is ShuffleHash Join? What is BroadCast Joins? What is bucketing and how to use it for better performance?
Chapters
00:00 - Introduction
00:48 - How Spark Joins Data ?
03:25 - Shuffle Hash Join
04:20 - Sort Merge Join
04:59 - Broad Cast Join
07:50 - Optimize Big and Small Table Join
13:32 - Optimize Big and Big Table Join
16:09 - What is Bucket in Spark ?
18:39 - Optimize Join with Buckets
Local PySpark Jupyter Lab setup - • 03 Data Lakehouse | Da...
Python Basics - www.learnpython.org/
GitHub URL for code - github.com/subhamkharwal/pysp...
The series provides a step-by-step guide to learning PySpark, a popular open-source distributed computing framework that is used for big data processing.
New video in every 3 days ❤️
#spark #pyspark #python #dataengineering

[@user-ye2be7kn3o]

very nice , so far best vid for beginners on join

[@easewithdata]

thanks ❤️

[@sureshraina321]

Most expected video😊
Thank you

[@chetanphalak7192]

Amazingly explained

[@pcchadra]

@easewithdata As per the video Shuffle Hash and Broad Cast Join both you are used to join small data set with large data set. For Broad cast we are storing small dataset in memory . Can you explain how it behaves is Shuffle Hash ? Some source is saying Shuffle Hash applicable when both data set is large and apply re-partition concept. Can you please elaborate?

[@prathamesh_a_k]

nice explaination

[@easewithdata]

Thanks please make sure share with your network on LinkedIn ❤️

[@DEwithDhairy]

PySpark Coding Interview Questions and Answer of Top Companies
czcams.com/play/PLqGLh1jt697zXpQy8WyyDr194qoCLNg_0.html

[@Aravind-gz3gx]

@23:03, the tasks showed only 4 tasks here , usually it will come's up with 16 tasks due to actual config in the cluster, but only 4 tasks is being taken due to the data is being bucketed before reading. Is it correct ?

[@easewithdata]

Yes, the bucketing would restrict the number of tasks to avoid shuffling. So it's important to decide number of buckets.

[@Abhisheksingh-vd6yo]

how 16 partition(task) is created because partition size is 128mb and here we have only 94.8 MB OF DATA
.. @please explain please

[@easewithdata]

Hello
Number of partitions for data is not only determined using partition size, there are some other factors too
checkout this article blog.devgenius.io/pyspark-estimate-partition-count-for-file-read-72d7b5704be5

[@subhashkumar209]

Hi,
I have noticed that you use "noop" to perform an action. Any particular reason to not use ".show()" or .display()?

[@easewithdata]

Hello,
show and display doesn't trigger the complete dataset. Best way to trigger complete dataset is using count or write. And for write we are noop.
This was already explained in past videos of the series. Have a look.

[@keen8five]

Bucketing can't be applied when the data resides in a Delta Lake table, right?

[@easewithdata]

Delta lake tables doesnt supports bucketing. Please avoid using it for the delta lake tables. Try to use other optimization like z ordering while dealing with delta lake tables.

[@svsci323]

@@easewithdata So, in real-world project bucketing need to be applied on rdbms table or files?

[@ahmedaly6999]

how i join small table with big table but i want to fetch all the data in small table like
the small table is 100k record and large table is 1 milion record
df = smalldf.join(largedf, smalldf.id==largedf.id , how = 'left_outerjoin')
it makes out of memory and i cant do broadcast the small df idont know why what is best case here pls help

[@Abhisheksingh-vd6yo]

df = largedf.join(broadcast(smalldf), smalldf.id==largedf.id , how = 'right join') may it will work here

[@avinash7003]

high cardinality --- bucketing and low cardinality --- partition?

[@easewithdata]

Yes

[@alishmanvar8592]

Hello Subham, why did not cover Shuffle hash join practically over here? as I can see here you have explained only in theory

[@easewithdata]

Hello,
There is very less chance that some will run into issues with Shuffle Hash Join. The majority of challenges comes when you have optimize Sort Merge which is usually used for bigger datasets. And in case of smaller datasets now a days everyone prefers broadcasting.

[@alishmanvar8592]

@@easewithdata suppose we don't choose any join behavior then u meant to say shuffle hash join is by default join?

[@easewithdata]

AQE would optimize and choose the best possible join

[@alishmanvar8592]

@@easewithdata Hello Subham, can u please come up with session where u can show how can we use delta table (residing on golden layer) for power bi reporting purpose or import into power bi