Master Reading Spark Query Plans
Vložit
- čas přidán 26. 07. 2024
- Spark Performance Tuning
Dive deep into Apache Spark Query Plans to better understand how Apache Spark operates under the hood. We'll cover how Spark creates logical and physical plans, as well as the role of the Catalyst Optimizer in utilizing optimization techniques such as filter (predicate) pushdown and projection pushdown.
The video covers intermediate concepts of Apache Spark in-depth, detailed explanations on how to read the Spark UI, understand Apache Spark’s query plans through code snippets of various narrow and wide transformations like reading files, select, filter, join, group by, repartition, coalesce, hash partitioning, hashaggregate, round robin partitioning, range partitioning and sort-merge join. Understanding them is going to give you a grasp on reading Spark’s step-by-step thought process and help identify performance issues and possible optimizations.
📄 Complete Code on GitHub: github.com/afaqueahmad7117/sp...
🎥 Full Spark Performance Tuning Playlist: • Apache Spark Performan...
🔗 LinkedIn: / afaque-ahmad-5a5847129
Chapters:
00:00 Introduction
01:30 How Spark generates logical and physical plans?
04:46 Narrow transformations (filter, select, add or update columns) query plan explanation
09:02 Repartition query plan explanation
12:57 Coalesce query plan explanation
17:32 Joins query plan explanation
23:23 Group by count query plan explanation
27:04 Group by sum query plan explanation
28:05 Group by count distinct query plan explanation
33:59 Interesting observations on Spark’s query plans
36:56 When will predicate pushdown not work?
39:07 Thank you
#ApacheSpark #SparkPerformanceTuning #DataEngineering #SparkDAG #SparkOptimization
🔔🔔 Please remember to subscribe to the channel folks. It really motivates me to make more such videos :)
This is the most detailed explanation I have ever seen.
Thanks a bunch. To my knowledge, no one has explained Spark explain function this detailed level. Very in-depth information.
Absolute gem ❤❤ would like to have video on handling real time scenarios (handle slow running job, oom etc)..
Proud of you brother, looking forward to more of such videos. Great job!
Explanation is so good
This is one of the best video about Spark I have seen recently!
Afaque, THANK YOU SO MUCH FOR THESE VIDEOS!!
They are so amazing for a fast paced learning experience.
Hope you soon upload much more!!
One of the best videos I have seen on Spark, waiting for your Spark Architecture Video
Thanks for such an in-depth overview!! helps a lot to grow!!
You are a gem bro. The content that you bring here is terrific. ❤❤❤
Thanks man, @yashwantdhole7645. This means a lot!
Amazing content.. I am a newbie into Spark but I am hooked.. Sir plz post the continued series.. awaiting for your video posts.. Amazing teacher
no one teaches detailed way complex things like you no matter what please spread you're knowledge to world i am sure there must be people learn from you , remember you as master life long who settled in it job like me
Explained the concept really well!
Very useful and explaining complex things in easy manner . Thanks and expect more videos from you
This takes me back to me YaarPadhade times. Great work Bhai much love!
rare content! please don't stop making these
My today's well spent 40 mins. Thanks for the knowledge sharing.
It's great to see such useful contents in spark... an its helpful to understand clearer with your notes! you rock.... Thankless thanks !!
Bhai mera bhai 😍 Abto hazaro students aayenge bhai ke pass par Apne sabse pehle student ko mat bhulna bawa😜
Very proud of you bhai... And i can guarantee every1 here that he is the best teacher that there is❤️
Beautifully explained. Many concepts got cleared. thanks a lot.Keep going.
Great content with practical knowledge. Hats off to you !!!
After looking for some time for best material which truly explains this topic, and try to dig deep enough you clearly delivered, thanks Afaque.
Glad it was helpful, appreciate it :)
One of the cleanest explanation I ever come across on the internals of Spark. Really appreciate all the effort you are putting into making these videos.
If you don't mind, May I know which text editor are you are using when pasting the Physical plan?
It's a great video with a great explanation. Awesome. Thank you for such a detailed explanation. Please keep doing such content.
one of the best videos i came across on spark query plan explanation. Thank you! :)
Appreciate it @myl1566, thank you!
Very useful, video man, thanks for explaining things in so much details, keep doing the good work.
Thank you so much for making this video. this is really very helpful.
Great explanation!!Keep uploading such quality content bro
This is really good, thanks so much for this explanation!
Bro, you dropped this👑
This is pure gold, congrats bro , keep the good work
Thank you @garydiaz8886, really appreciate it! :)
This is really informative, such details are not even present in the O'Reilly Learning Spark Book. Please continue to make such content. Needless to say but I have already subscribed.
please do more vedios bro. love this one
Thank you @varunparuchuri9544, really appreciate it :)
Bhai bhot bhadia content banaate ho. Love your vdos. Please keep it up. You have great teaching skills.
Bohot shukriya bhai sahab!
By far best content i have seen on explain query thing!!! Keep it brother. Good luck!
Glad, you liked it, thank you! :)
Great content brother. Please post more 😁
Bro, I am beginner but i was able to understand everything. Really great content and ur explanations was also amazing. Please continue doing such great videos. Thanks a lot for sharing .
@thecodingmind9319 Thanks for the kind words, means a lot :)
By watching your first 15mins of youtube video and I am awed beyond my words.
What a great explanation @afaqueahmad. Kudos to you!
Please make more videos of solving real time scenarios using PySpark & Cluster configuration. Again BIG THANKS!
Hey @VenuuMaadhav, thank you for the kind words, means a lot. More coming soon :)
"God bless you! Great video! Learned a lot"
Underrated pro max!
Amazing content! Thank you for sharing!
Thank you @crazypri8, appreciate it :)
Great explanation.
This was awesome!
this is gold. Thank you very much!
@user-meowmeow1 Glad you found it helpful :)
Great video thanks for sharing. I definitely subscribe
Great Content. Nice and Detailed!!
Thank you @shaheelsahoo8535, appreciate it :)
Excellent job 🙌
Thanks @prasadrajupericharla5545, appreciate it :)
Excellend content, please make more videos like this with deep understanding of "how stuff works"... Highly Appreciate it. Love from 🇵🇰
Thank you @MuhammadAhmad-do1sk for the appreciation, love from India :)
I am sure that down the line, in a few years, you will cross 100k subscribers. Great content BTW.
Hey @CoolGuy , thanks man! Means a lot to me :)
Very Good explanation...Keep Going
Thank you!
Great Video!
Appreciate it @jjayeshpawar, thank you!
You are awesome man❤
Thank you for taking the time to create such an in depth video for Spark Plans. This is very helpful !
Would you also be able to explain Spark Memory Tuning ?
How do we decide how much resources to allocate (driver mem, executors mem , num executors , etc for a spark submit ?
Also Data Structures Tuning, Garbage Collection Tuning !
Thanks again !
Thanks for the kind words @crystalllake3158 and the suggestion; currently the focus of the series is to cover all possible code level optimization. Resource level optimisations will come in much later, but no plans for the upcoming few months :)
Thanks ! Please do keep uploading, love your videos !
quality content
Thank you!
Bro please make more videos !!!
I loved your explanation and understood it very well. Could you help me to understand at 23 mins, if we have join key as cid and group by region. how the hash partitioning works. will that consider both?
Great explanation! I love the simplicity of it! I wonder what is the app you use for having your Mac as a screenshot that you can edit with your iPad?
Thanks @kvin007! So, basically I join a zoom meeting with my own self and annotate, haha!
You were too good!
Thank you!
Great explanation man! Thank you! What's the editor that you use in the video to read query plans?
Thanks @venkateshkannan7398, appreciate it. Using Notion :)
Hi Afaque.
Do we have any library or can we create a UDF for understanding why some records got corrupt while reading file?
I have a nested XML file with large number of columns and I want to understand why some columns are going into corrupt. Couldn't find anything helpful online.
This video would be greatly appreciated.
At the very end of the video 38:36, we see that the cast("int") filter is present in the parsed logical plan and Analyzed logical plan. I am a little confused as to when we refer those plans. Can you please explain?
I am doing coalesce(1) and getting error as : Unable to acquire 65536 bytes of memory, got 0.
But when i am doing repartition(1), it worked. Can you please explain what happens internally in this case?
if it is doing local aggregation before shuffling the data then why it will throw out of memory error while taking count of each key when the column has huge distinct values
Just 10 minutes into this notebook and I am awed beyond my words.
What a great explanation Afaque. Kudos to you!
Please make more videos of solving real time scenarios using Spark UI and one on Cluster configuration too. Again BIG THANKS!
Hi @mohitupadhayay1439, really appreciate the kind words, it means a lot. A lot coming soon :)
Can you please prepare a video showing storage anatomy of data during job execution cycle? I am sure there are many aspiring spark students who may be confused about the idea of RDD or dataframe and how it access data through apis (since spark is in memory computation) during job execution. It will help many upcoming spark developers.
Hey @udaymmmmmmmmmm, I added this video recently on Spark Memory Management. It talks about storage and responsibilities or each of memory components during job execution. You may want to have a look at it :)
Link here: czcams.com/video/sXL1qgrPysg/video.html
Thanks for the content and when can we expect new video?
Coming soon, in the next few days! :)
In exchange hashpartitioning what is the significance of number 200? what does that mean?
200 is the default number of shuffle partitions. You can find the number here in this table by the property name "spark.sql.shuffle.partitions" spark.apache.org/docs/latest/sql-performance-tuning.html#other-configuration-options
I have doubt when the data will be distributed to executor is it before scheduling the task or after scheduling the task and who assign the data to executor
Hey @sangu2227, this requires an understanding of transformations/actions and lazy evaluation in Spark. Spark doesn't do anything (either scheduling a task or distributing data) until an action is called.
The moment an action is invoked, Spark creates a logical -> physical plan and Spark's scheduler divides the work into tasks. Spark's driver and Cluster manager then distributes the data to the executors for processing :)
Hello @afaqueahmad7117, thanks for the great video. While explaining repartition, you mentioned you’ve a video on the AQE. Please can you link that as well?
Thanks @nijanthanvijayakumar, yes that video is upcoming in the next few days :)
Can't wait for that@@afaqueahmad7117
These CZcams videos are so much more helpful. Hats down one of the best ones that explain the Spark performance tuning and internals in a very simplest of forms possible. Cheers!
Fab Cotenet
Hi Sir, you mentioned that you referred AQE before. Can I get that link ? I want to know about AQE
Yes, I will be releasing the video in the next few days. :)
@@afaqueahmad7117 Thank you sir.
You mentioned that for coalesce(2) shuffle will happen, but later you mentioned that shuffle will not happen in case of coalesce hence no partitioning scheme. Could you please explain it in detail?
So, coalesce will only incur a shuffle if its a very aggressive situation. If the objective can be achieved by merging (reducing) the partitions on the same executor, it will go ahead with it. In case of coalesce(2), its an aggressive reduction in the number of partitions, meaning that Spark has no other option but to move the partitions. As there were 3 executors (in the example I referenced in the video), even if it reduced the partitions on each executor to a single partition, it would end up with 3 partitions in total, therefore it incurs a shuffle to have 2 final partitions :)
@@afaqueahmad7117 Thanks for clarification.
Hi Afaque, how can I download the data files you are using? I want to try it hands on :)
Should be available here: github.com/afaqueahmad7117/spark-experiments :)
What drawing board are you using for those notes?
Using "Notion" for text, "Nebo" on iPad for the diagrams
@@afaqueahmad7117cool thx!
Hi sir i came across a doubt
Consider the executor size 1gb/executor. We have 3 executors and intially 3 gb data gets distributed across 3 executors each executor is having 1gb partition after various transformations we came across a requirment to decrease the number of partitions to 1 partition for that we will use repartition(1) or coalesce(1). In this scenario all the 3 partitions will merges to 1 partition each partition is having size of 1 gb approximately. Collectively all the partitions size is 3 gb approximately. When repartition (1) or coalesce(1) all the 3 gb data should sit in 1 executor having capicity of 1gb only. So here the data is execeeding the executor size what happens in this scenario. Could you please make video on this requesting sir.
Hi @bhargaviakkineni, In the scenario you described above where the resulting partition size (3 GB) exceeds the memory available on a single executor (1 GB), Spark will attempt to spill data to disk. The spill to disk is going to help the application from crashing due to out-of-memory errors however, there is going to be a performance impact associated, because disk IO is slower.
On a side note, as a best practice, It’s best to also think/re-evaluate the need to write to a single partition. Avoid writing to a single partition, because it generally creates a bottleneck if the sizes are large. Try to balance out the partitions with the resources of the cluster (executors/cores).
Hope that clarifies :)
Local distinct on cust id doens't make sense and couldn't understand. How globally it does distinct count if the count is already computed. The reasoning behind why cast doens't push down predicate is not clearly explained and just as it's mentioned in the doc
Great Work buddy keep it up .... love your content, very simple to understand @Afaque Ahmed
Thanks a ton!