Video

Really waiting to see if you can add some real world use cases to your videos to strengthen our understanding. It will be appreciated a lot man!

Bro do the videos regularly on spark it will be very helpful. Thank you

Great explanation!

Amazing. This is just too good. Will share with my team also.

This is awesome stuff..The executor Tuning concept is explained at a very granular level.

sir can you please bring python and sql series for prep of interviews and also basics of it , remaining of the content is just great!

I don't think this kind of videos are available on Spark anywhere else. Great work Afaque!

What drawing board are you using for those notes?

Hey @afaque, this is top class stuff, thanks for putting in all the effort and making it available for us. Keep going :)

This is excellent Spark content videos. It is prefect explanation on Spark performance concept.

Top Class brother! Simple, Amazing and impactful. You deserve great appreciation to bring these internals. May God bless you with great health, peace, mind and prosperity! Keep growing.

You are a gem bro. The content that you bring here is terrific. ❤❤❤

you talked about yarn application master. is it driver which contain application master container right? means we are assigning driver memory as 1 gb. right?

if we have already alloting 1 core and 1 gb ram for yarn/os deamons then why do we need to allot seperate 1 core and 1 gb or one executor for yarn resource manager?

Content is useful. Please make more video 😊

Hi Afaque, it is was a really nice video. Never got such detailed understanding anywhere. Do you also provide 1:1 session? If yes, I am highly interested.

Thanks and please provide contact details .Also do you take classes?

I think its a bit dumb for spark to keep this value static... why not rather have a "target shuffle size(mb/gb)" config in spark. I wish the spark planner was a bit more sophisticated.

@Afaque thank you for making these videos. Very helpful. I have questions how do we estimate the data size? We run our batches/jobs on spark and each batches could be processing varying size of data. Some batches could be dealing with 300Gb and some could be 300Mb. How do we calculate optimal number of shuffle partitions?

No clarity is provided on when job is created. The stages are result of shuffle. The task is just a unit of execution

Local distinct on cust id doens't make sense and couldn't understand. How globally it does distinct count if the count is already computed. The reasoning behind why cast doens't push down predicate is not clearly explained and just as it's mentioned in the doc

Great Video!

Lovely!

Will that final actual executor memory again split into user,reserve, unified, overhead memory??

By watching your first 15mins of youtube video and I am awed beyond my words. What a great explanation @afaqueahmad. Kudos to you! Please make more videos of solving real time scenarios using PySpark & Cluster configuration. Again BIG THANKS!

Hi Afaque. Do we have any library or can we create a UDF for understanding why some records got corrupt while reading file? I have a nested XML file with large number of columns and I want to understand why some columns are going into corrupt. Couldn't find anything helpful online. This video would be greatly appreciated.

Hi Afaque, What if the join from Orders is on Product_id and customer_id in products and customers table, will this be still excutor local operation? Also if the Orders table is bucketed on product_id and if the join happens on Customer_id then bucketing the table on product_id alone does not make sense as it will not meet other join columns from orders table as we always not join in product_id in a real world scenario.

Very informative video. I have one question. I’m having databricks cluster and auto scaling is enabled. Will calculations change in that case?

Just 10 minutes into this notebook and I am awed beyond my words. What a great explanation Afaque. Kudos to you! Please make more videos of solving real time scenarios using Spark UI and one on Cluster configuration too. Again BIG THANKS!

hi sir, great content as always. just a question on the last part of the video: if i correctly understood you said to repartition(3) the big table so that rows are evenly repartitioned across the 3 executors and then apply the broadcast join. But in the code part you only performed a broadcast join without repartition(3). Why that? I am a little bit confused about that part. thanks a lot

Thanks Afaque. Terminology wise, Is this the same as Filter pushdown which you explained during the Query Plan video?

Hi Afaque, thank you again for this gold content. I believe that a similar series on Delta Lake optimizations would greatly benefit everyone. I realized that when i couldnt do bucketing with delta tables while following your guide.I hope you have plans to make one :)

Thank you so much again! I have one follow up question about partition during writes. If I use a df.write but specify no partitioning column or use repartition, could you pls let me know how many partitions does spark write to by default? Does it simply take the number of input partitions (total input size / 128m) or assuming if shuffling was involved and the default shuffle partitions being used were 200 , does it use that shuffled partition number ? Thank you

Precious 30 minutes, quality content

Wow! Thank you Afaque, this is incredible content and very helpful!

Absolute gem ❤❤ would like to have video on handling real time scenarios (handle slow running job, oom etc)..

Great work, going good. I hope you cover 2 more topic of driver oom and executor oom. Why it happens and how we can tackle it.

Really great content, all of your videos. Thannk you!! Just had a question out of curiousity - Does AQE only coalesce shuffle partitions or depending on the need, increase the shuffle partitions beyond 200?

what if the values are unique in join 1 to 1 join? will it create skew

I really enjoy your videos. Thanks for sharing your knowledge. I have a question about how you create these videos. It is an amazing way to create tutorial videos. Do you mind share what tools you use to make these videos? Thanks

Please make a dedicated video on shuffle partition... how it behaves when it's increased or decrease from 200

At 16.39 , when u use repartition(3) , why there are 6 files?

Excellent.. pls keep posting on internals of spark

Great explanation man! Thank you! What's the editor that you use in the video to read query plans?

Thanks Afaque for this great tutorial. This will really help while working on Spark Optimization. It would be of great help if you can tell how do you deal with this type of questions: - spark cluster size -- 200 cores and 100 gb RAM data to be processed --100 gb give the calculation of spark for driver memory, driver cores, executor memory, overhead memory, number of executors

Excellent job 🙌

Very nice explanation! Thank you for making this video.

Hell Afaque, your tutorials are excellent and I learnt so much about optimization techniques. I am wondering if you can add some real world use cases to your videos to strengthen our understanding. It will be appreciated a lot.

I have two accounts in CZcams and subscribed in both, Reason is you are putting some serious effort into the content. Beautiful Diagrams clear explanation accurate information is beauty of your content. Thanks, Afaque Bhai

After 3.0 salting is not useful