25 AQE aka Adaptive Query Execution in Spark

Video explains - What is Adaptive Query Execution in Spark ? What is AQE? What Optimizations does AQE provides with Spark ?
Chapters
00:00 - Introduction
00:51 - What is AQE and What it Offers?
02:13 - Join without AQE
05:16 - Join with AQE
05:23 - AQE Coalesce Post Shuffle Partitions
05:48 - AQE Skew Partitions Optimization
09:14 - AQE SortMerge to BroadCast Join
Local PySpark Jupyter Lab setup - • 03 Data Lakehouse | Da...
Python Basics - www.learnpython.org/
GitHub URL for code - github.com/subhamkharwal/pysp...
The series provides a step-by-step guide to learning PySpark, a popular open-source distributed computing framework that is used for big data processing.
New video in every 3 days ❤️
#spark #pyspark #python #dataengineering

[@PradyutJoshi]

Beautiful and simple explanation of AQE! Loved how clearly you have written the commands on jupyter notebook.
Thank you!
Keep up the good work. 🙌

[@easewithdata]

thanks 👍 Please make sure to share with your network.

[@user-dj4ht7rg2f]

Love your content :) I have one small question.. At 4:10 Spill memory is of 137MB and Spill Disk is of 77.2MB. If 137MB is spilled from memory why only 77.2MB is written in disk? Shouldn't it be 137MB? Can you please clarify this?

[@easewithdata]

Data written on disk are serialized and the data in memory is in deserialized format. Thus the amount will be less on disk. This is majir tradeoff when you are reading data from disks.
Please make sure to share with your network if you love this content ❤️

[@user-dj4ht7rg2f]

@@easewithdata Thanks for the quick response!! Sure, will recommend my mates.

[@rayees_thurkki]

After spark can you teach Airflow 🤗

[@easewithdata]

Yes only if you Like Subscribe and Share this with your network 😉

[@Consytec_Infraestructura_IT]

El video suena interesante pero que duro es tratar de seguir o entender el inglés con esta pronunciación 😢 🤷🏻‍♂️

[@easewithdata]

Please turn on subtitles to follow.