Video

It would have been Great if each step in cell was explained instead of whole cell. Thanks anyways

EaseWithData content is amazing. Thank you. Please help on this one. has anybody able to execute this successfully, I am getting java.lang.ClassNotFoundException: org.postgresql.Driver exception, few questions, 1) Will the jar download automatically in the path when spark code cell is run? is the jar version very old which is not found in it's site or is this version of postgres db still relevant? 2) Do we need to establish the network bridge manually one time so that two containers spark and postgres can talk 3) if the bridge network needs to be created then is it done in adhoc inline script before running the container in cmd screen or it needs to be done in the docker compose file? 4)I am not able to go further until I clear this section, since most of the stuff after depends on postgres db, please help. Thanks

Hi Subham, @Ease With Data- I followed the installation procedures in other playlist. But seems it is not working as expected. Can you help me to fix it? I can share more details. Basic chapters are fine with any of the code repository. but to run code with below line "18Optimize join" is not working. even jobs UI with 4040 port also not working spark = ( SparkSession .builder .appName("Optimizing Joins") .master("spark://f6d8b23a8515:7077") .config("spark.cores.max", 16) .config("spark.executor.cores", 4) .config("spark.executor.memory", "512M") .getOrCreate() )

Nice :)

Nice explanation 👍🏼

Love your content :) I have one small question.. At 4:10 Spill memory is of 137MB and Spill Disk is of 77.2MB. If 137MB is spilled from memory why only 77.2MB is written in disk? Shouldn't it be 137MB? Can you please clarify this?

I am using your pyspark-jupyter-lab docker file, But when creating spark session ,I am getting java runtime error. Java gateway process exited

how to optimize naukri if you are doing career transition into data field Civil Experience: 3.5 Education: MCA I am a bit confused with resume summary also. as recruiters only consider relevant experience, it litrally feels like have i wasted my career or 3.5 yrs. I am stuck can you please guide?

Nice explanation thanks for the knowledge sharing 🙂 please continue this playlist it is very helpful.

Got the "scram authentication is not supported by this driver" error while trying to connect to postgres. This is driving me nuts.

Hello, what if I wanted to do this in VSCode instead of Jupyter Notebook with docker as shown in the video?

What is invited to apply in naukri?and how to reply?

so in deployement mode the driver program is submitted inside a executer which is present inside a cluster. am I rignt?

Thanks a lot for sharing. It will be very helpful if you add data directory in git hub repo

Great 👍

thanks for this valuable insight, expecting the same video for apache iceberg and hudi in future

Very nice video

need to understand one thing why yyyy and dd not in capital letter is there any reason for that

Thanks for your lessons! You covered all the gaps between the main concepts.

Thanks for such a detailed work. You're a Hero.

Sir can u make a video to setup spark and hadoop with docker

Can a fresher become data engineer?

please make a video on creating resume for senior data engineers and please share the template thanks

Thanks, pls share the senior data engineer resume template.. will help

spark's standalone cluster is where on docker or any where please tell me my cluster execution codes are not running why?

Thanks bro :)

how to contact you sir

great work

thank you for your efforts

thanks for your efforts it helps lot

can accumulator variables be used to calculate avg as well? as when we are calculating the sum it can do for each executors but average wont work in the same way.

thank you in real time do we use cluster node or cline mode which you are using now?

localhost:4040 is not working when I done ".master("spark://e75727ddf432:7077")" how can be solved?

Why you made 32 shuffle partition if you have 8core, If one partition is going to process on single core, from where it will get other remaining 24 cores?

when I am refreshing my spark ui is giving error how can be solved giving this "spark://6b16b66805db:7077" and on "localhost:4040" also not working I give this".master("spark://6b16b66805db:7077")" how can be solved please.

Thanks for the Explanation, Very useful

From the vidoe : Select, where, group by etc ate transformations. We have narrow transformation and wide transformation. Wide transformation are those when data has to move or interact with data of other partitions in next stages

Again from video itself: executors are jvm processes, 1 core can do 1 task at a time, above pic we have 6 cores, so 6 tasks were possible

Shuffle is the boundary which divides job into stages

Great explanation.. liked the illustration that 2 counts happened and the fact that after local count and before global count, some shuffling happened

Awesome. Super explanation . I love it

why are we not using withCloumn instead of expr?

You are awesome bro.. Thanks a lot

What is the use case of slide duration in streaming app... any real world example ?

You are awesome man. I was trying to setup spark and kafka in docker for long time. done today. thank you very much

Hi, I have a doubt: How can we check if a stream has multiple sink from spark UI?

what if both the tables are very small like one is 5 MB and other is 9 MB then which df is broadcasted across executor?

one of the best explanation in depth, Thanks :) Could you please make a video on "end to end Data engineering" project, from requirement gathering to the deployment.

Hello, and thanks for the sharing of these useful videos. How to handle the writing in delta tables: Because the best practice is that the size of each parquet file should be between 128 MB to 1 GB. How to handle this situation while each batch has very less than the size that is mentioned? or how to handle to collect the number of batches and to reach the mentioned size and finally to write in deltalake.