Data Validation with Pyspark || Real Time Scenario

In this video will discuss about , how we are going to perform data validation with pyspark Dynamically
Data Sources Link:
drive.google.com/drive/folder...
#pyspark #databricks #dataanalytics #data #dataengineering

[@mohitupadhayay1439]

Amazing content. Keep a playlist for Real time scenarios for Industry.

[@ajaykiranchundi9979]

Very helpful! Thank you

[@listentoyourheart45]

Nice

[@vamshimerugu6184]

Great explanation ❤.Keep upload more content on pyspark

[@DataSpark45]

Thank you, I will

[@ArabindaMohapatra]

I just started watching this playlist. I'm hoping to learn how to deal with schema-related issues in real time.Thanks

[@DataSpark45]

Thanks a million bro

[@skateforlife3679]

Cool, but is it like this every time ? Like you have a reference df containing all columns and file name / path and you have to iterate over it to see if its matching ?

[@lokeswarreddyvalluru5918]

Yes

[@World_Exploror]

how did you define reference_df and control_df

[@DataSpark45]

we defined as a table in any DataBase. As of know i used them as a csv

[@OmkarGurme]

while working with databricks we dont need to start a spark session right ?

[@DataSpark45]

No need brother, we can continue with out defining spark session, i just kept for practice