Implementing Pyspark Real Time Application || End-to-End Project || Part-2

In this video we covered about Data Processing (cleaning) and validating
Data Clean for year_of_exp column using regex_extract::
pattern = '\d+'
idx = 0
df_presc_sel = df_presc_sel.withColumn('years_of_exp', regexp_extract(col('years_of_exp'), pattern, idx))
part1:
• Implementing Pyspark R...
link for file::
drive.google.c...
#azuredatabricks
#dataanalysis
#dataengineering
#pyspark
#pythonprogramming
#dataengineering
#dataanalysis
#pyspark
#python
#sql

[@kaushikvarma2571]

To solve header error, replace csv code to this
"elif file_format == 'csv':
df = spark.read.format(file_format).option("header",True).option("inferSchema",True).load(file_dir)"

[@memesmacha61]

Thank ypu bro

[@sachinmittal5308]

Hello Sir, Link is not working to download the full code from google drive?

[@Amarjeet-fb3lk]

Why all null coulumn count is zero,when you dropped only two null value column

[@DataSpark45]

Hi Amarjeet, their i purposely did that, in the next part we will relive that...Thanks for watching

[@skateforlife3679]

It is not good that for every transformations we eneed to execute all the code again end again. So what is the best practice ? Do in a notebook cell by cell ? And then develop the production code in py files when all tested in notebook ?

[@nikhilgr7539]

Still getting same header error even after reformatting

[@Vidush05]

Hi nikhil, Use the below line the issue will be resolved.
df = spark.read.format("csv").option("header", header) .option("inferSchema", inferSchema).load(file_dir)

[@balaa2670]

In the ingest.py file replace (header=header) and (inferschema=inferschema) to ("header", header) and ("inferschema", inferschema)

[@yogeshpalegar9269]

Hi sir how can i contact you for the coarse u not mentioned any contact?????

[@DataSpark45]

Hi Yogesh you can reach out to me in LinkedIn Lokeswar Reddy Valluru