Implementing Pyspark Real Time Application || End-to-End Project || Part-2
Vložit
- čas přidán 7. 09. 2024
- In this video we covered about Data Processing (cleaning) and validating
Data Clean for year_of_exp column using regex_extract::
pattern = '\d+'
idx = 0
df_presc_sel = df_presc_sel.withColumn('years_of_exp', regexp_extract(col('years_of_exp'), pattern, idx))
part1:
• Implementing Pyspark R...
link for file::
drive.google.c...
#azuredatabricks
#dataanalysis
#dataengineering
#pyspark
#pythonprogramming
#dataengineering
#dataanalysis
#pyspark
#python
#sql
To solve header error, replace csv code to this
"elif file_format == 'csv':
df = spark.read.format(file_format).option("header",True).option("inferSchema",True).load(file_dir)"
Thank ypu bro
Hello Sir, Link is not working to download the full code from google drive?
Why all null coulumn count is zero,when you dropped only two null value column
Hi Amarjeet, their i purposely did that, in the next part we will relive that...Thanks for watching
It is not good that for every transformations we eneed to execute all the code again end again. So what is the best practice ? Do in a notebook cell by cell ? And then develop the production code in py files when all tested in notebook ?
Still getting same header error even after reformatting
Hi nikhil, Use the below line the issue will be resolved.
df = spark.read.format("csv").option("header", header) .option("inferSchema", inferSchema).load(file_dir)
In the ingest.py file replace (header=header) and (inferschema=inferschema) to ("header", header) and ("inferschema", inferschema)
Hi sir how can i contact you for the coarse u not mentioned any contact?????
Hi Yogesh you can reach out to me in LinkedIn Lokeswar Reddy Valluru