Data Engineer Mock Interview | SQL | PySpark | Project & Scenario based Interview Questions
Vloลพit
- ฤas pลidรกn 26. 08. 2024
- ๐๐จ ๐๐ง๐ก๐๐ง๐๐ ๐ฒ๐จ๐ฎ๐ซ ๐๐๐ซ๐๐๐ซ ๐๐ฌ ๐ ๐๐ฅ๐จ๐ฎ๐ ๐๐๐ญ๐ ๐๐ง๐ ๐ข๐ง๐๐๐ซ, ๐๐ก๐๐๐ค trendytech.in/... for curated courses developed by me.
I have trained over 20,000+ professionals in the field of Data Engineering in the last 5 years.
๐๐๐ง๐ญ ๐ญ๐จ ๐๐๐ฌ๐ญ๐๐ซ ๐๐๐? ๐๐๐๐ซ๐ง ๐๐๐ ๐ญ๐ก๐ ๐ซ๐ข๐ ๐ก๐ญ ๐ฐ๐๐ฒ ๐ญ๐ก๐ซ๐จ๐ฎ๐ ๐ก ๐ญ๐ก๐ ๐ฆ๐จ๐ฌ๐ญ ๐ฌ๐จ๐ฎ๐ ๐ก๐ญ ๐๐๐ญ๐๐ซ ๐๐จ๐ฎ๐ซ๐ฌ๐ - ๐๐๐ ๐๐ก๐๐ฆ๐ฉ๐ข๐จ๐ง๐ฌ ๐๐ซ๐จ๐ ๐ซ๐๐ฆ!
"๐ 8 ๐ฐ๐๐๐ค ๐๐ซ๐จ๐ ๐ซ๐๐ฆ ๐๐๐ฌ๐ข๐ ๐ง๐๐ ๐ญ๐จ ๐ก๐๐ฅ๐ฉ ๐ฒ๐จ๐ฎ ๐๐ซ๐๐๐ค ๐ญ๐ก๐ ๐ข๐ง๐ญ๐๐ซ๐ฏ๐ข๐๐ฐ๐ฌ ๐จ๐ ๐ญ๐จ๐ฉ ๐ฉ๐ซ๐จ๐๐ฎ๐๐ญ ๐๐๐ฌ๐๐ ๐๐จ๐ฆ๐ฉ๐๐ง๐ข๐๐ฌ ๐๐ฒ ๐๐๐ฏ๐๐ฅ๐จ๐ฉ๐ข๐ง๐ ๐ ๐ญ๐ก๐จ๐ฎ๐ ๐ก๐ญ ๐ฉ๐ซ๐จ๐๐๐ฌ๐ฌ ๐๐ง๐ ๐๐ง ๐๐ฉ๐ฉ๐ซ๐จ๐๐๐ก ๐ญ๐จ ๐ฌ๐จ๐ฅ๐ฏ๐ ๐๐ง ๐ฎ๐ง๐ฌ๐๐๐ง ๐๐ซ๐จ๐๐ฅ๐๐ฆ."
๐๐๐ซ๐ ๐ข๐ฌ ๐ก๐จ๐ฐ ๐ฒ๐จ๐ฎ ๐๐๐ง ๐ซ๐๐ ๐ข๐ฌ๐ญ๐๐ซ ๐๐จ๐ซ ๐ญ๐ก๐ ๐๐ซ๐จ๐ ๐ซ๐๐ฆ -
๐๐๐ ๐ข๐ฌ๐ญ๐ซ๐๐ญ๐ข๐จ๐ง ๐๐ข๐ง๐ค (๐๐จ๐ฎ๐ซ๐ฌ๐ ๐๐๐๐๐ฌ๐ฌ ๐๐ซ๐จ๐ฆ ๐๐ง๐๐ข๐) : rzp.io/l/SQLINR
๐๐๐ ๐ข๐ฌ๐ญ๐ซ๐๐ญ๐ข๐จ๐ง ๐๐ข๐ง๐ค (๐๐จ๐ฎ๐ซ๐ฌ๐ ๐๐๐๐๐ฌ๐ฌ ๐๐ซ๐จ๐ฆ ๐จ๐ฎ๐ญ๐ฌ๐ข๐๐ ๐๐ง๐๐ข๐) : rzp.io/l/SQLUSD
30 INTERVIEWS IN 30 DAYS- BIG DATA INTERVIEW SERIES
This mock interview series is launched as a community initiative under Data Engineers Club aimed at aiding the community's growth and development
Our highly experienced guest interviewer, Ankur Bhattacharya, / ankur-bhattacharya-100... shares invaluable insights and practical advice coming from his extensive experience, catering to aspiring data engineers and seasoned professionals alike.
Our talented guest interviewee, Praroop Sacheti, / praroopsacheti has a remarkable approach to answering the interview questions in a very well articulated manner.
Link of Free SQL & Python series developed by me are given below -
SQL Playlist - โข SQL tutorial for every...
Python Playlist - โข Complete Python By Sum...
Don't miss out - Subscribe to the channel for more such informative interviews and unlock the secrets to success in this thriving field!
Social Media Links :
LinkedIn - / bigdatabysumit
Twitter - / bigdatasumit
Instagram - / bigdatabysumit
Student Testimonials - trendytech.in/...
Discussed Questions : Timestamp
1:30 Introduction
3:29 When you are processing the data with databricks pyspark job. What is the sink for your pipeline?
4:58 Are you incorporating fact and dimension tables, or any schema in your project's database design?
5:50 What amount of data are you dealing with in your day to day pipeline?
6:33 What are the different types of triggers in ADF?
7:45 What is incremental load ? How can you implement it through ADF ?
10:03 Difference between Data Lake and Data Warehouse?
11:41 What is columnar storage in a data warehouse ?
13:38 What were some challenges encountered during your project, and how were they resolved? Describe the strategies implemented to optimize your pipeline?
16:18 Optimizations related to Databricks or pyspark ?
20:41 What is broadcast join ? What exactly happens when we broadcast the table ?
23:01 SQL coding question
35:46 PySpark coding question
Tags
#mockinterview #bigdata #career #dataengineering #data #datascience #dataanalysis #productbasedcompanies #interviewquestions #apachespark #google #interview #faang #companies #amazon #walmart #flipkart #microsoft #azure #databricks #jobs
For incremental laod why we go about MERGE or UPSERT. MERGE or UPSERT we use to implement SCD types. For incremental load what we want is to copy newly arrived data in ADLS. For which we keep track of some reference key, through which we can recognize the new data. For example, in an Order fact table lets say it is Order_ID which keeps on increasing whenever we get a new order.
Hi Folks, below is the solution to the PySpark problem written in >>SCALA
Please provide the interview feedback in few mins at the end to help more with this.
Please attach the questions list link(in view mode) that are asked in mock interview in description
Great Initiative Sumit Sir !
thank you. A big thanks to people who are participating in this.
Good initiative. This is quite helpful on how to answer the scenario based questions, with an example. Thank you sir, Ankur and Praroop! ๐
So nice of you
Sir please make videos on topics like " Someone working in Tech Support from past 5 years and now moving to Data Engineer" What they should write in their resume like in experience section... Whether should give try as fresher or whatever
Sir, I also have the same question.
Yes that is very valuable. As most of the people are working in different roles but being in support roles in data field we are interested to switch into data engg.
surely will release a video on this soon
Great video for new data engineers like me.
Glad you enjoyed it
great content! very insightful questions and answers!
Glad you enjoyed it!
Please make videos for freshers as well, because these days no one is looking for freshers for data engineering roles...
will make a video for sure
Please also some video regarding what kinds of problems data engineer face in their day to days working
noted, will bring a video on this soon
Solution for Pyspark Problem
def location_f(loc):
if loc == 'CHN':
return 'CHENNAI'
elif loc == 'AP':
return 'ANDHRA PRADESH'
elif loc == 'HYD':
return 'HYDERABAD'
else:
return loc
re_location = F.udf(location_f, StringType())
df1 = df.withColumn('ref_id1', F.split('ref_id','\DIV-|\_')).drop('ref_id')
df2 = df1.withColumn('ref_id', F.col('ref_id1')[2]).withColumn('location', re_location(F.col('ref_id1')[1]))
df3 = df2.select('name', 'ref_id', 'salary','location')
df3.show
from pyspark.sql.functions import col, lit,when
df_employee.withColumn("LOCATION",
when(col("REF-ID").like("DIV-CHN%"), "CHN-CHENNAI")
.when(col("REF-ID").like("DIV-HYD%"), "HYD-HYDERABAD")
.when(col("REF-ID").like("DIV-AP%"), "AP-ANDHRA PRADESH")
.when(col("REF-ID").like("DIV-PUNE%"), "PUNE-PUNE")).show()
Hi Sir, Thanks for this series, very insightful. Just a query, does majority of the interviews goes till coding part or majority cases its theory only? or is it mix and match?
Yes they do
thank you so much sumit sir its really helpful
Happy to share more such informative videos for the community!
Good Work Praroop โค
Praroop has rocked it.
Hi Sir ,Request you to please upload more videos on Data engineer mock interview
one video daily for next 30 days
sir please make complete video on sql and mock interviews too
Definitely, will be covered in the upcoming videos
Please make interview session
for fresher.
surely
can u make a video for aws cloud as of azure
surely
I want to give mock interview.
wahh
๐
Really?
df_new = df.select(col("name"),col("refid"),col("salary"),split("refid","-")[1].alias("l"),split("l","_")[0].alias("loc")).drop(col("l"))
final_result_df = df_new.withColumn("location",when(col("loc")=="CHN","CHENNAI")\
.when(col("loc")=="HYD","HYDERABAD")\
.when(col("loc")=="AP","ANDRA_PRADESH")\
.when(col("loc")=="PUN","PUNE") ).drop("loc")