Getting into Data Engineering - Courses, Projects, and The Reality
VloĆŸit
- Äas pĆidĂĄn 9. 07. 2024
- Join My Data Engineering Courses - datavidhya.com/courses
How did I become a data engineer? What are the skills needed for data engineering? How to get a job as a data engineer and many more?
Data engineering is a new branch of engineering, but the data analysis field is growing rapidly and so is the demand for professionals who can handle this field.
Data scientists are redefining how businesses use data by helping them access, analyze, and interpret data in their everyday lives.
In order to be successful in this field, there are many skills you need to learn. Data engineering, or the ability to store and access data, is one of them. In this video, Darshil Parmar talks about how he became a data engineer and what skills he learned on the job.
0:00 - I was stuck in the hype
1:34 - How courses become meaningless
3:31 - My first data internship
5:12 - Projects I have worked on
6:44 - How is the job of a Data Engineer?
7:15 - How can you become a Data Engineer?
===================Follow Me Here===================
đŠđ» My Linkedin - / darshil-parmar
đ· Instagram - / darshilparmarr
đŻTwitter - / parmardarshil07
===================About Me===================
Freelance Data Engineer and Solution Architect.
I love learning about technology and teaching thing I know to other people. The goal of creating this channel was to teach everything I know to the world, I have spent most of my time learning things online and I want to share all my learning on this youtube channel.
I will be posting content related to data engineering, architecting systems, data science, machine learning, and interview preparation.
Thank you for watching the video, see you in the next video :)
My Accessories
đ Mic -amzn.to/3Bnu53n
đ± Camera (I use my phone) - amzn.to/3wOogIU
đ§ Headphone - amzn.to/3BsJxLB
âš Mouse and Keyboard - amzn.to/3zeF1yq
đ„ Monitor - amzn.to/3zk0Dts
#dataengineer #datascience #machinelearning
Sorry there was some editing issue in video, did not realise it but content is good
Thanks for this guideance
Hey darshil would you recommend MacBook for data engineer?
Bro0, u can make ur videos in mixed languages coz majority of learners are from non - eng background!... otherwise ur content is gudđđđ
Nice info
Bro what is full stack In data engineering
This was very humbling. Iâm learning by myself right now and this clarified a lot b/c I was getting a bit overwhelmed by all of the roadmaps I saw on how to become a DE. Thank you. (:
Thanks Man! I can really relate to whatever you're saying since I am just starting my career as a DS. I reckon, it's about patience and engaging more and more with hands on project. Keep up the good work buddy!
Thanks man
I relate so much with you when you mentioned about certification loop. However I am struggling in that as well
I have detailed video on this topic too, you can check my channel for more
Great work Darshil. Keep going. Kudos to you
100% relatabe. Thanks a lot for making this video :)
You've put it up Fantastically, Darshil bro!
An admirable summary of your roadmap. Thanks for sharing
This is very relatable! I found out that we do not learn by taking all the courses offered out there but we really learn by actual doing our own projects!
I agree, most people learn by doing...
Not everyone become an expert just going to the University...
Problem is how you know which projects will actually work for you, which projects could show your best skills.
Since every data related project has a different context and background, it's really hard to find the best one which can help you give 100%.
I never felt so related to any video this much.†this is purely my journey till first half.
Thank you so much. I needed this so badly
You are welcome
A Very honest and Informative Video !
Your content is really good glad to subscribe.
Loved your journey.Keep sharing !!your content is really awesome.
Thank you Aniket
Great video...Same experience...
Need to implement projects
I am currently working on data migration with plsql development. From last month in my organization I have aspired for big data engineer role, for that there is lot of learning they assigned to me. And I found your videos really helpful and motivated for the same aspiration. Thank you.
All the best Amit
Thanks, buddy, for sharing your useful learnings
You are welcome
Being a Data Engineer who started off as Data Scientist due to hype ... can relate very much
â
@@DarshilParmar Same here đ Started with Data Science and trying to move into Data Engineering. But what if Data Engineering also gets Saturated like Data Science? đ€đ
@@patrickbateman7665 sir what about data analyst ? Is data analyst good career option ?
@@YOUTHCAREERHUB Yes
@@patrickbateman7665 It will. But skills and expectations are very high from DE.
Great video Darshil. It has provided me a lot of clarity. Looking forward to more such amazing videos.
Thank you
Thanks man you are doing great đđ
Thank you
Inspiring Video.
Exactly my story. Right now i am in the certification loop. Hope I can follow your path.
All the best
Thank you so much sir!!! đđ»
Most welcome!
This is a really inspiring story. I have subscribed to learn more about data science and data engineering
Thank you
Almost the same story:
I started with Bg data engineering but didn't how and where to start didn't even know what is data engineering at that time, then learned Data Scinece did a few internship on it where I learned about data engineering. then moved to backend with Python. from there to learned a bit of DevOps. then learned all the technologies including Pytohn, SQL, Flask, Docker PySpark, Hadoop, Hive Kafka, AirFlow, PowerBi, and data warehousing.
Its been 4 years but I am really grateful of what I am now.
This is great content. I am going to mad over this decision over time. I see being a Data Engineer makes more sense in the long term.
All the best
this is some thing which i am looking for.....thank you so much..đđ
Most welcome đ
Indeed true...Agree with your viewpoints 101%!!! Accidently stumbled upon your series and was amazed at the way you have explained Spark in a crisp, concise manner with real-time examples/illustrations & found it helpful...!!! Thanks a lot Darshil!!!
Regarding your Data Science journey---a piece of advice to you---kindly focus on Statistics for data interpretation which would help you from the business perspective in the long run!!!
Tutorials regarding Computer Vision and other Data Science topics would be welcome!!!
Cheerz once again for sharing your valuable Gyaan.....indeed helpful!!!
Thank you Darshilđđđđ
This video has a potential to change my life.....
Thank you very much... â€â€
Keep going ..
Thanks you :)
Great..nice content ..good work and keep it up
Thank you, I will
The most honest answer you could find about Data Science hype out there. Kudos to you brother.:)
Thank you
Very precise
Thanks bro
Thank you
Amazing insights Darshil...Thanks a ton brother...erspect from Banga;lore...
I want to practice on projects for ETL, web scraping, and a use case to solve it and draw an ERD diagram to improve my skills as a junior Data Engineer. any advice on where could I find these things.
Same bro , started with data science now I'm a data engineer ,I took like â” courses, I can mirror myself in you.
Wow such an informative video!
Glad you liked it
Best strategy to learn programming -
Learn fundamentals
Start a project
Learn things that you need in the project
Correct
Tats true..... I have around 13 years of experience in etl projects ...specialist in informatica... currently am trying hard to learn machine learning. Started with python I learnt all concepts implemented but wanna do project In python can anybody help it.
â@@varuns4472hii
Thank you for the information on your path. I am starting with Python, but I was not sure it was the right starting. I found you advise very valuable...
Thanks for sharing...
Hey Santiago,
You are most welcome and you can also check out my Data Engineering roadmap
I am a data engineer. We rarely do coding with python or with any other language. We use all time SQL.
although you need coding understanding. But it's Not required. All this bootcampers are misleading you.
Thanks for the informative video. I am in a worse loop than this where I go to a site, buy a course and never return to it. Even if I do, I hardly complete it. I am a data engineer but need to upgrade to get into MAANG. Which type of projects should I target?
I'm starting my journey in data science. I've completed my B.E. petroleum and now looking to get in DS. I'm starting with python.Hope this channel helps people like me.
All the best, I will do my best here
Excellent video, more practical and less opinionated. Thanks bro...!! This is enlightening
Glad you liked it!
Sir what about data analyst ? Is data analyst a good career option in india or all over the world ?
Is it possible to do intensive while working in MNC
Could you share your duplication images deletion projects link?
Thanks. Greetings from Ecuador
Welcome!
Totally relate to you.Applied for Data scientist role and got Data Engineer role
Same here
Some Links to practice , tutorial to learn?
Very nice Vid! Can I have some project ideas while learning Apache Spark?
Thank you!
Check my project ideas to understand more
is i5 10th gen a good laptop configuration for using all sutffs to become a data engineer? or should i go for with Ryzen 5 series?
@Darshil Parmar suggest me good laptop for data engineering and Machine learning projects
can a guy passed out in 2018 from eee become the data engineer? Do they care about degrees in interview or the projects? or both?
Working as network engineer but wanted to grow my career in data engineer what should I do? Where will I get online classes so that I can get good command....
Nice insightsđđ
Thank you dharmil bhai
What are the tools needed to bcme data engineer
thnku brother â€
bro its really helpfull to me i'm too confuse to decide what can i do
I have two years of experience as a MySQL DBA. Am I eligible to land in data engineering and get internship ??
The Aero got međ
Subscribed :)
Hey Darshil, Brother i am an experienced professional working with ETL(so i have good understanding on SQL and RDBMS) since 8 years, I have learned myself python & Tableau and also i have done 5-6 projects on Data-Analysis but i am still not very sure if i should look to switch my career to Data-Analyst or to an Data Engineer.
After watching your video now i feel that i can more relate with Data Engineer than a Data Analyst according to my skill-set.
Any suggestions Brother ?
Sir meri bhi help kar dijiye please... Life barbaad ho rhi h.... Please sir guide me
Where do you get data engineer internships??
I was technically very good in machine learning as good as I have published one research paper in ieee.
I gave 6-7 interviews and in all of them I was rejected mainly due to my speaking skills which can be improved but what if it is in my genes i.e I am an extreme case of introvert like I have all the technical knowledge of the subject but It is too hard for me to express that to group of people.
Hence I had to let go of that dream.
You have to work on it, I am an introvert too, it took me lot of courage and trail to start this and many other things.
Go out, talk and get uncomfortable. That's the only way to go forward
Don't let it go bro. Keep learning and practice soon u will see different
Hello can you help with AWS beginners projects !!! Please
Hi. Do you take classes for data engineer?
Superb!
Thanks a lot!
I'm subscribing from this video because i find this video informative
Thank you
Hey, I would love to know more about your photo deleting model to see how it actually perform. Do you have any video or anything of that model?
I don't have video but it is on my Github - github.com/darshilparmar/Exam_Notes_Detection
@@DarshilParmar perfect thanks đđ»
My journey is similar. No idea of what to do and following the trend. Landed a job as data engineer with not much idea. Learnt a lot from my 1st project but then in next ended up feeling like there's no growth. It sucks! Feel like mind is not stable, gets bored of learning through courses. Aimless and wandering in search of satisfaction.
Hi ,r u working as data engineer in product based company.
Great Video ! Can you please share some thoughts on other technologies as well, how to choose between DS/Salesforce/Blockchain.
Thanks
All these 3 things are different, it depends on your goal
Hello!
I have been preparing to make a career switch to analytics from a Non-Stem background. I did Google Data Analytics Certificate but it taught me R. I am now stuck in R vs Python hell. I talked people and almost every person who is in Indian region tells me to Focus on Python libraries instead of R.
After completing the course I tried to learn R from various sources(book & online articles) because the R covered in the course wasn't good enough.
Again, I am getting stuck and in dilemma whether to get deeper into R and continue making projects in R or dive into python numpy, pandas etc. & gain skills to use on the projects.
I am not able to find the way.
Please throw your. 2 cents on me.
Learn python it has much broader scope and community support
Yes please learn python
Please make a video on how to get started from the beginning. As my story is same as yours i do some of the courses but i cannot solve some problems. So please make a video related to it
Please check my channel and you will find everything
How you complete this project without knowing anything still confused . delete pics which we dont need
Need a suggestion bro, right now I am into etl informatica i want to go for data engineer roles what all additional skills i require please answer the question if you have idea
I have roadmap, that will help you. Please check that on my channel
Hi Darshil, how can i apply for internships?
Hi Darshil,
I am from Commerce background and planning to learn data science, I am completely newbie and want to start from scratch.
Would you suggest something and do you think I should give it a try?
Thanks
Yes, you can do it but you will have to work really hard. It's not just about learning programming language and tools you will also need to understand computer science fundamentals
@@DarshilParmar Thanks.
I want to start my career as data analyst and how can I transform to data engineer
I'm having 2 year of experience in software engineering but looking for change in big data. Find difficult to enter as fresher in big data. Can anyone suggest me what should i do
I have a roadmap of this, just see that and fill your skill gaps
I know how to do all the projects on your channel, understand data structures, ETL pipelines and its still hard to get a job because they want you to have experience
Try internship first and then go about jobs
Also if you know how to do then build some projects and showcase that.
Projects=Experience too
is it possible to do some bigquery, google cloud projects? since this seems like a growing demand with the google cloud certificate being highly rewarded.đȘ
FIRST OF ALL
Good explanation bro
Thank you
Hello Darshil,
I m not techincal person with 15 yrs of Finance Background, i hv taken brake & Power BI n Mysql... do u think it wud b feasable for me to b Data Analyst..
Hi Darshil Parmar,
Thank you so much for the information and suggestions.
Please let me know, What level of DBMS skill is necessary for a Data Engineer role?
Thanks again
I have a complete roadmap on my channel on this, please check it
Agree. Finishing the course about a technology or even earning a cert about it would not guarantee that you're capable of actually implementing it, not mentioning in the production environment. BUT...before start a project, at least you need to know the basics about it. Follow step by step tutorials to build a project wouldn't help much either.
Besides, really?
6 months - a year for a decent internship? That sounds scary!
finished watching
We have same journey đ
Project is very important.
Suggest some great tensorflow resources before starting off with projects, please....
Why tensorflow?
Hii i am also interested on data science.. Can you help me
Currently i'm working on data migration project in my company we are moving oracle db to postgres
Going good
you applied for internship even after if you have web development experience ? is it possible to get job directly without internship if u have prior eperience example web development?
I didn't directly apply to a new internship directly, I spent around 1 year learning about Data Science/ML and then applied for that.
Getting a job without experience or internship is difficult but possible if you have skills and projects to back the same
Thanks Darshil for the informative video, I am working as data Engineer in Azure Cloud, I want to work in AWS as Data Engineer, can you guide me how can i find the internships on internet, or linkedin or xing or any other platform to achieve it so that i can learn aws while working in project??
You can do certification AWS SAA and get started easily
@@DarshilParmar hello darshil, can contact you on your mail and discuss on this in details, if possible ??
Hi , actually am working as java developer..exp is 2 years . I want to move in data engineer. Is it good way?
I am on the same path, currently working with 2nd internship and it is paid (25k). Your video suddenly pop-up in my CZcams recommendation. I can see my future here. đ
Welcome brother
Avinash Kumar What is your education qualification? Where did you start your journey?
@@pratikshakhalane9484final year B.E
@@AvinashKumar-ym2zq from where did you learn it? Were you self taught?
@@pratikshakhalane9484 Yes, most of the things are self tough as @Darshil have mentioned all the ways.
Great video. Thanks for sharing your journey. Hope you don't mind me sharing the following 2 points about this video.
Please loose the sound effect when you show a slide in your video. It does not help the watching experience.
The other point is about your delivery. It would be so much better if you spoke a bit slow, so that we connect with the technical details you are talking about.
On the whole, excellent work bro, and please keep it up.
Hello,
Thank you for watching this video and taking your time to comment, I will keep in mind these points and I already started working on it!
Thank you for valuable feedback
Hey, i want to know that in Andrew Ng course should i follow him, and learn octaves or try other programming language....
When I did that course I did everything in python
This video might help
How I Would Learn Machine Learning (If I Had to Start Over)
czcams.com/video/HU8K6o8nrJA/video.html
@@DarshilParmar ok thank you
What is Your college name for btech?
I completed B. Tech . Now I am a lead generation specialist. But I want to become data Engineer. , Your content is really best for that
Thank you
Python, sql, link please.. Tutorial
i don't know but i am still confused from where to start, should i start python or R or anything else first?
Start with Python, if you don't have knowledge of any other programming language yet, else it's upto you
Python, because it has a better user community
Yes go with python, don't think much