Fastest way to Start Your Data Engineer Journey in 2024 - 100 Days Of Data Engineering Crash Course
Vložit
- čas přidán 15. 06. 2024
- How do you become a data engineer.
There are lots of skills you need to learn that range from Python, SQL, Data warehousing, etc.
Can you learn all of that in 100 days? The short answer is no.
But if you are looking to become a data engineer in 2024, this video will help you set a baseline! Also check out the checklist below.
100 Days Of Data Engineering Checklist
docs.google.com/spreadsheets/...
If you enjoyed this video, check out some of my other top videos.
Becoming A Data Consultant In 2024
• The Ultimate Guide To ...
Top Courses To Become A Data Engineer
• Top Courses To Become ...
What Is The Modern Data Stack - Intro To Data Infrastructure Part 1
• What Is The Modern Dat...
If you would like to learn more about data engineering, then check out Googles GCP certificate
bit.ly/3NQVn7V
If you'd like to read up on my updates about the data field, then you can sign up for our newsletter here.
seattledataguy.substack.com/
Or check out my blog
www.theseattledataguy.com/
And if you want to support the channel, then you can become a paid member of my newsletter
seattledataguy.substack.com/s...
0:00 Intro
2:19 How To Use The Data Engineering Checklist
3:59 Kicking Off Your 100 Days Of Data Engineering
4:12 Days 1-10 Of Data Engineering The Basics
6:02 Days 11-40 Deeper Dive Into Python, SQL, Data Warehousing
7:57 Days 41-50 Building A Mini Data Project
8:52 Days 51-70 Survey Of Data Engineering Tools
10:32 Days 71-100 Data Engineering Project
Tags: Data engineering projects, Data engineer project ideas, data project sources, data analytics project sources, data project portfolio
_____________________________________________________________
Subscribe: / @seattledataguy
_____________________________________________________________
About me:
I have spent my career focused on all forms of data. I have focused on developing algorithms to detect fraud, reduce patient readmission and redesign insurance provider policy to help reduce the overall cost of healthcare. I have also helped develop analytics for marketing and IT operations in order to optimize limited resources such as employees and budget. I privately consult on data science and engineering problems both solo as well as with a company called Acheron Analytics. I have experience both working hands-on with technical problems as well as helping leadership teams develop strategies to maximize their data.
*I do participate in affiliate programs, if a link has an "*" by it, then I may receive a small portion of the proceeds at no extra cost to you. - Zábava
If you guys want to learn more about data engineering, then sign up for my newsletter here seattledataguy.substack.com/ or join the discord here discord.gg/2yRJq7Eg3k
inv is invalid :
Also upload videos on same format on
Data Analyst
and
Data Scientist
I have 300 days to learn 🕶️
This is really funny because this year my company offered me the posibility to enter the Data Department (I'm a Software Dev.). I'm kind of fed up with coding everyday and for now I'm learning Elastic Search, Apache Nifi and Airflow, then I'll head to Spark and AWS. This video is pretty intereseting to serve as a guidebook. Thanks for stressing the part of "What have you learned so far?"
How is the work life balance in this career?
What do you think about ai replacing software and data jobs?
@@JKyadav111 About as good of a chance that AI will replace doctors and engineers
This couldn't have come at a better time for me, and a free sheets document to track all of this? So valuable, thanks!
glad you're finding it helpful!
Thanks for the 3m plan. Exactly 👍🏼 just what I need!
One potential key topic is how to bounce back from difficult client/project that did not go as successful as desired (especially in data consulting if you have had that experience). For me, because of this experience, it made me lose confidence during a client screening to try & explain this experience & re-frame it as positive.
Hmm, this does sound like a solid video topic! Also I hope you feel confident again!!
@@SeattleDataGuy thanks. U the best :)
looking into this now, solid background in data primarily SSIS, informatica, Alteryx with a good amount of "BI" work. Thanks for this guide, we use Azure in our current work environment but mostly only at the corporate level. I'm going to challenge myself to follow this guide so I can transition into being a data engineer!
Thanks for sharing brother! I'm just going to stick with this plan.
You can do it!
Will be following your map for the next 6 months as I am into 9-5 routine.
Thanks for the video. Another video idea would be one for finding / getting your first DE job. It's a bit of a struggle finding entry DE positions
That would be a great video, i can add it to the line-up
Like number 1,000🎉 Great video! I’m going to start this challenge, you gave us a lot good information😎
glad you liked it!
Data Engineering means different things to different companies. For example, you don't list SAP as a core competency, but some companies do. WIth that said, isn't it better to pick a lane and stay in it? I.E. one or two of the cloud providers, AWS or Azure or GCP, and master data engineering, data analysis, machine learning, AI services, etc.
I have a question? I am interested in becoming a data engineer but I want to ensure I go about things diligently. I see a lot of boot camps where I live in the UK and also the option to go to university. I’m open to both but what would be better for a learning time frame of 3 years?
Would pet projects be enough to show some knowledge of Spark/Databricks or Azure? I feel like without previous real work experience it's impossible to even land an interview for any data engineering related position.
Hi Ben, I recently got laid off and looking into following your guide here. My background is in google analytics and Looker but we didn’t use too much SQL. Debating between Data Engineering or Cloud Engineering. I do love to build and create new things. Do you have any advice? I need to find a job quick and concerned data engineer could be saturated. What do you think?
Thanks alot for the roadmap man! I appreciate. I just have a question for the cloud services like GCP, AWS courses should I take them all or just pick one. Also many are suggesting GCP. What's your take
Start with AWS. My suggestion would be to be book an AWS associate exam (Solutions Architect Associate ) 6months/1year from now and start preparing for different services through udemy course by stephaane Marekk. I did the same and cleared the above exam in 5 months (won't recommend to do it though as it takes a lot of effort in preparing , i did this only because my company was paying for the certification which i had to pass anyhow).
@@darshantawte7435 Thanks alot
This was more meant to give you an overview, AWS is a solid place to start!
Hello what course , would you recommend for study ?
I’m up for the challenger
Starts at 3:58
Yeah I didn't realize how long my intro was!
Would you recommend any cloud platform course like acloudguru or cloud academy?
Hey this is awesome. Thank you!
you're welcome!
Thank you
You're welcome!
Hi! I'm newbie in this field. Now, I'm following your roadmap in the video posted 2 years ago. Is this new roadmap better? Should I change my current roadmap to this video?
if you've gone through most of the prior roadmap, i'd finish that. I just wanted to create a timeline vs. the old roadmap that doesn't have a clear time constraint
I have a video idea for you.
What would be an example of the minimum project portfolio to begin applying for jobs without previous engineering experience? Like 5 legit projects that include ____ & _____ & ____? I have no idea what's a realistic amount of coding to be considered for a coding entry level job....
Should I learn a lot about database administration or just writing SQL?
Can you tell more about the project 11:51 "transcript some videos, podcast and then summarize it with chatgpt"
It wasn't my project but they set up a bot to scrape podcast and videos and then used chatgpt to shorten it.
can we get into data engineering with no experience? I'm good with Python and SQL.
its always a challenge, but often I see people go from data analyst to data engineer a lot.
@seattledataguy i have started the datacamps data engineer track. completed about 6 courses in there. what should i do . start this one or continue with that one
Which datacamp have you started attending?
Is data integration a part of data engineering or what exactly?
Yes it is. It is part of the "ecosystem" i.e. one of the things you might be required to do.
Thanks for the video!! I have a question: what is the book's name at 7:03 ?? Thank you!
The Data Warehouse ETL Toolkit: Practical Techniques for Extracting, Cleaning, Conforming, and Delivering Data
@SeattleDataGuy Can AI effect totally to job role of data engineer
Great awesome...best wishes from kiranu😢
Thank you but why the tears
Amaaaazzziiinnngggg
Glad you're finding it helpful!
is Data Integration engineer the same as Data Enineer
I think it comes down to semantics. Some companies might call it ETL Engineer but it is essentially the same thing. Think of it this way, companies have pain points and they usually search for someone with a specific skill set to fix that specific problem (or a set of problems). It's like companies who are looking for a "python developer", "MySQL Developer", "PostGres Admin" etc.... but when you read the job description, the skills requirements listed are basically a Data Admin, Data Engineer, or even Data Analyst. Just know in the IT industry in general there is a lot of crossover in skills and duties. A Data Engineer is a "Software Engineer", "Database Admin", "Data Analyst" (some times), " Data Cloud Engineer/Cloud Engineer" etc...just read the job description and determine if it something you can do. Do not get hung up on job titles.
But you told in your vedio that DE will end in a decade....so why this roadmap?
I don't recall saying DE will end, but it will change. When i started everyone was on hadoop, now everyone wants to be on snowflake, databricks, spark, etc.
is data engineer a recession proof or layoff proof job?really worry if I should learn this under economy downturn😢
I don't think any job is recession proof. It's a reliable job but you never know when a company decides to cut teams
just curious.can you land on data engineering intern with this 100 days knowledge.
It likely depends on your background. Perhaps, but you'd have to focus on making sure you could pass an interview
@@SeattleDataGuy thanks I'm actually a Computer systems engineering student currently in 4th year have basic knowledge on Databases and cloud computing I'm hoping to enter Data engineering field in the future
The first sql video uses SQL Server management studio but I use Ubuntu. Is It a must I learn it or install windows VM?
You could also use a different database like mysql
can't access that spreadsheet :(
Were you able to access it? I have tested it and it should work
is there a way to learn cloud tools without spending a ton of my own money?
I believe most cloud solutions offer a free tier or trial. For example snowflake does like 30 days free or aws has its free tier - aws.amazon.com/free/
@@SeattleDataGuy hmm okay thanks I'll probably make a new account if my project takes too long 🤣
Fabric is the future…
Luffy reference at 2:04 🐐validates the entire video!! Really appreciate the list breakdown, Benjamin!
hahha glad you liked it!
I am doing the IBM Data Engeneering course from.Coursera. I hope to be ready to create a project and get hired for a first job after that.
GOod luck!
But it’s hard to crack data interview now
It can be difficult, but i believe you can do it!
the discord invite in the Google sheets doc is invalid :(
the link in the pinned comment works.
let me add it
Which is Kim's book? czcams.com/video/9FVchWw3EbU/video.htmlsi=JkVtysrM5iKPSNr4&t=423
there are a few references in the excel link but you can find the book here 160592857366.free.fr/joe/ebooks/ShareData/The%20Data%20Warehouse%20Toolkit.pdf
really like your content but your body shakes quite a lot and that is quite distracting
THIS GUY! 🤣 LOOK AWAY THEN. HAHAHAA
guess i have to turn on audio-only mode
@@malcorub