I'm actually currently trying to move away from data science and into data engineering, which is apparently a rare path to take. Personally, I don't really love the exploratory nature of data science (though I used to think that would be awesome)... I'm just the kind of person who likes well-defined tasks/problems that have a real solution - so much of data science is just seeing what's there, and sometimes, there's nothing useful at all (explaining to the business why you just did two weeks of work and have nothing to show for it sucks). I also really don't care about applications of the theory or how the data supports business goals, I simply have no interest in business at all. So engineering really appeals to me - all the tech without the boring business things! Also, less talking and meetings lol :P
That's fair enough, it's great that you're so self-aware about your own preferences! I hope that you'll enjoy data engineering, now is definitely a good moment to pivot (given the rising demand for DEs)!
@@karolinasowinska Thanks! It definitely feels like the right move for me, and your videos have made me feel more confident in that decision, qnd identify the skills I need to grow. Thanks so much for this content! ♥️
You make such good points @Karolina. It's so true that by being in a support function, data engineers are often invisible unless things break. But the counterpoint is that the work is less rushed, more long-term and so much more satisfying. This makes for an awesome work-life balance.
Definitely, like you said, there are always two sides of a coin! Being in the spotlight all the time basically means worse work life balance, being in the shadows means awesome work life balance :)
Hi Remi! I work as an analyst at one of the decacorn at Southeast Asia, and I'm pretty sure that if you work at a company who has "move fast and break things" culture, there's no work-life balance in there :) Don't get me wrong, this is also the condition that suitable for you if you want to learn more. I agree with your point on "more long-term" and "satisfying", that's also why I envy data engineers and programmers because they can focus on the deep work. Just my 2cent.
3 minutes into the video: “as a data engineer you are not at the heart of the business you’re behind the curtains” That is music to my ears. I am an analyst and I don’t know what I got myself into. I HATE talking to clients and I hate talking business. I enjoy my job the most when I get to build automation tools for myself or others and feel that’s where I would keep my sanity; being ETL focused. I want to be a data engineer
It's so good to be aware of you like and what you don't like! And it sounds like data engineering will be the perfect career for you - you can consider yourself lucky, given how in-demand those skills are now! ;)
@@karolinasowinska any advice on progressing other than reading and doing tutorials (just picked up designing data intensive applications last night). No much opportunity to get mentored i. My current role
exactlyyyyy, I am currently an analyst and I am here because I realized I much more enjoy building tools and pipelines than analysing and searching for insights :D
As a data engineer/architect this is spot on! Some analogies on reason #1 - data is your plumbing, while your data engineers are your plumblers. You dont see the plumbing but you are thankful that you have clean water in the house. It may be taken for granted at times but any leaks or burst pipes is a big problem. Another analogy is in the body data is the blood. Its not a main organ but organs need it (clean blood) to function properly. On #3 true same with data science or ML where they now can be a black box provided as a ready to use product. But in the future someone still needs to know when and how to use these tools. Great video, more power!
Hey I just started my internship as an etl developer and while going through various articles online I found it comes under data engineering. And while research about that I found your profile and I'm so glad to have found you!
I can relate so much to this. I'm a data analyst being bribed so much to keep doing data engineering, I gave up on it and enjoying being a hot bun 🤑 btw part of my learning process was your ETL videos, so thank you! That's why I keep coming back to watch your channel!
I am an introvert but I am in consulting. Talking to clients drains me big time. After 31, I am now looking into starting all over again and shift to data engineering.
You know, I have realized few weeks ago, that my current job is actually Data engineering :)). It has a very weird name, but I actually build data pipelines in clinical trials, and I love it. I'm happy with my job, and I got here without knowing how fancy it will be in 2021 :D
There are a few more examples where you can be closer to the business as a data engineer like myself! Companies where data is their main asset can be a huge one e.g. Opta who are the leading sports data provider, Nectar who collect data about customers to analyse their shopping habits and even Bloomberg/Reuters who have amassed so much financial data that they can sell it for tonnes of money! I'm also hoping technologies like Denodo and Databricks will just change the role of a data engineer instead of getting rid of it. Great video as always and the new 4K looks great :)
Oh, YT put this comment in "held for review" for some reason! I completely agree, how crucial and "central" you are as a data engineer depends on what industry you're in. And I also hope that Denodo/Databricks will only make data engineers' lives easier! :)
As a data engineer student I have to say that your videos help a lot!!! There a lot of questions that you just easily answer. I hope one day I’ll be as talented as you.
You had me at work-life balance is great. I like to do many things outside of career, so I am sticking to the field (currently 95% finished with my DE Bootcamp). I also think that one can go from Data Engineer to ML Engineer later on (if they have data science background already, but the job market is too competitive).
Hey, Thank you for the video. At first i just wanted to stop the video when you said "these 3 reasons are just your thoughts". But later on as you speak and explained the unbiased scenarios 'giving the listener a chance to decide on their own knowing the + and - of the field', i loved connecting to this fact. and yes, i am also trying to switch to Data Engineer role from Business Analyst role and thats what pulled me to watch this video. Again thank you for honest and helpful thoughts and contributing to the community.
Thank you for adding your perspective and experience to this. I have been presented a choice to work either as a Data Engineer or a Cloud Engineer and I'm still deciding which to go for. I have barely started my career as a cloud engineer, but I've build quite a bit of knowledge over the past 2 years to prepare myself for a cloud engineer role, only to be presented with the choice of becoming a data engineer now. Both seems attractive to me because I like to grow as a technical professional, I like the hands-on and practical aspect of both roles.
I thought I wanted to be a Data engineer. I mean, it sounds pretty cool, but I like more machine learning in my life. Now I know I have to be ML engineer to have a little of both. Very helpful!
The point regarding Machine learning... it depends. Today we have MlOps and AutoMl Tools. I'm pretty sure these things are starting to be more related to data engineers than data scientist.
I landed on this video by searching "why it is difficult to comprehend Data Engineering".. Actually I'm a Data passionate person with a Data Science degree, on a career change, realized that Data Engineering was the core of data and many companies would have requirements for it in the future. But unlike Data Science I feel DE demands a lot of IT roots. As I'm from Finance background with business experience, getting overwhelmed with hundreds of jargons is sometimes annoying. But, if I think of what I can accomplish with DE, I keep moving. Thanks for this video. It has motivated me to continue.
I'm moving into a DE role from a Data Analyst role. I have a degree en economics, so I get what you're saying. You are not the only one struggling with this.
Great Video. About the third reason, I myself work to a big retail chain in Brazil and they are migrating the ETL to DataBricks, this is been made by an entire data engeneer team
Saying data engineering will be irrelevant because of automation is like saying web dev is good for nothing because there is web builder. Anyway, love the video!
In you opinion how will the demand for data scientist positions be in the future? There're also plenty of platforms that aim to reduce or simplify the data science tasks. What career between data science and data engineering will have the best future?
Like I mentioned in the video, no job guarantees the security that it'll be in-demand for decades. I wouldn't think about it too much. Go for what you enjoy doing most, and new opportunities will emerge for your skillset in the future. :)
If you *do* want to pursue a career in data engineering, visit: dataappreciationsociety.com/ (also don't forget to switch on 4K for the best experience (I finally got a new camera!) haha)
I actually think it's quite nice to be 'behind curtains' , knowing that you're indispensible. Just like so many people who work on designing, building and maintaining physical infrastructure we use every day. There are a lot of jobs this pandemic has highlighted as crucial for our normal functioning. These workers were often underpayed immigrants, very vunerable to pandemic. So, I see this as much as the consequence of our ignorace and lack of empathy for our fellow workers. THAT has to change.
Do you think that some data engineers would benefit greatly from knowing machine learning because they could be building a platform for machine learning to be able to be done very efficiently?
I took a course in SQL and did well, but I don't want to work with that tool again. I want to apply machine learning to robotic controls. She makes it very clear to me that I do NOT want to be a Data Engineer working for a bank or finance company. I want to continue working in the aerospace industry applying machine learning algorithms to flight controls, or robotic controls.
@@karolinasowinska about 20 years ago I heard about this technology called "sub vocal communication". They used it to help divers communicate underwater. They placed a transducer over the vocal cords to pick up the nerve impulses from normal speech. The system had to be trained so that they could connect the electrical pattern with known words. Very cool stuff BEFORE the term machine learning was widely used. I wanted to use it to help someone like Dr. Hawking recover speech and have some mobility. I feel this should be my life's work!
I’m trying to get into data engineering. But by watching your video, now I’m in a doubt. I’m from management background. And I have zero knowledge about IT field. I have a couple of questions to you. Can anyone from non IT field, will able to do data engineering job or not? And, Why is it so hard to find data engineers compare to data science? Thank you,
From my research a lot of employers interchange a data science title for data engineering tasks. To get familiar with tech, go through the CompTia A+ concepts to get a foundation on general hardware/software/networking etc … then look for Data Engineering courses. Coursera has one. Hope this helps you out, good luck😄
love it and yes we do use hot bun saying, also the unification of the data sources would be great cos now everything is going to cloud so you just have to adapt.. data cloud engineer.. move from ETL to ELT.
Hey. This is a top notch video. I'm about of start off my career as a data engineer. I agree with the first point you make. I wanna become a Product manager in the long run. Do data engineer's transition from data to product?
Hi Karolina, I'm a university student in the US studying data science and I'm a huge fan of your videos! Given the high average salary of data engineers, I don't assume it is a job a new college graduate can get. What would you recommend for a prospective data engineer's first job out of college?
Probably the 4th reason is- the job becomes monotonous after 3 months ( if you know airflow, pyspark and snowflake). Typically you would do the same kinda transformation each time. But in ML you keep learning coz every month there's something new coming up. (take BERT and GAN for examples- they were like state of the art last year but this year there are so many improvements).
Good point! Although I'd say that there are new data engineering technologies coming out as well, or systems ar migrated to different environments like AWS. Obviously depends on the company you work for! If the company doesn't innovate, then you're right, writing ETL processes can get monotonous
@@karolinasowinska - Agreed. But what I am saying if you see the sheer variety of work in ML(let's say an anomaly detection, ranking algorithm, customer retention etc) is also more(which makes it more interesting and challenging) when compared to data engineering work. Great video by the way!!!!! I hope you reach 100K subscribers soon :) :)
I feel the Data Engineering role can also be quite diverse, depending on the company strategy and/or needs. So it's actually very possible to find a DE, that gets to do ML maybe in a startup. So it might not be a monotonous role, it depends. There are also alot of Big Data tools out there aswell. For a Data Engineer who wants to self-improve by learning all these tools will have alot to learn
Thanks for this informative video. I am a 4th year University student and I am currently struggling to decide between Data Engineering and Hadoop Developer career path.
Brilliant video and I'm seeing it at the right time. I've been a researcher and a data scientist for many years and now they are offering me to move towards data engineering. Something that I have no clue about. So your video was quite eye opening. I think I should probably reject the offer? What do you think?
hey, yr speak is very nice. my russian ears like yu) I am data engineer, but I`ve wanted be ds. BUT my job is better and easier and very interesting than I worked Data Scientist.
Hi Karolina. What do you thing about the people that haven't computer science background want to begin as DE? In my case, I'm a chemical engineer but, I'm really tired of my job and I need a change.
Interesting. As a data analyst, I was aiming towards becoming a data scientist but what you explained broadens my horizon. On a side lobster note, are you reading (or planning to read) Beyond Order?
I might order it. Although I haven't finished his first book yet, I couldn't get through very descriptive analogies and stories. But I appreciate the guy's mind.
Hello All, I am Jay, a Data enthusiast. I have a good knowledge and portfolio as a data analyst but I am concerned about the competition in this job role as diversed field of professionals are transitioning into it. For this reason I am trying to consider data engineering path as a career. I am currently in the canada and if someone experienced can give me insights about future prospects of data engineering then it would be very helplul.
Data Engineering is a really good one. Jay, do check out my other videos on the subject. I am also working on a dedicated course on how to launch a career in data. If you'd like me to send you an alert when it goes live, please share your e-mail address! All the best, Karolina
Some companies never use ML so they don't need DS or ML Eng. what they do, they prepare data for Analytics or just BI dashboards, sales funnels, and so on. How about that?
Hey, I was just curious about the live coding interaction within job interviews. What are some coding challenges(python,sql)/ data structures & algorithms questions you faced in interviews ? How did you approach them ? I’m a bit nervous about this portion😅.I’ll be looking for a junior DE role soon. Thanks for the great content btw😄
Did many DE interviews and was part of interviewing Jr DE candidates too. Here are 5 tips: 1. When you're asked to write a code/query to answer a question or to complete a task, you first want to explain what your approach would be: What functions will you use? What lines of code are you considering to write? Why? What do you think the output will be? Think out loud before typing anything. That way, even if you fail to complete the task or to completely get the desired answer, you're still showing that you understood what you were asked to do and that you knew how to get the desired output. +Most of the time, the interviewer will guide you as you are brainstorming too which can be very helpful. 2. Don't be afraid to Google stuff when asked to write a query or a code. It's better than spending a long time guessing. You're not expected to memorize coding lines or functions. For example, if you know you need to use a window function then say, "I will need to write a window function here", then google how to properly write it. What matters in this example, is that you know what we had to use to get the right answer. 3. Don't guess different methods to write your code until the interviewer confirms which one is right to use. Instead, explain the method you would use to get the answer or to complete the task then try it. If it's right, great. If the interviewer says it's wrong or your code shows it's wrong, don't be embarrassed. Instead think of another approach you can try, say it out loud, and try again. We all learn and perfect codes through trial and error. So, yes of course, the interview would love the right answer on the first try. But if you're not getting the right answer the first time, it's your chance to show your problem-solving skills and your tenacity. Show that you're not afraid to make mistakes, to troubleshoot, and to keep trying until you figure it out. 4. If you really don't know how to write the code/query the interviewer is asking for, then be honest about it. You can say, "I've never done this before", or, "I'm not familiar with this". But don't stop there. Always follow with, "But I can try to figure it out". The interviewer may move on to another question or they may give you a chance to try. If you can try, I would suggest to Google what you're asked to do. You can try to complete the task with instructions you find online or if it's too hard for you, at least share what you understand from the instructions. This shows you're willing to learn and try unfamiliar tasks. 5. Be friendly and approachable during the interview. Even when you nailed all the questions, if another candidate also nailed all or most of the questions and on top of that, they were friendlier, the other candidate will be picked. Even during technical interviews, we look at the personality. No one wants to work with cold people or hire someone the team may not be comfortable with. Most importantly, every interviewer is asked to find a candidate that's a good fit for the company culture. Good luck!
Really enjoyed this channel for learning about data jobs. I career changed from being a traditional scientist into being a DE which I discovered by accident. Going from a field which is, allegedly, oversaturated into a field in demand has been quite the shock. I personally have no interest in DS although would you consider creating a mini ML course for people interested in it? On the subject of DS, I think if DE and DS had the same job prospects, a lot of people would prefer to be a DS. What do you think draws people to DS over DE? Love the channel, keep it up.
My theory on why DS is more popular is that most people want to make discoveries / draw conclusions rather than simply building the infrastructure that lets other people do that; they wanna analyze and build models and find answers - it just sounds more exciting! The thing is you're still solving problems and using logic when doing DE, its just not as "on the surface" if that makes sense.
Thanks for updating. Just wanted to know is there any age barrier if one wants to enter data industry. Thanks in advance. I have done UG in Biotechnology and master in Food science and no experience.
Hi @karolina sowinska Do we need to have a compulsory professional certification for entering data Engineer domain as a fresher or can we do it with courses and certificates from Udemy and all?
Great video 👍 I have all the basic skills to qualify for entry level data analytics job so far unsuccessful to land a job. Do you have any suggestions? Thank you
I am a data engineer and really like the content here, most of the points were correct. Except for the fact that DataBricks or any other ETL automation platform can only have as much impact on Data Engineering as the up-and-coming AutoML will on Data Science. So that's not a reason to not become a DE and pursue DS.
For someone with a Bachelor in Industrial Engineering that wants to expand to technology, which path do you think it would be better? Data Science, Data Analytics, ML, SE, Data Eng...?? Meanwhile, I'm learning Python and SQL. Looks like they fit in any of those. :-) Cheers!
I wonder how stable the demand for these careers is. By the time I learn enough marketable data engineering skills in a few years, will the field be as “saturated” as data science allegedly is? A bit scary to think about. I know you mention this on the scale of one’s entire lifelong career. But it feels like data science went from highly in-demand to oversaturated in only a few years, never mind a few decades.
Data in general isn't going anywhere, its growing. If you know SQL even at least and have good problem solving skills, you're sorted you can easily get another position within the field. Careers are fluid sometimes, it can be exciting.
I have been kind of doing Data Engineering for years because I write ETLs and Automation tools in Python but I have not used tools like Airflow and Spark professionally for self toy projects. On Reddit I was advised to polish SQL, especially Windowing functions etc. As an aspiring Data Engineer, what roadmap would you recommend for me?
2:14 *(1)* You are behind the curtain, not at business core (in majority of cases) 5:57 *(2)* This work has nothing to do with data science 10:03 *(3)* In the future (medium term) there will be fewer job offers. Today all jobs in technology are transformed.
5:55 I don't get it, why it is bad to pursue a career just cause you need money. Isn't that what 80% of the population do? And what would be a better choice? I think in software engineering it would be harder to earn a big buck when you are not interested in technologies. And about work-life balance: i've heard totally opposite. That data engineers are used to work on saturdays and sundays, just cause business users often need access to the data on monday. Is this true?
Because if the career you picked doesn't utilise your strengths and talents, you'll find it hard to progress, and other, more talented people will outperform you. So it's a better strategy to pursue what you're naturally good at - the money will surely come.
IS just me or she doesnt known how to not sell this job? She just love the job so much that it is noticeable. She listed more pro's than cons and the cons (as many already mention) doesnt seem like that at all. Love the energy of this video, i totally vibes this way :D
I'm a Dev DBA and often my work involves DE type activities, but as I move towards cloud, I am getting more involved with DevOps. I'm in a situation now where I have to choose a new job and I am getting interviews with both DE roles and DevOps, but actually choosing is not so easy. What are your thoughts about these career paths?
@@adamo1262 Well I somehow found the way to incorporate both. I went to a DevOps focused consultancy but with my data background they allowed me to take on cases in both areas when they arise. The majority of my work since I began is DevOps though (ELK stack, Kubernetes, Helm charts, Azure DevOps Pipelines, connecting to Sonarqube, and other CI/CD tools). I see a benefit in both because no matter what direction you take you are becoming more experienced in software development and its lifecycle which wasn't something I was really exposed to as a DBA. I think DE is definitely more focused than DevOps because there are so many moving pieces in DevOps, and it will take longer to understand how everything fits together. Sometimes I miss working with Data though, I've only had a few DE cases and when I have them I realise how much I enjoy that type of work. If I was doing it again, I would probably take a dedicated DE role. I say it because for me I find some DevOps work to be quite trivial (at least how I interpret the value of the work), with DE I see immediately the output of my efforts and I find that quite rewarding.
9:33 But wouldn't a data engineer that transitions into being a data scientist make for a better data scientist since they have more of a technical skillset/mindset. Like I think its easier to go from being a data engineer to a data scientist (Assuming you have the speaking skills/personality for it) than it is to go from being a data scientist to being a data engineer. I'd love some feedback. Right now I am in a bootcamp to become a data engineer. In the future I would like to work more on the business side, sort of like being a consultant.
11:45 - interesting thought, however once a given person earn big amount of money and can't progress further, with job as a whole - as a position - will going to be less and less wanted, then it isn't difficult to imagine someone wouldn't want to switch their career path to completly unrelated field. I couldn't imagine a Data Engineer moving into a spot of Frontend developer. Furthermore, i bet it would be a hilarious interview. Ofc this one is a bit less propable but even backend, or some weird mix of different things, not even connected to web dev. This is something that is comparable to losing half of the skills build over the years in a blink. Coding is coding, but the problem solving skills that were data-specific won't be of any use anymore, unless you somehow move into data science, but again i'm not eperienced so i don't even know if those two have something in common.
Wow, many of those cons sounded like pros to me hahaha
Haha I know right! ;)
yeeees!!!!
@@sanchesrfl it's a great industry!
Exactly what I thought...😁
Same here. Let me code and build pipeline and I am happy
I'm actually currently trying to move away from data science and into data engineering, which is apparently a rare path to take. Personally, I don't really love the exploratory nature of data science (though I used to think that would be awesome)... I'm just the kind of person who likes well-defined tasks/problems that have a real solution - so much of data science is just seeing what's there, and sometimes, there's nothing useful at all (explaining to the business why you just did two weeks of work and have nothing to show for it sucks). I also really don't care about applications of the theory or how the data supports business goals, I simply have no interest in business at all. So engineering really appeals to me - all the tech without the boring business things! Also, less talking and meetings lol :P
That's fair enough, it's great that you're so self-aware about your own preferences! I hope that you'll enjoy data engineering, now is definitely a good moment to pivot (given the rising demand for DEs)!
@@karolinasowinska Thanks! It definitely feels like the right move for me, and your videos have made me feel more confident in that decision, qnd identify the skills I need to grow. Thanks so much for this content! ♥️
> likes well-defined tasks/problems that have a real solution
> no interest in business at all
is'n it a contradiction?
You make such good points @Karolina.
It's so true that by being in a support function, data engineers are often invisible unless things break. But the counterpoint is that the work is less rushed, more long-term and so much more satisfying. This makes for an awesome work-life balance.
Definitely, like you said, there are always two sides of a coin! Being in the spotlight all the time basically means worse work life balance, being in the shadows means awesome work life balance :)
Hi Remi! I work as an analyst at one of the decacorn at Southeast Asia, and I'm pretty sure that if you work at a company who has "move fast and break things" culture, there's no work-life balance in there :) Don't get me wrong, this is also the condition that suitable for you if you want to learn more. I agree with your point on "more long-term" and "satisfying", that's also why I envy data engineers and programmers because they can focus on the deep work.
Just my 2cent.
3 minutes into the video: “as a data engineer you are not at the heart of the business you’re behind the curtains”
That is music to my ears. I am an analyst and I don’t know what I got myself into. I HATE talking to clients and I hate talking business. I enjoy my job the most when I get to build automation tools for myself or others and feel that’s where I would keep my sanity; being ETL focused. I want to be a data engineer
It's so good to be aware of you like and what you don't like! And it sounds like data engineering will be the perfect career for you - you can consider yourself lucky, given how in-demand those skills are now! ;)
@@karolinasowinska any advice on progressing other than reading and doing tutorials (just picked up designing data intensive applications last night). No much opportunity to get mentored i. My current role
Precisely like I think.
exactlyyyyy, I am currently an analyst and I am here because I realized I much more enjoy building tools and pipelines than analysing and searching for insights :D
Its so nice to see that i'm not the only who thinks like this.
As a data engineer/architect this is spot on!
Some analogies on reason #1 - data is your plumbing, while your data engineers are your plumblers. You dont see the plumbing but you are thankful that you have clean water in the house. It may be taken for granted at times but any leaks or burst pipes is a big problem.
Another analogy is in the body data is the blood. Its not a main organ but organs need it (clean blood) to function properly.
On #3 true same with data science or ML where they now can be a black box provided as a ready to use product. But in the future someone still needs to know when and how to use these tools.
Great video, more power!
Data Engineering is where is at guys :) , we are the backbone of Machine Learning and AI
Hey I just started my internship as an etl developer and while going through various articles online I found it comes under data engineering. And while research about that I found your profile and I'm so glad to have found you!
And I'm so glad that you've found me! Welcome aboard! ;)
The first reason sounds like an advantage to introverts.
I thought so too!
I can relate so much to this. I'm a data analyst being bribed so much to keep doing data engineering, I gave up on it and enjoying being a hot bun 🤑
btw part of my learning process was your ETL videos, so thank you! That's why I keep coming back to watch your channel!
Haha you might as well! If you enjoy what you're doing ;)
I am an introvert but I am in consulting. Talking to clients drains me big time. After 31, I am now looking into starting all over again and shift to data engineering.
All the best, you can do it!
I am a Data Engineer now. Lets see how far I can take the load. LOL. Thanks for the wonderful video Karolina.
You know, I have realized few weeks ago, that my current job is actually Data engineering :)). It has a very weird name, but I actually build data pipelines in clinical trials, and I love it. I'm happy with my job, and I got here without knowing how fancy it will be in 2021 :D
Haha nice one! Definitely, if you're building data pipelines then you're a genuine data engineer! ;) I'm happy that you enjoy it!
There are a few more examples where you can be closer to the business as a data engineer like myself! Companies where data is their main asset can be a huge one e.g. Opta who are the leading sports data provider, Nectar who collect data about customers to analyse their shopping habits and even Bloomberg/Reuters who have amassed so much financial data that they can sell it for tonnes of money!
I'm also hoping technologies like Denodo and Databricks will just change the role of a data engineer instead of getting rid of it.
Great video as always and the new 4K looks great :)
Oh, YT put this comment in "held for review" for some reason! I completely agree, how crucial and "central" you are as a data engineer depends on what industry you're in. And I also hope that Denodo/Databricks will only make data engineers' lives easier! :)
As a data engineer student I have to say that your videos help a lot!!! There a lot of questions that you just easily answer. I hope one day I’ll be as talented as you.
wishing you a belated Happy Birthday :)
A fellow Data Engineer who is inspired from your videos :)
Thank you so much! :) 😊
I just got rejected from my data engineering internship, so this video came at a good time 😃😭
I'm sorry to hear that. If you enjoy DE, don't give up! :)
You had me at work-life balance is great. I like to do many things outside of career, so I am sticking to the field (currently 95% finished with my DE Bootcamp). I also think that one can go from Data Engineer to ML Engineer later on (if they have data science background already, but the job market is too competitive).
That's awesome!
Exactly!!
Always good to hear the cons of a career from someone who is in that career. In this case, some of the cons can be viewed as pros :)
That was my impression as well!
Hey, Thank you for the video.
At first i just wanted to stop the video when you said "these 3 reasons are just your thoughts".
But later on as you speak and explained the unbiased scenarios 'giving the listener a chance to decide on their own knowing the + and - of the field', i loved connecting to this fact.
and yes, i am also trying to switch to Data Engineer role from Business Analyst role and thats what pulled me to watch this video.
Again thank you for honest and helpful thoughts and contributing to the community.
Love the quality of the camera, and thanks for the content!
Thank you for adding your perspective and experience to this. I have been presented a choice to work either as a Data Engineer or a Cloud Engineer and I'm still deciding which to go for. I have barely started my career as a cloud engineer, but I've build quite a bit of knowledge over the past 2 years to prepare myself for a cloud engineer role, only to be presented with the choice of becoming a data engineer now. Both seems attractive to me because I like to grow as a technical professional, I like the hands-on and practical aspect of both roles.
Greetings from Brazil, @Karolina. You are being completely honest. Your video is fully true.
Thanks, I do try to!
I thought I wanted to be a Data engineer. I mean, it sounds pretty cool, but I like more machine learning in my life. Now I know I have to be ML engineer to have a little of both. Very helpful!
Good points! As a DBA I'm totally in the shadows. Your video helped me a lot to find my way in a carreer change. Thanks
The point regarding Machine learning... it depends. Today we have MlOps and AutoMl Tools. I'm pretty sure these things are starting to be more related to data engineers than data scientist.
I landed on this video by searching "why it is difficult to comprehend Data Engineering".. Actually I'm a Data passionate person with a Data Science degree, on a career change, realized that Data Engineering was the core of data and many companies would have requirements for it in the future. But unlike Data Science I feel DE demands a lot of IT roots. As I'm from Finance background with business experience, getting overwhelmed with hundreds of jargons is sometimes annoying. But, if I think of what I can accomplish with DE, I keep moving. Thanks for this video. It has motivated me to continue.
I'm moving into a DE role from a Data Analyst role. I have a degree en economics, so I get what you're saying. You are not the only one struggling with this.
Great Video. About the third reason, I myself work to a big retail chain in Brazil and they are migrating the ETL to DataBricks, this is been made by an entire data engeneer team
Saying data engineering will be irrelevant because of automation is like saying web dev is good for nothing because there is web builder. Anyway, love the video!
Hah well, true, I guess!
Thank you for that.
I'll watch it again later.
Liked, subscribed and activated the bell.
My pleasure! ;)
In you opinion how will the demand for data scientist positions be in the future? There're also plenty of platforms that aim to reduce or simplify the data science tasks. What career between data science and data engineering will have the best future?
Like I mentioned in the video, no job guarantees the security that it'll be in-demand for decades. I wouldn't think about it too much. Go for what you enjoy doing most, and new opportunities will emerge for your skillset in the future. :)
If you *do* want to pursue a career in data engineering, visit: dataappreciationsociety.com/
(also don't forget to switch on 4K for the best experience (I finally got a new camera!) haha)
Done! Wow amazing quality!
My display cannot do this(
No need 4k when we view karolina
surely, the most important tip :D I can see every little wrinkle now! 😂
Now we have four times more than just one Karolina.
I actually think it's quite nice to be 'behind curtains' , knowing that you're indispensible. Just like so many people who work on designing, building and maintaining physical infrastructure we use every day. There are a lot of jobs this pandemic has highlighted as crucial for our normal functioning. These workers were often underpayed immigrants, very vunerable to pandemic. So, I see this as much as the consequence of our ignorace and lack of empathy for our fellow workers. THAT has to change.
just discovered your channel!!!! Thx for the great video!!!!! Really insightful and helpful
Your cons are totally pros for me) thanks
You're welcome:)
great video, definitely didn't discourage me
“Reason number two: it has nothing to do with machine learning “. That’s exactly why I like it 😂
Do you think that some data engineers would benefit greatly from knowing machine learning because they could be building a platform for machine learning to be able to be done very efficiently?
True. Although often MLs and DEs collaborate in teams, so that knowledge shouldn't be necessary. Definitely useful though, especially at start ups.
Haha 😂 you really made the cons sounds like pros!
I can't agree more! Thank you for sharing your experience so openly!
I'm glad that this is relatable! :)
I took a course in SQL and did well, but I don't want to work with that tool again.
I want to apply machine learning to robotic controls. She makes it very clear to me that I do NOT want to be a Data Engineer working for a bank or finance company.
I want to continue working in the aerospace industry applying machine learning algorithms to flight controls, or robotic controls.
I'm sure there is a demand for those kinds of roles. If there's a will, there's a way, so good luck with getting your place there!
@@karolinasowinska about 20 years ago I heard about this technology called "sub vocal communication". They used it to help divers communicate underwater. They placed a transducer over the vocal cords to pick up the nerve impulses from normal speech.
The system had to be trained so that they could connect the electrical pattern with known words. Very cool stuff BEFORE the term machine learning was widely used.
I wanted to use it to help someone like Dr. Hawking recover speech and have some mobility.
I feel this should be my life's work!
Thank you for your time and video. I would appreciate a video of resources to learn data engineer. I love your videos. Thank you for your help.
Great idea, I'll think about it! ;)
I’m trying to get into data engineering. But by watching your video, now I’m in a doubt. I’m from management background. And I have zero knowledge about IT field. I have a couple of questions to you.
Can anyone from non IT field, will able to do data engineering job or not?
And, Why is it so hard to find data engineers compare to data science?
Thank you,
From my research a lot of employers interchange a data science title for data engineering tasks. To get familiar with tech, go through the CompTia A+ concepts to get a foundation on general hardware/software/networking etc … then look for Data Engineering courses. Coursera has one. Hope this helps you out, good luck😄
haha that cieple buleczki analogy-love it!
Haha ciepłe bułeczki, dokładnie one chodziły mi po głowie! ;D
love it and yes we do use hot bun saying, also the unification of the data sources would be great cos now everything is going to cloud so you just have to adapt.. data cloud engineer.. move from ETL to ELT.
Thanks for the video.i need more video about data engineer.love ❤️❤️ from india
Another great video. Thank you.
Thanks! :)
hey Karolina you look like a smart person thanks to your honesty
Where can I get entry level jobs for data engineering working remotely ?
Hey. This is a top notch video. I'm about of start off my career as a data engineer. I agree with the first point you make. I wanna become a Product manager in the long run. Do data engineer's transition from data to product?
Yes. But being a general software engineer is probably an even more direct route.
No 1. Absolutely right. You’re not making architectural decisions. That’s not the job.
Hi Karolina, I'm a university student in the US studying data science and I'm a huge fan of your videos! Given the high average salary of data engineers, I don't assume it is a job a new college graduate can get. What would you recommend for a prospective data engineer's first job out of college?
You can do it straight out of college! And if not, I suggest general software enegineering :)
Probably the 4th reason is- the job becomes monotonous after 3 months ( if you know airflow, pyspark and snowflake). Typically you would do the same kinda transformation each time. But in ML you keep learning coz every month there's something new coming up. (take BERT and GAN for examples- they were like state of the art last year but this year there are so many improvements).
Good point! Although I'd say that there are new data engineering technologies coming out as well, or systems ar migrated to different environments like AWS. Obviously depends on the company you work for! If the company doesn't innovate, then you're right, writing ETL processes can get monotonous
@@karolinasowinska - Agreed. But what I am saying if you see the sheer variety of work in ML(let's say an anomaly detection, ranking algorithm, customer retention etc) is also more(which makes it more interesting and challenging) when compared to data engineering work. Great video by the way!!!!! I hope you reach 100K subscribers soon :) :)
@@samparkpradhan8849 I see what you mean, you're definitely right! :) And thank you! :)
If you are doing the same transformations across multiple projects, automate the process(es) and let the computer(s) do the work.
I feel the Data Engineering role can also be quite diverse, depending on the company strategy and/or needs. So it's actually very possible to find a DE, that gets to do ML maybe in a startup. So it might not be a monotonous role, it depends. There are also alot of Big Data tools out there aswell. For a Data Engineer who wants to self-improve by learning all these tools will have alot to learn
The big lobsters sounds better than the head cheese, I am going to start using that one.
Thanks the vid is soo informative
Data engineers are the IT of the data world.
Thanks for this informative video. I am a 4th year University student and I am currently struggling to decide between Data Engineering and Hadoop Developer career path.
I think in products or solutions that are heavily relied on data or are so called "data driven", data engineer role is main business function.
True! (Unless the firm only advertises itself as data driven)
Brilliant video and I'm seeing it at the right time.
I've been a researcher and a data scientist for many years and now they are offering me to move towards data engineering. Something that I have no clue about. So your video was quite eye opening. I think I should probably reject the offer? What do you think?
If you actually enjoy data science side of things, then don't do it! ;)
Thanks Karolina! Really appreciate to know your idea.
Any Certifications that you would recommend someone who would like to get a job as a Entry Level Data Engineer?
I recommend the courses I mentioned in the description:)
hey, yr speak is very nice. my russian ears like yu) I am data engineer, but I`ve wanted be ds. BUT my job is better and easier and very interesting than I worked Data Scientist.
What a beauty❤️ ? Beauty & brain. What a deadly combo😘
Ohh, how very flattering :>
Hi Karolina. What do you thing about the people that haven't computer science background want to begin as DE? In my case, I'm a chemical engineer but, I'm really tired of my job and I need a change.
I know plenty of people from STEM (not just computer science) background who are data engineers :) You'll be fine!
@Karolina Sowinska Thanks for the inspiration! I hope we will be colleagues soon.
So helpful thankyou
I really liked your "hot bun" comment :-D Thanks to your videos I discovered that data engineering actually is pretty awesome, you're completely right
Haha I discovered that English say "hotcakes" instead, I was close! :D
Interesting. As a data analyst, I was aiming towards becoming a data scientist but what you explained broadens my horizon. On a side lobster note, are you reading (or planning to read) Beyond Order?
I might order it. Although I haven't finished his first book yet, I couldn't get through very descriptive analogies and stories. But I appreciate the guy's mind.
Hello All,
I am Jay, a Data enthusiast.
I have a good knowledge and portfolio as a data analyst but I am concerned about the competition in this job role as diversed field of professionals are transitioning into it. For this reason I am trying to consider data engineering path as a career.
I am currently in the canada and if someone experienced can give me insights about future prospects of data engineering then it would be very helplul.
Data Engineering is a really good one. Jay, do check out my other videos on the subject. I am also working on a dedicated course on how to launch a career in data. If you'd like me to send you an alert when it goes live, please share your e-mail address! All the best, Karolina
Great video with awesome advice. Do have recommendation for ETL tools for some just getting into this. Thanks in advance!
Hey, I have a data engineering course for beginners on my channel :)
Some companies never use ML so they don't need DS or ML Eng. what they do, they prepare data for Analytics or just BI dashboards, sales funnels, and so on. How about that?
Even if companies dont use ML, they still at least need someone to take care of the data infrastructure,.ie. data engineers.
The video is quite informative I suggest you change the video title because it is kina misleading, but thanks for the information :)
Hey, I was just curious about the live coding interaction within job interviews. What are some coding challenges(python,sql)/ data structures & algorithms questions you faced in interviews ? How did you approach them ? I’m a bit nervous about this portion😅.I’ll be looking for a junior DE role soon. Thanks for the great content btw😄
Did many DE interviews and was part of interviewing Jr DE candidates too. Here are 5 tips:
1. When you're asked to write a code/query to answer a question or to complete a task, you first want to explain what your approach would be: What functions will you use? What lines of code are you considering to write? Why? What do you think the output will be? Think out loud before typing anything. That way, even if you fail to complete the task or to completely get the desired answer, you're still showing that you understood what you were asked to do and that you knew how to get the desired output. +Most of the time, the interviewer will guide you as you are brainstorming too which can be very helpful.
2. Don't be afraid to Google stuff when asked to write a query or a code. It's better than spending a long time guessing. You're not expected to memorize coding lines or functions. For example, if you know you need to use a window function then say, "I will need to write a window function here", then google how to properly write it. What matters in this example, is that you know what we had to use to get the right answer.
3. Don't guess different methods to write your code until the interviewer confirms which one is right to use. Instead, explain the method you would use to get the answer or to complete the task then try it. If it's right, great. If the interviewer says it's wrong or your code shows it's wrong, don't be embarrassed. Instead think of another approach you can try, say it out loud, and try again. We all learn and perfect codes through trial and error. So, yes of course, the interview would love the right answer on the first try. But if you're not getting the right answer the first time, it's your chance to show your problem-solving skills and your tenacity. Show that you're not afraid to make mistakes, to troubleshoot, and to keep trying until you figure it out.
4. If you really don't know how to write the code/query the interviewer is asking for, then be honest about it. You can say, "I've never done this before", or, "I'm not familiar with this". But don't stop there. Always follow with, "But I can try to figure it out". The interviewer may move on to another question or they may give you a chance to try. If you can try, I would suggest to Google what you're asked to do. You can try to complete the task with instructions you find online or if it's too hard for you, at least share what you understand from the instructions. This shows you're willing to learn and try unfamiliar tasks.
5. Be friendly and approachable during the interview. Even when you nailed all the questions, if another candidate also nailed all or most of the questions and on top of that, they were friendlier, the other candidate will be picked. Even during technical interviews, we look at the personality. No one wants to work with cold people or hire someone the team may not be comfortable with. Most importantly, every interviewer is asked to find a candidate that's a good fit for the company culture.
Good luck!
Really enjoyed this channel for learning about data jobs. I career changed from being a traditional scientist into being a DE which I discovered by accident. Going from a field which is, allegedly, oversaturated into a field in demand has been quite the shock.
I personally have no interest in DS although would you consider creating a mini ML course for people interested in it?
On the subject of DS, I think if DE and DS had the same job prospects, a lot of people would prefer to be a DS. What do you think draws people to DS over DE?
Love the channel, keep it up.
Actually a great idea, I can definitely think about a short ML course! :)
My theory on why DS is more popular is that most people want to make discoveries / draw conclusions rather than simply building the infrastructure that lets other people do that; they wanna analyze and build models and find answers - it just sounds more exciting! The thing is you're still solving problems and using logic when doing DE, its just not as "on the surface" if that makes sense.
Thanks for updating. Just wanted to know is there any age barrier if one wants to enter data industry. Thanks in advance. I have done UG in Biotechnology and master in Food science and no experience.
Of course not! Go for what you love, never put limitations on yourself! :)
Hi @karolina sowinska
Do we need to have a compulsory professional certification for entering data Engineer domain as a fresher or can we do it with courses and certificates from Udemy and all?
Great video 👍
I have all the basic skills to qualify for entry level data analytics job so far unsuccessful to land a job.
Do you have any suggestions?
Thank you
Do you have a portfolio of personal projects which demonstrates your technical and communication skills?
Good suggestion above!
It is like devops, no one cares of you until something gets wrong
Thanks for this advices 😇😇
Pleasure:)
Karolina you have a good pronunciation. Nice to learn English. :)
Thanks!
Lobster Part, Made my day:) Thank you
I am a data engineer and really like the content here, most of the points were correct. Except for the fact that DataBricks or any other ETL automation platform can only have as much impact on Data Engineering as the up-and-coming AutoML will on Data Science. So that's not a reason to not become a DE and pursue DS.
What's your LinkedIn Vipin?
Im 15 and want to become a data ingeneer
For someone with a Bachelor in Industrial Engineering that wants to expand to technology, which path do you think it would be better? Data Science, Data Analytics, ML, SE, Data Eng...?? Meanwhile, I'm learning Python and SQL. Looks like they fit in any of those. :-)
Cheers!
I'd go for an engineering role, so data engineering or software engineering in general ;)
I wonder how stable the demand for these careers is. By the time I learn enough marketable data engineering skills in a few years, will the field be as “saturated” as data science allegedly is? A bit scary to think about. I know you mention this on the scale of one’s entire lifelong career. But it feels like data science went from highly in-demand to oversaturated in only a few years, never mind a few decades.
Data in general isn't going anywhere, its growing. If you know SQL even at least and have good problem solving skills, you're sorted you can easily get another position within the field. Careers are fluid sometimes, it can be exciting.
I have been kind of doing Data Engineering for years because I write ETLs and Automation tools in Python but I have not used tools like Airflow and Spark professionally for self toy projects.
On Reddit I was advised to polish SQL, especially Windowing functions etc.
As an aspiring Data Engineer, what roadmap would you recommend for me?
Hey, for the roadmap visit www.dataappreciationsociety.com
It would be awesome to see some videos doing data engineer things
Wow. Being in the shadows, work-life balance and good salary.
Sign me up!
Your content is amazing
2:14 *(1)* You are behind the curtain, not at business core (in majority of cases)
5:57 *(2)* This work has nothing to do with data science
10:03 *(3)* In the future (medium term) there will be fewer job offers. Today all jobs in technology are transformed.
5:55 I don't get it, why it is bad to pursue a career just cause you need money. Isn't that what 80% of the population do? And what would be a better choice? I think in software engineering it would be harder to earn a big buck when you are not interested in technologies.
And about work-life balance: i've heard totally opposite. That data engineers are used to work on saturdays and sundays, just cause business users often need access to the data on monday. Is this true?
Because if the career you picked doesn't utilise your strengths and talents, you'll find it hard to progress, and other, more talented people will outperform you. So it's a better strategy to pursue what you're naturally good at - the money will surely come.
IS just me or she doesnt known how to not sell this job?
She just love the job so much that it is noticeable. She listed more pro's than cons and the cons (as many already mention) doesnt seem like that at all.
Love the energy of this video, i totally vibes this way :D
Is it correct say that data analyst are more involved into business than data engineer?
At the end a D. E is a developer in data field, right?
Yep that's very correct!
Okay my mmmmmmmnmmmmm m mm m m m m m
this background music! super groovy.
Ha, thanks! ;)
Wow, this really made me more confused about my career choices. Maybe I just like making stuff for myself.
The really con part is to maintain that pipeline. Maintaining it sometimes on weekend😢
Ouch :( I feel that :(
I love tech, but I like Machine Learning and stathistics as well.
The good news is that in order to do Machine Learning, you obviously need to work with tech;)
I Love it
I'm a Dev DBA and often my work involves DE type activities, but as I move towards cloud, I am getting more involved with DevOps. I'm in a situation now where I have to choose a new job and I am getting interviews with both DE roles and DevOps, but actually choosing is not so easy. What are your thoughts about these career paths?
What did you choose in the end and why? I would really appreciate your insight if you don't mind
@@adamo1262 Well I somehow found the way to incorporate both. I went to a DevOps focused consultancy but with my data background they allowed me to take on cases in both areas when they arise. The majority of my work since I began is DevOps though (ELK stack, Kubernetes, Helm charts, Azure DevOps Pipelines, connecting to Sonarqube, and other CI/CD tools). I see a benefit in both because no matter what direction you take you are becoming more experienced in software development and its lifecycle which wasn't something I was really exposed to as a DBA. I think DE is definitely more focused than DevOps because there are so many moving pieces in DevOps, and it will take longer to understand how everything fits together. Sometimes I miss working with Data though, I've only had a few DE cases and when I have them I realise how much I enjoy that type of work. If I was doing it again, I would probably take a dedicated DE role. I say it because for me I find some DevOps work to be quite trivial (at least how I interpret the value of the work), with DE I see immediately the output of my efforts and I find that quite rewarding.
What exactly is a data engineer. I have one more semester of computer programming. How big of a difference is that?
Why do you prefer being a data eng. over being a software eng. or a dev? Thanks!
I see data engineering as a type of software engineering, with a focus on data. :)
Ahan, ok thanks
this time , I am little affried and confused about my path choice ... In India it is difficult to get jobs from one to another ...
What job were you interested in?
@@karolinasowinska I definatly looking for long term career & good stability, that's y i thought Data engg will be good option..
@@ankushojha5089 then go for it! ;)
9:33 But wouldn't a data engineer that transitions into being a data scientist make for a better data scientist since they have more of a technical skillset/mindset. Like I think its easier to go from being a data engineer to a data scientist (Assuming you have the speaking skills/personality for it) than it is to go from being a data scientist to being a data engineer.
I'd love some feedback. Right now I am in a bootcamp to become a data engineer. In the future I would like to work more on the business side, sort of like being a consultant.
11:45 - interesting thought, however once a given person earn big amount of money and can't progress further, with job as a whole - as a position - will going to be less and less wanted, then it isn't difficult to imagine someone wouldn't want to switch their career path to completly unrelated field. I couldn't imagine a Data Engineer moving into a spot of Frontend developer. Furthermore, i bet it would be a hilarious interview. Ofc this one is a bit less propable but even backend, or some weird mix of different things, not even connected to web dev. This is something that is comparable to losing half of the skills build over the years in a blink. Coding is coding, but the problem solving skills that were data-specific won't be of any use anymore, unless you somehow move into data science, but again i'm not eperienced so i don't even know if those two have something in common.