Everyone's Data Infrastructure Is A Mess - The Truth About Working As A Data Engineer
Vložit
- čas přidán 28. 08. 2024
- Is everyone’s data a mess?
Recently, I came across a post in the data engineering subreddit that asked the question.
The answer is yes, but no.
As someone who has seen data infrastructure at FAANGs, Enterprises, start-ups, and every other company in between, all companies need to make some concessions that can build up and become messy over a long period of time.
So let’s discuss some of the causes of data infrastructure becoming messy and how some companies are trying to deal with it.
Also, I forgot to cover a very important topic!
That is all of the mess often starts at the data source.
You can read the fuller version of this topic here
seattledataguy...
If you need consulting help, set up some time with me here -
calendly.com/s...
If you enjoyed this video, check out some of my other top videos.
Top Courses To Become A Data Engineer In 2022
• Top Courses To Become ...
What Is The Modern Data Stack - Intro To Data Infrastructure Part 1
• What Is The Modern Dat...
If you would like to learn more about data engineering, then check out Googles GCP certificate
bit.ly/3NQVn7V
If you'd like to read up on my updates about the data field, then you can sign up for our newsletter here.
seattledataguy...
Or check out my blog
www.theseattle...
And if you want to support the channel, then you can become a paid member of my newsletter
seattledataguy...
Tags: Data engineering projects, Data engineer project ideas, data project sources, data analytics project sources, data project portfolio
_____________________________________________________________
Subscribe: / @seattledataguy
_____________________________________________________________
About me:
I have spent my career focused on all forms of data. I have focused on developing algorithms to detect fraud, reduce patient readmission and redesign insurance provider policy to help reduce the overall cost of healthcare. I have also helped develop analytics for marketing and IT operations in order to optimize limited resources such as employees and budget. I privately consult on data science and engineering problems both solo as well as with a company called Acheron Analytics. I have experience both working hands-on with technical problems as well as helping leadership teams develop strategies to maximize their data.
*I do participate in affiliate programs, if a link has an "*" by it, then I may receive a small portion of the proceeds at no extra cost to you.
If you guys want to learn more about data engineering, then sign up for my newsletter here seattledataguy.substack.com/ or join the discord here discord.gg/2yRJq7Eg3k
Based on the thumbnail, I really expected this to be a 3 second video where you just record yourself saying “yes.”
Hahaha - yes
3 secs is too long to spit yes, 0.01 sec is all you need to say "Уe"
There are other layers to this problem. E.g. regulatory compliance. Who can/cannot access the data? What and where can you access the data? Think GDPR. Other factors include external vs internal data. Is there a cost to accessing/collecting the data? Most companies are not even close to having good Data Governance fundamentals, and many of them may never meet a high standard given the constant turnover. The Data Engineer role will evolve into a greater one of overall Data Governance.
I find this unlikely. Data engineers tend to be highly technical order takers. It’s in there interest to use technologies that lead to high salaries. It’s not in their interest to have data quality 4 quarters from now.
On top of that, data governance is a topic that never goes below C Suite at most companies. I’ve seen job requirements follow the trend you predict, but I think that’s just about technical knowledge of metadata rather than business skills.
@@edwardmitchell6581 You are describing the current state. I am talking about future state.
I have noticed that automation needs quite a bit of manual maintenance
hahaha...if you've ever had to backfill a table....
Ha soooo True!
Everywhere I've worked its been a mess!
its just always a fight to try to bring it to some level of sanity
For those who want to get into consulting, assuming you're starting from 0 exerpince, how many years of experince would you say is good before you can reasonable make a living off of consulting?
Consulting is entertained mainly with the logic that someone with MORE experience than them is going to help solve an unsolvable problem. So if you have 0 experience, what in your opinion is it that you would be even offering to them?
@@KshitijPatil1 I think k you missunderstood my question, I'm asking howany years is good to start consulting in general
@@ceejay1353 My bad. So you're asking how many years does it take to make a living off of consulting gigs, should you leave your current job, right?
@@KshitijPatil1 Yeah!
@@ceejay1353 Got it. So my assumtions about getting into these types of career paths is that you already have your first 2-3 clients when you start. This means the people you've worked with, trust and respect your contribution are happpy to commit their company's dollars on a weekly/monthly basis. This helps you to a) anchor your price and b) provide references for your potential clients to get social proof from. The reason point a) is important is so that you know how much is the max you can earn per month, and deduce the number of clients you need to juggle. Point b) helps you to go on an aggressive client acquition excercise, because till you get your schedule packed, there's no financial upside to this excercise.
So with companies data governance being such a mess, would you say that the field of data engineering & governance still has a future for atleast a decade? Or will it all be automated since automation seems a huge part of Data Engineering already?
How can you automate data strategy or data management?
@@edwardmitchell6581 Today in a data engineering team of 10, about 1-2 people take care of data strategy and data management. The remaining 8 build and maintain the solution. Their main job is to automate the engineering solution. If data engineering can be automated sufficiently then the total amount of DEs required will go down.
There is plenty of work to do. I can't speak in decades, but 5 years, yeah probably
Palantir seems to be the only solution
we'll see! From a stock perspective I am still waiting to break even although i bought at like $18 so it was there a while back
U look like drake n that’s y I clicked n YK Which drake.
i don't know which drake hahaha