Data Scientist vs. AI Engineer
Vložit
- čas přidán 3. 06. 2024
- Get the guide to generative AI → ibm.biz/BdmSiA
Explore the technology → ibm.biz/BdmSi9
Breakthroughs in generative AI have given rise to the growth of an emerging AI Engineering role that is differentiating itself from traditional data science. Do these two disciplines focus on the same problems? Is there any overlap in techniques and models? In this video, Isaac Ke, a former data scientist turned AI engineer, explains key differences and similarities between the two fields, along with some of the emerging trends gripping the AI landscape.
AI news moves fast. Sign up for a monthly newsletter for AI updates from IBM. → ibm.biz/BdmSiQ
Thank you for the explanation. But I feel they are not even on the same level. To me AI Engineer is a subtype of MLE who focus ML application which uses LLM. I would compare between DS vs MLE. And to me the comparison boils down to compare science vs engineering. Each has a totally different mindset when tackling the same problem. While engineer approach a problem from a system perspective, scientist approach a problem from an inference perspective.
I see the AI Engineer (at least in the context explained above) as a SWE that builds AI applications. Not that there’s anything wrong with that. When I think of AI Engineer, I typically think of someone actually building the LLM.
Thats true. If we compared the roles: The AI engineer, deploys the model, prebuilded as service (SaaS) form GCP, AWS, Azure, etc from the requirements from the R&D team. On the other hand, the Data Scientist (MLEng) role is focused in build the intern arquitecture to build the algorithms or make finetunning to the models using diferents methods from the bussiness requirements (Data Analyst).
If youre in a Hybrid Role, maybe youre Data Scientist focused on building AI products, or Machine Learning Engineer with Bussiness background or something similar, but the Science is not equal to the Engineering. The perpspective is different. you´re right.
Hmmm. Im a data scientist and there seems to be some concepts that I find wrong or misleading.
1) data scientists can also do prescriptive tasks aside from prediction and classification tasks. In fact the last project that I worked on was in the prescriptive analysis domain
2) data scientists also deal with texts and media data. From my experience that largest I handled so far is around millions of these data
3) data scientists are not limited to traditional ML models and Neural Networks. In fact, pretrained models are also used to speed up the training process with some fine tuning involved.
I think that one has to be a data scientist first in order to be an ai engineer. The reason is that you can’t engineer something that you don’t understand like a data scientist from the ground up. That being said, data scientists wear many hats and the ai engineering role can be included. I think the only difference seems to be that the engineering side demands more complex data and doesn’t do a lot of structured data analysis.
@@DanielK1213th that’s my goal, I am trying to get a data science job, then move to a machine learning job after a few years.
I am also a data scientist. I have been using LLMs to help with my feature engineering. What was mostly useless free form text fields can now easily be cleaned and standardized. One example I am working on takes a duration, which can be anything from “one week” to “oh idk maybe half a month” and converts it to integer day value. The LLM is able to transform into a 7 and 15 respectively and note whether it’s an estimate based on ranges or language. Pretty cool stuff.
I agree, I used to mostly create models based on unstructured data when I was working in text analytics.
Great presentation. Super clear. I can’t wait to watch more of your talks. Thanks
Great effort. I think it's a discussion that we should be having over the next few years. But it's definitely premature. Just like data science became a field long after people were actually practicing data science, we will only realize the differences a bit in retrospect.
literally the perfect video for me right now
Thank you so much for the video! I'm learning Gen AI so it really helped me understand the differences between data scientists and AI engineers.
Thank you so much for the clarity!.. What a Wonderful video!
I am learning AI, but it is pretty slow for me as I am an old truck driver although I did computer repair and builds for 12 years. My wife is a clever engineer like you and she can also write backwards fluently like you did here, but in real-time (not post-production). She is also learning AI now.
He isn’t writing backwards: the video has been mirrored; the same goes for all the videos I have seen on this channel.
To verify, confirm that they all appear to be left-handed, which is very unlikely.
@@solsospecialyeah I always came to this same conclusion.
Really well explained and summarized! 😊
I am currently working on my bachelor's thesis and can absolutely confirm that I am currently using (almost) all techniques from both sides. The overlap in my area/subject is extremely large and quite often I have to be very creative when it comes to obtaining and processing information... so definitely both sides... 😅
Keep these vids coming!! 🔥🔥🔥
wow great breakdown, thanks professor Isaac, I learned a lot 🤔
You are a great teacher. I love your analysis: top-notch
Thank you, I build RAG applications as an intern and never really knew how to qualify my job. I do some data science like scraping and cleaning data but I also do prompt engineering among other things. I don't train the models per say though or even fine tune them (for now), so was reluctant to say I'm an AI engineer but given your description I guess it's coherent.
Well explained! THANK YOU.
Pretty interesting.
I'm gonna start learning Data Analysis.
Very helpful info.
Best explanation on the topic
Very nicely explained
From scientist to engineer to technician. Since I mostly use NLP I'm excited of the possibilities of llms but fear the models will become so good that we will shortly simply have to take the back seat.
Dude, GPT 4o can’t even generate simple code correctly without mistakes. Your job is safe.
@@superuser8636You must not be using it correctly then. If you prompt it correctly then it can get great results on many different types of tasks
You know what IBM. YOUR COMPANY WAS DREAM COMPANY. WITH HELP OF THE SHORT CONTENT WHICH EASED ME LANDED IN FRESHER DEVOPS JOB . THANKS
Congrats!
Hi,
Thank you for such a huge clarification. However, can you please shed some more light on these regarding AI Engineering:
1. What are the sub-fields/areas under AI Engineering?
2. How much math is required to become AI Engineer?
3. Where can I learn the fundamentals/essentials to become an Applied AI engineer?
TIA
1. generative AI, or big new models that use multiple stuff to classify or make regression. Also, robotics.
2. A LOT, learn math and statistics, the rest doesn't matter
3. Internet. Start with datascience, math and statistics. Within datascience you need to learn about common models (MLP, SVM, etc). After that, start understanding LSTMs, CNNs, dropout and batch normalization. In the end, after around 1 or 2 years, start learning transformers, visual transformers, and also diffusion generative models.
Start with any calculus and basic math videos, also basic statistics. After that, use a course from udemy and youtube that talks about sklearn. And then go through computer vision with deeplearning and time series prediction algorithms... it is a possible way.
@@vitorpmh Thanks for the response.
1. I know about these GAI. Any other type of sub-areas/fields based on different criteria.
2. Any fields/areas that requires less math. I heard, interoperability is one areas where no/less math. But not sure if it can be considered AI engineer. Also, prompt engineering. Any thing else?
3. I just finished Google AI essentials from Coursera. I'm coming from Social science background but has STEM background as well. So, expecting some AI related skillsets (but not hardcore) and I also have Biology/life-science related domain knowledge. Any suggestions?
Thank you for sharing this
enjoyed video wondering how you do annotation of your notes
They just changed your title dude, it’s the same thing
As you are an example of DS pivoted to AIE, how would you transition from one role to another? I am really interested in what you describe as AIE, but recently landed a job in DS, so I was curious what steps could I follow in the long term to shift my carrer to what I really want to do. Thank you!
"AI engineers" are just software engineers who dabble with OpenAI API calls.
Not even close.
Consider engineers who are prepare OpenAI APIs.
Great! Thank you.
The DS scope is only EDA, feature engineering, giving business insight and story telling. More than that is area of MLE and AIE.
Data Science is generating insight from "data". Building the statistical analysis, gain thr business efficiency or profit. Mostly use SQL, Python, Sklearn. Working with Jupyter notebook.
ML Engineer is developing, serving, maintain the ML model. Sklearn basis. Pytorch. Tensorflow. NLTK. May use Python, C, Java, C# etc. Working with Postman, MlOps.
AI Engineer is Implementor or Enabler of AI solution that may combine either pretrained ML or AI or Gen AI. AI may be processing of language, image, audio, artificial voice, ocr. May use Python, Java, C#. Working with Docker, Linux server.
It all clear.
Thank you ♥
with GPT store in place . do we really need to work on foundation model to get the result we want?
Thanks
I appreciate this distinction. There are nuances but the inputs are different, tuning techniques and evaluation approaches are different. This view is opinionated and could offend a Data Scientist who knows neural networks very well (and can create foundation models rather than just use it). But you could have someone on the right who cant do the ones on the left. And someone on the left who despite knowing a lot needs to become familiar with techniques on the right. They can cross but given that additional work is needed, its reasonable to say they are different. There is enough work that I think we need the distinction and if you can do both then yey for you. Maybe it should be GenAI/TransformerAI Enginner rather than just AI engineer but we can keep it simple.
great video
ML engineers are data scientists that develop scalable ML pipelines and bring research to production following MLOps standards (they work together with data scientists) and know the math and SE. Being a ML engineer includes being able to deploy models as microservices that get consumed by multiple “AI” applications. One thing is the model and another are applications that consume the models and apply certain business logic
In my opinion the new “AI engineer” is a very misleading term for backend software engineer that knows how to connect/use to AI apis
Issac Ke my GOAT!!!!!
This video is informative. However, I feel Prescriptive capability or 'Prescriptive' analytics has always been part of Data Science. I have seen Data Scientists with exceptional domain knowledge, building Prescriptive Analytics systems. However, in this video, I was surprised to see, how 'Prescriptive' analytics switched sides - as it too got heavily influenced in the newfound AI (or GenAI) rage. On the other hand, I feel - AI is more towards Applications, and specially GenAI with a promise of productivity booster.
Cheers!
Awesome 👍
excellent
The AI Engineer part only talks about LLMs (ChatGPT,Gemini types of models) only, which feels heavily misleading. Reducing the whole field of AI just to something that has been popular for the last few years is not really understandable. Also I think using the term Generative AI for LLMs is another misleading thing. We can also generate videos, audio, images, 3D structures with AI. Back in the day when image generation was popular people used to use generative term for images. Another problem in the video is that we don't always use "Foundation models". The video shows as if AI Engineers mostly finetune (adapt) these foundation models. Don't let this video think that AI is just finetuning LLMs. We have lots and lots of stuff to do in the field of AI :)
What are the differences between a ML engineer , AI engineer and Datascientist
Im so surprised that this video felt like an oversell of AI engineer and GenAI stuff. Most of the usecases he compared are wrong. DS side is almost like a process if all ML applications while AI eng side just appliactions. Also where is evaluation? Explain ability ?
The 'Prescriptive' capability or specifically Prescriptive Analytics has always been part of Data Science. I found in this video, it switched side. And as you mentioned AI is more seen from Application side, specifically GenAI for its ability as productivity booster.
I’m interested in going into the field of data science/ machine learning. I’m currently a self taught programmer who have done text book math in years.
Can anyone in this profession tell me what the math requirements are for these roles?
Whoop yessir Isaac
I like the comparison but I do note that Data Science was not presented fairly, he could've said that Data Scientists lately do work on million of rows data, using Deep Learning algorithms, just a side note*. But thanks for the video!
Great job.
I'm A.I engineer...!!! Amen...!!!
Interesting
Where does fine tuning fit in all these ?
Fine tuning is usually referred to in connection to nural networks when one takes a base model of some sort and continues training the model on a specific domain of the problem at hand
PEFT
It was more of a data scientist vs generative AI engineer
Great
Basically, every five years a new career is dropped. When that happens, all the other new drops from the past 15 yr like data science, business intelligence, ML engineer, start to look more like SQL queries. In 2034, we’re all going to say we’re “sentient robot engineers”
I must disagree. I just finished an MSc in AI, and we learned everything you mentioned in the Data Science section and the AIe section, but nothing you mentioned, we learnt math behind the algorithms, etc.
So basically I switched from Data Scientist to Ai Engineer without even knowing.
I’m a bit surprised to hear this from IBM … it sounds a bit wrong, I didn’t know IBM competence on AI has dropped this much
Hi , is data scientistit requirement to become ai engineer . I am from devops
@@babasathyanarayanathota8564 AI engineer in this context has no meaning, what they say in the video it’s wrong
@@babasathyanarayanathota8564you need data experience to get a job as a mle
@@babasathyanarayanathota8564what’s required is that you must know ML and when to use it as “AI engr” is an applied field. Data science isn’t required but is a plus.
I am very disappointed in this video as well.
I study data science... the AI Enginering seems need more people to work.... data science can be done by one SCIENTIST...?
Anything is possible?
Data scientists train the models and ai engineers choose the required models for each step of whatever ai tool they are building.. is that a good analogy?
Fin analyst vs portfolio manager relation?
Btw didn’t mention langchain
Also with dspy, prompt engineering shld be dead. Dspy adds reasoning to prompts
This is one handsome fella😍
Writing backwards is an AI Engineer-type flex
The video is horizontally flipped.
Particularly when you do it with your left hand 😂
Confusion..🙄
This is more like Data Scientist vs Generative AI Engineer
I'm still an undergraduate, any tips to land on a big comapany(Google, IBM, etc.) as an AI engineer.
Good question 🤣 A good internship a good knowledge of artificial intelligence and good projects on the portfolio
I am a Chief Generative AI DataDevSecFinMLOps Cybersecurity Architect Scientist Engineer Officer 😅
Also a saiyan
I am not sure if people know what is our capability they would let us use it because it is just like omni potent and omnipresent which is God like and we can even decide what individual fate is to be or not to be which may not be for humam biased
AI Eng. -> applied level
We just get our ai guy to do both jobs (and sort our website out all the time), no one show him this video or else he might ask for more money 🤫
Explained with good concepts, but...
Data Science: name of a career fundamented on statistics and computer science that already existed and has had updates over the years.
While AI Engineer is the name of a vacant position.
A data scientist is capable of doing everything you describe on the right side of the board and beyond, why? knows Statistics and data, and the fact that it is not structured is still data.
You are comparing the man who knows how to build a car with the man who drives it.
This does not sound right. Sorry, IBM.
Relating AI Engineer to Gen AI (2 years old field) is obviously wrong. If this is the case, then 90% of today's Data Scientists are also AI engineers, and this distinction does not make sense anymore😮
I really don't think the details of this are right. Plenty of data scientists work in generative AI / LLMs / unstructured data.
"prompt engineering" lol.. do you use chatGPT to help you come out with an "engineered prompt" ? The new form of engineer, prompt engineer!
AI Engineering
"Use Machine learning, such as regression".
Lol. Regression is machine learning now? Blimey.
It is
It is part of available ML models (i.e. linear regressor models,ridge regressor models, and lasso regression models)
I think it is more Generative AI Engineer ..
So many opinionated and false statements in one video 🤦🏻♂️
Wouldn't expect this from an official video from IBM
agreed would have been better from fly on the wall not fly that moved walls
Cannot call someone as AI engineer if they are just using others models.
avoid any title using Data ..prefer to be Software Engineers ...you dont want to be begging for compute just to productize your models....
I stopped watching at "DATA STORYTELLER" 😆😆😆