Will Quant Finance End Up Like Data Science
Vložit
- čas přidán 25. 02. 2023
- A subscriber wanted to know if quant finance will end up like data science where undergrads will be hired as the new normal. The short answer is no but the answer is a bit complicated. Quantitative finance started out like data science as a trendy degree where every quant was a full stack quant. As the quantitative finance industry matured the job got split into quant (also known as quant researcher or model developer), quant dev (implementation), and trader (business user). Data science has had a similar full stack start but in the last few years has really started to segment into data engineering, data science, ML engineer, and dev ops. Even these new categories seem undecided on what they really mean. Quant finance is more mature as an industry that data science however BOTH still have a long ways to go.
Website:
www.FancyQuantNation.com
Support:
ko-fi.com/fancyquant
Quant t-shirts, mugs, and hoodies:
www.teespring.com/stores/fanc...
Connect with me:
/ dimitri-bianco
/ dimitribianco
Data science nowadays can mean anything between making pizza and proving deep statistical theorems
this comment is food for YT algorithm
Thanks! CZcams Analytics actually has a message for this video stating due to more viewer engagement it is getting more views.
This is sooo validating to hear!! So frustrating joining a team (or interview) and being asked ridiculous questions about overly complicated 'trendy' models or packages, and essentially being forced to ask why they don't do something much more straightforward, simple, and maintainable. Very hard to always be providing that feedback. Folks don't want to hear it!
That's great to hear others can relate!
I deal with this often and it drives me crazy.
Thanks for answering my question, Dimitri. This clears up a lot of the things I was wondering. It seems like a Master's degree isn't "the new bachelor's degree" but rather jobs have hijacked titles that used to have Master's degree requirements.
True...at the same time though. Masters and phd folks have also hijacked some jobs that previosly did not require these degrees...I think.
Separating curve fitters from actual data scientists is the industry equivalent of separating men from the boys. It’s kinda needed tbh. I’m a statistician turned data scientist and now looking to transition into the quant space. I feel your frustrations, Dimitri.
Do you have a PhD?
Well said. Very well said
Hi Dimitri, can you make a video about how the new Chatgpt4 can impact or facilitate different types of quant?
This is gold. Thank you Dimitri!
Thank You! Finally, someone said it as it is. I am a data scientist and I hate the way it has evolved.
Why
@Diego0wnz from other comments it seems people took the video message as I don't like data science. I like data science as a field however there have been a lot of bad approaches that have developed most likely not from an academic perspective. The rigor needs to increase in data science and there are too many data scientists doing it very rigorously but there are also massive amounts of people and books that have gone down the wrong path. Many good data scientists agree there needs to be improvements however they often get spoken over. I would guess this is why the comment above was made.
An issue when modeling phenomena which evolve through time is that you can't know for sure it won't blow up. My understanding is that, if you try to do forecasting, you either have a set of variables serving as a state which captures how your response evolves, or you have to rely on things like trend extrapolation, which feels like a gamble. The first option is ideal but rarely attainable, and for the second option, I'm not quite sure how you could avoid the pitfalls.
Hey great video, I had to comment, because currently I'm a graduate student in a Masters of Data Science and Artificial Intelligence. So everything you're talking about is something I think about almost daily. The split you describe is even apparent in my university, where there is a program in the CS department(my program) and one in the Math Department called "Statistical Data Science". And my program really pushes the ML and programming aspect, but I'm finding that a lot of my classmates don't really understand how the models work on a mathematical level, and it makes me really skeptical. So my focus in my program has been to shore up my statistical skills as much as possible and fill up my courseload with as many Mathematical Statistics, model development, and higher level statistics classes as possible. It strikes me as really strange that a programmer would know how to program A.I. to solve a problem, yet can't do a basic linear or logistic regression and it makes me a bit uneasy. So my goal is to avoid that if possible. I am teaching myself finance by the way since my undergrad was not in the financial field, and am reading books like "Quantitative Financial Analystics" and "Option Volatility and Pricing"(with the workbook :) ).
That's actually good problem You've pointed out. I got hired in credit risk department in a large consulting firm. They're naming themselfs as the quants but the tools being used are precisely data science'ish. Mostly classification algos. Of course, they're not working on investment risk, it's just a banking, thus sophistication of tools being used is largely limited by regulation entities. But still. I'm undergrad with a DS course done, after that I've enrolled Financial Maths MSc, but offer was too good too wait for the end of the MSc. IMHO situation on the market is that it is better to hold on a decent job offer rather than pursuing masters, but it's a good idea to acktually pursue it finally and I'm sure about this.
Hi Dimitri! Thank you for your video :) I recently got admission from University of Maryland with Quantitative Finance masters program. Do you think is it helpful to get a job as a quant in the United States? I’m an international student from Korea 😂
Another issue that can be seen LinkedIn job postings is that majority of Data Scienctist jobs are strictly just Data Engineering jobs, and very few actual model building jobs.
I have noticed this as well.
Most companies do not need data scientest they need data engineers, vast majority of companies have very poor data quality. Also IMO data engeeries are a much more needed role than a data scientist, I do not see that changing this decade either.
@FanTaaGoesHD I completely agree! Without quality data both quants and data scientist are useless. Data engineers oddly get overlooked.
@@DimitriBianco IMO this is because the work isnt sexy, you arent creating cool models. When you read the DE job descritpion, who would want to do that over a DS. Only salary can change that
Great jon
Hi Dimitri
Have you too noticed that most who claim to work as data scientists lack the mathematical background to work on the same projects?
Are you saying that data science roles will sort of bottom out in 20 years and then pick up?
Given Quants have a more structured approach to problems and model building, do you believe it to be the reason why we still do not have an option pricing model based on machine learning?
I think current ML models shine where the environment is sort of stable, with less uncertainty. With that much uncertainty in the financial market, you need some statistical and probability theory, even for simple binomial tree models. You can't really learn the future from data unless the past predicts the future quite well. There is a lot of probability used in option pricing because the future is uncertain...and current ML will not replace that, I think. Even when ML models are incorporated into option pricing, probability and statistics will never go away because the future will always be uncertain.
The majority but NOT all data scientists tend to lack math and stats. Many of the programs have started adding in more however they are often too computer science focused.
I think in 20 years we'll finally see the roles settle into something meaningful.
Option pricing has and can be done with a variety of techniques including trees. This is helpful if speed of pricing is the goal and not accuracy. There is always a tradeoff between speed and accuracy. There is also the aspect of understanding the model structure which conclusions can be drawn from. The quant space is a bit isolated in the sense that quants are just science people applying science to finance. If a quant moved into tech they would be labeled an ML Engineer or Data Scientist.
@@DimitriBianco So what I get from this is a 2 year Masters in Statistics (1st year) and Data Science (2nd year) is not a bad idea
yeah good take i remember going to a hackethon meeting and one of the participates who identified who was a data scientist it was honestly a data analyst role mostly python some data pipelines and tableau.
Hey do you have a video on job opportunities for people who just have or want a undergraduate degree?I understand quant jobs are out of reach (b.s economics minor in math, know python and have used it in internships)
Dimitri doesn’t believe it but frankly he is wrong, if your coding and stats math skills are good enough to get you through the coding tests then you’ll get the job but it’s probably one of the most competitive field in the world.
A really interesting area is operations. Banks have these teams which do analytics and strategy on how to run the business.
If Data Science/ Data scientist was design for the purpose of making business environments more efficient using and interpreting data, How is it that they could choose to ignore Econometrics as a tool applied to economic and financial theory? Applied Econometrics to economic theory, financial theory and the business environment is essential in order to interpret the data in a business sense, otherwise your just operating like a chicken with it’s head cut off, doing a whole bunch of nothing. It seems to me that data science/scientists are more caught up with the trend rather than the scholarship of being someone who is train to solve real-world problems.
Thanks Dimitri for your input on this topic.
Not all observable patterns in every data has some well known theory underlying it. Both data science and econometrics are useful. To say one is garbage is a naive position to take...You don't need theory to discover patterns. In data science...the focus is first discovering some patterns, and coming up with some theory to explain it. The theory is not necessarily some well known economics or finance theory. In traditional econometrics...theory first, then look for data to test the thoery. In data science, you are not testing any theory...but trying to discover hidden patterns.
@emmanuelameyaw6806 you need a hypothesis first. Without first defining a hypothesis it is no longer science but data exploration. The problem is no matter what pattern or relationship you find, a business user can create a story to go with it. This is the same issue with p hacking and thinking correlation is causation. You must have a hypothesis and business insight before you try fitting anything.
True. I agree. But having a hypothesis does not mean you are doing science. Economists set up hypothesis all the time when they write papers, it doesn't mean they are scientists. And a scientists can explore data...it doesn't mean he is not a scientist. I see no problem with data exploration...and yes, it is a big part of data science and ML. If you have unstructured with no well known theory to work with. You can only explore the data to see if some patterns emerge. And then you can try to explain that pattern if it exist. Data science pretty much is learning from data with limited theory...no formal testing, no robustness checks as in econometrics models. But that is fine. The two are different and they have different purposes. This is clearly seen, for example, in how linear regression is taught in econonetrics vs data science. In econometrics, linear regression, for example, is about estimating the parameter of interest, adding useful control variables, checking the estimated parameter is unbiased, consistent and robust to different specifications. And at economics graduate school, yes, you do a lot of econometric theory too...but I never heard stuff like gradient descent in econometrics...although it is the same as data science folks do. That is optimize some cost function...but in data science, they stop there because they are not really trying to build a causal model. And that is fine, it doesn't mean it is useless, just different. This video seems to trash data science models and glorify econometric/statistical models in quant finance. Neither is better or worse...just different models, I think.
@@emmanuelameyaw6806 The purpose of Data Science was to improve efficiency of operations in the business environment using and interpreting data. This business environment includes thousands of small business's that make up more than half of the private sector in the economy and many of which who can't afford the luxury of hiring someone just for the sake of getting lost in the data via exploration and not formulating hypothesis and conclusions that are rooted in some kind of social/consumer or economic theory. Simply exploring data year round WITHOUT hypothesis and making assumptions that real scientist do to then prove through testing sounds like a waste of my money as an employer, and as a small business who needs to improve production and efficiency
if you know SQL, you are now a Data Engineer, all job titles are becoming a joke. Irrespective on the job title the focus should be on the actual job/task
I am an undergrad who has taken courses like graduate string theory, quantum field theory, have experience with stochastic calculus and probability theory. I don't come from school per se but has a super famous theoretical physics department (Yang institute) and I worked with many of those Professors. How do I approach a company with a preparation but not much life maturity? I have lot of coding and some ML skills.
The key to finding work is being able to present the skills you have that match the jobs. Often when people from different backgrounds write resumes they list really cool skills like string theory but those of us hiring don't know exactly how that applies to our work. If you can list our models, theories, and tools that we know it makes it much easier for us to interview you and make a job offer.
@Dimitri Bianco can you share what some of the most important models to know are?
@Kevin Hammon all the basic ones. OLS, WLS, logistic, SARIMAX, GARCH, CART (decision trees), RF, and GBM. Depending on your problems you might build splines, markov chains, and monte Carlo.
Is it worth getting a MFE or MSCF after a MS stats for quant finance?
I don't think so but there are scenarios where it would help. Learning to market your skills is crucial for finding any job.
I'm a consultant who did his master's in Operations Research. I also work with Corporate Finance as a domain.
I agree that the democratization of data science has led to a general degradation of discourse in the space. I can not believe I got hired simply because I could clearly explain a linear regression. The bar is low if you have been rigorous in your academic life. I have seen software devs who do data science not be able to explain what would be basic reasoning taught in an econometrics course, but will use the OpenAI APIs to sell their deliverables. It's sad.
Do you think a 'Co-op BBA (Management & Finance) and Co-op BSc (Statistics - Quantitative Finance)' degree would be enough to get a job in quant finance?
In a competitive finance city (NYC, Chicago, London, Hong Kong, and etc.), no. You need a master or PhD level of math and stats. The topics all build so you really need the undergrad material plus a deeper look and more advanced topics from a graduate degree.
Hey Dimitri, thanks for all the information! I'm about to start my masters in data science and artificial intelligence, and I'm curious about the relationship between quants and data science. Is it possible to transition from a data science role to becoming a quants professional after completing my masters? I'd love to hear your insights on this. Thanks!
It is possible however from my experience I have found most data science degrees to lack the math and statistics rigor. For example, many programs focus on NLP and image processing. For finance these have some fringe applications however I need really strong model developers who understand probability theory as the distributions that data is pulled from make a huge difference. As a specific industry example, I had a team of data scientists from one of the banks viewed as top data scientists in finance. They were working on a time-series model however they split their training, val, and testing data such that each set had a mix of data from all years. I failed their model as this is data leakage. Even after multiple meetings of trying to explain this they couldn't understand. Just from personal experiences however I think many data science programs are skipping much of the base theory in favor of programming and application.
Short conclusion, you can do it but make sure to take meaningful classes on theory and not just blind application classes.
@@DimitriBianco Thank you, Dimitri, for your insightful response! I appreciate your perspective on the math and statistics rigor in data science programs. It's clear that a strong foundation in probability theory and understanding the underlying distributions of data is crucial, especially in the finance industry.
I understand the importance of theory and not solely focusing on programming and application. As I embark on my masters in data science and artificial intelligence, I'll make sure to seek out meaningful classes that provide a solid theoretical background.
Your personal experiences with data leakage and the challenges faced by the data science team from the bank highlight the significance of practical knowledge combined with a deep understanding of foundational concepts.
Once again, thank you for sharing your valuable insights and emphasizing the importance of a well-rounded education in both theory and application. It's given me a clearer perspective on how I can prepare for a potential transition from data science to the quants field. I look forward to learning more from your videos! Keep up the great work!
@@DimitriBiancoSo what books would you recommend to bridge that type of gap in knowledge as I'm also will begin my masters in data science
@@Simba365 I'll make a video about this. Keep eye an out for it in the next few weeks.
@DimitriBianco looking forward to it
Really informative video. Out of curiosity, do "junior quants" typically progress into senior quants in firms without educational programs, or do they end up leaving after a few years?
Typically all the education is done before your first quant job (Jr quant) and then you work your way to senior quant and management roles.
I'm in an undergrad DS program and all the points in this video are analysed and over analysed so we can actually implement the models and undestand what is happening under the hood, because why re invent the wheel, just build a car instead.
The fact data scientists do not test hypotheses like econometricians do does not mean they do not have any specific questions they want to answer. Econometrics and statistics are like, I think or believe, or theory says A affects B, I want to test that hypothesis. Data science is more like, I don't have a theory, but I want to predict the future value of A...using several hundred or thousands of variables that I think affect A. Eventually, unimportant features would be dropped or given less weight. An econometrics model that can be tested does not necessarily mean it is a better predictor. And if you have hundreds or thousands of features with no formal theory, and prediction as the goal, maybe ML models are a better choice. What you have done in this video is trying to put econometrics/statistics models above data science models in quantitative finance. All your negative comments were on data science models, and all your positive comments were on econometrics/statistical models. These are two completely different models, doing different things. One should not try to compare them and choose one over the other. Econometric models want to estimate and test an effect given some predefined theory or hypothesis. Data science models, on the other hand, are about prediction when you have a lot of features with no formal predefined theory or hypothesis. If you are gonna criticize the data science field because of fake data scientists, why not also criticize the econometrics/statistics field because of fake econometricians and statisticians? Because there are fakes in any field, it does not matter whether data science or econometrics or statistics. I agree, though, data science has become easier, but so is econometrics and statistics. Defining a hypothesis is not rocket science...and anyone can come up with a testable hypothesis and use statmodels or STATA, SaaS or R, etc. One reason why data science has become popular (and not econometrics or statistics) is that you can find new insights from massive data with no predefined theory or hypothesis. But in your view, I guess that advantage is rather a flaw and not an advantage. Well, I would say do not compare oranges to apples. Besides, I also think data science has become popular because the resources to do data science are free...unlike econometrics/statistics in the past, where you needed paid software (SaaS, MATLAB, STATA, EVIEWS, SPSS, etc.). If these were free in the past, maybe, econometrics/statistics would have been like data science, where everyone can do it with almost no cost. Your trashing of the data science field here is largely unjustifiable because the same arguments also apply to the econometrics and statistics fields. In any case, testing a hypothesis is no rocket science...and it can even be easier than looking for meaningful insights in a huge data set. Just a thought. I am a fan of your channel. .... an economist trying to switch to quantitative finance.
You should review some of my past videos. I have criticized economics and quantitative finance. Data science can be done rigorously and there are forms doing it. My team specifically is building machine leading, math, and stats models which are all being used in production. The data science field is very immature because it is not very old in comparison to other fields. Quant finance is also having a melt down currently due to a variety of changing factors including outdated curriculum which was criticized in the last video.
@@DimitriBianco
Statistics is not a branch of mathematics. It's an empirical field of study, like physics. Rigorous proof (by the standards of mathematicians) is neither required nor even desirable.
Any "theorems" in statistics require such narrow and precisely defined hypotheses that they only apply in very restrictive situations (for instance, the Neyman-Pearson theory of hypothesis testing).
MLE is just finding the maxima of a function. Usually this is just standard vector calc stuff: take the gradient, find the critical points. If you know measure theory and Ito calculus, you have more than enough background to understand MLE.
@@DimitriBianco Take, for example, the very commonly taught rule of thumb: When the sample size is 30 or more, the sampling distribution of the sample mean will be approximately Normal. Apart from being simply incorrect, there’s typically little or no justification for where this rule comes from. I have to think this really gets under the mathematician’s skin. Of course, rules like this come from “experience,” and it’s simply loosely true for most populations… whatever I mean by most. This kind of loose, vague language really affronts the mathematician’s sensibility.
Statistics is not a branch of mathematics. It's an empirical field of study, like physics. Rigorous proof (by the standards of mathematicians) is neither required nor even desirable.
Any "theorems" in statistics require such narrow and precisely defined hypotheses that they only apply in very restrictive situations (for instance, the Neyman-Pearson theory of hypothesis testing).
Take, for example, the very commonly taught rule of thumb: When the sample size is 30 or more, the sampling distribution of the sample mean will be approximately Normal. Apart from being simply incorrect, there’s typically little or no justification for where this rule comes from. I have to think this really gets under the mathematician’s skin. Of course, rules like this come from “experience,” and it’s simply loosely true for most populations… whatever I mean by most. This kind of loose, vague language really affronts the mathematician’s sensibility.
Haha, in so called date scientist. I totally agree with you. Date science can be anything from working with Excel to building models in python etc. It's super washed up. I like to call my self a "days scientist focusing in insurance pricing" just because ds is about nothing haha
What is you opinion on data science masters that dont require quantitative background?
Not good. You might end up with the degree but you'll be doing business analytics or applying models not knowing what is really going on behind the scenes.
There always is the possibility that you personally take the time to fill the gaps however from a hiring perspective, I wouldn't hire someone from a weak data science program.
No? Not really. A great model usually comes down to doing everything yourself from Researching to Implementing or a team of people that's reduced into different roles, even then.. it's a gamble. You could have a team that half asses everything or disagrees without the same vision in the place.
After talking to a Data Scientist recently, they are just told what their suppose to do but without validation. That's like light hearted plagiarism without being detected by the AI Bots in school. Most of them, don't even care, If it works... slap a model, make sure it looks good and calling it a day.
There are great Data Scientist but usually become Quants... if they ever dabble into Financial Markets
So do you suggest going into a Finance PhD? I'm really interested in doing so and further my knowledge in Quant Finance. What do you suggest?
They’re not as quantitatively rigorous but you can get decent jobs elsewhere
@@incertosage you mean with a Finance PhD?
@@akkshheyagarwaal7629 yeah
@@akkshheyagarwaal7629 yes
Honestly all of this is just playing with language. Of course real Data Science requires rigorous scientific thinking and solid software engineering. And it's the same for Quant Finance. Just ignore all the fake it till you make it people that think they can work in a technical role without understanding what they are doing. Imagine a mechanical engineer developing critical parts of a car without understanding the models...
Statistics is not a branch of mathematics. It's an empirical field of study, like physics. Rigorous proof (by the standards of mathematicians) is neither required nor even desirable.
Any "theorems" in statistics require such narrow and precisely defined hypotheses that they only apply in very restrictive situations (for instance, the Neyman-Pearson theory of hypothesis testing).
Take, for example, the very commonly taught rule of thumb: When the sample size is 30 or more, the sampling distribution of the sample mean will be approximately Normal. Apart from being simply incorrect, there’s typically little or no justification for where this rule comes from. I have to think this really gets under the mathematician’s skin. Of course, rules like this come from “experience,” and it’s simply loosely true for most populations… whatever I mean by most. This kind of loose, vague language really affronts the mathematician’s sensibility.
data science is just a beginner friendly mode of computational statistics
Should be taken with a serious grain of salt. It’s an N of 1 viewed through an incredibly narrow scope. The man himself should’ve taken a correlation it’s not causation approach/tone to his critique.
"Data science is a joke". I'm sold. I'm a data scientist, and you're right, the field is pathetic. It's also basically impossible to break into since I've been out of university for many years. And since the job these days is so incredibly easy, it's hyper competitive, and pretty much impossible to break in. If I ever want to work in the field again, I'll have to go back to school to get a degree where i'll learn absolutely nothing new.
What's the difference between financial engineering and quant finance masters?
They typically mean the same. Many people use them interchangeably. Technically financial engineering is someone who builds derivative products. Quant finance is just a more general term to catch all.
@@DimitriBianco thanks for answering. So , quant finance people can also form derivative products or its more of statistical modelling ? What should one choose for doing statistical modelling?
@kits1111 financial engineering is just a specialty within quant finance programs. For statically modeling I would look at the program curriculum and see how many stats classes they have.
Wait until ai take over that job
8:06 😂😂😂
U are not a quant
Im a math undergrad and I knew from the jump that data science was just a subset of stats but this has convinced me I should just focus on learning the stats. I don't think black box models are going to hold my attention for very long. I can't deny not "needing" a masters for data science is pretty damn attractive though 😅 that might just be my laziness