My guess is that RNNs appear to have lower absolute error for extreme values than AR/-MA models at 13:27 because financial data, like daily stock returns, aren't normally distributed, but rather they follow heavy-tail distributions, like Cauchy. Since NNs are universal approximators, they can be used to capture the parameters of any distribution, whereas using AR/-MA models implies that the data are generated by a stationary gaussian process; thus they miss out on the extreme values of the true distribution.
the vast majority of content available on youtube is related to the application of ML models to price prediction when those of us who are dedicated to this know perfectly well that the area where it is most successfully applied is in portfolio rebalancing and the classification of market regimes. It would be great to make a comparison of different models such as RNN, GMM, HMM, RF, DBSCAN and KNN + Lorentzian distance, to see which of them is able to better classify market regimes using as features the first n PCA extracted from inputs such as volatility, LR slopes, rsi, stochastic, etc.
All the predictions are positives. Both models have simply learned that stocks tend to go up. This is a common results when you train your model on stocks listed in the SP500, which introduces survivorship bias (you are implicitly introduction lookahead bias, because those are stock that are still listed and ignored those which disappeared).
Hey so the 10 stocks chosen for each model were indeed chosen because they had predicted positive returns. However, if we look back to the overall distributions of predictions, there are an equal number of positive and negative predicted returns by all the models. I do agree with the point of training on stocks in the sp500 and it might make sense to broaden that set for next time. Thanks for the suggestion!
getting your hands on a survivorship bias free dataset from the S&P500 is terribly hard to do. There are some websites that offer these sets, but they tend to cost a few thousand $ per year. Gathering them independently is near impossible when you've not got good connections at the NYSE or with other data providers. But if you ever intend to do it please let me know i need it haha@@ritvikmath
I would've liked to see the base line you usually use. The return that you would've made if you simply just invested X$ every day over the course of the experiment. Overall great video though, love your work
Yes it would be great to see a follow up video with the baseline. If investing in a random scattershot of 10 stocks nets you 0.4% increase, then it's evidence the random market hypothesis is true and fitting models to the stock market doesn't really do anything useful!@@ritvikmath
Only recently discovered your channel and enjoy following your work. You show the distribution of predicted returns for the three models. I would have liked seeing another plot showing the actual realized returns. I expect those realized returns would look very much like the AR returns - highly leptokurtic. It appears to me that the RNN models are hugely overestimating the market volatility. To me, it seems an effective predictive model would have to match the distributional properties of the returns for one to have any confidence in its application to individual stocks. A possibly interesting experiment would be to generate some artificial stock data with an AR process and see how well those dynamics are captured by the RNN model. Keep up the good work.
Thanks! That’s a good call out for sure. I usually align the density axis only when I’m plotting multiple distributions on the same chart since, in this case, we mainly wanted to know which was widest. It is more correct though to align the density axis as well to make it more clear these distributions all integrate to 1 and are over the same observational units.
Great video idea!. BUT I'd love to have a comparison with the index (How much would a direct investment in the S&P have made) and maybe some metrics for preformance (sharpe, sortino, ...). Keep up the good work.
There was no mention of the vanishing/exploding gradient in your RNN model. That is, because there was a difference in prediction results between the AR model and the RNN model, it would have helped to see how close or far the largest eigenvalue is in the activation function is from 1. Intuitively, it would have helped to isolate the two models to one stock's time series to see if the RNN model has either an exploding or vanishing gradient. Specifically, the AR model can be iterated in discrete steps to model the time series. This, then, should be able to be replicated by the RNN model. If the discrete steps veer towards a sink or an attractor, that is, towards infinity or zero, then the problem is inherent with the RNN model and not with your data.
Would love a video on the actual maths behind estimating the AR model parameters (e.g. Yule-Walker etc.). The model form is easy to grasp but how it actually works - not so much
Four things: 1. Would a weighted average of the two models have the same outcome? It looks like GEHC is the only overlap. 2. Would stacking the two models improve the result? 3. Would adding another year or two of training data improve either model? That would probably dampen the response, but could help pick up seasonality. 4. It would be interesting to run a naive regressor (like next value = previous value) on the data to see if there's a general population trend.
1-3. Interesting idea and something I’d like to experiment with for future videos 4. Definitely, am trying to be more diligent about using common baselines for future videos in this series
If you want to make these much more realistic, get the returns per period (instead of one single avg) and calculate some measure of risk for them, especially maximum drawdown. From my data analysis, my models may not improve my returns over buy and hold strategy that much but they do decrease the max drawdown significantly.
So what I would love to see is your thought process behind the the setup. How did you choose the particular AR(IMA) model that you ran against the RNN models. Why did you choose a regularised RNN model? While I'm not interested in stocks, I do find time series stuff really interesting, so thanks for the entire playlist you've been working on. I found that it really supplemented the course content that I was working with.
Hey, it's just marvellous...please upload more! If possible can you show the above concept by doing it on Python etc It will be really wonderful. Thanks a lot, I really learnt a lot from your videos they are amazingg 💯
Very nice, i wonder how well this performs in a falling market though. Since from march 2023 - march 2024 the sp500 increased quite a bit! (And more than .4%!)
Great video, very interesting and informative! I was wondering if you would mind sharing the code by chance? I want to try something similar but need some inspiration on the coding front.
What was the time horizon used to forecast? As someone else already mentioned, I think it would also be nice if you could explore using tree based methods (XGB, LGBM, RF) for time series forecasting and compare te results to these kinds of methdods (AR, RNN).
could you possibly do this again for a VAR model or ARIMA model if you haven't already? I would be interested in seeing a comparison to some more model analysis
Very nice! Im not at all adequately equipped to critique or question the RNN model strengths and weaknesses so I’ll just pose this as a question instead. Wouldn’t another weakness be that it inputs previous predictions? Meaning if previous predictions were poor, this will negatively impact future predictions?
Very interesting question; this is certainly true when using any kind of time series model to predict more than one time step into the future. The error accumulates as you move through time. This model however predicts just one day ahead at a time and so any error comes from just the current time step. Of course, given it’s the stock market, this error is already very significant.
How long did you run the system on real money for? Many models are positive at the start becuse the training data resembles the current market - then as soon as dynamics or regime changes they fall off a cliff. So I would be skeptical to conclude anything from the fact it was briefly positive - but good vid. Where did you get those models from btw? (the equations) Also regarding overfit, have you heard of k fold cross validation? Its super simple - you can do basic cross validation in 15 minute in py.
Hey great intuition about models performing better the closer you are to the validation data. I didn't get into the specifics here but the method was basically using N-1 days for training and then validating on the next day's return. We roll a window like that forward through time to understand the average performance of these models. The equation for the AR model is the usual one you'd find in any econometrics textbook. The equation for the RNN is the way the SimpleRNN class is implemented in Keras and is a bit different from how you might have seen RNN's usually taught. Specifically, in Keras the hidden layer and output layer are the same but in more theoretical approaches they are distinct.
13:14 Aren't you conditioning on the outcome variable here? The actual return the next day is unknown at the time of prediction. I think it would be more useful to condition on whether the *predicted* return was >1% to know ex-ante as an investor whether the prediction from AR or RNN is more reliable. Informative video though, thank you!
Interesting question. I think they’re both useful measures in the same way that precision (which calculates correct predictions among all positive predictions) and recall (which calculates correct predictions among all positive labels) are both useful measures. The charts I showed help us understand how well each model does on truly extreme instances and the charts you propose help us understand how well each model does on truly extreme predictions.
Great video - I've been using LSTMs for over 2 years to try and create a robust model capable of predicting movements on the market, trained on the SP500 stocks, not dissimilar to your video! Would be interested to see your setup for pre-processing, feature enhancement, model architecture and lookback parameters. Also subscribed!
.4 is 40%, .04 is 4%, 4% of $1000 is $40. --To only make $4, the result would be .004 not .4…. Did this do better than you thought or was there a mistake in the graphic for the video? Because the result of .4 means the you made $400.
I'm curious what kind of training/validation loss you were training to? I'm a physics major that has been playing around with different model architectures for my own stock prediction model. Mine looks at 120 day time steps and trys to predict the next two weeks of closing prices. Ive seen some good results in training my transformer model, I've also found that the more training data I've added the better my model performs. Takes about 2hrs per epoch on my laptop though 😵💫
you could look into batch normalisation layers and dropouts to improve time. and did u try to experiment with batch-sizes in general? sometimes performance doesnt really go down but the training process speeds up by alot by varying the parameters like batch-size, learning-rate, etc. do you already use your gpu to run tensorflow? you could also think of using some kind of provider like google colab to run it on their infrastructure aswell
so the regularized model puts constraints on how big or small the coefficients can get which attempts to stop the model from learning the training data too well
Love that idea! I actually just tried it and if we only considered stocks where the AR and RNN both had positive return predictions and the absolute difference between their predictions was less than 0.001, the experiment return would have been basically flat (we lost $0.54 on the $1000 investment). Super awesome idea for future vids!
@@ritvikmath but then when you got the model's outputs, did you place those predictions manually or did you use some application to do it automatically? surely it must have been the latter, no?
@@bleizthomas4707I don’t think he did this in actual live market environment; wasn’t it just an experiment? The actual infrastructure you would need to execute this strategy safely are most likely beyond the means of your average retail trader. There are some retail brokerage platforms that will allow you to execute trades with an API but I would seriously avoid them due to the many risks involved (I.e., not only can normal mishaps with your infrastructure kill you but it’s also highly likely you see price slippage that kills your trades as well). You should never execute intraday trading on your own - leave it to the RenTechs, Two Sigmas, etc. of the world that can afford the tools to do this correctly. Just my two cents.
![The moment we stopped understanding AI [AlexNet]](/img/n.gif)
My guess is that RNNs appear to have lower absolute error for extreme values than AR/-MA models at 13:27 because financial data, like daily stock returns, aren't normally distributed, but rather they follow heavy-tail distributions, like Cauchy.
Since NNs are universal approximators, they can be used to capture the parameters of any distribution, whereas using AR/-MA models implies that the data are generated by a stationary gaussian process; thus they miss out on the extreme values of the true distribution.
Thanks for that analysis ! Really insightful
Hmm, then RNNs might be more suitable to model stocks with higher volatility and AR more suitable for lower volatility stocks.
the vast majority of content available on youtube is related to the application of ML models to price prediction when those of us who are dedicated to this know perfectly well that the area where it is most successfully applied is in portfolio rebalancing and the classification of market regimes. It would be great to make a comparison of different models such as RNN, GMM, HMM, RF, DBSCAN and KNN + Lorentzian distance, to see which of them is able to better classify market regimes using as features the first n PCA extracted from inputs such as volatility, LR slopes, rsi, stochastic, etc.
Great suggestion!
100% regime shift discovery is the true alpha
@@ritvikmathLike that idea too!
All the predictions are positives. Both models have simply learned that stocks tend to go up. This is a common results when you train your model on stocks listed in the SP500, which introduces survivorship bias (you are implicitly introduction lookahead bias, because those are stock that are still listed and ignored those which disappeared).
Hey so the 10 stocks chosen for each model were indeed chosen because they had predicted positive returns. However, if we look back to the overall distributions of predictions, there are an equal number of positive and negative predicted returns by all the models. I do agree with the point of training on stocks in the sp500 and it might make sense to broaden that set for next time. Thanks for the suggestion!
getting your hands on a survivorship bias free dataset from the S&P500 is terribly hard to do. There are some websites that offer these sets, but they tend to cost a few thousand $ per year. Gathering them independently is near impossible when you've not got good connections at the NYSE or with other data providers. But if you ever intend to do it please let me know i need it haha@@ritvikmath
Videos related to trading are quite informative and really helpful. Would like to see more of such content.
More to come!
Love these videos too
I would've liked to see the base line you usually use. The return that you would've made if you simply just invested X$ every day over the course of the experiment. Overall great video though, love your work
Hey great callout; I think I need to be better about using the same baselines in all the videos in this series. Glad you liked it!
Yes it would be great to see a follow up video with the baseline. If investing in a random scattershot of 10 stocks nets you 0.4% increase, then it's evidence the random market hypothesis is true and fitting models to the stock market doesn't really do anything useful!@@ritvikmath
Exactly, like comparing it to a random sample. Is it better than random?
Best video on these two models I've ever seen. Simple, clear, concise, with calculus and theory
Wow, thanks!
Only recently discovered your channel and enjoy following your work.
You show the distribution of predicted returns for the three models. I would have liked seeing another plot showing the actual realized returns. I expect those realized returns would look very much like the AR returns - highly leptokurtic. It appears to me that the RNN models are hugely overestimating the market volatility. To me, it seems an effective predictive model would have to match the distributional properties of the returns for one to have any confidence in its application to individual stocks.
A possibly interesting experiment would be to generate some artificial stock data with an AR process and see how well those dynamics are captured by the RNN model.
Keep up the good work.
Thanks for the analysis and suggestions!
Nice video, love your content! I liked how you aligned the x axis on the video, but I think you forgot to align the y-axis on density.
Thanks! That’s a good call out for sure. I usually align the density axis only when I’m plotting multiple distributions on the same chart since, in this case, we mainly wanted to know which was widest. It is more correct though to align the density axis as well to make it more clear these distributions all integrate to 1 and are over the same observational units.
Great video idea!. BUT I'd love to have a comparison with the index (How much would a direct investment in the S&P have made) and maybe some metrics for preformance (sharpe, sortino, ...). Keep up the good work.
Great suggestion!
There was no mention of the vanishing/exploding gradient in your RNN model. That is, because there was a difference in prediction results between the AR model and the RNN model, it would have helped to see how close or far the largest eigenvalue is in the activation function is from 1. Intuitively, it would have helped to isolate the two models to one stock's time series to see if the RNN model has either an exploding or vanishing gradient. Specifically, the AR model can be iterated in discrete steps to model the time series. This, then, should be able to be replicated by the RNN model. If the discrete steps veer towards a sink or an attractor, that is, towards infinity or zero, then the problem is inherent with the RNN model and not with your data.
This is highly informative. More of such videos!
More to come!
Wow, first video i have seen from you and i really liked the way you explained the concepts in a easy to understand yet detailed way!
Thanks!
Great video! I'm going to college next year for data science + machine learning and your whole channel so informative!
Love to hear it; wishing you well for college!
Would love a video on the actual maths behind estimating the AR model parameters (e.g. Yule-Walker etc.). The model form is easy to grasp but how it actually works - not so much
Good suggestion!
Four things: 1. Would a weighted average of the two models have the same outcome? It looks like GEHC is the only overlap. 2. Would stacking the two models improve the result? 3. Would adding another year or two of training data improve either model? That would probably dampen the response, but could help pick up seasonality. 4. It would be interesting to run a naive regressor (like next value = previous value) on the data to see if there's a general population trend.
1-3. Interesting idea and something I’d like to experiment with for future videos
4. Definitely, am trying to be more diligent about using common baselines for future videos in this series
If you want to make these much more realistic, get the returns per period (instead of one single avg) and calculate some measure of risk for them, especially maximum drawdown. From my data analysis, my models may not improve my returns over buy and hold strategy that much but they do decrease the max drawdown significantly.
So what I would love to see is your thought process behind the the setup. How did you choose the particular AR(IMA) model that you ran against the RNN models. Why did you choose a regularised RNN model? While I'm not interested in stocks, I do find time series stuff really interesting, so thanks for the entire playlist you've been working on. I found that it really supplemented the course content that I was working with.
Thanks for the feedback!
Hey, it's just marvellous...please upload more! If possible can you show the above concept by doing it on Python etc It will be really wonderful. Thanks a lot, I really learnt a lot from your videos they are amazingg 💯
Awesome to hear!
Very nice, i wonder how well this performs in a falling market though. Since from march 2023 - march 2024 the sp500 increased quite a bit! (And more than .4%!)
Interesting question! Will keep that angle in mind for future videos in this series
Can you please make a video on prediction for cryptos??
Interesting suggestion!
Great video, very interesting and informative! I was wondering if you would mind sharing the code by chance? I want to try something similar but need some inspiration on the coding front.
What was the time horizon used to forecast? As someone else already mentioned, I think it would also be nice if you could explore using tree based methods (XGB, LGBM, RF) for time series forecasting and compare te results to these kinds of methdods (AR, RNN).
Seems that AR predicts pretty well stocks with low-volatility. Then the question is why to even predict, if a basic quant analysis can do it ?
could you possibly do this again for a VAR model or ARIMA model if you haven't already? I would be interested in seeing a comparison to some more model analysis
Great suggestion!
Well done.
Thanks!
Very nice! Im not at all adequately equipped to critique or question the RNN model strengths and weaknesses so I’ll just pose this as a question instead. Wouldn’t another weakness be that it inputs previous predictions? Meaning if previous predictions were poor, this will negatively impact future predictions?
Very interesting question; this is certainly true when using any kind of time series model to predict more than one time step into the future. The error accumulates as you move through time. This model however predicts just one day ahead at a time and so any error comes from just the current time step. Of course, given it’s the stock market, this error is already very significant.
@@ritvikmath Thanks for taking the time!
Excellent video! Does the equation on the right apply to LSTMs as well?
How long did you run the system on real money for? Many models are positive at the start becuse the training data resembles the current market - then as soon as dynamics or regime changes they fall off a cliff. So I would be skeptical to conclude anything from the fact it was briefly positive - but good vid. Where did you get those models from btw? (the equations) Also regarding overfit, have you heard of k fold cross validation? Its super simple - you can do basic cross validation in 15 minute in py.
Hey great intuition about models performing better the closer you are to the validation data. I didn't get into the specifics here but the method was basically using N-1 days for training and then validating on the next day's return. We roll a window like that forward through time to understand the average performance of these models. The equation for the AR model is the usual one you'd find in any econometrics textbook. The equation for the RNN is the way the SimpleRNN class is implemented in Keras and is a bit different from how you might have seen RNN's usually taught. Specifically, in Keras the hidden layer and output layer are the same but in more theoretical approaches they are distinct.
Love it!
Thanks!!
The real take-away is that in the US, $4 buys you only half a cup of coffee.
So true
13:14 Aren't you conditioning on the outcome variable here? The actual return the next day is unknown at the time of prediction. I think it would be more useful to condition on whether the *predicted* return was >1% to know ex-ante as an investor whether the prediction from AR or RNN is more reliable. Informative video though, thank you!
Interesting question. I think they’re both useful measures in the same way that precision (which calculates correct predictions among all positive predictions) and recall (which calculates correct predictions among all positive labels) are both useful measures. The charts I showed help us understand how well each model does on truly extreme instances and the charts you propose help us understand how well each model does on truly extreme predictions.
The models traded for a single day. I suppose that's not a good way to evaluate their capabilities, right?
Great video - I've been using LSTMs for over 2 years to try and create a robust model capable of predicting movements on the market, trained on the SP500 stocks, not dissimilar to your video! Would be interested to see your setup for pre-processing, feature enhancement, model architecture and lookback parameters. Also subscribed!
That’s awesome to hear; always love hearing about folks trying to model the market with ML methods! And thanks for the sub 🙏
.4 is 40%, .04 is 4%, 4% of $1000 is $40. --To only make $4, the result would be .004 not .4…. Did this do better than you thought or was there a mistake in the graphic for the video? Because the result of .4 means the you made $400.
So it’s 0.4% which means we made $1000 * 0.4 / 100 = $4
great vid :)
Thanks!
I'm curious what kind of training/validation loss you were training to?
I'm a physics major that has been playing around with different model architectures for my own stock prediction model.
Mine looks at 120 day time steps and trys to predict the next two weeks of closing prices. Ive seen some good results in training my transformer model, I've also found that the more training data I've added the better my model performs.
Takes about 2hrs per epoch on my laptop though 😵💫
you could look into batch normalisation layers and dropouts to improve time. and did u try to experiment with batch-sizes in general? sometimes performance doesnt really go down but the training process speeds up by alot by varying the parameters like batch-size, learning-rate, etc. do you already use your gpu to run tensorflow? you could also think of using some kind of provider like google colab to run it on their infrastructure aswell
at 12:00 you said you used regularization but sound like you gave description of normalization, can you clarify? thanks.
so the regularized model puts constraints on how big or small the coefficients can get which attempts to stop the model from learning the training data too well
@@ritvikmathL2 or L1?
Would it be possible to share the code?
I wonder the performance of RNN in forecast weekly return
Great direction for future videos!
What are the features that you considered? Is it only the stock price itself.?
Yes for these models
Also can you share the code you used to make these models?
Is there a mistake in the way you wrote the AR model sum at the start, as the derivative w.r.t x_i wouldn't quite be \beta_i ?
Thanks for pointing it out, yes the notation needs to be adjusted a bit
could you please do a video on bayesian structural time series!!
Great suggestion!
What would the error look like if you only used the picks where both models concur? I wonder if that would help or hurt your returns in the end 🤔
Love that idea! I actually just tried it and if we only considered stocks where the AR and RNN both had positive return predictions and the absolute difference between their predictions was less than 0.001, the experiment return would have been basically flat (we lost $0.54 on the $1000 investment). Super awesome idea for future vids!
Is there any way to message you directly?
Why not marry both and use the deepAR architecture? :)
Interesting direction!
Github code share? 😊 please
Awesome video!
Have you tried transformers?
To come !
Gotta pay taxes on that $4 though 😅😅
Mannn make that a quarter coffee 😂
Or maybe, y’know… everything makes money when the market goes up 🤷🏻♂️
Can you tell us which tech stack you used for the automatic trading ?
hey this was coded in python, specifically using the yfinance library to get all the relevant stock price data
@@ritvikmath but then when you got the model's outputs, did you place those predictions manually or did you use some application to do it automatically? surely it must have been the latter, no?
@@bleizthomas4707I don’t think he did this in actual live market environment; wasn’t it just an experiment? The actual infrastructure you would need to execute this strategy safely are most likely beyond the means of your average retail trader. There are some retail brokerage platforms that will allow you to execute trades with an API but I would seriously avoid them due to the many risks involved (I.e., not only can normal mishaps with your infrastructure kill you but it’s also highly likely you see price slippage that kills your trades as well). You should never execute intraday trading on your own - leave it to the RenTechs, Two Sigmas, etc. of the world that can afford the tools to do this correctly. Just my two cents.