ROC and AUC in R
Vložit
- čas přidán 16. 12. 2018
- This tutorial walks you through, step-by-step, how to draw ROC curves and calculate AUC in R. We start with basic ROC graph, learn how to extract thresholds for decision making, calculate AUC and partial AUC and how to layer multiple ROC curves on the same graph.
You can get a copy of the code from the StatQuest GitHub, here:
github.com/StatQuest/roc_and_...
NOTE: This StatQuest builds on the example in the original ROC and AUC StatQuest:
• THIS VIDEO HAS BEEN UP...
Also, if you're curious, here are some links to StatQuests about...
...Logistic Regression
• StatQuest: Logistic Re...
...and Random Forests...
• StatQuest: Random Fore...
For a complete index of all the StatQuest videos, check out:
statquest.org/video-index/
If you'd like to support StatQuest, please consider...
Buying The StatQuest Illustrated Guide to Machine Learning!!!
PDF - statquest.gumroad.com/l/wvtmc
Paperback - www.amazon.com/dp/B09ZCKR4H6
Kindle eBook - www.amazon.com/dp/B09ZG79HXC
Patreon: / statquest
...or...
CZcams Membership: / @statquest
...a cool StatQuest t-shirt or sweatshirt:
shop.spreadshirt.com/statques...
...buying one or two of my songs (or go large and get a whole album!)
joshuastarmer.bandcamp.com/
...or just donating to StatQuest!
www.paypal.me/statquest
Lastly, if you want to keep up with me as I research and create new StatQuests, follow me on twitter:
/ joshuastarmer
#statquest #ROC #AUC
You can get a copy of the code from the StatQuest GitHub, here: github.com/StatQuest/roc_and_auc_demo/blob/master/roc_and_auc_demo.R
Support StatQuest by buying my book The StatQuest Illustrated Guide to Machine Learning or a Study Guide or Merch!!! statquest.org/statquest-store/
Hi Josh,
Love your content. Has helped me to learn a lot & grow. You are doing an awesome work. Please continue to do so.
Wanted to support you but unfortunately your Paypal link seems to be dysfunctional. Please update it.
The code would not run when I downloaded it from github
@@ryanmckenna2047 What part didn't run? I just re-ran it and worked fine.
did u make this in python too??
@@ashishdayal172 not yet
"The only man who never makes mistakes is the man who never does anything."
Thank you ;)
No, thank you! You're comment was very helpful and spared me a lot of future embarrassment. The video was only seen by 100 or so people (not 1,000s) before you pointed out the error.
Dude, your videos are great. I never found something so clearly on the internet. Congratulations!!!
Crazy how good you are at explaining. You explain the little things I always start to struggle with other teachers/tutors! Thank you so much for these Videos
Happy to help!
Complex things in simple and understandable language. I have never met a better teacher!
Thank you very much! :)
Such an awesome channel I came across! ....gonna share it with everyone under my umbrella !!! You are doing really great bro!
Thank you! :)
Impressive video. Theory and examples with software are the best way to learn. There is much going on this video, one of the best of ever. Thank You, Josh, greetings form Italy for a happy new year for you, your beloved ones and for all the people which follow your amazing lessons.
Wow, thanks!
Your explanation of the process and logic behind each function and line are so helpful. I hope you'll make more of these videos. Thank you so much, this content is uniquely valuable.
Thanks!
Great explanation of everything including each parameter in the graphs. Loved it!
Thank you!
Best song ever, Josh. StatQuest keeps gettin’ better and better! Many thanks.
Thank you so much! :)
Hey Josh Ty again, while my studies I reproduced everything using R Colab (Really recommend for who is studying Josh's codes in R)
Good job and well-done. I like your style of teaching, it's great!!!
Thank you! 😃
Thank you for helping me with my credit risk class :)
Thank you for this informative video. It helped me a lot. Great work!
Glad it helped!
Wonderful tutorial!!.....thank you so much Josh :)
Thanks! :)
These are the best videos. When I need to relax, I watch your videos
Glad you like them!
@@statquest If you have two output neurons in a ANN (for a two class classification problem {1,0; 0,1}, it is okay to build the ROC just by comparing output of any one of those neurons with its corresponding target?
Thanks Josh, I changed it to {1,0} as output as the AUC for the two neurons {1or0} in the {1,0;0,1} architecture were not the same.
Hey..... Love the way you present ❤️
Thank you so much 😀
You solve my headache. Thanks a lot
Happy to help!
Thank you sooooo much Josh! You are a life saver!!😄
Happy to help!
You are amazing man, thanks for the video and keep making more videos like these. BAM!!
Double BAM!!!! Thanks for the encouragement! :)
Double Bam!!
Thanks for your wonderful and detailed videos!
Thank you so much for supporting StatQuest! BAM! :)
Incredibly helpful, thank you!
Thanks!
Dear ,i haveenjoyed ur video ,very much clearity of thoughts
Thank you so much 🙂
Thanks man, very clear and helpful
Thanks! :)
YOU ARE MARVELOUS,EXTRAORDINARY .I WISH YOU COULD HAVE EXPLAINED IN PYTHON
One day I will.
Thank you sooooo much for your lessons. Super helpful
Thanks!
Many Thanks Josh!
You're welcome! :)
hey bro, i love your videos so much, please hang in and i will continue to support you!
Thank you very much!
thank you for such an informative tutorial
Glad it was helpful!
Thank you very much!
The silly songs, the calm voice and the bams gives this vibes as if the course is narrated by Forrest Gump.
Love it.
Thanks! :)
VERY helpful - thank you!
Glad it was helpful!
You are amazing, man! Thanks!!!
Thanks
Keep up the good work .. Thank u🤩
Thanks!
Thank you so much for this video!
Glad it was helpful!
Thanks a lot sir! You are very helpful!
Most welcome!
So good Thanks for the video
Glad you enjoyed it!
waaa.. i'm so thankful found this video. Thanks a lot. Stay healthy cool people :)
Thanks!
Thanks a lot !
I thank God I found this channel 2 years ago... 😇
bam!
@@statquest 😄😄😄
Great video thanks
Thank you!
Thank you sir
great video!!
Thank you! :)
Great video! One quick question. Do you know how to plot ROC-AUC graph for SVM and adaboost?
This video is absolutely amazing! but how can i determine the threshold/cut off weight from threshold probability that decides whether the subject is obese or not using code and not by direct extrapolation from the logit curve?
Bam! Good tutorial.
Thanks! :)
I mean, the stats tutorial is indeed very well done, but the intro song was already enough to make me immediatly click on the like button.
bam! :)
Great again! I would be interested to see how to make combined ROCs for, say 2-4 different biomarker candidates. This would be to see if their combined use would result in higher AUCs than that of individual markers.
Noted
Thank you!
:)
Is there anyway to suppress plotting the top and right axes? I tried bty='n' and axes=FALSE to add them later using axis(1) and axis(2) but neither of those worked.
thank you soooo much!!!
:)
Excellent videio
Thanks!
Hey Josh, great videos on ROC curves, your teaching is refreshingly concise and clear. I just have one question that I hope you could expand on. When we first generate 100 samples from a normal distribution, why do we need to sort them from low to high? And what would the dangers be if we didn't do this?
Thanks for the great content!
What time point in the video, minutes and seconds, are you asking about?
@@statquest roughly around 2:55
@@user-io2em1ld4n Technically, you don't need to sort them, but it makes it easier to look at the data. When we print out the values for the "obese" variable at 4:11, the output is way easier to interpret because the values for weight were sorted.
Thank you so much!
Do you consider make a video about limited dependant variables models (tobit, heckman...)?
It will be very helpful for us! All the best.
OK. I'll put it on the to-do list, but it will be a while before I get to it.
Thank you! This is a short bibliography about the topic:
J. Scott Long, Regression Models for Categorical and Limited Dependent Variables
Alfred DeMaris, Regression With Social Data: Modeling Continuous and Limited Response Variables
Wooldrige, Introductory Econometrics
I can share you the books if needed.
@@Davidravaux OK. However, just know that my to-do list is huge (it has about 200 things on it - I get about 3 or 4 requests every day), so it might take me a long time to get to it. However, if a lot of people start asking for a certain topic, that topic gets moved closer to the top of the to-do list. So, if you know of a ton of people interested in this subject, you should have them add to this comment.
Ok, I totally understand, thank you for clarifying.
Please make more Videos with R! :)
:)
Ah, the pirate's favorite programming language!
:)
Thanks for the video and explanations! What statistical test would you use to compare 2 ROC curves?
There are a bunch of options. This tool (in R) implements them: bmcbioinformatics.biomedcentral.com/articles/10.1186/1471-2105-12-77
You are the savior of the little humans we are, thanke you god! I have a silly question, sometimes you use
R is funny about the "
I got an error, Error in roc.data.frame(trainData, fitModelTrai$votes[, 1], plot = TRUE, :
'response' argument should be the name of the column, optionally quoted. the only difference between your code and mine is that I have many parameters/columns/features (approx 35) not only one (weight)
That ROC you had really tied the room together
:)
Thank you
:)
Thank your great lectures! The thresholds that you derive here are between 0 and 1. Can we translate these thresholds to the actual cut-off values?
In these examples, the thresholds are the actual cut-off values. In other words, if the logistic regression predicts that the probability that a mouse is obese is 0.9, then we would compare that to the threshold that we obtained from the ROC graph to make a final classification.
i am a big fan of you! can you make a survival anaylsis video?
Yes! I will make one this spring. Many people have asked for this topic, so it is at the top of my to-do list.
thanks you bro
You're welcome! :)
THanks for the videos Josh! I have a question about AUC. Even though in this video AUC for random forest is lower than logistic, isn't forest a better alternative here as there exists a threshold that generates higher true positive rate for the same false positive rate compared to logistic. This makes the significance of AUC subjective in comparison
What you have to do is pick a range of thresholds that are acceptable. Once you do that, you can compare the AUC between those thresholds to determine which method is best.
Love your content! Quick q: from a conceptual standpoint, are you just testing the hypothesis that the underlying distribution of the weights (which you defined as a gaussian) is not a uniform distribution
ROC graphs give us a sense of how accurate or models are given different thresholds for making decisions. For more details, see: czcams.com/video/4jRBRDbJemM/video.html
Hi Josh, your videos are great! I have one question about choosing best method based on ROC overlapping graph. If we compare Logistic Regression and Random Forest we see that Logistic Regression is better because of bigger AUC. Bur does it make more sense here to choose Random Forest because one specific instance of Random Forest (with one specific threshold) gave us best confusion matrics? I assumed here that accurately classifyng positive and negative class are equally important.
It really depends on your goals. In general, Logistic Regression performs better. However, depending on what threshold works best for you, you may still choose Random Forests if it performs better at that threshold.
Hi Sir, your videos are very helpful. Hope that you can make a video on mean decrease Gini of Random Forest
I'll keep that in mind.
thank you very much !!!! 😁😁
Hi and thanks for your great videos! Could you please elaborate about the obese variable and specifically about the "test" part in that code line. What if I already know who is obese and who is not (let's say based on some external medical profile, let's say "real") and I want to estimate the prediction of the model which is based on a some score (let's say "score") that each individual has. Would I just do glm(real ~ score).? What if I wanted to find the best score - the score that above it I classify someone as "obese" and below it "not obese". what's between the probability threshold in ROC curve and a thresholding of the score itself. Thanks!
In order to draw this ROC graph, we have to know who is obese and who is not to begin with. So the situation in this video is no different from yours. If you want to find the "best" score, you have to then decide what percentage of false positives and false negatives you are willing to live with - the ROC graph will help you decide that. You can then find the corresponding value by looking at the thresholds and the probabilities predicted for from your model with different scores.
Thank you for the video. It was very easy to follow. May I know how do i obtain optimal cut off points using the ROC curve?
I answer that question in my video that explains ROC and AUC: czcams.com/video/4jRBRDbJemM/video.html
@@statquest Thank you for your reply! I was actually wondering how to interpret the threshold numbers seen on 09:51. After head(roc.df), you get a list of TPP, FPP and thresholds. For example in the 2nd row TPP 100 FPP 97.77, what does threshold of 0.01349 mean?
I also have a separate question, I am curious if it is always necessary to always create a linear model first for the ROC curve? For example I am comparing the ROC curves of age and co-morbidities against non-cancer mortality, do I have to create a linear regression for age using glm()?
savior
Many thanks Josh, you are doing a great job.
In my study, I would like to calculate and plot pROCs for a couple of maxent scenarios and glm model scenarios using 1000 iterations and a 5% omission error using pROC package in R, would be really grateful if you can guide me a bit. Thanks in advance.
Let me know how it goes! :)
@@statquest May I get the R code for the scenario I mentioned? I am still trying to figure out how to prepare data from the maxent output and then use it with pROC package to calculate and plot AUCs. I am relatively a newbie in R. Theory wise I think I am pretty clear, but struggling with codes and commands to get this job done with pROC package.
@@rahulg1504 The code for this video is here: github.com/StatQuest/roc_and_auc_demo/blob/master/roc_and_auc_demo.R
Hello for the video, really useful, in this example you come up with a method to classify obese and not obese , what about when you don't know a threshold for the initial classification of obese or not obese ? Does the pROC function test different thresholds ?
That's the whole idea of an ROC graph to being with - it's used to determine the optimal threshold.
Hey Josh, is there a way to make inferences on more than two ROC and to perform multiple comparisons? (a generalization of DeLong's test? and maybe a method to adjust alpha for multiple comparisons too?)
Good question! Off the top of my head I don't know if there is or not.
You gotta stop saying BAM!!! it's really funny :D
Great vedio! Very helpful. BTW, there is a discrepancy between this clip and the code shared in your website about the obj roc.df (line 78). Nothing has been assigned to the obj yet so when we run the line 78 gives an error msg. Overall, very clear and handy. Thank you!
Thanks for catching that! The problem had to do with how wordpress interprets the the ">" and "
@@statquest I see. Good to know! Thank you~ :>
Loved the video! How do you relate the threshold back to the data? I.e. make a statement like the threshold between obese and not obese is 140lb
First, you find the threshold you are interested in (these are in roc.df), then we look at weight associated with the largest glm.fit$fitted.values < the threshold. For example, if the threshold is 0.5, then the weight is: max(weight[glm.fit$fitted.values < 0.5])
Love You
:)
How d you calculate the p-value of the AUC (obtained from logistic regression model)?
See: stats.stackexchange.com/questions/386468/does-auc-roc-curve-return-a-p-value
YOU DA BEST
Thank you!
is it possible to print the cutoffs at seq(0.1, by=0.1) in this curve with roc() function? ...Awesome videos btw!!!!!!!!
I'm not sure this would be easy to do, since the thresholds may not exactly equal 0.1, 0.2, 0.3 etc. For example, in this video, the thresholds start at 0, then the next one is 0.013, then 0.032, ..., 0.088, 0.1004, 0.119, etc. So you see, there is no threshold that is exactly 0.1. So you'd have to calculate the differences from different thresholds and print the one that has the smallest difference.
I Have a problem with this. The ifelse function is giving me a different value each time i run it. ¿Do you have a clue why is this happening?
Hey! Wonderful video. I had just one doubt- I used a similar code that you used in my Rstudio. And as the runif function is generating random numbers, I could have very well expected that the values in the obese variable is different from the ones generated in your machine. However, eerily enough, it came out to be exactly the same. What sort of sorcery is this? 😮
Did you set the seed of the random number generator? If so, we'll get the same random numbers every time.
Hey josh great video. I'm having a small doubt. Is there any way to plot ROC graph for multiclass ? I ran a multinomial logistic regression model on my dry bean dataset which has 7 classes. Is there a way to plot ROC graph for this ?
I don't know how to do that.
Hahaha.. That cute confession that you have a hard time remembring what sensitivity and specificity mean, made me laugh.. Because it is so confusing to me also.. These really are confusion metrics🤣
Since I had so much trouble remembering about sensitivity and specificity, I wrote a little song to help me out: czcams.com/users/shortsPWvfrTgaPBI
Wow.. You are so creative at making things easy.. I am impressed!
i am following python for data science so far and got stuck after saw this video , best person like you using R language instead of python so what should i do and which one is best for data science and also in future purpose R program or python kindly let me know and enlighten me
thanks in advance ..! little BAM
They are both very useful. Python is a great language used in a lot of different situations and has a lot of good machine learning libraries. In contrast, R is very useful for doing statistics.... So I would recommend learning both if you have time.
Thank you for another great video. I have a question, what if we have multiple problems for classifications? Not only two classifications (obese and not obese). For example, we want to classify 10 cell types (let's say cell type 1, cell type 2, ..., cell type 10) whether these cell types are present or not in the tissue sample? How can we use this roc() function to plot the ROC curve?
To be honest, I don't know the answer to that off the top of my head.
@@statquest I have made my own function to plot the ROC curve with similar condition I mentioned. However, I need to make another function to calculate the AUC and was hoping I could use the roc() function which seems providing more information and can include much more information, such as AUC and partial AUC as well. 😰
thank you for your video. btw, can you make one for python?
I'll work on it. I'm doing a lot more Python coding these days, so it makes sense.
@@statquest A year later, I suddenly wake up to StatQuest. Python implementation please. Perhaps SciKit Learn also has built-in computations for these and other metrics. I'll check...
Is it expected that the number of sensitivity/specificity values determined by the roc function (that we stored in the data frame) may not match the number of predictor/response values that I input? For example, my input predictor/response vectors contained 46 objects, but the roc function returned only 12 sensitivity/specificity values.
I believe this is possible if there are fewer thresholds that make a difference. In other words, some thresholds might result in the same number of false positives, true positives etc. and in that case, those "duplicate" thresholds will be omitted.
@@statquest Okay great this is exactly what I thought was happening--just wasn't sure if that was a possible outcome. Thanks so much for your reply and for all the great videos!!!
Is is possible to compute the significance level of AUC using the pROC package? Sth similar to the SPSS output
As far as I know, you can do confidence intervals. For more details, see: www.rdocumentation.org/packages/pROC/versions/1.16.2
Hi, Josh! A big fun of yours! Thanks for so many wonderful videos! Wonder if I can ask for help. I am using glmer function in R to fit a mixed effect logistic regression to my longitudinal data. However, I am having trouble extracting fitted value for my model to draw a ROC, like what you did with glm.fit$fitted.value. I have been searching about it but in vain. Appreciate it if you can give me a clue! Thanks very much!
This might help: stats.idre.ucla.edu/r/dae/mixed-effects-logistic-regression/
@@statquest Thanks for your kind reply! The web page didn't solve my problem directly, but it's very informative! Thanks!
could you do the same in python too?
Hello! Any ideas on how to plot the Precision-Recall Curve?
I'll keep that topic in mind.
What does par(pty=‘M’) do? You said it’s maximum but does it change the shape of the plot ?
Yes. It uses up all available space to draw the plot, regardless of the shape of that space (so if that space is rectangular, your plot will be rectangular). In contrast, setting pty='s' forces the plot to be square.
Great video! I was wondering if it is possible to plot this graph in a Multinomial Logistic Regression?
Hmmm...I'm not sure.
@@statquest Ah okay, what about a multiple logistic regression? Any ideas about that one?
@@mathiasschmidt93 As long as your predicted value is binary, it shouldn't matter how many variables you use to make predictions - the process is the exact same as illustrated in this video. To see how it is done in R, see: czcams.com/video/qcvAqAH60Yw/video.html
Is there a way to increase the font size of the text and numbers? Great job btw
Yes. See: stackoverflow.com/questions/4241798/how-to-increase-font-size-in-a-plot-in-r
BAM !!!!!!! Indeed
:)
BAM!!!
Right on! :)
I love u so much
:)
I installed and loaded pROC, it says couldn't find function roc... Which editor u used? I used rstudio, help pleaseeee?
I used RStudio as well. Sorry you'r having trouble.