Singular Value Decomposition (SVD): Dominant Correlations
Vložit
- čas přidán 18. 01. 2020
- This lectures discusses how the SVD captures dominant correlations in a matrix of data.
These lectures follow Chapter 1 from: "Data-Driven Science and Engineering: Machine Learning, Dynamical Systems, and Control" by Brunton and Kutz
Amazon: www.amazon.com/Data-Driven-Sc...
Book Website: databookuw.com
Book PDF: databookuw.com/databook.pdf
Brunton Website: eigensteve.com
This video was produced at the University of Washington - Věda a technologie
![SVD: Image Compression [Matlab]](http://i.ytimg.com/vi/QQ8vxj-9OfQ/mqdefault.jpg)
![SVD: Image Compression [Matlab]](/img/tr.png)
I cannot put in words how much talent you have for teaching. It is just impressive. Thank you so much.
You do these fantastic explanations with your original behind-the-whiteboard technique AND you're giving the book this information comes from for free?
??????????
What a gift you are to the internet, thank you so very much. I love your teaching style.
Wow, thank you!
If they had a Nobel prize for teaching on CZcams, this guy would be one of the top contenders
czcams.com/video/0Ahj8SLDgig/video.html
I could complain about these videos being to fast-paced and technical for a more novice audience, or that the explanations are not very thorough, or even about the idiosynchratic notation (m for number of columns, like come on matrices are mxn by default). But it still wouldn't change the fact that these videos motivated me to learn more about SVD and served as a invaluable resource for getting introduced to the maths I need for my job. Thank you. I hope there is more content in the making.
This guy is the refreshed version of professor Gilbert Strang. Professor Steve Brunton, you're an amazing linear algebra teacher. Glad I found these videos. Going to recommend to my entire computational linear algebra classmates.
Thank you so much
Strang will always be the GOAT.
OMG.... These videos really moved me. I haven't had the feeling to learn knowledge with excitement and refreshment, thank you so much for being such a good teacher. I feel the beauty of math.
this series is like a gift, thank you so much. Will definitely buy the book when I can :)
Hope you enjoy it! (check out databookuw.com/databook.pdf until then)
Engaged delivery with exceptional content clarity. If Mr Brunton isn't tenured it would be an indictment of the US university system. Cheer!
Professor Brunton is great. I saw his SVD slightly covered in his in-class lecture. Now he dedicate his time to make this video lecture. Super!
Usually some of the explanations can be skipped using Einstein notation but it is often ignored how very educational it can be to explicitly walk though the process. Respect!
the best grad school prof ever!
I learned SVD 4 years ago, and nobody ever explained it so well! Will recommend this series to every math student!
a real genius can dissect complex subject matter to explain it in a simple and intuitive way. and this guy is just a splendid example of such genius
Sir, do you even have a slightest idea what a superawesomely-superawesome teacher you are?!!!
Thank you, SO MUCH!
🙏🙏🙏
One of the best things to have happen in Jan 2020 is seeing you share more of your knowledge! Much Appreciated!
Wow, thank you. I just found your channel today, and I'm grateful for these intuitive explanations. Cheers!
Every one is applauding the wonderful lesson, which is very true!!
BUT No one is applauding Steve's wonderful ability to write inverted.
Thank you for this fantastic series, definitely among the best educational content i have come across on youtube!
Your descriptions here are, by far, the best I have seen. Maybe it is because you teach in the way I learn but these have been really useful.
Thanks a lot.
I haven't watched the series yet, but I already know it's fantastic.
I already liked your classroom course on the same topic.
One of the best CZcams explanation i have ever seen.
Dominant correlations? More like "Dang good pieces of information!" Being serious, this is some of the absolute best content on CZcams.
This is as good as it gets, thank you.
I got the book last week, and the Python implementation from the website is a great addition. Thanks again for these priceless videos.
This is so good. We need more teachers like this!
thanks a lot ❤.. youve made someone in Lebanon get SVD in a very clear way :)))....
That was so clear and well explained. It's really connecting a lot of dots in the maths I've had to learn for modelling in engineering
This is one of the best Stuffz I've seen on the internet.....explicit and was taught with so much passion, rigor, Aggressiveness in presentation, clear intuition etc. With these stepwise approach, even the blind is set and equipped to be a Genius and solve real time problems. You don't want to know how grateful I am. Thanks
best video on SVD topic, concept crispy clear, and the video is movie quality, really enjoy
Amazing lecture, thank you for making the SVD series! I appreciate the importance of SVD much more after watching your lectures. Thank you again!
Making my research ever so meaningful! Thank you.
wow these tutorials needs high level of attention!
thanks for the extraordinary lectures
Thank you so much ! This seemed so complicated and you made It so clear and easily understandable , amazing !
Thank you for making such excellent learning material freely available, you are a godsend.
As always, It is really nice to have more prof. Brunton content to learn from. This is the nudge I needed to get your new book and dive deeper to data-driven modeling and machine learning control. Thanks, prof. Brunton. I just finished the series, I am really looking forward to the rest of it.
If the [X]T[X] represents the correlation between the columns, does that mean that the diagonal has the largest elements? And does [X][X]T represent the correlation between the row? Can that be interpreted in any meaningful way?
Thank you very much for these amazing lecture videos, I bought the book after watching a few of them. I am also very happy that you made it available as an ebook as well, it works great on my Kindle :)
Thank you so much for uploading these amazing lectures!
I don't think I would ever see this sort of intuitive explanation of U and V matrices in terms of Eigen values and Eigen vectors. Thanks a lot Professor for helping humanity in understanding SVD.
You are the the best teacher I have seen
God Bless you, Steve because you make a big difference in how to teach ML salut from Brazil
This video is incredible. Such a great explanation
Incredibly well explained. Thank you!
Your teaching is so impressive and awesome.
I am not clear on how X^tX is a correlation matrix, which I understand to be a matrix of correlation coefficients between variables. Really enjoying this series.
Yeah that's my problem too! The video is incredible but I don't get why those matrixes are correlations matrixes
It technically isn't a correlation matrix. If you center each variable (column) by subtracting a mean, then X'X yields a covariance matrix (if you divide the whole result by n-1). If you normalize each column in X by its standard deviation, then X'X yields a correlation matrix. I think he leaves out these details to simplify.
@@neurochannels ty .. that seems plausible
Thank you for the great video!
Couple questions:
1. Is the column space of the matrix X what people usually refer to as the feature space?
2. How does it relate to the standard basis on R^n?
3. When you take the inner products in (X^T)(X), is that the inner product in R^n? or some other inner product? i.e pearsons correlation on the space of random variables.
Your lectures are treasure for us ...........👍👍❤️❤️
So nice of you!
Very good videos you are making! Keep it up!
I honestly understood nothing at all when my professor tried to explain SVD and stuff...i really don't like the lecture style of my university because it is basically: yeah the SVD is a thing and you have the decomposition of input and you can use it and i mean it is easy to see that this are eigenvalues and this are the eigenvectors and you easily get to that formula
Like this is how the professor says it..but without even writing something down or showing anything..or if he writes something it is literally the sentences he said. So i was just like: wtf is this and how does it work and why are these eigenvalues and why this formula and stuff. And sure at a university you should always ask if something is unclear but there is always waaaaay to much to ask about simply because the explanations are a big amount of BS.
And now watching this i understood sooo much..and it even only took 11 minutes..i really wish my university would explain like that :(
anyways i am glad there are people like you so in my freetime i just learn all the stuff on my own 🤷♂️
For this amazing videos you are going to heaven.
Thank you, your explanation is beautiful.
Excelent content, really love it! thanks
Great thanks professor!!! At last I understood what SVD is
very clear. I love it professor
you just got a genius for teaching complex things.
You're awesome. I wish I could watch this video a few years earlier.
greatly put. again, linear algebra is magic.
First SVD I've ever liked and like spreading it.
This lecture cleared a misunderstanding I had about SVD. Thank you so much !
Glad it was helpful!
For sure, the best explanation I'd seen about the whole topic. I'm very surprised on the production and staging. Prof. Brunton seems to be writing in a transparent surface with actual pens. However, it's not possible to film it from the perspective all we are seeing unless he is doing mirror writing, like Leonardo Davinci. Involved mathematics, fluent speaking and mirror writing, at the same time! too much, even for a sharp mind. So, how was done? Maybe the scene is reflected in a physical mirror and the camera is pointing that mirror, and some linear algebra to correct the perspective 🙂, or maybe everything is advanced postprocessing. Please, we want to know how did you shoot this fantastic video. Greetings from Spain!
Thank you so much, your videos helped a lot !
Beautifully explained :)
Great explanation! thanks for sharing
My pleasure!
Nicely explained again
Great lecture Steve. You say that we usually want the economy SVD. In what situations would we want to compute the full svd please?
thank you very much! you video is most intuition!!!😊
Thank you so much! So clear, this is what you need to be a good engineer/scientist in any fields.
Great interpretations!
Why would you say U is the eigenvectors of the column space of the data when U is corresponding to X @ X.T which is the correlation between rows of X.
Hello professor, many thanks for the video's. I have watching both your electrical engineering (laplace transform) videos, as well as this one on SVD. I am bit confused where the approximation you state (towards the end of the video) comes from. Is the original SVD itself approximate ? In the case of truncated SVD, we are just dropping the columns that would be zero'd anyway correct ?, so at what point of time do things become approximate.
Awesome videos!
THAT WAS A W E S O M E!
Very clean and clear explanation of the computational process for SVD. However, it doesn’t explain much the intuitive concepts of SVD in terms of what it is conceptually as a decomposition of a linear transformation with basis change.
Thanks for the video!
Does the data have to be centered/standardized for transpose(X)*X to be the "correlation matrix" ?
Brilliant. I think there is one little error at 6.10. Sigma has the size mxn, so Sigma Transpose is of the size nxm. They both have sigma1,..,sigma_m in the diagonal. So instead of Sigma^2 we get another mxm matrix. It doesn't change end result,but I thought it was important to mention to avoid confusion.
CZcams should add in option to put multiple likes
Thank you for the great content, I have a question:
At 3:54 you say "if these are people's faces then the ij-th entry of this matrix ( X^t X ) is the inner product between person i and person j 's face, so if there's a value of this matrix that is large that means that those two people had a large inner product, their faces are similar, they have the same basic face structure. If you have a small value of this inner product that means that they are nearly orthogonal and they're very different faces".
The inner product (as usually defined for a Euclidean space) of two vectors gives you an idea of the "similarity" of the two vectors, in this case each vector is a "list" of numbers each representing a pixel of a picture of a face. The idea of "if the two vectors are similar then the two faces they represent are similar" supposes that the pictures were somehow taken such that the distribution of the values of the pixels was related to the facial characteristic, correct? Otherwise it's not clear to me how could the similarity between two faces be "detected" (for example two photos of the same face in different lighting/position would be associated to two very different vectors).
Thanks!
Very impressive and didatic.
Hi. Great explanantion! Just had a slight confusion. Isnt it like Inner product signifies similarity but wont be able to give a correlation since 'technically' correlation could mean a measure of linear reltionship between 2 variables (That is, how much a value cahnges in response to the other)? Bu tin the video, it is marked as a correlation.
This is gold.
Hi, so I'm just wondering, does a column in X here represent a feature column, and the rows represent the different samples, or do the columns represent the different samples and the rows represent the different features?
In the book (p. 13) you say:
"_This provides an intuitive interpretation of the SVD, where the columns of
U are eigenvectors of the correlation matrix XX* and columns of V are eigenvectors of X*X".
Isn't the rows of
U are eigenvectors of the correlation matrix XX* and rows of V are eigenvectors of X*X, is it?
I performed XT*X for V and X*XT for U on a matrix X. But X does not equal to the U*S*VT. I even converted the values of U and V to orthogonal. Still does not equal to the orignal X. What am I doing wrong?
I am greatful for such a good series of videos with you Steve as your are really talented. I have a question that might be basic : to obtain a proper correlation matrix, one should center X and reduce it afterwards ? I asked chatgpt but trust you more on this ^^'
I know this specific video is about intuition, I'm just asking to confirm my understanding thanks :)
Thanks, I'm glad you like them! Yes, typically you would center X (subtract the column-wise mean or row-wise mean first, and then take inner products with either all pairs of columns, or all pairs of rows). We discuss this in databookuw.com/databookV2.pdf if you want more details.
Ok super thanks!@@Eigensteve
Hi Prof. Steve,
I don't understand why the eigenvectors given by the right singular vectors do not correspond to the same vectors when I compute Eigenvectors function (in Mathematica software) of the matrix X^T.X. Could you please help me?
I have a singular matrix X ={{-1, 0, -1}, {1, -1, 0}, {0, 1, 1}}, with rank =2, with the economy SVD I get V= {{1/Sqrt[2], -(1/Sqrt[6])}, {0, Sqrt[2/3]}, {1/Sqrt[2], 1/Sqrt[6]}}. The relationships are clearly satisfied as the definition of eigenvectors, i.e., X^T.X.v=3.v.
However, when I run the Eigenvector[ X^T.X] it returns {{1, 0, 1}, {-1, 1, 0}, {-1, -1, 1}}, where the first vector is ok, and corresponds to 1/Sqrt[2]*{1, 0, 1}, but the remain ones have no correlation with v2= {-(1/Sqrt[6]), Sqrt[2/3], 1/Sqrt[6]}. However, I know that this last eigenvector {-1, -1, 1} corresponds to the (right) null space of X. Am I doing something wrong?
Maybe it is because this matrix has the four spaces and that is the reason for these vector are not correponding??
So, my question is, why I got different eigen vectors when considering two different techiques?
Shouldn't they span the same space?
Thank you
Hi Prof,
at 6:43 , it is substituted U^ (U^)T = I. But U^ is an economy matrix you, earlier it is mentioned that they are no more unitary matrices once truncated. Please clarify this point.
In the previous lecture, he mentioned that for truncated matrices, U'U=I, but it is not the case that UU'=I. In this lecture, he only made use of the first fact. So you are correct they are not "full" unitary, but you only need "half" unitary for this proof to go through.
Why dot product of X is correlation matrix? Did we substructure the mean somewhere?
The class is just incredible, and you are a very good teacher. But I just want to know, how do you write it in front of you and the text and math symbols are not inverted to us? Thats a nice magic trick.
@7.28 can we say that svd= v€2VT is equally represented by cholesky decomposition LDLT?
Awesome!!
Great video, thanks for it, but I have a question regarding your explanation of positive semidefinite in minute 4:36. You said that X*X is symmetric and positive semidefinite because the elements are inner products. Nevertheless, according to the definition of an inner product, it is only positive definite when and it does not occur for all the elements of that matrix. Can anyone explain to me if that was a mistake in the video or if I am wrong, please?
Hi! Yes, off-diagonal elements of a matrix can be negative, but this does not mean that the matrix is not positive semi-definite. Positive semi-definite ensures that the eigenvalues are not negative. You can google about the Gram matrix :)
At first I also had this question )
I think, If nxm X has n features of m faces, then (X-p)(X-p)^T , where p is nx1 vector of row means(here p is broadcast from nx1 to nxm), is the nxn covariance matrix of the features among those faces.
(X-q)^T (X-q), where q is the 1xm row vector of column means (again broadcast to nxm), is the mxm covariance between the faces among those features.
After dividing by the number of data points - 1.
Hi. What isn't a hat V there since we are using the truncated SVD? Thanks.
is V the single vectors of the correlation matrix or the data matrix? 8:12
Mind blowing
I have a question. Does anyone know why the value of inner product is related to the correlation ?
Just to note, it is not the correlation matrix but the covariance matrix which is gained by the X^T X etc. Otherwise great video, I really needed this type of intuition for SVD:s.
I know that your post is a year old, but I was surprised that you seem to have been the only person that noticed the "correlation"/"covariance" confusion. Too bad that Steve Brunton did not respond to your post.
thank you so much
what kind of board do you use ??
Good video but it would also be handy if you marked the array sizes of the SVD factors (both full & economy), not just those of the correlation matrix factors.
Why are the values of the correlation matrix inner products of the columns of X and not of the rows of X? Don't we want to see the correlations between different features of the faces, not between the different faces themselves?
Great question. We dig into this more in the other videos in the playlist. But short answer is that you can look at the row-wise or column-wise correlation matrices, and they will have the same non-zero eigenvalues (really quite surprising when you first think about it). Both of these can be used to extract the SVD.
@@Eigensteve ahh ok thanks so much for the reply! Will watch the whole playlist now. Great videos btw
Amazing
Is it a guarantee that X^T X is positive semi-definite?