The Kernel Trick in Support Vector Machine (SVM)
Vložit
- čas přidán 8. 05. 2022
- SVM can only produce linear boundaries between classes by default, which not enough for most machine learning applications. In order to get nonlinear boundaries, you have to pre-apply a nonlinear transformation to the data. The kernel trick allows you to bypass the need for specifying this nonlinear transformation explicitly. Instead, you specify a "kernel" function that directly describes how each points relate to each other. Kernels are much more fun to work with and come with important computational benefits.
---------------
Credit:
🐍 Manim and Python : github.com/3b1b/manim
🐵 Blender3D: www.blender.org/
🗒️ Emacs: www.gnu.org/software/emacs/
This video would not have been possible without the help of Gökçe Dayanıklı. - Věda a technologie
I listened to my lecturer and I was convinced that not even she understood something of her lecture...You clarified a lot for me only in three minutes...
More common than you'd like to think, I'm also convinced my lecturers don't understand it either.
Your vidoes are absolutely amazing. Please keep making these, eventually the serious view numbers will come!
Wow! I can't believe I didn't find this channel until now- your videos are amazing! As a creator myself, I understand how much work must go into this, so HUGE props!! Liked and subscribed 💛
Incredible video, no messing around with long introductions, not patronising and easy to follow. It should be used as a guide for other people making educational videos!
This video is perfectly clear. I learnt SVM in class while I was confused by the lecture, and it is much clearer now.👍
seriously !!!! simplest explanation of Kernel in SVM ever seen , just wow
thank you so so much bro for the hard work you are doing to make such great videos ;*
The best and most concise tutorial on Kernel tricks and SVM.
Absolutely incredible explanation 👏
Please keep making these contents. They are so intuitive. I don't understand why such channel don't grow on the other hand shitty contents are growing exponentially.
so amazingly simple and clear explanation, thank you so much !
You saved me for my Data mining exam tomorrow
🙏
Thanks a lot! I struggled to understand the difference between conventional dimension lifting and the "trick". Now it's crystal clear! Great explanation.
Wonderful!!
Please don't stop doing videos you are helping a lot of people
excellent video, short but informative!
understood fully. thank you. giving a code sample is like a bonus. Awesome explanation.
just subscribed after watching this video only. Hoping to find more good content as these in your channel
Great Visuals and explanation. Got it in One go. Thanks
I went through class video for 1 hour didnt understand a thing..thank god you thought me in 3 min ..you are a legend bro
Thank you for this clear explanation.
super smooth explanation. Thanks!
Very good video! Nice visualization! :)
Wow, I am sharing this everywhere bro. Fantastic videos, we will grow together !!
Simply AMAZING! Thanks a lot!!!
Such an amazing explanation!
Amazing videos by explaining different concepts in simple words.
Please have long vides as well..
Keep doing the Good work 👏
Wow...this is for free.
Amazing visuals!
thankyou so much you have nailed it it is crystal clear about kernel after watching your video thanks again
Wow, very nicely explained
Gonna try the Kernel Trick!
Amazing Man.. 😍
Just Awesome...
Wow! Great video, thanks :)
Fantastic video, but I think you should mention at the end why the "kernel trick" isn't practical with lots of data (i.e. why deep learning is used much more than the "kernel trick" in this age of big data): given a kernel, you need to store the entire matrix of data inputs into the kernel. There are ways to mitigate this a bit (for example Random Fourier Features and Nystrom method) but still this is a huge issue that no one has seemed to figure out how to fix.
On the other hand, if you have a small amount of complicated data then the kernel trick is very useful! For example a medical researcher might only have access to the horrifically complicated genomic data of patients at their hospital.
incredible explanation!
I am korean so i am not good at english, but your teaching is very clear and easy to understand. Thank you teacher!
Super good explanation
fantastic contribution
Thank you!! Nice video :)
i love it ! thanks for the explanation
At 2:11 the shown Kernel and Transformation function do not match. The transformation function is missing the element ,1 as its last component and needs a scaling factor of sqrt(2) innfront of first 3 elements
👍
Amazing explanation
Thank you so much, excellent content.
I really like this video. Thanks!~
Fantastic video
Great Video Sir
Great.....Great presentation...This is what I mean when I say use Visuals graphics to explain a concepts
Really great video, thanks for sharing! Out of interest, what do you use to produce your videos?
Thank you!! I use Blender3D and manim
amazing !! subscribed
that's so powerful to understand
Something that took days to understand was well understood within a few mins. Cannot emphasize enough the importance of visualizations in MATH.
Awesome! Thanks a ton!
great video!
great work
to the point, concise, easy to understand, and even with code sample
thanks!
I wish I could like this video twice.
Just fantastic explanation, i was wondering how much time takes to make such a high quality video, and what software he is using to do it. ? ? anyone knows ?
Holy shit these are good videos!
Wonderful clear
Thanks, Sir. Your explanation is incredibly amazing
So nice of you! You are most welcome!
@Visually Explained It helps so much. I'm waiting for the next video
i have never understood kernel trick in SVM better.
Bravo! Great lecture! How the hell do you do this interactive function animations?
Hmmmm!
Grant Sanderson is getting a serious contender right here
Fantastic video, thankyou
Thank you too!
awsm video please make a video explaining K-Nearest Neighbors Algorithm also
Quick question: How do we choose the gamma parameter in the RBF kernel at 3:00? By, say, cross validation?
AWESOME!!
i hope u come back , i really like this content , pls
perfect video
really a superbe video ! Thank you.
So nice of you :-)
Amazing video! How do you animate your videos?
Thanks! I use manim and Blender3D
Thanks! How do you know which gamma to use?
Excellent
THANK YOU!!!
Great video! How does this differ from kernal ridge regression?
kernel ridge regression is just that: regression using the kernel trick. Namely, instead of a hyperplane of best fit, you do the kernel trick to implicitly nonlinearly map the data into a high dimensional (sometimes infinite dimensional) space. But like with SVM, we don't need this map, and just need the kernel matrix at all the data points to practically perform algorithms.
"Ridge" just refers to adding an l2 penalty to avoid overfitting. "Lasso" refers to l1 penalty, and I think in practice people even use l1+l2 penalties.
wow! that's so cool
Can you make a video on unique game conjecture / unique label cover ? That would be very helpful
What kind of software did you use to create such beautiful illustrations?
I have added a list to the video description
@@VisuallyExplained appreciated!
Amazing
excellent
What did you use for visualization ?
Blender, manim library, and after effect
you are amazing, you saved me !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
YAYY!
Im confused, how to display the hyperplane with polynomial kernel? Help me please?
did you use the Unity to produce this video?
he said blender
Crystal clear
Thank you
So easy to visualize
thanks!
In witch programmulka can doing this
How to I use this when my decision boundary needs to be spiral?
What software do you use to make these videos? and why have you stopped making videos!? and why did you start?
why did you have "1 + " term in your polynomial kernel?
Ty!
Loved your video..can you Mentor me?
Still don't really understand, but I'm closer, thanks!
i dream about this visualisations
I'd like to subscribe again
Great visualization.
As I see, kernel-based regression is a type of symbolic regression
I really would have liked to know why you claim that we only need to computer inner products. Does it arise from the dual problem? If i remember correctly that problem features such scalar products. And why is that better?
If you write the dual problem (e.g., page 4 of www.robots.ox.ac.uk/~az/lectures/ml/lect3.pdf) you can see indeed that it only depends on inner products of the training points. There is, however, a more "intuitive" way to see why SVM only cares about the inner products without actually writing the dual problem. If I give you the matrix X = (xi^Txj) of pair-wise inner products of n points, you can recover the coordinates of the n points "up to rotation" by computing the matrix-square root of X for example. In other words, all sets of n points that have the inner product matrix X are obtained by rotating the n points whose coordinates are the columns of sqrtm(X). Now, if you want to separate the n points with a hyperplane, your problem doesn't fundamentally change if all the points are rotated in some arbitrary way (because you can just rotate the separating hyperplane in the same way). So the knowledge of the matrix of inner products X is sufficient for SVM to do its job.
As to why that's helpful, let's say have 100 points, and each point has a million features (which can easily happen if you "lift" to the data). That's a 100 million numbers you need to store. However, the matrix of inner products will be 100x100 only, which is a huge saving!
@@VisuallyExplained this is my other Account. The first one is for work, didn't realize i used that one.
Thanks for taking the time to answer :). Interesting, not quite trivial, that the matrix contains all relevant information.
I have already read some more sources on it now. Since you seem to understand involved math, maybe u can help me with another Question. The Question is, why exactly we use the kernel trick instead of simply using a usual transformation into another vectorspace and then use usual linear svms. This seems like it would work so there has to be a motivation for the kernel trick. I already read that this has better performance. But even the book "hands on machine learning" only says that it "makes the whole process more efficient" which says practically nothing about the motivation. One thing one can easily notice is that since the dual problem optimizes only for the lagrange multipliers,, we have to calculate the kernel only once before training. This also seems to be the reason why the kernel trick only works for the dual problem. But i was wondering weither this is the whole Motivation or if there is some more magic that i missed here?
@@crush3dices There are basically two main reasons. The first that you already alluded to, is for performance reasons. It's more efficient to compute k(x,x') than the transformation f(x) if f is very high dimensional (or worse, infinite dimensional). The second reasons is practical: sometimes, it is easier to choose a kernel than a transformation f
@@VisuallyExplained alright thanks.
wow!