at 5:10 it seems that max pooling and conv layers are accidentally swapped. The max pooling layer has a smaller dimension than the leading layer and usually comes after a convolution.
Are there such thing that is similar to word embeddings? Or you simply take your pixel data as patches and run it through the dense layer to get projections?
In this video, there is no explanation for the output of a vision transformer. In NLP transformers, the output is a probability distribution over the vocab but in vision transformers, I guess it is over a code book. But what this code book is and how it is aligned to the input image is not clear. Thanks a lot for this video but it is incomplete
The output from an NLP transformer is a set of token-level embeddings not a probability distribution over the vocab. The probability distribution over the vocab that you’re referring to is actually an extra component (a head) that is used for Masked Language Modeling (MLM). ViT doesn’t use MLM for pretraining (unlike NLP transformers) so a equivalent head isn’t used. So the output for the ViT is actually the same as the NLP transformer, it is a set of token (for ViT this is patch)-level embeddings. I hope that makes sense? Thanks!
@@jamesbriggs Yes it is very useful. You are right the output is a token embedding but the output for MLM is a probability distribution over the vocab in NLP. I guess MIM is used as a pretraining in vision transformer. In this case, there should be a code book if I am not mistaken
Před rokem+1
Excellent video, James. Thank you! I have a question, how do you compute the 9.8 MM comparisons at 10:09 ?
hey, thanks a lot. i have come from TensorFlow. so can u please answer, is it training the whole vit model for our dataset or freezing the vit pre trained part and training classification head only (like trainable=false in tf)?
Thanks a lot for the video. I cant find any precise explanation about the function of self-attention layer and MLP layer in the encoder modules. Could you maybe add some information about that?
no fun using huggingface transformers library. you should have explained vision transformers using a more basic implementation, than a high level library
This was extremely clear and I was able to understand ViTs better. Thank you so much!
Really enjoyed every bit. Trying to setup the transformer for an Audio Regression task, the ViT has shown amazing performance in classification
Hi, do you suggest any articles that allow me to use transformers with audio classification? I would appreciate any help.
Great video! I've watched quite a few videos and read papers about Transformers, but your video really made me understand the concept
glad it was helpful!
The clarity of your discourse is unmatched and it's always a pleasure to follow your videos. Kinda a side effect of your passion for the domain!?
thanks a ton, I'm glad it helps - and yep it's definitely a bonus doing the videos in such a cool domain
@@jamesbriggs I think the content is so clear that you can imagine a full courses bundle ^^
It is part of a (free) course/ebook :) www.pinecone.io/learn/image-search/
James is the top G in deep learning
The is by far the best explanation of the paper that I could find. Thanks a lot!
Great explanation, unique on YT. Thanks!
you're welcome!
Very good introductory video. Thanks for sharing.
Oh man, really great explanation, easy to digest. Keep it up!
Incredible content! Thx James!
Thanks!
Great explanation. Very fine details. Great work
Another great video of yours. So clear and clarifying. Thx !
Glad it helped!
You are an inspiration james
at 5:10 it seems that max pooling and conv layers are accidentally swapped. The max pooling layer has a smaller dimension than the leading layer and usually comes after a convolution.
Thank you so much for this video :)
You're welcome :)
Thank you for the effort ur putting here in your explanations. :)
Great video man. Keep it up👍👍
Well, Bag of Words and Bag of Visual Words WAS a merger of NLP and Computer Vision, back in the day (2010s)
Thanks a lot for this ! Amazing amazing explanation!
Great video man cheers! Do you have a video about using a dataset made up of your own images on the vision transformer?
Are there such thing that is similar to word embeddings? Or you simply take your pixel data as patches and run it through the dense layer to get projections?
In this video, there is no explanation for the output of a vision transformer. In NLP transformers, the output is a probability distribution over the vocab but in vision transformers, I guess it is over a code book. But what this code book is and how it is aligned to the input image is not clear. Thanks a lot for this video but it is incomplete
The output from an NLP transformer is a set of token-level embeddings not a probability distribution over the vocab.
The probability distribution over the vocab that you’re referring to is actually an extra component (a head) that is used for Masked Language Modeling (MLM). ViT doesn’t use MLM for pretraining (unlike NLP transformers) so a equivalent head isn’t used.
So the output for the ViT is actually the same as the NLP transformer, it is a set of token (for ViT this is patch)-level embeddings.
I hope that makes sense? Thanks!
@@jamesbriggs Yes it is very useful. You are right the output is a token embedding but the output for MLM is a probability distribution over the vocab in NLP. I guess MIM is used as a pretraining in vision transformer. In this case, there should be a code book if I am not mistaken
Excellent video, James. Thank you!
I have a question, how do you compute the 9.8 MM comparisons at 10:09 ?
I think it should be 65.5k comparisons
hey, thanks a lot. i have come from TensorFlow. so can u please answer, is it training the whole vit model for our dataset or freezing the vit pre trained part and training classification head only (like trainable=false in tf)?
Thanks a lot for the video. I cant find any precise explanation about the function of self-attention layer and MLP layer in the encoder modules. Could you maybe add some information about that?
go watch nanoGPT video by andre karpathy
please can i have access to your powerpoint
it currently gives the error 'no module named 'datasets', anybody has a fix?
`pip install datasets` should work
This video is really helful. Thank you!
great stuff!
Glad to hear!
thank you sir
Most welcome
no fun using huggingface transformers library. you should have explained vision transformers using a more basic implementation, than a high level library
Nice one..! This content desperately needs "Promo SM"!