I found no explanation for this anywhere and when reading the paper missed the detail that each tokens positional encoding consists of multiple values (calculated by different sine functions). Your explanation and visual representation finally made me understand! Fourier transforms are genius and I'm amazed in how many different areas they show up.
@@AICoffeeBreak this method relocates the embeddings in a specific direction in the embeddings space, so that new position in the relevant embedding cluster have "another" meaning to (say there is another instance of the same word later) other words of "same kind". But that place should be reserved other semantics, else the space is literally filled with "second position" coffee and "tenth position" me, "third position" good etc etc. This can go wrong in soooo many ways. Don't get me wrong, I am a clear cut "Chinese Room Experiment" guy, I don't think you can translate "he is a good doctor" before imagining an iconic low resolution male doctor and recall a memory of satisfaction and admiration of consumatory reward, but again, the "he" in "he did again" and "man, he did it again" should literally have the same representation in the network to start discussing things.
You are entirely right. I was short in my comment because I commented on the same issue in Cristian Garcia's comment. But there is no way you would have seen it, so I will copy paste it here: 😅 "Concatenating has the luxury of extra, exclusive dimensions dedicated to positional encoding with the upside of avoiding mixing up semantic and positional information. The downside is, you can afford those extra dimensions only if you have capacity to spare. So adding the positional embeddings to initial vector representations saves some capacity by using it for both semantic and positional information, but with the danger of mixing these up if there is no careful tuning on this (for tuning, think about the division by 10000 in the sine formula in "attention is all you need")."
And you correctly read between the lines, because this was not explicitly mentioned in the video. In the video I explained what an act of balance it is between semantic and positional information, but you identified the solution: If adding them up causes such trouble, then... let's don't! 😂
@@AICoffeeBreak new to AI, but what do you mean by the word "capacity"? Do you mean RAM? Do you mean that if we concat positional encodings to original vector instead of adding, it will take up more RAM/memory and therefore make the training process slow?
This is the most intuitive explanation of the positional encoding I have come across. Everyone out there explain how to do it, even with code, but not the why which is more important. Keep this up. You have earned my subscription.
I was browsing through some channels after first stopping on Sean Cannells and I noticed your channel. You got a great little channel building up here. I decided to drop by and show some support. Keep up the great content and I hope you keep posting :)
What a great video, I'm studying my Ph.D. at NLU, so this came in very handy. Thank you very much and greetings from Ensenada Baja California Mexico :D!
i watched bunch of videos on the positional embedding , but this video makes me very clear intuition behind the positional embedding , thank you very much for this great video , 😊
Thank you. One thing is to know the formula and applying it and other thing is to understand the intuition behind it. You made it very clear. All the best
Thanks Letitia! A vid on relative positional embeddings would be nice 😃 Implementations seems a bit involved so I've never used them in my toy examples.
Concatenating has the luxury of extra, exclusive dimensions dedicated to positional encoding with the upside of avoiding mixing up semantic and positional information. The downside is, you can have those extra dimensions only if you have capacity to spare. So adding the positional embeddings to initial vector representations saves some capacity by using it for both semantic and positional information with the danger of mixing these up if there is no careful tuning on this (for tuning, think about the division by 10000 in the sine formula in "attention is all you need").
Hi Letitia. Thank you so much for your wonderful video! Your explanations are more intuitive than almost anything else I've seen on the internet. Could you also do a video on how positional encoding works for images, specifically? I assume they are different from text because images do not have the sequential pattern text data have. Thanks!
Thanks for the suggestion. I do not think I will come to do this in the next few months. But the idea of image position embeddings is that those representations are most often learned. The gist of it is to divide the image into patches, let's say 9. And then to number them from 1 to 9 (from the top-left to bottom right). Then let gradient descent learn better representations of these addresses.
Gemini: This video is about positional embeddings in transformers. The video starts with an explanation of why positional embeddings are important. Transformers are a type of neural network that has become very popular for machine learning tasks, especially when there is a lot of data to train on. However, transformers do not process information in the order that it is given. This can be a problem for tasks where the order of the data is important, such as language translation. Positional embeddings are a way of adding information about the order of the data to the transformer. The video then goes on to explain how positional embeddings work. Positional embeddings are vectors that are added to the input vectors of the transformer. These vectors encode the position of each element in the sequence. The way that positional embeddings are created is important. The embeddings need to be unique for each position, but they also need to be small enough that they do not overwhelm the signal from the original data. The video concludes by discussing some of the different ways that positional embeddings can be created. The most common way is to use sine and cosine functions. These functions can be used to create embeddings that are both unique and small. The video also mentions that there are other ways to create positional embeddings, and that these methods may be more appropriate for some types of data.░
In another video I've seen, apparently it doesn't matter if positional embedding are learned or static. It seems as thiugh the rest of the model makes accurate deductions regardless. This is why I was not surprised that Fourier transforms seem to work nearly as well as self attention.
This was awesome! I don't have a complete understanding but it definitely pushed me to the side of understanding. Did you make a video about relative positions?
This is the best and most intuitive explanation of positional embeddings. THANKYOU so much for this video. btw: what software did you use to create these lovely animations?
Thanks, glad you like it! . For everything but Ms. Coffee Bean, I use the good old Powerpoint (morph and redraw functionality FTW). The rest is Adobe Premiere or kdenlive (video editing software).
I found no explanation for this anywhere and when reading the paper missed the detail that each tokens positional encoding consists of multiple values (calculated by different sine functions). Your explanation and visual representation finally made me understand! Fourier transforms are genius and I'm amazed in how many different areas they show up.
love how the "Attention is all you need" paper appears with an epic-like bgm
It wasn't on purpose, but it is funny -- in hindsight 😅🤣
This video is a clear explaination of why you shouldn't add your positional encoding but concat.
Extra dimensions dedicated exclusively to encode position! Sure, but only if you have some extra to share. 😅
@@AICoffeeBreak this method relocates the embeddings in a specific direction in the embeddings space, so that new position in the relevant embedding cluster have "another" meaning to (say there is another instance of the same word later) other words of "same kind". But that place should be reserved other semantics, else the space is literally filled with "second position" coffee and "tenth position" me, "third position" good etc etc. This can go wrong in soooo many ways. Don't get me wrong, I am a clear cut "Chinese Room Experiment" guy, I don't think you can translate "he is a good doctor" before imagining an iconic low resolution male doctor and recall a memory of satisfaction and admiration of consumatory reward, but again, the "he" in "he did again" and "man, he did it again" should literally have the same representation in the network to start discussing things.
You are entirely right. I was short in my comment because I commented on the same issue in Cristian Garcia's comment. But there is no way you would have seen it, so I will copy paste it here: 😅
"Concatenating has the luxury of extra, exclusive dimensions dedicated to positional encoding with the upside of avoiding mixing up semantic and positional information. The downside is, you can afford those extra dimensions only if you have capacity to spare.
So adding the positional embeddings to initial vector representations saves some capacity by using it for both semantic and positional information, but with the danger of mixing these up if there is no careful tuning on this (for tuning, think about the division by 10000 in the sine formula in "attention is all you need")."
And you correctly read between the lines, because this was not explicitly mentioned in the video. In the video I explained what an act of balance it is between semantic and positional information, but you identified the solution: If adding them up causes such trouble, then... let's don't! 😂
@@AICoffeeBreak new to AI, but what do you mean by the word "capacity"? Do you mean RAM? Do you mean that if we concat positional encodings to original vector instead of adding, it will take up more RAM/memory and therefore make the training process slow?
This is the most intuitive explanation of the positional encoding I have come across. Everyone out there explain how to do it, even with code, but not the why which is more important.
Keep this up. You have earned my subscription.
Wow, thanks a lot! :)
I had my morning coffee with this and will make an habit!
What a cool comment! Thanks.
+ 1 for video on relative positional representations!
i've read numerous articles explaining the positional embedding so far.. however, it is surely the greatest & clearest ever
Thanks a lot! Glad we could help. :)
Multi Dimensional Spurious Corelation Identifying Beast 🔥🔥
That's what I am calling transformers from now on
Bro
Where have you been hiding all this time?
This is next level explaining
Probably the clearest explanation for positional encoding:D
Thanks!
+1 for more vids on positional encodings.
You've solid understanding of Mathematics of Signal Processing
Just watched this again for a refresher; thee best video out there on the subject!
Wow, thanks!
Make Ms. Coffee Bean spill the beans on positional embeddings for images
😂
BEST AI channel on CZcams!!!!!
Very intuitive. I know there is sine cosine positional encoding but it is actually effective that I got it here..👍👍
Great stuff :) Would love to see more of that, especially for images or geometry!
Noted!
Most insightful explanation I have found on this subject so far. I was looking for it for days... Thank you! Keep going, you rock!
Thank you a lot! Also for the super thanks!
Great explanation !! And Yes Yes Yes.
Amazing! This is the best explanation for positional encodings period. Subscribed!!
Thank you!
OMG, such a amazing video to explain Positional Embedding....
thank you!
Great explanation of the intuition of positional encodings used in the Transformer!
Dang it, i learned something and my morning coffee isn't even finished.
Its going to be one of *those* days.
Sound like a good day to me! 😅
Whish you a fruitful day!
I was browsing through some channels after first stopping on Sean Cannells and I noticed your channel. You got a great little channel building up here. I decided to drop by and show some support. Keep up the great content and I hope you keep posting :)
Thanks for passing by and for the comment! I appreciate it!
Great explanation - It might be the best. I think I finally figured it out. I highly appreciate it.
Letitia's Explanation Is All You Need!
And comments like these!
+1
"A multi-dimensional, spurious correlation identifying beast..." ... wow. Douglas Adams would be proud of that.
Hope so. 😁
What a great video, I'm studying my Ph.D. at NLU, so this came in very handy. Thank you very much and greetings from Ensenada Baja California Mexico :D!
Thanks, thanks for visiting from so far away! Greetings from Heidelberg Germany! 👋
great explanation of positional embeddings. Just what I need.
The best explanation of how exactly position embeddings work !
Amazing content. After seeing this all the articles and research papers makes sense.
This is probably the best explanation of this topic on CZcams! Great work!
Thanks so much! 😊
I finally understand the positinal encoding! Thanks
Happy to help. :)
the most brilliant positional encoding explanation EVER that was GOD Level explanation
Haha, thanks.
Nice explanation
Thanks for appreciating!
This explanation is incredible
Super intuitive explanation, nice!
amazing explanation!!! I have been looking for a clear explanation on how the positional encodings actually work and this really helped! thank you :)
Brilliant video. This was the best explanation of positional encodings I have seen. It helped a TON!!!
I also make AI videos. I am more into the image side(convolutions and pooling) so it was great to see more AI educators.
i watched bunch of videos on the positional embedding , but this video makes me very clear intuition behind the positional embedding , thank you very much for this great video , 😊
Thanks, that's great to hear! ☺️
Super clear and amazing (as always) explanation of sines and cosines positional embeddings! 🙌
Thanks! Always happy when you visit!
I loved your simple and explicit explanation. You've earned a sub and like!
Thanks, see you again soon!
great explanation :)
Thanks! 😁
Thanks for the explanation
Interested to know about rotary position embedding
This was very intuitive, thank you very much! Needed this, please keep making videos
Thanks, as always, for the great explanation!
It was Ms. Coffee Bean's pleasure! 😅
Thank you. One thing is to know the formula and applying it and other thing is to understand the intuition behind it. You made it very clear. All the best
Well said! -- Humbled to realize this was put in context with our video, thanks.
Thanks for watching!
I feel lucky to have found your channel. Simply amazing ❤️
Happy to have you here!
Thanks Letitia, great explanation
Great explanation - thanks very much for sharing this.
Glad you enjoyed it!
Explained really well, thank you 😊
Thanks, so nice of you!
This is a great video! Just found your channel and glad I did, instantly subscribed :)
Thank you so much, you honour us!
Great! It took me forever, before I had found your videos, but finally I understand it. Thank you soooo much!
You're very welcome!
So helpful! Thank you a lot!
Very well explained! Thank you so much!
best video about position emb i have ever heard
Thanks!
This was so insightful!
Sooo good!
Excellent! Great work!
Lifesaver! Thank you for the explanation.
Really Thank you! Your video is just amazing!
Your explanation was great! I got everything I wanted to know about positional embedding. thank you:)
Awesome, thanks for the visit! ☺️
this video is a gem. thanks!
Just an amazing explanation ...
Hey, Great Explanation :). Love to see more videos.
Glad you found us! 😁
great explanation. really loved it. Thank you
These videos are priceless, thank you!
Amazing explanation! Thank you so much !
Nice explanation! I’d love to hear more about multidimensional and learned position encodings
Noted!
this helped me so much!! thank you!!!
Very good explanation!! Thanks for this 🙏🙏
Thank You for your wonderful message!
Thanks Letitia! A vid on relative positional embeddings would be nice 😃
Implementations seems a bit involved so I've never used them in my toy examples.
Regarding this topic, I've seen positional embeddings sometimes being added and sometimes being concatenated with no real justification for either 😐
Concatenating has the luxury of extra, exclusive dimensions dedicated to positional encoding with the upside of avoiding mixing up semantic and positional information. The downside is, you can have those extra dimensions only if you have capacity to spare.
So adding the positional embeddings to initial vector representations saves some capacity by using it for both semantic and positional information with the danger of mixing these up if there is no careful tuning on this (for tuning, think about the division by 10000 in the sine formula in "attention is all you need").
Wow, thanks so much! This is so well explained!!
Glad it helped. :)
Hi Letitia. Thank you so much for your wonderful video! Your explanations are more intuitive than almost anything else I've seen on the internet. Could you also do a video on how positional encoding works for images, specifically? I assume they are different from text because images do not have the sequential pattern text data have. Thanks!
Thanks for the suggestion. I do not think I will come to do this in the next few months. But the idea of image position embeddings is that those representations are most often learned. The gist of it is to divide the image into patches, let's say 9. And then to number them from 1 to 9 (from the top-left to bottom right). Then let gradient descent learn better representations of these addresses.
Thanks, your videos helped the most
Awesome. Thank you for making this great explanation. I highly appreciate it.
Great video, many thanks!
Thanks, this was so clear! Finally understood position embeddings!
Fantastic!
Gemini: This video is about positional embeddings in transformers.
The video starts with an explanation of why positional embeddings are important. Transformers are a type of neural network that has become very popular for machine learning tasks, especially when there is a lot of data to train on. However, transformers do not process information in the order that it is given. This can be a problem for tasks where the order of the data is important, such as language translation. Positional embeddings are a way of adding information about the order of the data to the transformer.
The video then goes on to explain how positional embeddings work. Positional embeddings are vectors that are added to the input vectors of the transformer. These vectors encode the position of each element in the sequence. The way that positional embeddings are created is important. The embeddings need to be unique for each position, but they also need to be small enough that they do not overwhelm the signal from the original data.
The video concludes by discussing some of the different ways that positional embeddings can be created. The most common way is to use sine and cosine functions. These functions can be used to create embeddings that are both unique and small. The video also mentions that there are other ways to create positional embeddings, and that these methods may be more appropriate for some types of data.░
Best explanation I've seen - good work!
Thank you so much! 🫨
Excellent explanation. Will certainly like to learn about other encodings in areas like image processing.
Loved your explanation, thank you for this video!
Glad you like it!
I really enjoy your videos 😄!
I would like a video about using transformers with time-series data.
Noted!
amazing stuff! so clear and intuitive, exactly what I was looking for :)
Thanks for watching and appreciating! 😊
love your explanations !!!
🤗
Beautiful explanation
Glad you think so! :)
In another video I've seen, apparently it doesn't matter if positional embedding are learned or static. It seems as thiugh the rest of the model makes accurate deductions regardless.
This is why I was not surprised that Fourier transforms seem to work nearly as well as self attention.
COuld you please elaborate using an example maybe?
Great work! Clear and deep explanation!
excellent explanation!
What a lovely explanation & video!.. Thank you!
Glad you enjoyed it! Thanks for the visit and leaving a comment.
@@AICoffeeBreak Thank you again!.. subscribed!!
@@bingochipspass08 Oh, great, then I'll see you on future videos as well.
This was awesome! I don't have a complete understanding but it definitely pushed me to the side of understanding. Did you make a video about relative positions?
Yes, I did! czcams.com/video/DwaBQbqh5aE/video.html
Great content! Can you do a video about Tabnet please? :)
Nice madam ...what a video !... outstanding
🙏
This is the best and most intuitive explanation of positional embeddings. THANKYOU so much for this video. btw: what software did you use to create these lovely animations?
Thanks, glad you like it! . For everything but Ms. Coffee Bean, I use the good old Powerpoint (morph and redraw functionality FTW). The rest is Adobe Premiere or kdenlive (video editing software).
This is an amazing explanation! Tku!!!🤓🥳🤩
Thank you for the explanation! Can you please make a video on Positional information in Gated Positional Self Attention in ConViT paper?
Noted!
so helpful, appreciate it!
you are a genius!!
Cool explanation
Really great!