@@TheAIEpiphany Id be really interested in anything related to the autoregressive model part also mentioned in this video, maybe something like training a transformer?
I truly appreciate your explanations, especially PyTorch implementation part, which reduce the gap between concepts and real world implementations. Finding this channel is like finding treasures to me, I've recommended this channel to all my friends. Look forward to your weekly update, thanks :)
Appreciate your work! Both paper and code parts are very helpful. Two suggestion to make the code more concise - pytorch has built in function to calculate pairwise distance `torch.cdist`. - directly using `index_select` to get the quantized matrix may be more convenient.
Thank you for such a great explanation, adding code into this format is really helpful to digest the concepts more intuitively. Please keep them coming the same way.
Amazing explanation! Thank you very much. I was a bit troubled about understanding how this model can be used to generate new images but after reading around I think I get it now
Loved the explanation, especially the part where you covered all the important aspects and showed them in the code. Subscribed and looking forward to more of this content!
00:01 VQ-VAE is a crucial model for AI research and used in various novel works. 02:02 Variational autoencoders use a stochastic bottleneck layer. 05:58 VQ-VAEs impose structure into the latent space for continuous and meaningful interpolation. 07:50 Discrete representations are a natural fit for many modalities and enable complex reasoning and predictive learning. 11:40 Using l2 norm to find closest vector and approximate posterior 13:33 The likelihood assumption and the loss terms in VQ-VAEs 17:18 Conversion of bchw tensor to standard representation 18:59 VQ-VAEs use flat input and an embedding table to find distance to codebook vectors. 22:27 Explanation of implementing straight through gradient 24:09 The approximate posterior z given x is a deterministic function. 27:37 The model is an autoregressive token predictor for generating novel images. 29:11 VQ-VAEs compress data to a discrete space with code size k=512. 32:24 VQ-VAE v2 has hierarchical structure for better reconstructions 34:04 VQ-VAEs capture high-resolution images with some distortion Crafted by Merlin AI.
One thing that confused me is -> why do they convert BCHW to BHWC and then combine BHW x C => (16K, 64)? Should the quantization be done per image in the batch? It seems the entire batch is merged and quantized instead.
@TheAIEpiphany Man, that's such an epic explanation. Thank you so much for your help! One thing that I am struggling with, is 28:00 - by tweaking the prior, does that mean that we can trick the model about what "was" in the image? (what is is expected). The concept of predicting the next token is easy for me, - but what are we predicting? a next discrete-embedding vector from the table? But these vectors weren't guaranteed to be in any order... Or are we predicting the next word? In that case, how do we associate word token to the discrete-embedding vector? During teaching this autoregressive model, how do we know which one is the target/correct vector, that we want to be predicted?
If anyone else has this question, the autoregressive model is an addition which doesn't "improve" the quality of the VQ-VAE. But, we can swap it instead of the encoder+codebook, and use it to produce new images. So basically, "autoregressiveModel+decoder". You have to remember that once VQ-VAE is learned, the codebook vectors will be frozen forever. They will not be shuffled etc. So, when deployed into production, the Autoregressive model doesn't care what encoder does. Instead, the autoregressive model has learnt to look at the few code-book indices (we pick them arbitrarily), and to generate remaining indices of codebook that it thinks will be relevant. For example, if we gave it and index describing sky, it might decide that a following index describing a cloud will be more likely, than, say, of a fish. Once the autoregressive model produced all the needed indices, we feed the chosen codebook-vectors into the decoder. This allows us to generate images.
Thanks a lot, it was an awesome explanation. And yes the code part is necessary as far as I think, and would highly recommend that. Moreover, it would be great if you can also make some content regarding these distributions, because I have tried to understand them, but still, they sound quite fuzzy to me. Thanks again!
You’re smashing it. Take some pauses. Pacing conveys a lot / gives space to digest content. Consider you want to cause people to have a light bulb moment. You can’t give people the answer so quickly. I’m looking forward to pytorch stuff. Maybe do some meditation before you record / stillness. Pause.
Hi, What's the application you are using to write on the PDF? i mean the way you write something in side with the original pdf in the black side of the pdf?
Thanks for your amazing&simple explanation. It realy helpful. In some paper based on VQVAE, they use perplexity for measurement. But i can not understand what perplexity means in VQVAE model. So if you are not busy can i request explain 'what perplexity means in VQVAE?' Thanks again for your wonderful explain!
Thank you so much for the explanation! I wanted to ask how you get to understand some of the details that are not mentioned in the paper, like how the KL Div ends up being equal to log K?
Overall great explanation. One thing I find confused though. In the paper, loss2 and loss3 are something between Codebook(embeddings vector) and the encoding after CNN. However, in the code, it is something between Quantized encoding after CNN and the encoding after CNN. Can you explain why they are the same thing?
thank you for your explaining and code. When run the code there is an ERROR: Could not open requirements file: [Errno 2] No such file or directory: 'requirements.txt'. it seems there is no requirement.txt file there
This is a good explanation of VQVAE. I do have a question though. OpenAI's Jukebox is based on VQVAE and they pass gradients through the latent space in their loss function. So is there any difference or what do you think is going on?
I have been thinking about the VQ-VAE for generating music and it seems to me that one large limitation of quantizing your latent vectors is that you lose the ability to see interesting results that lay between clusters of latent vectors. For example, I train my model on both reggae and death metal songs and the resulting latent space shows two clusters. It would be nice to then hear songs that interpolate between the 2 clusters but it seems that the quantizing step will force any new vectors (our desired hybrid) to adopt the established codebook vectors which are only representative of the "pure" songs. Am I correct in this line of thinking? Has anyone seen any more info on this at all?
Hello, great video! I had a question regarding the token prediction training. Can this be used to generate images from a text description? If so, where in the code is this implemented? I'm having trouble understanding this last part
Combining the code with the paper explanation helps the understanding immensely. Allows for concept as well as application. Thank you
I wish I had one of these videos for every paper I read, awesome work
Glad to hear that man, thanks!
Great explanation! Especially useful explanation of the code! Please keeping doing the code part! You are a life saver!
Super valuable thanks! I'll consider maybe doing a walk-through of some code feel free to suggest something!
@@TheAIEpiphany Id be really interested in anything related to the autoregressive model part also mentioned in this video, maybe something like training a transformer?
I truly appreciate your explanations, especially PyTorch implementation part, which reduce the gap between concepts and real world implementations. Finding this channel is like finding treasures to me, I've recommended this channel to all my friends. Look forward to your weekly update, thanks :)
Thanks man! 🙏 Yup I am getting back on track with CZcams I had a weird period over the last month. 😄
Code part is pretty good.It has made paper more clear.
Awesome thanks for that feedback man!
This was some nice explanation
Thanks for explaining it very clearly. Code explanation makes the concept more robust.
Appreciate your work! Both paper and code parts are very helpful.
Two suggestion to make the code more concise
- pytorch has built in function to calculate pairwise distance `torch.cdist`.
- directly using `index_select` to get the quantized matrix may be more convenient.
Not my implementation - I agree why not reuse the existing library code when possible
Your suggestions are really neat 👌
Thank you for such a great explanation, adding code into this format is really helpful to digest the concepts more intuitively. Please keep them coming the same way.
Great! Also the code section was a very good idea. Saved a lot of time thanks.
Brilliant, never got so close to understand what's going on. Really well done
Glad to hear that man!
Thanks for the amazing video... You can make them longer and more detailed if needed... Really fun to watch
Amazing explanation! Thank you very much. I was a bit troubled about understanding how this model can be used to generate new images but after reading around I think I get it now
Loved the explanation, especially the part where you covered all the important aspects and showed them in the code. Subscribed and looking forward to more of this content!
00:01 VQ-VAE is a crucial model for AI research and used in various novel works.
02:02 Variational autoencoders use a stochastic bottleneck layer.
05:58 VQ-VAEs impose structure into the latent space for continuous and meaningful interpolation.
07:50 Discrete representations are a natural fit for many modalities and enable complex reasoning and predictive learning.
11:40 Using l2 norm to find closest vector and approximate posterior
13:33 The likelihood assumption and the loss terms in VQ-VAEs
17:18 Conversion of bchw tensor to standard representation
18:59 VQ-VAEs use flat input and an embedding table to find distance to codebook vectors.
22:27 Explanation of implementing straight through gradient
24:09 The approximate posterior z given x is a deterministic function.
27:37 The model is an autoregressive token predictor for generating novel images.
29:11 VQ-VAEs compress data to a discrete space with code size k=512.
32:24 VQ-VAE v2 has hierarchical structure for better reconstructions
34:04 VQ-VAEs capture high-resolution images with some distortion
Crafted by Merlin AI.
Finally a good explanation on how the autoregressive prior part works :X
You're a great teacher! Glad you came back to this paper and love the format with the code walkthroughs. Very thorough!
best explanation ever, unbiased comment
Instead of argmin on the distance to the closest embedding, couldn't we just use a softmax instead?
Very nice code part. Truly helped me to understand, what is happening
awesome presentation, thanks.
Fantastic in every way, including the code explanation as well!
Thank you. Love the code, love the in depth explanation! Explaining the math is also great for a beginner like me.
Thank a lot sir.
Simple and concise explanation by covering the related basics also.
the distance looks like (a - b)^2 19:30
Great choice of the article, thank you, was very interesting!
Thanks man!
Great, we also need VQ-GAN, TransGAN and GANsformer
VQ-GAN coming soon as well as DALL-E. I'll add the other 2 to my list. 😂 Thanks!
Can you also do clip+ vqgan
@@varunsai9736 Sure I'll see whether I can cram it into VQGAN video
Great explanation!!! Thank you!
One thing that confused me is -> why do they convert BCHW to BHWC and then combine BHW x C => (16K, 64)? Should the quantization be done per image in the batch? It seems the entire batch is merged and quantized instead.
@TheAIEpiphany Man, that's such an epic explanation. Thank you so much for your help!
One thing that I am struggling with, is 28:00 - by tweaking the prior, does that mean that we can trick the model about what "was" in the image? (what is is expected).
The concept of predicting the next token is easy for me, - but what are we predicting? a next discrete-embedding vector from the table? But these vectors weren't guaranteed to be in any order...
Or are we predicting the next word? In that case, how do we associate word token to the discrete-embedding vector?
During teaching this autoregressive model, how do we know which one is the target/correct vector, that we want to be predicted?
If anyone else has this question, the autoregressive model is an addition which doesn't "improve" the quality of the VQ-VAE.
But, we can swap it instead of the encoder+codebook, and use it to produce new images. So basically, "autoregressiveModel+decoder".
You have to remember that once VQ-VAE is learned, the codebook vectors will be frozen forever. They will not be shuffled etc.
So, when deployed into production, the Autoregressive model doesn't care what encoder does.
Instead, the autoregressive model has learnt to look at the few code-book indices (we pick them arbitrarily), and to generate remaining indices of codebook that it thinks will be relevant.
For example, if we gave it and index describing sky, it might decide that a following index describing a cloud will be more likely, than, say, of a fish.
Once the autoregressive model produced all the needed indices, we feed the chosen codebook-vectors into the decoder.
This allows us to generate images.
Thanks a lot, it was an awesome explanation.
And yes the code part is necessary as far as I think, and would highly recommend that.
Moreover, it would be great if you can also make some content regarding these distributions, because I have tried to understand them, but still, they sound quite fuzzy to me.
Thanks again!
Nice video. Please do more videos like this. 👍🏻
This video is invaluable. Thank you
much helpful!
You mentioned posterior and prior, can you provide some reference, why they model it in this way ?
THANK YOU BROTHER AMAZING
thank you a lot. i like the format with code
Love the Pytorch code!
thank you very much for this explanation.
I would like to know how the creation of the codebook is going
You’re smashing it. Take some pauses. Pacing conveys a lot / gives space to digest content. Consider you want to cause people to have a light bulb moment. You can’t give people the answer so quickly. I’m looking forward to pytorch stuff. Maybe do some meditation before you record / stillness. Pause.
Thanks for the feedback! I agree I need to work on me being less hectic haha I guess.
Hi,
What's the application you are using to write on the PDF? i mean the way you write something in side with the original pdf in the black side of the pdf?
Thanks for your amazing&simple explanation. It realy helpful.
In some paper based on VQVAE, they use perplexity for measurement. But i can not understand what perplexity means in VQVAE model. So if you are not busy can i request explain 'what perplexity means in VQVAE?'
Thanks again for your wonderful explain!
A great video! Thanks for sharing!
Thanks Tristan!
Thank you so much for the explanation! I wanted to ask how you get to understand some of the details that are not mentioned in the paper, like how the KL Div ends up being equal to log K?
🙏 Well, analyzing these I bring in my understanding and background from elsewhere to better understand what is going on in this particular paper.
Thank you
Overall great explanation. One thing I find confused though. In the paper, loss2 and loss3 are something between Codebook(embeddings vector) and the encoding after CNN. However, in the code, it is something between Quantized encoding after CNN and the encoding after CNN. Can you explain why they are the same thing?
Code is nice
Which software are you using for paper review? One side paper and you can draw and put code next to it.
OneNote
How do we generate new images from the VQ-VAE model. Can you do a tutorial on the pix2pix model for generating new image samples? Thanks
awesome
thank you for your explaining and code. When run the code there is an ERROR: Could not open requirements file: [Errno 2] No such file or directory: 'requirements.txt'. it seems there is no requirement.txt file there
cool video
Very good explanation. And with an implementation to support it.
Thanks a lot!
This is a good explanation of VQVAE. I do have a question though. OpenAI's Jukebox is based on VQVAE and they pass gradients through the latent space in their loss function. So is there any difference or what do you think is going on?
thanks!! it really likes me, very usefull!
I have been thinking about the VQ-VAE for generating music and it seems to me that one large limitation of quantizing your latent vectors is that you lose the ability to see interesting results that lay between clusters of latent vectors. For example, I train my model on both reggae and death metal songs and the resulting latent space shows two clusters. It would be nice to then hear songs that interpolate between the 2 clusters but it seems that the quantizing step will force any new vectors (our desired hybrid) to adopt the established codebook vectors which are only representative of the "pure" songs. Am I correct in this line of thinking? Has anyone seen any more info on this at all?
Hello, great video! I had a question regarding the token prediction training. Can this be used to generate images from a text description? If so, where in the code is this implemented? I'm having trouble understanding this last part
Hi, thank you for your work. Can you explain how they incorporate pixelcnn (or wavenet)?
+1 on code part
i love you