as per my understanding and from the LayerNorm code in pytorch, in NLP for an input of size [N, T, Embed], statistics are computed using only the Embed dim, and layer norm is applied to each token in each batch. But for vision with an input of size [N,C,H,W], statistics are computed using the [C,H,W] dimentions
Thank you so much for providing solid examples and calculations to explain these concepts. I have seen these concepts elsewhere but couldn't make sense of them until I saw how you computed the values. Great video!
Another great video - I like the structure you use of summarising the concept and then diving into the implementation. The code really helps bring it together as some others have commented. I look forward to seeing more of this series and would love to see a longer video of you deploying a transformer on some dummy data (perhaps you already have one - still going through the content)!
Thanks so much on commenting on all the videos! I really appreciate it. And yea, going to be introducing the code behind the encoder and decoder in the coming sessions!
Great! Since you’re covering transformer components, I would love to see TransformerXL and RelativePositionalEmbedding concepts explained in the upcoming videos! ☺️
Nice series on transformer. Really liked it. Btw interesting design choice for the video to use a landscape layout of the transform architecture during intro :D
if our batch has 2 vectors of 2 words x 3 embedding size say: [[1,2,3],[4,5,6]] and [[1,3,5], [2,4,6]] For layer normalization, Is nu_1 = mean(1,2,3,1,3,5) and nu_2 = mean(4,5,6,2,4,6) Just wanted to clarify. Keep up the great work, brother. Like the small bite sized videos.
meaning , the statistical calculation for first vectors of all batches are done together? like how you calculating mean using the first vectors of both the batches?
Great video! I find these very informative, so please keep them going! Question on the output dimensions though. In your transformer overview video the big diagram shows that after the layer normalization, you have a matrix of shape [batch_size, sequence_len, dmodel] (in the video 30x50x512 I believe.) However here you end up with an output matrix (out) of [sequence_len, batch_size, dmodel] (5x3x8). Do we need to reshape these output matrices again to [batch_size,sequence_len,dmodel], or am I missing something? Thanks again for all the informative content!
10:30 when layers normalizations are "computed across the layer and also the batch", are their means and std connected as if the batch boundaries aren't there? So that means there's a different learnable gamma and beta parameter for each word?
“Means and std connected as if the batch boundary isn’t there” ~ yes But we have the same gammas and betas for the same layer across ALL words in the dataset
Where can I find the reason why we need to calculate mean and standard deviation over the parameter shapes? In Pytorch, they just caculate over the last dimension, hidden size.
I am actually not sure why it is okay to reshape your input from (B, S, E) to (S, B, E) @ 10:10. It doesn't matter in this case as B=1, but in general wouldn't you be changing the your data? In particular, you would change the number of words in each batch, i.e., changing the maximum sentence length.
Have an actual question this time! While trying to understand the differences between layer and batch normalization, I was wondering whether it’s also accurate to say you are normalising across the features of a vector when normalising the activation function - since each layer is a matrix multiply across all features of a row, would normalising across activation functions be similar to normalising across the features? In the same thread, can/should layer and batch normalization be run concurrently? If not, are there reasons to choose one over the other?
Good questions. From what I understand, we normalize the activation values across the layer (I.e make sure the values across the layer follow a bell curve of sorts). While in batch normalization, we do the same exact thing but across the batch dimension instead The only issue i see with batch normalization is that it is dependent on the batch size (which is typically an order magnitude smaller than the size of layers) If we apply a normalization to a small number of items, we might get erratic results (that is the mean and standard deviation of a small number of numbers simply don’t lead to values that are truly “normalized”). One remedy would be to increase your batch size to a sizable number (like in the hundreds). I have heard this is an issue with NLP problems specifically. But I would need to do my own experimentation to see why.
Great Video !! Just a question, why do we need to swap the dimensions and all that other stuff ? Why can't we do something like this # assuming inputs is of format (batch_size,sentence_length,embedding_dim) mean=inputs.mean(dim=-1,keepdim=True) var = ((inputs - mean) ** 2).mean(dim=-1, keepdim=True) epsilon = 1e-5 std = (var + epsilon).sqrt() y = (inputs - mean) / std y
I think your code is not correct, LayerNorm will not be over the batch. Try and think what it means to take normalisation over the batches. For a layer norm each entity of the batch is unique, and each should be normalised across the layer of the MLP (essentially everything is MLP).
In theory, that’s what’s happening. In code, we make “8 parallel heads” by introducing another dimension. For example, vectors of shape 30 (max sequence length) x 10 (batch size) x 512 (embedding size) would form query key and value tensors of shape 30 x 10 x 8 x 64. So there is a new dimension for heads that essentially acts like another batch dimension for more parallel computation
I wouldn’t necessarily say that is the case. Tensor flow and PyTorch are frameworks we can use for building these complex models. PyTorch might be easier to use since you don’t need to code out tensors themselves. But tensorflow (or even going as low as numpy) can be used to train models too
the diagram is GOLD
as per my understanding and from the LayerNorm code in pytorch, in NLP for an input of size [N, T, Embed], statistics are computed using only the Embed dim, and layer norm is applied to each token in each batch. But for vision with an input of size [N,C,H,W], statistics are computed using the [C,H,W] dimentions
Thank you so much for providing solid examples and calculations to explain these concepts. I have seen these concepts elsewhere but couldn't make sense of them until I saw how you computed the values. Great video!
I really like your voice and delivery. It’s quite reassuring, which is nice when the subject of the videos can be pretty complicated.
Thanks so much! Very glad you liked this. There will be more to come
Another great video - I like the structure you use of summarising the concept and then diving into the implementation. The code really helps bring it together as some others have commented. I look forward to seeing more of this series and would love to see a longer video of you deploying a transformer on some dummy data (perhaps you already have one - still going through the content)!
Thanks so much on commenting on all the videos! I really appreciate it. And yea, going to be introducing the code behind the encoder and decoder in the coming sessions!
@@CodeEmporium I love your content so will do what I can to comment , like and spread!
Great! Since you’re covering transformer components, I would love to see TransformerXL and RelativePositionalEmbedding concepts explained in the upcoming videos! ☺️
i help with relpos, its the same as usual posenc but instead of fixed positional encoding its in relation to each other, pairwise.
very clear and sound explanation for a complex concept, thumb up for the hard works!
Nice series on transformer. Really liked it. Btw interesting design choice for the video to use a landscape layout of the transform architecture during intro :D
the python example really helped to solidify understanding.
Super glad it did
if our batch has 2 vectors of 2 words x 3 embedding size say:
[[1,2,3],[4,5,6]] and [[1,3,5], [2,4,6]]
For layer normalization,
Is nu_1 = mean(1,2,3,1,3,5)
and nu_2 = mean(4,5,6,2,4,6)
Just wanted to clarify. Keep up the great work, brother. Like the small bite sized videos.
meaning , the statistical calculation for first vectors of all batches are done together? like how you calculating mean using the first vectors of both the batches?
EXCITED!!!!!
Me too :)
best explanation ever!
Thanks Man ,
you deserve more than a like
Thanks a ton for the kind words :)
Great video! I find these very informative, so please keep them going! Question on the output dimensions though. In your transformer overview video the big diagram shows that after the layer normalization, you have a matrix of shape [batch_size, sequence_len, dmodel] (in the video 30x50x512 I believe.) However here you end up with an output matrix (out) of [sequence_len, batch_size, dmodel] (5x3x8). Do we need to reshape these output matrices again to [batch_size,sequence_len,dmodel], or am I missing something? Thanks again for all the informative content!
I wonder the same. Do you know the reason for it?
10:30 when layers normalizations are "computed across the layer and also the batch", are their means and std connected as if the batch boundaries aren't there? So that means there's a different learnable gamma and beta parameter for each word?
“Means and std connected as if the batch boundary isn’t there” ~ yes
But we have the same gammas and betas for the same layer across ALL words in the dataset
appreciate!
Value Matrix should be on the right side of the multiplication with Attention Weights matrix
Amazing your expiation are so clear. Can this help with exploding gradients?
Thanks so much! And yea. Later normalization should help exploding and vanishing gradients. So training is stable
@@CodeEmporium Perfect thanks.
Great lectures.
Where can I find the reason why we need to calculate mean and standard deviation over the parameter shapes? In Pytorch, they just caculate over the last dimension, hidden size.
I' m sorry, I just check the Pytorch code, there is normalized_shape like parameter_shape in your code.
I am actually not sure why it is okay to reshape your input from (B, S, E) to (S, B, E) @ 10:10. It doesn't matter in this case as B=1, but in general wouldn't you be changing the your data? In particular, you would change the number of words in each batch, i.e., changing the maximum sentence length.
Thanks! This vedio helps me a lot
Glad!
Thank you very much friend, very good!!
Thanks so much for watching !
Have an actual question this time! While trying to understand the differences between layer and batch normalization, I was wondering whether it’s also accurate to say you are normalising across the features of a vector when normalising the activation function - since each layer is a matrix multiply across all features of a row, would normalising across activation functions be similar to normalising across the features?
In the same thread, can/should layer and batch normalization be run concurrently? If not, are there reasons to choose one over the other?
Good questions. From what I understand, we normalize the activation values across the layer (I.e make sure the values across the layer follow a bell curve of sorts). While in batch normalization, we do the same exact thing but across the batch dimension instead
The only issue i see with batch normalization is that it is dependent on the batch size (which is typically an order magnitude smaller than the size of layers) If we apply a normalization to a small number of items, we might get erratic results (that is the mean and standard deviation of a small number of numbers simply don’t lead to values that are truly “normalized”). One remedy would be to increase your batch size to a sizable number (like in the hundreds). I have heard this is an issue with NLP problems specifically. But I would need to do my own experimentation to see why.
Beautifully done video but isn't layer normalization essentially batch normalization layer?
Wow. Amazing channel
Thanks so much!
Great Video !! Just a question, why do we need to swap the dimensions and all that other stuff ? Why can't we do something like this
# assuming inputs is of format (batch_size,sentence_length,embedding_dim)
mean=inputs.mean(dim=-1,keepdim=True)
var = ((inputs - mean) ** 2).mean(dim=-1, keepdim=True)
epsilon = 1e-5
std = (var + epsilon).sqrt()
y = (inputs - mean) / std
y
I have the same doubt . did you get it ?
I think your code is not correct, LayerNorm will not be over the batch. Try and think what it means to take normalisation over the batches. For a layer norm each entity of the batch is unique, and each should be normalised across the layer of the MLP (essentially everything is MLP).
Given that we have 8 heads, Is it 512 / 8, which is 64? Are we actually going to split the 512 into 8 equal 64 length parts?
In theory, that’s what’s happening. In code, we make “8 parallel heads” by introducing another dimension. For example, vectors of shape 30 (max sequence length) x 10 (batch size) x 512 (embedding size) would form query key and value tensors of shape 30 x 10 x 8 x 64. So there is a new dimension for heads that essentially acts like another batch dimension for more parallel computation
🔥🔥🔥
:) thank you
Is pytorch better for NLP task than Tensorflow?
I wouldn’t necessarily say that is the case. Tensor flow and PyTorch are frameworks we can use for building these complex models. PyTorch might be easier to use since you don’t need to code out tensors themselves. But tensorflow (or even going as low as numpy) can be used to train models too
@@CodeEmporium Thanks for clarifying
Really want to hear indian accent of thai guy 😅😂