Weight Initialization explained | A way to reduce the vanishing gradient problem
VloĆŸit
- Äas pĆidĂĄn 9. 07. 2024
- Let's talk about how the weights in an artificial neural network are initialized, how this initialization affects the training process, and what YOU can do about it!
To kick off our discussion on weight initialization, we're first going to discuss how these weights are initialized, and how these initialized values might negatively affect the training process. We'll see that these randomly initialized weights actually contribute to the vanishing and exploding gradient problem we covered in the last video.
With this in mind, we'll then explore what we can do to influence how this initialization occurs. We'll see how Xavier initialization (also called Glorot initialization) can help combat this problem. Then, we'll see how we can specify how the weights for a given model are initialized in code using the kernel_initializer parameter for a given layer in Keras.
Reference to original paper by Xavier Glorot and Yoshua Bengio:
proceedings.mlr.press/v9/gloro...
đđŠ VIDEO SECTIONS đŠđ
00:00 Welcome to DEEPLIZARD - Go to deeplizard.com for learning resources
00:30 Help deeplizard add video timestamps - See example in the description
09:42 Collective Intelligence and the DEEPLIZARD HIVEMIND
đ„đŠ DEEPLIZARD COMMUNITY RESOURCES đŠđ„
đ Hey, we're Chris and Mandy, the creators of deeplizard!
đ Check out the website for more learning material:
đ deeplizard.com
đ» ENROLL TO GET DOWNLOAD ACCESS TO CODE FILES
đ deeplizard.com/resources
đ§ Support collective intelligence, join the deeplizard hivemind:
đ deeplizard.com/hivemind
đ§ Use code DEEPLIZARD at checkout to receive 15% off your first Neurohacker order
đ Use your receipt from Neurohacker to get a discount on deeplizard courses
đ neurohacker.com/shop?rfsn=648...
đ CHECK OUT OUR VLOG:
đ / deeplizardvlog
â€ïžđŠ Special thanks to the following polymaths of the deeplizard hivemind:
Tammy
Mano Prime
Ling Li
đ Boost collective intelligence by sharing this video on social media!
đ Follow deeplizard:
Our vlog: / deeplizardvlog
Facebook: / deeplizard
Instagram: / deeplizard
Twitter: / deeplizard
Patreon: / deeplizard
CZcams: / deeplizard
đ Deep Learning with deeplizard:
Deep Learning Dictionary - deeplizard.com/course/ddcpailzrd
Deep Learning Fundamentals - deeplizard.com/course/dlcpailzrd
Learn TensorFlow - deeplizard.com/course/tfcpailzrd
Learn PyTorch - deeplizard.com/course/ptcpailzrd
Natural Language Processing - deeplizard.com/course/txtcpai...
Reinforcement Learning - deeplizard.com/course/rlcpailzrd
Generative Adversarial Networks - deeplizard.com/course/gacpailzrd
đ Other Courses:
DL Fundamentals Classic - deeplizard.com/learn/video/gZ...
Deep Learning Deployment - deeplizard.com/learn/video/SI...
Data Science - deeplizard.com/learn/video/d1...
Trading - deeplizard.com/learn/video/Zp...
đ Check out products deeplizard recommends on Amazon:
đ amazon.com/shop/deeplizard
đ” deeplizard uses music by Kevin MacLeod
đ / @incompetech_kmac
â€ïž Please use the knowledge gained from deeplizard content for good, not evil.
Machine Learning / Deep Learning Tutorials for Programmers playlist:
czcams.com/play/PLZbbT5o_s2xq7LwI2y8_QtvuXZedL6tQU.html
Keras Machine Learning / Deep Learning Tutorial playlist:
czcams.com/play/PLZbbT5o_s2xrwRnXk_yCPtnqqo4_u2YGL.html
Data Science for Programming Beginners playlsit:
czcams.com/play/PLZbbT5o_s2xo_SRS9wn9OSs_kzA9Jfz8k.html
your explanation is great -- can you please make videos using pytorch in your channel?
why are the random weights normally distributed with mean of 0 what is the intuition behind it
God sent you to help machine learning learners.
This series is A hidden gem ! You deserve more views !! Thank you
Just amazing!
I'll always be thankful to you for providing us these astounding videos!
You have put together a great, concise accessible series for the uninitiated in the field of Deep Machine Learning. Thank you!
This video is perfect, THANK YOU SO MUCH!
have never come across a video which explains this concept so well. awesome!
This playlist is really good. Grateful to you for your effort! :)
Thank you very much for this video! I really enjoyed learning about initialization and how it connects with everything else! Great that it can be used to tackle the vanishing/exploding gradient problem!
Thank you a lot for the detailed explanation!
Amazing video as always! Thank you for your contribution to the machine learning community, it's very valuable and we learn a lot from you.
Some explanatory remarks on the equation var(weights) = 2 / (n_in + n_out):
The mentioned var(weights) = 1/n (or = 2/n when using ReLU) turned out to be good for the forward pass of the input (magnitude of the activations are kept approximately constant). However, with this the problem of vanishing gradients still exists during backpropagation. From the perspective of backpropagation it would be ideal to have var(weights) = 1 / n_{nextLayer}. Thus, in an article by Glorot and Bengio (2010) a compromise was found: var(weights) = 2 / (n + n_{nextLayer}), whereas n / n_{nextLayer} is the number of neurons in the layer before / after a considered neuron.
This is from the lecture I've attended on deep neural networks.
overall, which initalization technique must we use? formula?
@@shreyajain1224 I'm not sure anymore. But if I remember correctly you initialize based on the normal distribution, whereas you specify the SD of the normal distribution as explained in my comment above (you of course need to transform that variance into an SD).
Maybe @deeplizard can help?
WOOW I love the new style! PLEASE add more videos!
your way of explanation is so clear.
Thanks.
KEEP IT UP!
Thank you! I'm glad you're liking the new style. đ
New videos are coming soon. Hoping to release the next one tonight!
And here we go!
czcams.com/video/HetFihsXSys/video.html
Thank you for the video, very clear and easy to understand
Awesome work. Keep doing it !!
Never stop teaching us !
Your angel to me thanks for saving my time and also for reducing my stress levels to understand the concepts
Thank you very much! Now I understand better what goes on with Xavier initialization!
This series is really awesome. Thanks a lot for the detailed explanations. This certainly took a lot of effort so I ( and I guess everyone else who is watching and subscribing to this) highly appreciate that. Could you also cover the other layers in keras e.g. Embedding, Timedistributed,...? :)
Appreciate that, Daniel! You're welcome.
Thanks for the suggestions. I'll make sure these items are included on my list of potential topics for future videos!
Great. Thanks a lot!
This youtube Channel is a blessing from god, if there is one :D. By sharing your knowledge in such a easy way, you are seriously doing so much GOOD. Thank you
The explanation given is great. I was expecting things much in deep but that's fine. Now I got the clarity on what I need to dissect and I will definitely explore the content in this channel. Love and respect from India. Keep up the good work :)
The explanation is so perfect and clear that there are no dislikes in the video!!. Loved your voice and way of explanation :)
Thanks, ravi!
very clear explanation :D thanks!
You are literally saving my life :)
Great videos ! Thanks !
I love your voice, it's so soothing to listen to.
clean and simple explanation thanks
You are my new hero
Thank you very much , this is such a clear explanation , it is very helfeul .
Nicely explained
Thanks much for the video
0:22 intro
0:54 how weights matter
2:48 bad weights can cause vanishing gradient
4:44 heuristic for initial weight
7:24 keras code
Great video :-) I have thoroughly enjoyed it, and I recommend it in my circle to follow your series. Out of curiosity, does vanishing gradient also depend on the activation function you choose, and if yes, what are activation functions where this issue happens less? Any suggestions based on your experience! Many thanks.
This is extremely useful
What I think is missing is the mathematical demonstration
thank you mam ! you are awesome
Loved thisâ€ïž
Please make a video on how ResNet helps in solving the problem of vanishing and exploding gradients??????
thank you so much for this wonderful explanation. What is the default weight initialization in PyTorch?
Before this insightful vid, I didn't know Keras silently doing so much for us without us even knowing. :D (y)
don't get this..."if the desired output for our activation function is on the opposite side from where it's saturated, then during training when SGD updated the weights and attempts to influence the activation output, it will only make very small changes in the value of this activation output barely even moving it in the right direction."
I guess it means that for some odd reason, your random initialisation of weights (while being completely random) would always trigger an activation 1, instead of 0 (In this case, 0 is what is needed) - it would mean that the network would never learn.
In very layman terms, the random weights give rise to a standard deviation which always makes the activation fire a neuron incorrectly that what *Learning* desires.
Thank you for this amazing content. However, I'm having trouble understand how the fact that Z can take on values significantly higher than 1 is going to make SGD very slow. I guess I'm not following this statement - "if the desired output for our activation function is on the opposite side from where it's saturated then during training when SGD updates the weights and ...". Could you clarify a little bit more on this? Thanks a lot!
But variance = sum( (x-x_mean) ^2 ) / N so it won't be 250 but 1
And also idk var(z) = 250 it seems to be that x != 1, more like x= 1* (random number between - 1 and 1)
Mind boggling, spellbound, one of the best Computer Science(not just AI/ML) channel on CZcams, hands down. I have watched your videos on Deep Reinforcement Learning and Conv Nets as well, they both are a masterpiece as well(although I wish you continued your series on Deep Reinforcement Learning, teaching other topics like Actor Critique and more stuff). And on top of that, you are funny as hell.
Thanks for these videos.
Spellbound... I like it! Thank you, Sourabh! Glad to hear you're finding value in all of the content and that you get my humor too :D Note that the RL series is still being developed with more content to come!
thanks for the video, I have a question. when we multiply weights by 1/sqrt(250), the variance problem of the activation function will be solved, but, the weights have the mean around zero and standard deviation around 1/sqrt(250), so weights are smaller than one and vanishing problem can occur for the first layers derivatives, can you help me to understand this problem?
If i initialize
the weight is it mean that every time the confusion matrix is same?
I faced a problem regarding confusion matrix as when i compile the my classification code for credit approval data set, every time it showed different confusion matrix. I fixed the randomization in the training and test set splitting.
glorious
How do we change the values of these weights and bias on each layer?
Hi! I have been learning a lot from your awesome video explanations. so first of all, thank you for that.
Secondly, I have a question about input_shape. At 7:30, the parameters for the input_shape = (1,5) means that input layer has 2 dimensions with 1 and 5 elements respectively. I am still not clear about that.
Thanks in advance.
Can you please explain why there is 2 instead of 1 in relu activation function?
why are the random weights normally distributed with mean of 0 what is the intuition behind it
in input_shape(1,5) what does 1 stands for and 5 for ?
very nice video as always! just have a quick question here, if we use the glorot initializer in keras, would it adjust to the activation function we are using? For instance, 2/n or 1/n for relu or sigmoid respectively.
Hey Ziqiang - According to their docs, it appears they've implemented glorot initialization to always use the 2 / (n_in + n_out) method mentioned at 6:38. keras.io/initializers/#glorot_normal
i guess if we take the weight from a different distribution, the classical CLT will come handy to retrieve the same result as in the Xavier initialization...o am i wrong?
Love your videos! What are some "symptoms" that indicate your model is being affected by vanishing gradients? Is there a way to determine this, aside from overall accuracy scores?
when vanishing gradients occur, one symptom is that the weights in the first layer get less, or no, update compared to the weights in the later (deeper) layers.
Thanks for awesome video.
I have couple of doubts.
1. Suppose I am training a MLP and I have data that I divided into batches b_1,b_2...b_n.
Lets say I do Xavier weights initialization. Now after I do back propagation over batch b_1 or after few batches
We update the weights , which is in some way proportional to gradients. There is a good
chance that this might distort the weights distribution (i.e not Xavier any more) . How do you guarantee the learning takes place after one/few back propagations?
2. To avoid vanishing/Exploding our weights needs to be around 1 (i.e 0.99 or 1.001) some thing like this. In any stage of the learning this has to be the case. My question is if the above statement is true then arent we restricting the weights and is learning actually taking place ?
I wonder can we say accumulated variance can contribute to vanishing gradient when we have ReLu as the activation?
Also can it contribute to the exloding gradient?
Whats the process when backpropagating?
This new intro theme music is much better.
Hi, great videos! I have a recommendation: Would be great if you could take us through Gradient Boosting in detail, just the way you have done with ANNs. Looking forward
Hello! You kicked ass!
Question - 3:44 "If the desired output for our activation function is on the opposite side from where it saturated, then during training when SGD updates the weights in an attempt to influence the activation output, it will only make very small changes in the value of this activation output."
What does this mean? how is it related ?
I also have a similar question, what does saturation mean here. A little guidance will be really helpful.
Thank you for creating such awesome videos.
The slope of the sigmoid is very small when its input is either very large or very small. So its derivative would be very small and it will contribute to the vanishing gradient.
But now in general ReLu is used far more often than sigmoid and this is not the case with ReLu.
3:45 i believe theres an error on the sigmoid function? Shouldnt it be: fn(x) = 1/1+e^-x ?
The two expressions are equal to each other :)
3:06,Please explain how we got 250 as variance
yes please
In the Glorot initialization I don't understand why this condition should be validated to avoid vanishing/exploding gradient problem:
"we need the gradients to have equal variance before and after flowing through a layer in the reverse direction"
Thank you
I think it's because if variance increases, the gradient would have to explode to keep up and shrink if variance decreases
by saying variance of random numbers you are actually implying variance of random variables right?
I have one more query input_shape=(1,5) means are there 1X5= 5 input nodes?
Hey Ayan - Yes to both questions.
deeplizard why are the random weights normally distributed with mean of 0 what is the intuition behind it
what about "Bios" i dont get it ,,, you didnt talk about bios and im so confused ,,, is it same for tuning bios as well?
bias
@@im-Anarchy helpful đ«€
I donÂŽt get how var(z) = 250, some help please?
thank you for the videos, I've been watchinh this list the last week and then I'll watch the on about Pytorch.
thank you again for this great work, you are helping a lot of students
Even I am not clear on this part. Can someone help us understand this.
She said that in this particular case we are considering we have 250 nodes in the input layer, all with a value of 1. If this is the case, then z is the sum of all the weights. The weights are normally distributed numbers with variance 1. The variance of the sum of normally distributed numbers is the sum of the variances. Since the variance of each weight is 1, the variance of z is then 1+1+1+... as many times as there are weights. Since there are 250 nodes in the input layer, there are 250 weights and hence the variance is 1+1+1+1+...=250.
@@luismisanmartin98 variance of each weight that is a single number should be zero right?
@@skadoosh7398 what do you mean? All weights are single numbers taken from a distribution with a certain mean and variance.
you had written that the variance of each weight is 1. By the term 'each weight', do you mean a single element of the weight matrix? for example: let the weight matrix be [1,-1], it has mean 0 and variance 1. So do you mean that variance of each weight is 1 i.e var(1)=1 and var(-1)=1. I think I am missing something important. Please clarify
Audio quality is not great. My laptop's speaker volume and the CZcams volume adjuster are at the highest level, but I can barely hear the sound (to be fair there are some background noise at my place, but still the audio should be way louder)
Thanks again for the feedback! We've been experimenting with the audio. Still tuning it trying to get it right. Check out one of the recently released videos, like the one below, and I'd appreciate your feedback on the audio quality on that one. Also, out of curiosity, are you listening through headphones? czcams.com/video/HEQDRWMK6yY/video.html
That video seems better, but could be louder. I'm not listening through headphones. Thanks
Thanks for checking it out and letting me know your thoughts!
what is the value of node ??
how will you define 'value of node' ??
why does node have any value ??
Start at the beginning of the course to have these points clarified:
deeplizard.com/learn/playlist/PLZbbT5o_s2xq7LwI2y8_QtvuXZedL6tQU
@@deeplizard yes, I have started from 1st lecture of this series. What I don't get it is, the 'value of node' is input of the 'input node' or activation output of 'input node' or kind of constant in activation function
The layer we focus on in this episode is the input layer, which consists only of the raw input data. In this case, we're saying the raw input data is a structure of length 250 with each element in that structure being of value 1.
@@deeplizard thank u đđfor your patience, but
i got all those things except this statement 'element in that structure being of value 1' i got that here element is input node and structure is the input layer but what i don't got is 'node of value 1'
how can an input node have any value ?? is that value '1' is assumed input value or assumed output value
what do you mean by 'input node of value 1' ??
The input value and output value of the input layer is the same. The input layer is made up only of the raw input data that is being passed to the network. In this case, we suppose we pass input data as a structure of length 250 with each element in that structure being of value 1. This is the input data. The data flows from the input layer to the next layer via connected weights. The input to an "input node" is the same as its output. It is simply the input data. This is not the case for subsequent layers, only for the input layer.
Wow.... new logo effect...
{
"question": "Can you identify a way \"Keras\" is helping to quickly train a network?
(Select the most appropriate answer)
",
"choices": [
"Keras defaults weight initializater attribute to 'glorot_uniform', which protects the variance from becoming larger.",
"Exploiting the GPUs.",
"Self adjusting the learning factor.",
"Keras automatically drops few neurons to reduce the load in training."
],
"answer": "Keras defaults weight initializater attribute to 'glorot_uniform', which protects the variance from becoming larger.",
"creator": "Anandachetan Elikapati",
"creationDate": "2020-08-19T12:01:45.585Z"
}
Thanks, Anandachetan! I changed the wording just a bit, but your question has now been added to deeplizard.com/learn/video/8krd5qKVw-Q :)
Hey, the content is dope! sincerely,, yet I would probably prefer less distracting background than the one you've used. Thx btw, thumbs up..
Agree :) We stopped using background noise during explanations in later episodes.
A start like a game loading, a beautiful woman's voice
4:40
Really awesome series, thank you. I must say your voice is very low in this particular episode. Please try speaking louder.
For a more or less normal, medium sized, WELL TRAINED neural network:
- What is the actual range (actual values for maximum and minimum) of the weights and the biases between input layer and first hidden layer, between last hidden layer and output layer, and between the hidden layers?
- What is the range of means and standard deviations for the three collections of values mentioned?
- Does it make sense to use these characteristics of weights and biases to set the random initial values for the weights and biases for the respective layers? Or has one always have to use the Xavier initialization for weights anyway, but not for biases, to allow for a sensible growth path towards the final values for weights and biases?
I truly would appreciate an answer to these questions, thank you very much!
0:31 - 0:52 -> Contents of the video.
Oh
So lets call a bunch of weights coming from ALL the neurons on the last layer into ONE neuron on the next, a SET
So its the SET that needs to have a variance of 1/N?
I thought the weights that needed this variance were the weights that came from ONE neuron to ALL the neurons of the next layer
Hey Reyner - Yes, it's the set of weights you described in your top sentence that needs to have variance of 1/N.
@@deeplizard thx for the enlightenment
Something you haven't explained so far in the series; in building the first layer, the parameter "input_shape" has not been properly explained. why is it having two parameters and what does each parameter mean?
I know it denotes our input data but how does the two parameter it takes works and when do we ignore one of them?
Thanks in advance
i write my own neural net so no initialization is done for me :D
Wow a video on machine learning in the middle of advertisements
A lizard's gotta eat!
deeplizard yeah I can totally understand, but 3 ads in a short video really hinders what you're trying to understand, especially if it's a complex topic. You can try to follow 3b1b: they don't show ads for the first month so that users can watch ad free for first month. Also try to use CZcams SEO tools to increase viewers. I see your channel has a lot of potential. Good luck :)
Hey Mayur - Thanks for the feedback and suggestions. Really appreciate it!
Speak louder please.
IF IT WAS POSSIBLE I WOULD MAKE YOU MY GIRL.
PEOPLE WANT MODELS
I WANT SOMEONE LIKE YOU WHO TEACHES YOU THOUGH LIFE