Tutorial 8- Exploding Gradient Problem in Neural Network
Vložit
- čas přidán 22. 07. 2019
- After completing this video, you will know:
What exploding gradients are and the problems they cause during training.
How to know whether you may have exploding gradients with your network model.
How you can fix the exploding gradient problem with your network
Below are the various playlist created on ML,Data Science and Deep Learning. Please subscribe and support the channel. Happy Learning!
Deep Learning Playlist: • Tutorial 1- Introducti...
Data Science Projects playlist: • Generative Adversarial...
NLP playlist: • Natural Language Proce...
Statistics Playlist: • Population vs Sample i...
Feature Engineering playlist: • Feature Engineering in...
Computer Vision playlist: • OpenCV Installation | ...
Data Science Interview Question playlist: • Complete Life Cycle of...
You can buy my book on Finance with Machine Learning and Deep Learning from the below url
amazon url: www.amazon.in/Hands-Python-Fi...
🙏🙏🙏🙏🙏🙏🙏🙏
YOU JUST NEED TO DO
3 THINGS to support my channel
LIKE
SHARE
&
SUBSCRIBE
TO MY CZcams CHANNEL
I never got such clear explanation for deep learning concepts.
I had Coursera deep learning. They make it more difficult to what it is.
Thank you Krish.
From now If anyone asked me about Vanishing Gradient Descent OR Exploring Gradient Descent I will not just answer and I even take a class to them
The best video I've ever seen
Exactly
i have a small doubt...in vanishing- the values where very small but here its high but both have the same eqn right? or is it due to the weights in the vanishing was normal and in exploding its high?....ur help is really appreciated
@@kiruthigakumar8557 even I have the same doubt if anyone can help it would be really appreciated
@@sargun_narula As he said when using sigmoid the values would be between 0-1 so if their weights are smaller when we initialise them but for a smaller network that is 1 or 2 hidden layer network vanishing won't be a problem but if it uses more like 10 layers then after some layers considering last 3rd layer when backpropagating the derivative will be decreasing with every layer and due to that optimizer will be so slow to reach the minima value and that's what vanishing gradient is. Talking about exploding gradient if weights are bigger and derivative increases after backpropagating than that may put our optimizer into diverging rather than reaching minima i.e exploding problem. Simply saying weights shouldn't be initialized so high or so low.
@@kiruthigakumar8557Irrespective of your activation function your weights causes the Exploding/Vanishing gradient descent problem. Weights shouldn't be initialized so high or so low. Here is the Andrew Ng video for the same czcams.com/video/qhXZsFVxGKo/video.html
Loving this playlist
Most of these abstract concepts are explained very elegantly
Thank you so much
This playlist is like a treasure.
Best explanation for EXPLODING gradient problem on the internet I have encountered so far. Awesome!
9:32 peak of interest! Happiness in explaining why it will not converge... I love that reaction!!!😍😍😍
yeah, the time where I smiled with respect :)
I really love your videos. Today only i started watching your tutorial. It was really helpful. Thank you so much for sharing your knowledge.
Deep Concepts are getting clear.
Thank you sir. Such a beautiful explanation
Sir, your videos are very educational and, you put a lot of energy into making them. They make the learning process easy, and it also lets me develop an interest in deep learning. That's the best I could have asked for and, you delivered it. Thank you, Sir.
Your classes are quite clear, thank you so much !!!!
how i missed the class all these years
how come you are able to simplify the topics.
👏
Congrats for a well explained topic. Now I know the effect of exploding gradients
Excellent Videos bro, I am getting clear picture on those concepts Thank you very much for making the video's with clear understandable manner.
I am follwing your every video.
Very well explained, and the writings and drawings are very clear too by the way
Love the explanation bro... I used to initialize weights randomly but after watching this, I came to know the impact of such initializations...
Exploding Gradient Problem is because of Higher Weights Initialization. If the weights are higher, then during BackProp gradients value will be higher which in turn affects the new weights to be vv small when updating weights [ Wnew = Wold - lr * Grad] Due to which the weight difference will be Varying a lot at every epoch and this is why Gradient Descent will never converge.
Pure passion,appriciate it.
Very passionate and articulate lecture well done
Best explanation so far. No doubt !!!
Awesome explanation! Best video I have seen for this problem.
Amazing explanation sir. I am going to learn whole deep learning from your videos only
That was one of the best explanations for Exploding gradient problem. But please mention the next video in the description box. I could find it hard.
YOU ARE JUST KIND DUDE. THANKS
Please keep making videos like this!
This is super krish, its like a story that you explain... at 9:35 minutes the whole picture jumps into your mind. neat explanation. Nice work krish... awaiting for more videos. meet you on satruday..till then cheers
hats off to you sir,Your explanation is top level, THnak you so much for guiding us...
love your video of machine learning algorithms, kudos
keep up the good work, disrupting the education system. Lots of love
Another Great Video. Namaste
best explaination... thanks for making this video
great work.. Kudos to u!!!!!!!!!!
Very very effective video sir 👍👍👍👍👍👍....my love and gratitude to you 🙏...
Very nice explanation.thanks
Excellent ..!!!
thanks Krish... nice explanations
awesome video, much respect
I love the energy
excellent and to the point explanation sir. Waiting for your future videos in Deep Learning.
Amazing In-Depth Explanation!
Best video. Hands down
Exploding GD explained nicely!
u doing great job man
super explanation sir !!
Awesome video!
You're great!
Do tutorial based on machine learning like regression ,classification and clustering sir
great video !
Excellent.
Awesome 😊👏👍
just excellent :-)
excellent krish
love to watch your videos
beautiful explanation
One correction: dL/dW'11 should be (dL/dO31. dO31/dO21. dO21/dO11. dO11/dW'11)
In tutorial 6 also there was a correction...!
is there an explanation
You are right @kueen, krish has missed out the first term in the chain rule.
yes you are right
but what will come in "dL" is that (y-Y) ^2 or log loss funtion will come in "dL"
just wanted to know... does the chain rule refer to partial derivative ??
Thanks a lot sir
Thank you.
Too good man !!! #BohotHard
Question:
Hi Krish. dO21/do11 is large because we mutliple the derivate of the sigmoid (btwn 0 to 0.25) with a large weight. However, in Tutorial 7 we didn't use this formula(chain rule derivation), we directly said dO21/do11 is between 0 to 0.25. Please can you clarify on this?
Even I have the same question, sir can you please explain this section?
even I have the same doubt.. can u explain this?
That is because O21 = sigmoid(ff21) and when we take the derivate of O21 with respect to any variable (be it O11), We know it will range between 0 and 0.25. Because the derivative of sigmoid(x) ranges from 0 to.25, and x can be any value.
Superb video once again.But need to study a little bit of theory.But still no idea how questions are framed in an interview in regards to deep learning.
the chain rule is a mistake please correct it.
Yeah I commented on it too
No. It is correct
No it is incorrect
yes derv L /derv o31 is missed
the activation function is denoted by phi, not to be confused with the symbol of cyclicc integral
At 08:30, the derivative of O21 wrt O11 is 125, but O21 is a sigmoid function. How can its derivative be 125 because derivative of sigmoid function ranges from 0 to 0.25.
very good content
Please see, the chain rule has missed something at 2:55. @krish naik
yes there is mistake is missing del L /del o31 onwards
@@omkarrane1347 yes this is a miss
Thanks krish
Write a model function to predict the y when given weights wi , input x where
y=w0+w1.x
how i can model this function??
Waiting for future videos on DL
Request for a video on side by side comparison of vanishing gradient and exploding gradient...
Great videoo
This likes turn into 1M likes after mid 2021. People do not understand the effort and hard work as they are also not doing anything right now. wait and watch
in this video ,5:30 u mentioned that w21' is this correct? i hope that is w11''? am i right or wrong ?So z=O11.w11''+b2will come .instead O11.w21+b2. am i right ?pls
Sir please make a video on bayes theorem and its concepts learning....
In the vanishing gradient you directly put values b/w 0 and 0.25 as derivative ranges in that range but why not put direct values here ?
I mean the same we could we have done in vanishing gradient as well i.e. expanding the equation and multiple by its weight ?
Even i am having the same doubt. After watching this video, I cannot understand why (d O21 / d 011) was directly put between 0 to 0.25 in Vanishing Gradient Problem video.
@krish naik sir, can you please help clarify this doubt
yes it made me confused too
hello sir,
In vanishing gradient problem you have mentioned that derivative of sigmoid is always between 0-0.25. When you did the derivative of sigmoid function result i.e derivative of o12 w.r.t o11 it must be in the range of 0-0.25 but when you expanded we got the answer as 125. I did not understand how did the derivative of sigmoid exceed the range of 0-0.25. It seems contradictory. Hope you can clear my doubt, sir.
I am having the same doubt. Can anyone please explain it?
Even I had this question
He multiplied 0.25 with initial value weight w21 which was 500. W21 is derivative of z wrt O11 in his case.
The same analysis can be done to explain vanishing gradients. So why do we say Relu solves vanishing gradients problem? low weights there can be a problem there as well when derivative is 1 right?
@7:47 d(w_21 * O_11) = O_11 dw_21 + w_21 dO_11 (why are you assuming w_21 is constant)
Doubt: the BIAS that is added, what constitutes this bias.
For instance Learning rate was found by optimization models, what methodology is used to introduce bias?
im getting confused as u said 3.20. why do u expand o21/o11 in this expolding gradient but y not expanded in vanishing gradient?.
Exploding Gradient Problem is only for sigmoid activation function or for all activation functions
love your videos and can't thankyou enough. Thankyou so much for theawesomest lessons
Really very good videos, One doubt - High value weights causing this exploding problem. But W-old also might be large vale right, if we do W-old - derivative L / dW not cause for big variance right. please help me.
How do you define O_11? in the first hidden layer?
Shouldn't the derivative be dl/ dw'11 = dl/dO31 and then the rest? Could someone please clarify? Thanks
You're right
So overall you're saying that if you choose high values of weights, it'll cause problem to reach or maybe will never reach global minima
On what basis are the weights initialises
👍👍
Krish Naik bester Mann!
Thanks Krish for the video, however I didn't understood how you replaced loss function with output of output layer, it should actually be real output minus predicted.pls suggest.
He has just shown that the predicted output will be made input to the loss function (not that predicted output is loss function as you have comprehended)
So basically Exploding and vanishing are dependent on how the weights are initialised?
sir, please note that in the last two videos there was the wrong application of chain rule. even our teacher who referred to the video has written the same mistake in her notes. ref del L /del o31 onwards
I probably made a mistake in the last part
can you explain what is wrong briefly. so I can understand
Which one is correct then one used in this video or the one used in the previous video ??
sir may be there is a problem in the chain rule that you explain. Here something is missing that is derivative of L with respect to O31
Very well explained thanks! I have a doubt tho: Are vanishing and exploding gradient coexistent phenomena? As they both happen in the BP does their happening depend exclusively on the value of the loss at a particular epoch? Hope my question is clear
Even I hv the same question. Appreciate if you can clear
Thanks for the great explanation. One small doubt/clarification would be helpful. Here since we have sigmoid if the Weight value is somewhat around 2 ( ex:2) dO11/dW11. then the dO21/dO11 value will be .25*2 = .5 then chain rule ((dO21/dO11) * (dO11/dW11)) will be again .5*.5 = 25 considering dO11/dW11 weight is also 2. Then instead of exploding it will be shirking. Can you please suggest what is the thought for this scenario
Thanks for the great explanation. One small doubt/clarification would be helpful. Here since we have sigmoid if the Weight value is somewhat around 2 ( ex:2) dO11/dW11. then the dO21/dO11 value will be .25*2 = .5 then chain rule ((dO21/dO11) * (dO11/dW11)) will be again .5*.5 = .25 considering dO11/dW11 weight is also 2. Then instead of exploding it will be shirking. Can you please suggest what is the thought for this scenario
a small doubt is that in another video you told that derivative of loss w.r.t. weight equals to derivative of loss w.r.t. output and etc... but in this video you considered directly from out on r.h.s could you please conform it
sir why your are not writing the term dL/d(o31) with other terms?
why we are multiplying O11 with weights
Thanks for this amazing video sir!
Just to summarize, can I say that only if my weight initialization would be very high and activation function is sigmoid and learning rate is also very high, I can experience this problem and no other such cases?
Activation function doesn't matter for exploding gradient decent to occur. High magnitude weights initialization alone can cause this problem.
deepan chakravarthi
Activation function is proportional to weights being applied.so exploding gradient indirectly depends on activation function and directly on weights.
The derivate should also be high
Sir, the only time when the exploding gradient problem occurs is when the weights is high and the time when vanishing gradient occurs is when the weights are too low, is my assumption correct?
Yes
@2.37 u have missed a derivate dL/d031 on the RHS.
why isn't there any weight like W31 for O21
How we will assign the weight value as 500. The normalized value is (-1,1).
I don't think that we will face Exploding Gradient problem ever because we use the standard way of initializing weights like Xavier/Glorot(sigmoid, tanh) and 'He_uniform/normal'(ReLU).