The Absolutely Simplest Neural Network Backpropagation Example
Vložit
- čas přidán 5. 06. 2024
- I'm (finally after all this time) thinking of new videos. If I get attention in the donate button area, I will proceed:
www.paypal.com/donate/?busine...
sorry there is a typo: @3.33 dC/dw should be 4.5w - 2.4, not 4.5w-1.5
NEW IMPROVED VERSION AVAILABLE: • 0:03 / 9:21The Absolut...
The absolutely simplest gradient descent example with only two layers and single weight. Comment below and click like! - Věda a technologie
![Eminem - Houdini [Official Music Video]](http://i.ytimg.com/vi/22tVWwmTie8/mqdefault.jpg)
GREAT, it was a perfect inspiration for me to explain this critical subject in a class. Thank you!
Really nice work. Thank you so much for your help.
Best video ever about the back propagation in the internet 🛜
Dude, this was just what I needed to finally understand the basics of Back Propagation
if you _Really_ liked his video, just click the first link he put on the description 👍
To understand mathematics, I need to see an example.
An this video from start to end is awesome with quality presentation.
Thank you so much.
Unreal explanation
@8:06 this was super useful. That's a fantastic shorthand. That's exactly the kind of thing I was looking for, something quick I can iterate over all the weights and find the most significant one for each step.
Hi I have question for you, at 3:42, you have, 1.5*2(a-y) = 4.5*w-1.51, how did you get this result?
... in case someone missed it like me - it's in the description (it's a typo). y=0.8; a=i*w = 1.5*w, so 1.5*2(a-y) =3*(1.5*w - 0.8) = 4.5*w - 3*0.8 = 4.5*w - 2.4 is the correct formula.
My long search ends here, you simplified this a great deal. Thanks!
Not kidding. This is the best explanation of backpropagation on the internet. The way you're able to simplify this "complex" concept is *chef's kiss* 👌
just perfect, simple and with this we can extrapolate easier when in each layer there are more than one neuron! thaaaaankksss!!
This was great. Removing non linearity and including basic numbers as context help drove this material home.
If you use relu there is nothing more that that
finally, a proper explanation.
Fantastic. This is the most simple and lucid way to explain backprop. Hats off
I had to write a comment and thank you for your very precise yet simple explanation, just what I needed. Thank you sir.
After a long frantic search, I stumbled upon this gold. Thank you so much!
I have to say it. You have done the best video about backpropagation because you chose to explain the easiest example, no one did that out there!! Congrats prof 😊
did you _really_ like his video? Then, i'd suggest you click the first link he put on the description 👍
I was just looking for this explanation to align derivatives with gradient descent. Now it is crystal clear. Thanks Miakel
I watched almost every videos of back propagation even Stanford but never got such clear idea until I saw this one ☝️.
Best and clean explanation.
My first 👍🏼 which I rarely give.
a 👍is very good, but if you click on the first link on the description, it would be even better 👍
@@webgpu 🆗
The best short video explanation of the concept0 on CZcams till now...
Very clearly explained and easy to understand. Thank you!
What a breakthrough, thanks to you. BTW, not to nitpick, but you are missing a close paren on f(g(x), which should be f(g(x)).
Thank you for your video. But I’m a bit confused about 1,5.2(a-y) = 4,5.w-1,5, Might you please explain that? Thank you so much!
I think this is how he got there :
1.5 * 2(a - y) = 1.5 * 2 (iw - 0.5) = 1.5 * 2 (1.5w - 0.5) = 1.5 * (3w - 1) = 4.5w - 1.5
@@user-gq7sv9tf1m dude thanks for that, I was really scratching my head over how he got there too
i am also confused this error
@@user-gq7sv9tf1m y is 0.8 not 0.5
You made this concept very simple. Thank you
Great video. Just one question, this is for 1 x 1 input and batch size of 1 right?. If we have, let´s say a batch size of 2, It is just to sum (b-y)^2 to the loss function ( C= (a-y)^2 + (b-y)^2) isnt it?, with b = w * j and j = the input of the second batch size. Then you just perform the backpropation with partial derivatives. Is it correct?
best on internet.
This makes more sense than anything I ever heard in the past! Thank you! 🥂
It beats the 1002165794 thing and 1001600474 jumping and calculating with 1000325836 and 1000564416. Much easier 😊
you are wrong: Say me what is deltaW?
Absolutly simple. Very useful illustration not only to understand Backpropagation but also to show gradient descent optimization. Thanks a lot.
I'm currently programming a neural network from scratch, and I am trying to understand how to train it, and your video somewhat helped (didn't fully help cuz I'm dumb)
@Mikael Laine even though you say that @3:33 has a typo. i cant see the typo. 1.5 is correct because y is the actual desired out put and it is 0.5. so 3.0 * 0.5 = 1.5
dude please make more videos. this is amazing
GOD BLESS YOU DUDE! SUBSCRIBED!!!!
Thank you bro! Its so easier to visualize it when its presented like that.
Great video, going to spend some time working out it looks for multiple neurons, but a demonstration on that would be awesome
4:03 Shouldn't 3(a - y) be 3(1.5*w - 0.8) = 4.5w - 2.4? Where have you got -1.5 from?
My maaaaaaaannnnn TYYYY
Thanks very helpful.
I think there is a mistake. 4.5w -1.5 is correct.
On the first slide you said 0.5 is the expected output.
So "a" is the computed output and "y" is the expected output. 0.5 * 1.5 * 2 = 1.5 is correct.
You need to correct the "y" next to the output neuron to 0.5.
Thanks! This is Awesome. I have I question, if we make the NN more complicated a little bit (adding an activation function for each layer), what will be the difference?
I am so happy that I can't even express myself right now
there's a way you can express your happiness AND express your gratitude: by clicking on the first link in the description 🙂
You made it easy to understand. Really appreciated it. You also earned my first CZcams comment.
Thanks for making this
thank you, this is exactly what I was looking for, very useful!
Excellent , please continue we need this kind of simplicity in NN
if we take directly the derivitive dC/dw from C=(a-y)^2 is the same thing right? do we really have to split individually da/dw and dC/da ???
I don’t get it you write 1.5*2(a-y) = 4.5w -1.5
But why? It should be 4.5w -2,4
Because 2*0,8*-1,5= -2,4
Where am I rong?
Thats sick bro I just implemented it
Bro this is awesome, I was struggling to understand chain rule, now it is clear
Thanks for the video! Awesome explanation
man 4:08 i dont undestrand how you find the valor 4.5, in expression 4.5.w-1.5,
It clicked after just 3 minutes. Thanks a lot!!
Great video. I believe there is a typo at 1:10. y should be 0.5 and not 0.8. That might cause some confusion, especially at 3:34, when we use numerical values to calculate the slope (C) / slope (w)
Thanks for pointing that out; perhaps time to make a new video!
yes, that should say a=1.2
+Mikael Laine I would be si glad if you could make more videos explaining these kind of concepts and how they actually work in a code level.
Did you have any particular topic in mind? I'm planning to make a quick video about the mathematical basics of backpropagation: automatic differentiation. Also I can make a video about how to implement the absolutely simples neural network in Tensorflow/Python.
Let me know if you have a specific question. I do have quite a bit experience in TF.
@@mikaellaine9490 How about adding that to description? Someone else asked that question.
Awesome dude. Much appreciate your effort.
Great illustrated, thanks
This is absolutely awesome. Except..... Where did that 4.5 come from???
You’ve probably figured it out by now but just in case: i = 1.5, y=0.8, a = i•w. This means the expression for dC/dw = 1.5 • 2(1.5w - 0.8). Simplify this and you get 4.5w - 2.4. This is where the 4.5 comes from. Extra note: in the description it says -1.5 was a typo and the correct number is -2.4.
Nice and clean. Helped me a lot!
best explanation i had ever seen, thanks.
Very helpful tutorial. Thanks!
Brilliant. What would be awesome is to then further expand if u would and explain multiple rows of nodes...in order to try and visualise if possible multiple routes to a node and so on...i stress "if possible...".
excellent video, simple & clear many thanks
This video is very well done. Just need to understand implementation when there is more than one node per layer
Have you looked at my other videos? I have a two-dimensional case in this video: czcams.com/video/Bdrm-bOC5Ek/video.html
So what is the clever part of back prop? Why does it have a special name and it isn't just called "gradient estimation"? How does it save time? It looks like it just calculates all derivatives one by one
This is the best tutorial on back prop👏
Thank you for sharing this video!
Great video
in the final eqn why it is 4.5w-1.5 instead it should be 4.5w-2.4 since y=0.8 so 3*0.8 =2.4
Yes you are right. I noticed too.
Thanks for a very explanatory video.
this was kicking my a$$ until i watched this video. thanks
Bro i just worked it through and it makes so much sense once you do the partial derivatives and do it step by step and show all the working
Exactly what i needed
Great video! One thing to mention is that the cost function is not always convex, in fact it is never truly convex. However, as an example this is really well explained.
I see.
As previously mentioned, there are a few typos. For anyone watching, please note there are a few places where 0.8 and 0.5 are swapped for each other.
That being said, this explanation has opened my eyes to the fully intuitive explanation of what is going on...
Put simply, we can view each weight as an "input knob" and we want to know how each one creates the overall Cost/Loss.
In order to do this, we link (chain) each component's local influence together until we have created a function that describes weight to overall cost.
Once we have found that, we can adjust that knob with the aim of lowering total loss a small amount based on what we call "learning rate".
Put even more succinctly, we are converting each weight's "local frame of reference" to the "global loss" frame of reference and then adjusting each weight with that knowledge.
We would only need to find these functions once for a network.
Once we know how every knob influences the cost, we can tweak them based on the next training input using this knowledge.
The only difference between each training set will just be the model's actual output, which is then used to adjust the weights and lower the total loss.
Absolutely amazing 🏆
Good content sir keep making these i subscribe
Thank you for the easiest expression for bacpropagation dude
thanks a lot... a great start for me to learn NNs :)
are you able to briefly describe how the calculation at 8:20 works for a network with mutliple neurons per layer?
The video shows what is perhaps the simplest case of a feedforward network, with all the advantages and limitations that extreme simplicity can have. From here to full generalization several steps are involved.
1.- More general processing units.
Any continuously differentiable function of inputs and weights will do; these inputs and weights can belong not only to Euclidean spaces but to any Hilbert spaces as well. Derivatives are linear transformations and the derivative of a unit is the direct sum of the partial derivatives with respect to the inputs and with respect to the weights.
2.- Layers with any number of units.
Single unit layers can create a bottleneck that renders the whole network useless. Putting together several units in a layer is equivalent to taking their product (as functions, in the set theoretical sense). Layers are functions of the totality of inputs and weights of the various units. The derivative of a layer is then the product of the derivatives of the units. This is a product of linear transformations.
3.- Networks with any number of layers.
A network is the composition (as functions, and in the set theoretical sense) of its layers. By the chain rule the derivative of the network is the composition of the derivatives of the layers. Here we have a composition of linear transformations.
4.- Quadratic error of a function.
---
This comment is becoming a too long. But a general viewpoint clarifies many aspects of BPP.
If you are interested in the full story and have some familiarity with Hilbert spaces please Google for papers dealing with backpropagation in Hilbert spaces.
Daniel Crespin
Thank you. Here is pytorch implementation.
import torch
import torch.nn as nn
class C(nn.Module):
def __init__(self):
super(C, self).__init__()
r = torch.zeros(1)
r[0] = 0.8
self.r = nn.Parameter(r)
def forward(self, i):
return self.r * i
class L(nn.Module):
def __init__(self):
super(L, self).__init__()
def forward(self, p, t):
loss = (p-t)*(p-t)
return loss
class Optim(torch.optim.Optimizer):
def __init__(self, params, lr):
defaults = {"lr": lr}
super(Optim, self).__init__(params, defaults)
self.state = {}
for group in self.param_groups:
for par in group["params"]:
# print("par: ", par)
self.state[par] = {"mom": torch.zeros_like(par.data)}
def step(self):
for group in self.param_groups:
for par in group["params"]:
grad = par.grad.data
# print("grad: ", grad)
mom = self.state[par]["mom"]
# print("mom: ", mom)
mom = mom - group["lr"] * grad
# print("mom update: ", mom)
par.data = par.data + mom
print("Weight: ", round(par.data.item(), 4))
# r = torch.ones(1)
x = torch.zeros(1)
x[0] = 1.5
y = torch.zeros(1)
y[0] = 0.5
c = C()
o = Optim(c.parameters(), lr=0.1)
l = L()
print("x:", x.item(), "y:", y.item())
for j in range(5):
print("_____Iter ", str(j), " _______")
o.zero_grad()
p = c(x)
loss = l(p, y).mean()
print("prediction: ", round(p.item(), 4), "loss: ", round(loss.item(), 4))
loss.backward()
o.step()
thanks a lot for that explanation :)
Where and how did you get the learning rate?
Thanks you
Helped me so much!
Thank you so much! I'm 14 years old and I'm now trying to build a neural network with python without using any kind of libraries, and this video made me understand everything much better.
No way me too
Brooo WW I ended up coding something which looked good to me but for some reason It didn't work so I just gave up on it. I wish you good luck man@@Banana-anim8ions
The error should be (1.2 - 0.5) = squared(0.7) = 0.49. So y is 0.49 and not 0.8 as it is displayed after minute 01:08.
man, thanks!
very clear
Ow you did not lie on the tittle.
Thanks
Thank you so much!
hmm, if y = .8 then should dc/dw = 4.5w - 2.4. Because .8 * 3 = 2.4, not 1.5. What am I missing?
why do we ever need to consider multiple levels, why not just think about getting the right weight given the output "in front" of it
This video is gold.
THIS IS SOO FKING GOOD!!!!
I thought it is just similar to LMS widely used in communication, right? LMS was developed by Bernard back in 60s.
Okay I am better with language than with maths, so I'll try to sum it up:
We basically look for a desired weight in order to get a certain output unit. And we get this desired weight by setting the weight equal to C, which again is the x-value of the minimum of some function that we get by deriving the function containing the original (faulty) output continuously (by steps determined by a "learning rate") until it is very close to zero. That correct?
Спасибо братан, наконец-то выкупил что после последнего слоя происходит:)
6:55 but it's NOT the same terms. Is that da0/dw1 term correct?
Looks like the w term turns into a each back step.
Very helpful
I apologize in advance. I don't quite understand, why can't we equate the derivative function to 0 instead of gradient descent. If it is nonlinear, it will have several zeros. Then you can choose the one that suits us
Deep learning is a numerical method of finding the features / classes appropriate for the given problem. You are correct in that here - in this naive example - you could calculate the closed-form solution, but in the general / complex case that would not be feasible.