Neural Network from Scratch | Mathematics & Python Code
Vložit
- čas přidán 26. 07. 2024
- In this video we'll see how to create our own Machine Learning library, like Keras, from scratch in Python. The goal is to be able to create various neural network architectures in a lego-fashion way. We'll see how we should architecture the code so that we can create one class per layer. We will go through the mathematics of every layer that we implement, namely the Dense or Fully Connected layer, and the Activation layer.
😺 GitHub: github.com/TheIndependentCode...
🐦 Twitter: / omar_aflak
Same content in an article: towardsdatascience.com/math-n...
Chapters:
00:00 Intro
01:09 The plan
01:56 ML Reminder
02:51 Implementation Design
06:40 Base Layer Code
07:55 Dense Layer Forward
10:42 Dense Layer Backward Plan
11:23 Dense Layer Weights Gradient
14:59 Dense Layer Bias Gradient
16:28 Dense Layer Input Gradient
18:22 Dense Layer Code
19:43 Activation Layer Forward
20:46 Activation Layer Input Gradient
22:30 Hyperbolic Tangent
23:24 Mean Squared Error
26:05 XOR Intro
27:04 Linear Separability
27:45 XOR Code
30:32 XOR Decision Boundary
====
Corrections:
17:46 Bottom row of W^t should be w1i, w2i, ..., wji
18:58 dE/dX should be computed before updating weights and biases
====
Animation framework from @3Blue1Brown : github.com/3b1b/manim
In the backward function of the dense class you're returning a matrix which uses the weight parameter of the class after updating it, surely you'd calculate this dE/dX value before updating the weights, and thus dY/dX?
Wow, you are totally right, my mistake! Thank you for noticing (and well catched!). I just updated the code and I'll add a comment on the video :)
I can't add text or some kind of cards on top of the video, so I pinned this comment in the hope that people will notice it!
@@independentcode Why can't you?
Did the youtube developers remove that awesome function too?
No wonder I've felt things have been off for so long!
Can you plz help me with this .. I want a chess ai to teach me what it learnt
czcams.com/video/O_NglYqPu4c/video.html
just curious what happens if we propagate the updated weights backward like in the video? Will it not work? Or will it slowly converge?
I like how he said he wouldn’t explain how a neural network works, then proceeds to explain it
This video, instead of the plethora of other videos on "hOw tO bUiLd A NeUrAl NeTwOrK fRoM sCraTcH", is the literal best. It deserves 84 M views, not 84 k views. It is straight to the point, no 10 minutes explanation of pretty curves with zero math, no 20 minutes introduction on how DL can change the world
I truly mean it, it is a refreshing video.
I appreciate the comment :)
@@independentcode Thank you for the reply! I am a researcher, and I wanted to create my own DL library, using yours as base, but expanding it for different optim algorithms, initializations, regularizations, losses etc (i am now just developing it on my own privately), but one day I'll love to post it on my github. How can I appropriately cite you?
That's a great project! You can mention my name and my GitHub profile: "Omar Aflak, github.com/omaraflak". Thank you!
I love the 3b1b style of animation and also the consistency with his notation, this allows people to learn the matter with multiple explanations while not losing track of the core ideas. Awesome work man
This might be the most intuitive explanation of the backpropagation algorithm on the Internet. Amazing!
This is basically ASMR for programmers
I almost agree, the only difference is that I can’t sleep thinking about it
@@nikozdevbruh I fall asleep and allow my self to hallucinate in math lol
I felt relaxed definetly :D
Probably the best explaination of neural network of CZcams ! The voice and the musique backside is realy soothing !
True
Not only was the math presentation very clear, but the Python class abstraction was elegant.
Thanks for making such great quality videos. I'm working on my Ph.D., and I'm writing a lot of math regarding neural networks. Your nomenclature makes a lot of sense and has served me a lot. I'd love to read some of your publications if you have any.
The best tutorial on neural networks I've ever seen! Thanks, you have my subscription!
Very clean and pedagogical explanation. Thanks a lot!
It is the best one I've seen among the explanation videos available on CZcams!
Well done!
Best tutorial video about neural networks i've ever watched. You are doing such a great job 👏
This could be 3Blue1Brown for programmers! You got yourself a subscriber! Great video!
I'm very honored you called me that. I'll do my best, thank you !
+1
@@independentcode +1 sub
This was the best mathematical explanation on CZcams. By far.
THANK YOU !
This is exactly the video I was looking for.
I always struggled with making a neural network, but following your video, I made a model that I can generalize and it made me understandexactly the mistakes I made in my previous attempts.
It's easy to find on youtube videos of people explaining singular neurons and backpropagation, but then quickly going over the hard part: how do you compute the error in an actual network, the structural implementation and how it all ties together.
This approach with separating the Dense layer from the activation layer also makes things 100x clearer, and many people end up smacking them both in the same class carelessly.
The visuals make the intuition for numpy also much much easier. It's always a thing I struggled with and this explained why we do every operation perfectly.
even though I was only looking for one video, after seeing such quality, I HAVE to explore the rest of your channel ! Great job.
Thank you so much for taking the time to write this message! I went through the same struggle when I wanted to make my own neural networks, which is exactly why I ended up doing a video about it! I'm really happy to see that it serves as I intended :)
This is such an elegant and dynamic solution. Subbed!
Thank you so very, very, very much for this video. I have been wanting to do Machine Learning, but without "Magic". It drives me nuts when all the tutorials say "From Scratch" and then proceed to open Tensor Flow. Seriously, THANK you!!!
I feel you :) Thank you for the comment, it makes me genuinely happy.
Absolutely astonishing quality sir. Literally on the 3b1b level. I hope this will help me pass the uni course. SUB!
Amazing approach ! Very well explained. Thanks!
This video really saved me. From matrix representation to chain rule and visualisation, everything is clear now.
Very well-done. I appreciate the effort you put into this video. Thank you.
This is a so high quality content. I have only basic knowledge of linear algebra and being a non-native speaker I could fully understand this
Impressive, lot of information but remains very clear ! Good job on this one ;)
Thank you very much for your videos explaining how to build ANN and CNN from scratch in Python: your explanations of the detailed calculations for forward and backward propagation and for the calculations in the kernel layers of the CNN are very clear, and seeing how you have managed to implrment them in only a few lines of code is very helpful in 1. understanding the calculations and processes, 2. demistifying the what is a black box in tensorflow / keras.
This video is the best on CZcams for Neural Networks Implementation!
jesus christ this is a good video and shows clear understanding. no "i've been using neural networks for ten years, so pay attention as i ramble aimlessly for an hour" involved
Thank you, that's the best video I have ever seen about neural networks!!!!! 😀
Wonderful, informative, and excellent work. Thanks a zillion!!
There are many solutions on the internet...but i must say this one is the best undoubtedly...👍 cheers man...pls keep posting more.
This is the best channel for learning deep learning!
Man, I love you. How many times i tried too do the multilayer nn on my own, but always faced thousand of problems. But this video explained everything. Thank you
one of the best video i have ever seen.
struggled alot to understand this and you have explained so beautifully
you made me fall in love with the neural network which i was intimidating from.
thank you so much.
Thank you for your message, it genuinely makes me happy to know this :)
Thank you so much, my assignment was so unclear, this definitely helps!
Thank you! Well done! Absolutely wonderful video.
This is so ASMR and well explained!
I loved the background music. It gives peaceful mind. I hope, you will continue to make videos, very clear explanation
Such a great video. Really helped me to understand the basics.
This is a very good approach to building neural nets from scratch.
best video, very clear-cut. Finally I got the backpropagation and derivatives.
That was helpful, thank you so much.
This is really dope. The best by far. Subscribed right away
Thanks you so much for your contribution in this field.
Very nice and clean video, keep it up
I think the last row's indices of the W^T matrix at 17:55 must be (w1i, w2i,...,wji).
Still the best explannation i have ever seen btw, thank you so much. I dont know why this channel is still so underrated, looking forward to seeing your new videos in the future
Yeah I know, I messed it up. I've been too lazy to add a caption on that, but I really should. Thank you for the kind words :)
actually,you saved my life, thanks for doing these
you are the best 🥺❤️..wow.. finally i able to understand the basics thanks
Thank you for really great explanation!
Wish you will make even more 😉
This is the best video i have seen so far ❤
Finally found the treasure. Please do more video bro. SUBSCRIBED
Amazing explanation!!
Amazing tutorial!
That was incredibly explained and illustrated. Thanks
Thank you! I'm glad you liked it :)
@@independentcode Most welcome!
Dude this is amazing
Awesome man!!
Thank you so much for the video!!!
You are the only youtuber I sincierly want to return. We miss you!
Keep it up .please make a deep learning and ml series for future.
this is a great video thank you so much
Thank you very very much for this video....
Content at it's peak
Amazing!!
Big Fan of you from today !
after 1000 videos watched, i think i get it now, thanks
Only 4 video and you have avove 1k subs,
Please continue your work 🙏🏼
Clear, to the point. Thank you. Like (because there are just 722, and have to be a lot more)
Well done
I love u , best ML video ever
Very educational
Whyyyy you don't have 3Million subscriptions you deserve it ♥️♥️
That is so satisfying
Best tutorial💯💯💯💯
your voice is calming and relaxing, sorry if that is weird
Haha thank you for sharing that :) Maybe I should have called the channel JazzMath .. :)
grate video
thank you
great stuff
I would like alot if u continue your channel bro
thanks a ton for this amazing video on neural networks, this is the best i have seen so far 😊. Can you please also give a hint how to update your code to make it a Binary Neural Network?
Awsome Video
BEST OF BEST THANK YOU
I developed my first neural network in one night yesterday. that could not learn because of backward propagation, it was only going through std::vectors of std::vectors to get the output. I was setting weights to random values and tried to guess how to apply backward propagation from what i have heard about it.
But it failed to do anything, kept guessing just as I did, giving wrong answers anyway.
This video has a clean comprehensive explanation of the flow and architecture. I am really excited how simple and clean it is.
I am gonna try again.
Thank you.
I did it ! Just now my creature learnt xor =D
amazing video. one thing we could do is to have layers calculate inputs automatically if possible. Like if I give Dense(2,8), then the next layer I dont need to give 8 as input since its obvious that it will be 8. Similar to how keras does this.
I think most of the ML PhDs dont aware of this abstraction. Simply the best.
I don't know about PhDs since I am not a PhD myself, but I never found any simple explanation of how to make such an implementation indeed, so I decided to make that video :)
@@independentcode I think you should keep going video seris and show how capable this type of abstraction. Implemnting easiliy almost every type of neural nets.
Thank you for the kind words. I did actually take that a step further, it's all on my GitHub here: github.com/OmarAflak/python-neural-networks
I managed to make CNNs and even GANs from scratch! It supports any optimization method, but since it's all on CPU you get very quickly restricted by computation time. I really want to make series about it, but I'll have to figure out a nice way to explain it without being boring since it involves a lot of code.
@@independentcode GANs would be great also you could try to do RNNs too and maybe even some reinforcement learning stuff :D
Yeah, this is awesome
this is an amazing video which explains so perfectly how neural networks work. I appreciate and thank you for all the effort energy you put in this video and it is shame that your work did not receive enough views that it deserves. I believe you use manim to make animations like 3b1b, dont you?
Thanks a lot for the kind comment 😌 I'm glad if the video helped you in any way :) Yes it is indeed Manim!
sir please keep up with your videos I learn a lot
In your code you compute the gradient step for each sample and update immediately. I think that this is called stochastic gradient descent.
To implement full gradient descent where I update after all samples I added a counter in the Dense Layer class to count the samples.
When the counter reached the training size I would average all the stored nudges for the bias and the weights.
Unfortunately when I plot the error over epoch as a graph there are a lot of spikes (less spikes than when using your method) but still some spikes.
My training data has (x,y) and tries to find (x+y).
Would you be able to share the code? This is where the part where I’m confused.
This is one of the best videos to really understand the vectorized form of neural networks! Really appreciate the effort you've put into this.
Just as a clarification, the video is considering only 1 data point and thereby performing SGD, so during the MSE calculation Y and Y* are in a way depicting multiple responses at the end for 1 data point only right? So for MSE it should not actually be using np.mean to sum them up?
whiteout any doubt best explanation of NN ive ever seen - why you stop your productivity my friend ?
Great stuff! I find it even better than the one from 3b1b. Can you think of any way the code can be checked with matrices outside the learning set?
Thank you!
If you mean to use the network once it has trained to predict values on other inputs, then yes of course. Simply run the forward loop with your input. You could actually make a predict() function that encapsulates that loop since it will be the same for any network.
Great tutorial . Btw what is your editor font ?
This is absolutely amazing thank you. Is there any chance you can open source the manim animations too?
I'm interested in that almost as much as the neural network library design!
how can we update this to include mini-batch gradient descent? Especially how will the equations change?
This indeed is the better explanation of the math behind the neural networks I've found on the internet, could I please use your code on github in my final work for college?
Thank you for the kind words! Other videos are coming up ;)
Yes of course, it is completely open source.
Any chance on a video modifying the training to use SGD/minibatch?
hi, i love this video. Only one question why in the DenseLayer.backward() in the bias and the weigths you use -= insted of =. Why we substract that value?
The rest is all clear :) Ty
In tensorflow they use weight matrix W dimensions i x j then take transpose in calculation.
when looking at the error and it's derivative wrt some y[i], intuitively I would expect that if I increased y[i] by 1 the error would increase by dE/dy[i], but if I do the calculations the change in the error is 1/n off from the derivative, does this make sense?
I noticed that you are using a batch size of one. make a separate Gradiant variable and ApplyGradiants function for batch sizes > 1
Note 1: also change "+ bias" to "np.add(stuff, bias)" or "+ bias[:,None]
Note 2: in backpropagation, sum up the biases on axis 0 (I'm pretty sure that the axis is 0) and divide both weights and biases by batch size
Thanks for the tip on the biases.
Thanks for the tip on the biases. (1)
Can you (or someone else) please explain to me what note 1 means.
Edit: As for note 2, I successfully implemented it (by summing on axis 1), so thanks for the tip.
in the case of mini batch / batch gradient descent, would the input to the first layer be a matrix of ( Number_of_Features * Data_Points ) ? in that case, do I need to compute the average of the gradients in back propogation in each layer?
@@nahianshabab724 I guess yes, I saw that in multiple videos, just add a 1/m in the MSE formula.
what is the use of layer class ? and a great video. Hope you keep posting stuff on your channel
It is the best It is the Beauty because the explanation is great
How output gradient is calculated and passed into the backward function?