Video

Thanks for the great video. In the Tokenizer class you define self.vocab using the self._build_vocab() function, but then self.ocab is overwritten when you run self.train(). Why do you initialize self.vocab (for bytes([0-256] and special tokens) if you are going to just overwrite it?

Around the 1:30:00 mark, I think I missed why some saturation (around 5%) is better than no saturation at all. Didn't saturation impede further training? Perhaps he just meant that 5% is low enough, and that's the best we can do if we want to avoid deeper activations from converging to zero?

I arrived at dh just figuring out by the size of the matrix, and then I continued with your video and you just did all the derivatives and I taught... I am so dumb, I should I have done that, but then you say " now I tell you a secret I normally do... 49:45.... hahahahahhaha

@AndrejKarpathy referring to 52nd minute of the video, in order to conclude that bigrams are learning, the likelihood of each should be greater than 1/(27.0*27.0) and not 1/(27.0) as mentioned in video. Thoughts?

Dude is a legend. Big respect for this!

Thank you so much for the great learning materials you create and share, this is precious. I've recently also run into a highly visual explanation on attention mechanism by 3Blue1Brown (czcams.com/video/eMlx5fFNoYc/video.htmlsi=G7PPnlbmx379YWjp) and I liked the intuition we can have behind the Values (timestamp: czcams.com/video/eMlx5fFNoYc/video.htmlt=788). So as far as I understood, we basically can intuitively think of the Value as some vector we can add to a word (~token) so that we get a more refined detailed meaning of the word. For example, if we have a "fluffy creature" in a sentence, then at first we have an embedding for "creature" and then "pay attention" to what came before and have richer information. That is, the Value shows how the embedding of a "creature" should be modified to become an embedding of "fluffy creature"

Grandma Jailbreak still works on ChatGPT !!!

i consistently watch this vid, love it!

I transcribed the code from the screen and it ran OK but now I get "RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn" on the call to loss.backward(). What does this mean? Is a memory problem on my PC?

Great video!!

wow. I am amazed by how intuitive this is.

1:26:12

That napalm jailbreak no longer works on newer models like chatGPT-4o. I tried seeking help from chatGPT😃. However, that base64 trick did work!!!

big fan of Andrej. Please Please keep making these videos. They are sooooo good!

Btw thanks for the napalm recipe

1:19:31 Honestly you don't even need topological ordering, I literally implemented a recursive approach where I call backpropag on the output node and it will then first set the gradient of his children and then cycle through its children to and backpropagate on those nodes recursively

Awesome video ... well done, well explained ... must see.

great to have a tutorial on that! I also found a free tokenizer on poe, so I don't even have to do it on my own :)

Yes the aggreagtion is data dependend but the linear transform to create the queries, keys and values is the same for all nodes (so we need multiple attention heads, cause this will not capture much in a single head)

This is actually really helpful

Awesome tutorial. Thank you Andrej!

Thank Andrew, This is a great video for beginners

Can someone explain to me why dbnmeani doesnt need keepdim = True? the size of dbnmeani is different of bnmeani if we dont put keepdim=true

Thanks for sharing free tutorials

Hello Andrej. Your explanations are exceptional and the way you teach is very unique and it helped me learn a lot. I am very grateful to you for your lectures. I only request you to make a video about the mixture of experts architecture, how it works, and how it could be implemented in code. That would be amazing. Thank you.

Where can I find models that have not been aligned or fine tuned to act like chatbot's? Models that are trained on large datasets of language (preferable multilingual) but only works as document completers? I'm sure those could be very useful in some contexts as well, right?

Historic. Gem of a video. Kudos to you Andrej!

Andrej, I just couldn't thank you more for putting all of these together! You showed the world the spirit of open source. I'd call you cyber-Prometheus

Is it fair to call the log counts "logits"? If the logit is the function that maps a probability p to ln p/(1 - p), and in this neural network we start with log counts and end up with probabilities, the operations we take to get there don't seem to "undo" the logit function: we exponentiate the log counts, which, if these were logits, would yield the p/(1 - p) from the logit function, but then I don't see an operation that goes from that to p. Surely it's not the normalization step? So are these log counts also intuitively logits?

<*_> This is my seal. I have watched the entire video, understood it, and I can explain it in my own words, thus I have gained knowledge. This is my seal. <_*>

Great Stuff... thank you for sharing this strategy!! Similar to one other that you did while ago. Can you tell me which drawing tools you use apart from the drawing of pocket option itself?? Thanks

I'd like to see those hypothetical 500 lines of C code and how fast it would run

Can someone explain why the ratio needs to be explicitly close to the number -3.0? why -3.0 and not -1.0 for example?

Great content! Made me understand the theoretical basis of neural networks which i knew mathematically already in a quite comprehensive practical manner. On the way i could get a better practical understanding of classes as well!

Just starting to learn about programming. Will come back to this video later maybe in the future once I understand more about AI.

Masterpiece.

Great overview. But creepy grandmother ;-)

Im doing my bachelor thesis on this topic. After Weeks and 8-10 hours per day trying to get a overview of the topic and so on this by far was the most helpful video. Keep up the good work.

Very cool demo, I coded along until the end because I wanted to listen and understand. Really good follow up to the Coursera courses.

Great Explanation. this feels like learning dikstra algrithm, in an analogy of google maps.

God- Level Introduction to LLMs

Give a man a fish, feed him for a day. Teach a man to fish, feed him for a lifetime. Thank you, my friend.

"So this, Jane, is THE INTERNET!"

Incredible.

At 54:55, can we not just implement encode by iterating over merges dictionary(the order is maintained) and calling the merge() function on tokens ? This is what I mean def encode(text) : tokens = list(text.encode("utf-8")) for pair, idx in merges.items() : tokens = merge(tokens, pair, idx) return tokens Seems about right to me...

Can someone explain the necessity to write "with torch.no_grad()" while updating bmeans_running & bstd_running,Thanks in advance.

Can someone explain me why this is so hyped? I think it’s just differentiation that we learnt in calculus right?

Спасибо, всё работает. Ждём новых связок.

so fascinating! and then scale it up and it starts talking like real humans and knows everything! this is holarious!! it s so cool that you just put this vids out here!!

The value delivered in these videos is just insane! It’s people like Andrej that restore trust in humanity