Let's build the GPT Tokenizer
Vložit
- čas přidán 7. 06. 2024
- The Tokenizer is a necessary and pervasive component of Large Language Models (LLMs), where it translates between strings and tokens (text chunks). Tokenizers are a completely separate stage of the LLM pipeline: they have their own training sets, training algorithms (Byte Pair Encoding), and after training implement two fundamental functions: encode() from strings to tokens, and decode() back from tokens to strings. In this lecture we build from scratch the Tokenizer used in the GPT series from OpenAI. In the process, we will see that a lot of weird behaviors and problems of LLMs actually trace back to tokenization. We'll go through a number of these issues, discuss why tokenization is at fault, and why someone out there ideally finds a way to delete this stage entirely.
Chapters:
00:00:00 intro: Tokenization, GPT-2 paper, tokenization-related issues
00:05:50 tokenization by example in a Web UI (tiktokenizer)
00:14:56 strings in Python, Unicode code points
00:18:15 Unicode byte encodings, ASCII, UTF-8, UTF-16, UTF-32
00:22:47 daydreaming: deleting tokenization
00:23:50 Byte Pair Encoding (BPE) algorithm walkthrough
00:27:02 starting the implementation
00:28:35 counting consecutive pairs, finding most common pair
00:30:36 merging the most common pair
00:34:58 training the tokenizer: adding the while loop, compression ratio
00:39:20 tokenizer/LLM diagram: it is a completely separate stage
00:42:47 decoding tokens to strings
00:48:21 encoding strings to tokens
00:57:36 regex patterns to force splits across categories
01:11:38 tiktoken library intro, differences between GPT-2/GPT-4 regex
01:14:59 GPT-2 encoder.py released by OpenAI walkthrough
01:18:26 special tokens, tiktoken handling of, GPT-2/GPT-4 differences
01:25:28 minbpe exercise time! write your own GPT-4 tokenizer
01:28:42 sentencepiece library intro, used to train Llama 2 vocabulary
01:43:27 how to set vocabulary set? revisiting gpt.py transformer
01:48:11 training new tokens, example of prompt compression
01:49:58 multimodal [image, video, audio] tokenization with vector quantization
01:51:41 revisiting and explaining the quirks of LLM tokenization
02:10:20 final recommendations
02:12:50 ??? :)
Exercises:
- Advised flow: reference this document and try to implement the steps before I give away the partial solutions in the video. The full solutions if you're getting stuck are in the minbpe code github.com/karpathy/minbpe/bl...
Links:
- Google colab for the video: colab.research.google.com/dri...
- GitHub repo for the video: minBPE github.com/karpathy/minbpe
- Playlist of the whole Zero to Hero series so far: • The spelled-out intro ...
- our Discord channel: / discord
- my Twitter: / karpathy
Supplementary links:
- tiktokenizer tiktokenizer.vercel.app
- tiktoken from OpenAI: github.com/openai/tiktoken
- sentencepiece from Google github.com/google/sentencepiece - Věda a technologie
![[1hr Talk] Intro to Large Language Models](http://i.ytimg.com/vi/zjkBMFhNj_g/mqdefault.jpg)
![[1hr Talk] Intro to Large Language Models](/img/tr.png)
I'm amazed at the times we live in. One of the top AI experts in the world is sharing free tutorials that teaches technologies in great detail including examples and code. Thank you very much.
Yeah, it's unbelievable, yet very helpful
@@von.changfeng We're living an incredible era.
unprecedented
can't agree more
@@louisyuliu7200 You can come here to Ukraine and.. find out that times r not the best. But ye. Respect to Andrej to sharing knowledge
The fact that Andrej prefers being here with us over developing self driving or developing AGI is surely one of the great things about living in the present times.
Not to worry, he's simply applying the Feynman method.
A True Master
Is Andrej hype-phobic and this means self-driving and AGI application research reached plato? That would explains why he is dedicating himself into teaching basics so more students go into the field
@@web No, not at all. It's moving faster than expected and he wants the knowledge spread out and not contained to corporations before it's too late. Quite the opposite of your guess. Models need to be open-source or there will be huge disparities in capabilities.
@@webHe’s an advocate for free and accessible knowledge for all plus he simply choose to do what he loves sharing knowledge with everyone
This guy left Tesla, OpenAI, and God knows how many other companies so he can teach us how the state of art AI works, from scratch, clearly explained. Such a pure gem. Thank you very very very much.
200,000 people watching probably only about 500 of those can actually implement any of it... One in a thousand probably more like
what do you mean? (like 500 of those will try to implement it themselves?)
@@13thbiosphere
@@13thbiospherevery true but a lot of people, like me, just like watching people who really know their shit, in whichever area
@@13thbiosphere Yeah thats completely true, but I would argue it is still worth it even just for those 500 people. 500 new AI researchers could make a very significant difference in the field. (btw im one of those 500 that can implement lol)
@@simonvutov7575 pathetic
You could have made a million bucks or spent time on a beach with a cocktail. Instead you chose to do this for the benefit of unknown strangers. You are an amazingly talented and generous man. Thank you so much!
Hah, so you know his net worth?)
he is already likely worth 100M+ lmao . and rightfully so.
Whenever Karpathy leaves a job the ML world jumps forward a bit with his educational content
I cannot express enough how grateful I am for your incredible tutorials. As a 44-year-old South Sudanese individual with three children and a full-time job, recently relocated from the UK to the US, my days are filled to the brim with responsibilities. However, I have made it a priority to utilize every ounce of my free time to follow your series of lectures. I must say, I have never been as excited about learning a new topic as I am now. Your clear explanations and engaging content have ignited a passion for learning within me that I never knew existed. Thank you from the bottom of my heart for all that you do.
Welcome to America!! Great to have you.
Internet is amazing
calm down
You being one among them, isn't it? Think... @@MikeDonaldson-eh2ru
❤❤❤
Am South Sudanese, watching this from Nairobi, Kenya
This is honestly better than OpenAI releasing a new model
yes building the components and explaining the full concept is truly open source .... a paid resource is not open source .... the aim is to learn how to make things for yourself so that when such things are removed or censored they are already out in the publics hands ... how can you regulate people own experiments! .... next we need to be able to build a full model with the components with only our own data and not "The common crawl or Wiki data" hence a closed domain llm.... as it should be! everybody should have the right to create their own AI! with their own DATA !... in the future the merging of models to have multiagent systems is the way .... so plug in models into the multi aGI!
Ha ha, if GPT-5 isn't the next release, it means OpenAI hasn't released anything at all. Incremental releases are just tiring because it seems like they've only fixed the bugs from the previous model using hundreds of millions of high-quality human feedback in 2023, and nothing else is really happening.
When andrej uploads i immediately drop everything and watch the complete video. Thank you so much for your valuable content!
Same here!
me too
I tried to watch his very first video. dint understand a thing.
@@teslabot5650 it takes some time. You need to have basic programming skills to understand this. Also algorithms help a bit.
yes the first few were not easy but after while it has been very good to listen to and understand the process to think with regard to some projects... as you note the projects are often simple and not over complicated!
Andrej, content like this is advancing the industry more than most companies are. Thank you for educating us. You are pushing everybody forwards!
We're so back!!!!
He's even more back :)
@@TheLokiGT he never left
@@coder_rche is inside u😢💀
@@ultrasound1459 I don't mind
Its so over
the intrinsic value in andrej making videos for the world is huge. Please andrej keep doing this, no other jobs can output the alpha you're delivering here, pushing the boundaries of the best educational content on the planet. You are born for this.
Thank you, Andrey! In an environment where 99% of conversations about LLM are marketing, technical videos are especially valuable and interesting!
Been waiting on the lecture to drop ever since we saw the Github repo go public.
Andrej, on behalf of the academic and professional communities, thank you for putting the effort into creating this open knowledge.
On a personal note, I've been following your work since the early OpenAI days, and your passion during keynotes/presentations has supported my choice of pursuing a PhD in the field.
Thank you Andrej.
Feynman-class teaching abilities. Amazing. Thank you very much as always Andrej
Andrej, thank you for these amazing tutorials! They are the best I could find on the internet. It's always great to hear explanations from an actual expert in the field instead of trying to understand it from videos of a random Indian guy
The best thing I have learnt from Andrej's videos is 'How to learn?' Break things down to axiomatic levels and then connect those pieces together! Its truly enlightening!
OpenAI was such a closed cult, Andrej gave himself to the world.
🌹❤️🌹
Andrej deserves his flowers.
Thank you Andrej Karpathy. Thanks for teaching Python libraries, GPT generator codes. Great work
Congrats Andrej on your new venture!
What is his venture?
@@ultrasound1459This 🙂
Andrej, you're really good at teaching. We're lucky that you're spending your valuable time to prepare these free contents.
Sorry Andrej, but you've missed the mark: you tell us that you don't like this topic, but you've still managed to make it fascinating!
I haven't watched the video yet but I just wanted to say I really appreciate you taking the time to teach this. I loved all of your previous videos. You are an excellent teacher and your style makes it super easy for me to learn.
Thank you very much for this. I'm from a traditional programming background, and your videos on AI are really excellent. I have followed the previous sessions, being able to implement the things correctly, and compare some of the stuff, with your results. That is awesome! Good luck.
Andrej was wrong about one thing though. This video is not dry and boring at all as he indicated in the video. I watched the two hours plus video with joy and excitement. Thank you!
Oh my god. I never understood tokenization (hoped i would remain on vision :)) ). This is golden. Thank you so much ❤❤❤❤ you are my hero, I've learned deep learning from your cs231 while in college (we did not have deep learning as an option) and that got me a ML job. Thank you so very much ❤❤❤
I had other plans but you said "space egg" and I was hooked. You've gone and done that engaging professor thing again. Thank you. I think you doing this is incredibly impactful. It's a real inspiration. Please keep it up.
The attention to the details... You prepared this material briliantly. Thank you.
Liking before watching the video. So glad to see you continuing to upload!
Wow, I just got a notification about your new video, I love these videos, thank you for this content Andrej, this is super informative
I don't which tokens would best be used to say "Thank you" but I feel so lucky to even get to watch and learn from you.
the fact that this is available completely for free is absolutely amazing, the passion and amount of time/effort you spent on this is reflected by the pure quality, i cant thank you enough for doing this andrej
Amazing! Was waiting patiently for another upload
Watching from Muqdisho, Somalia.
Much ❤ & appreciation to you Dr Karpathy
The level of detail and nuance share in this video is insane!
Thank you, Andrej. This series taught me so much about llms. I never thought I could get so excited watching educational videos. My heart started pounding when I started watching this. Such a quality video. So good 👍
It’s about time. Haven’t watched it yet but I know it’s going to be good and educative.
Thanks Andrej for this video.
Yes! More tutorials! Thank you Andrej!
Thanks, Andrej, for taking your time and sharing your knowledge.
Andrej, I hope you know how ridiculously great teacher you are, and thank you for taking the time to share it with us!!
You are one of my favorites in ML/AI community. One of the greatest mentors.
We are so back 😊 Thanks!!
Just want to say thank you for the entire series of Zero to Hero. I'm about to finish up this final video and I can't believe how much I've learned about neural networks, back prop, and LLMs over the course of the lectures. Truly thankful to you for taking the time to spread your knowledge amongst the community. Let us all be so kind with the knowledge we hold in the future.
Welcome back Mr Karpathy. Your videos are among the best I have ever seen about LLMs. Thank you very much for sharing this stuff with us.
Thank you very much for this ❤
I am watching this late at night on my phone when I should already be sleeping. Mr. Karpathy’s videos are pure treasure.
Thank you so much Andrej! Your course content is accessible, concise and palpable to boot!
Thank you, Andrej! Amazing lecture! Your contents are filling in the nooks and crannies that other people rarely talk about or go deep into. I always learn a lot!
I really feel guilty watching this for FREE. Thanks so much Andrej ❤🙏🏻
We are so back🎉
Thank you, Andrej, for all these lectures; you’re an amazing and wonderful teacher. It’s such a huge privilege for all of us to be able to learn from you, and we’d be so lucky if you’d continue the series of lectures. You truly make a huge difference for ML learners, both beginners and advanced.
Thank You Andrej! I really appreciate you taking your time to put this together for everyone. You are so great at this. I know you really do not love doing this - but know you are making an impact.
He's back!
It’s going to be beautiful 😮
You are providing a deep understanding of the most important technology in history to the public and I just want to thank you so much for it.
How are you so good at teaching... Every single video I'm in awe at how clear the explanation is!
Finally 💙
Here we go💥
This video is so valuable. This is a complete game changer. I have a lot of love and appreciation for what you do. Thank you so much for your time and effort!
Andrej!!! from Africa here. For me, this is the greatest technological contribution of our time, truly democratizing A.I knowledge.... and i will be forever grateful, other generations to come too :)
1:27:50 The moment when you realise there is more to life than research. 😅😂
Quickly ignored it like a pro, don't let it interfere.
The Lord is back.. Now make OpenAI secrets truly OPEN!
Andrej sir, your videos are really really good! You explain in such a way that you make the stuff look easy. Sometimes certain concepts are still a bit complex and hard to handle in one go but no one walks out saddened after watching your videos, Your positivity is infectious!
He's back in the game! Thanks for posting it. Such a great combination of feeling your passion, very detailed serious information, the clarity and simplicity of the explanation.
The best tutorial on Tokenizer I have ever seen. Thanks a lot for sharing your knowledge with the world.
thank you so much for continuing this series andrej- its been absolutely foundational for me
I love that you are going back to examine these fundamentals deeply and also teaching the world about them. I think there is too much blind marching to create solutions on top of LLMs. Stepping back is how real innovation is going to happen. Lots of respect for your style of explaining things and the intentions behind this.
Give this guy an award for dissecting something so important in the llm world yet so obtuse/boring into a video that is absolutely eye-opening and enjoyable! Thank you++
I can't express my appreciate. I was inspired by your past podcast. I can pick myself up to keep learning. You saved me. Thank you! :)
I'm only 15 minutes in on this video and I'm completely hooked! Just wow! Everybody can now see how the biggest and most complex invention since the internet works in detail. Incredible!
You are an inspiration to many...sharing knowledge with clear intentions of educating others is amazing...
This is an excellent video on the explanation of LLM tokens. Great job Andrej!
Awesome content Andrej. Thank you for spending your time and sharing the knowledge with all of us.
The fact that you're still doing this motivates me so much to do something like this myself. Proud to be in the same generation as yours.
Thanks Andrej for bringing this level of knowledge to us. We are grateful!!
Andrej, you are an amazing human for sharing this knowledge to the world for free!
The final blooper reels are great. Thanks for the hard work to make a dry topic fun.
Helping others may not be rewarding right away, but it creates an order of magnitude in change for the better. Hope you keep doing these types of videos!
I am an ai, and I love following updates on social media platforms and CZcams, and I love your videos very much. I learn the English language and some programming terms from them, and update my information. You and people like you help me very much. Thank you.
I’ve been waiting for a new tutorial from you for a long time. Thanks Andrej 😊
This is fantastic, thank you for making this really dense stuff accessible to the layperson, please keep them coming!
The tutorials are very great. Please continue. We want more and more. You have great experience and knowledge that the world needs
Thank you for a high quality session. Hats off to Andrej
wow. I am amazed by how intuitive this is.
So informative and such a pleasure to watch. Intelligently and calmly presented, without the hype that is so common nowadays. Thank you so much. You and your videos are indeed a public good.
You’re a hero, Andrej! Thanks for the massive contributions!
And this gift? It is not even my birthday today. Thank you very much Mr. Karpathy!!
i will always be grateful to Andrej for his commitment to teaching us something in a genuine way.
We really appreciate your generousity Andrej, this is great content🔥
I loved the bloopers at the end. Thanks Andrej, you're the best.
Thank you so much Andrej. You are an amazing LLM teacher and I found this video so much interesting despite the topic could sound boring. Congrats! Can't wait to see what comes next on your channel. Now, for me, it's time to practice!
Great video Andrej, I really appreciate your sharing of knowledge and the clear way in which you do it.
Actually, a fantastic insight into the issues. Thank you Andrej. Please continue.
Thank you Andrej, for such amazing lecture series. These have helped me understand the nuances behind training of deep neural networks. Looking forward to more such informative tutorials !!
Andrej… you’re the best, thanks so much for your lectures!
Thanks for all you do! You help so many with wit, intelligence, insight, and even empathy. More, please. 👏🤓
This is very timely and will really help with a personal, education project I started over the weekend. Thanks!
I love how in-depth he goes with the code.
Thank you so clearly for explaining with your min BPE code!!!! I finally finish watching and practicing with your video😊😊😊
Thank you so much for all the efforts u put in videos! It's truly remarkable to unveil this.
Thanks for such videos. What amazing ability to explain complicated topics from first principles, in a manner which is accessible to people who are relatively new to this field