Which Quantization Method is Right for You? (GPTQ vs. GGUF vs. AWQ)
Vložit
- čas přidán 31. 05. 2024
- In this tutorial, we will explore many different methods for loading in pre-quantized models, such as Zephyr 7B. We will explore the three common methods for quantization, GPTQ, GGUF (formerly GGML), and AWQ.
Timeline
0:00 Introduction
0:25 Loading Zephyr 7B
3:25 Quantization
7:42 Pre-quantized LLMs
8:42 GPTQ
10:29 GGUF
12:22 AWQ
14:46 Outro
📒 Google Colab notebook colab.research.google.com/dri...
🛠️ Written version of this tutorial maartengrootendorst.substack....
🤗 Zephyr 7B on HuggingFace huggingface.co/HuggingFaceH4/...
Support my work:
👪 Join as a Channel Member:
/ @maartengrootendorst
✉️ Newsletter maartengrootendorst.substack....
📖 Join Medium to Read my Blogs / maartengrootendorst
I'm writing a book!
📚 Hands-On Large Language Models www.oreilly.com/library/view/...
#datascience #machinelearning #ai - Věda a technologie
Thanks a lot for clarifying the main differences between quantization methods and also for sharing your code.
Amazing content! Most youtube tutorials just go into trying out the outputs of pre-made LLMs but rarely dive into this level of technical details.
Thanks!
Thanks didn't want to feel too unproductive on a thanksgiving but didn't want to commit to a full video series. Always releasing timely and great stuff!
I was struggling with quantization last weekend! very timely! thanks
Thank you for the differences and the code.
Thanks for including the colab, and I wasn't aware of AWQ before this video.
Would you consider making a video on the efficiency on each, especially when using gpu on gguf model?
Useful information and well made video.
Outstanding! (To put things in perspective: I've seen a LOT of praise for wrapping the obvious or marketing-only BS into lengthy videos and I'm not shy to speak my mind there too!)
Thank you for the informative video. I understood how I made a huge mistake using gguf when having the VRAM to use GPU primarily.
Great content! I am wondering whether nowadays we should choose LLMs over BERT models on most tasks or use seperately based on specific use cases? That could be an interest topic to discuss!
Really enjoyed this session! Any chance you can continue this by showing how to fine-tune this versions of the models?
Really enjoyed your video. It was very informative. Just wanted to know can finetuning be done on these pre-quantized models ??
Always fun to hear Dutch people talk English and American at the same time, haha.
Nice informative video .
Thanks for the video! What I don't understand is that people always say that AWQ is faster than GPTQ, but in my 3060 12gb they are usually quite slow, around 3t/s, while in gptq I can get from 5 to 20t/s
Thank you so much for the video, i would like to know which method is faster at inference time.
i would like to load the new codellama models on to 2 a6000's, is there any good guides on how to do that in python? im reaching out after soo many attempts and hours of research, also i woul dlike to run in windows 11 and linux but guides on loading and running models from my own python would be awsome!
Great video and great comparisons. Can you make a video on how to quantize a model oneself as well?
inference is tooo slow on the T4 gpu on collab, i fed a football commentary transcribe text line to llm the pre quantized one, it took 3 minutes to obtain the result
Thank you for the great explanations :)
Does it make sense to do this before trainning? Quantize the model with these tecniques befor doing peft, qlora, p-tunning etc?
It definitely helps with training if the full model does not fit on the GPU. With many of these methods, efficiency is important and quantization is seldom not used.
Hi, WHich one is faster ?
GGUF with CUDA or AWQ ?
Thanks for this video - was a great explanation on the difference between the three models. How's the support for AWQ now? Also I would love it if you could make a video on how to deploy these quantized models for production
Great video, how ever its quite frustrating trying to run this code in production the dependencies are never correct.
Good one..
Brilliant video; you have a style that explains things nicely. Thank you. Sub'd.
If you are looking for ideas, I think an overview of what "weights, biases and parameters" mean for models would be great.
RuntimeError: cutlassF: no kernel found to launch! - it what I'm getting when trying to run your example at step 4:
outputs = pipe(
prompt,
max_new_tokens=256,
do_sample=True,
temperature=0.1,
top_p=0.95
)
My understanding was GPTQ is recommended when quantized model can fit entirely in vRAM and that with GGUF one can still offload layers to a GPU. If I wanted to try mixtral dolphin ~40GB version (On a system w/4080 16GB VRAM, 64GB RAM) what would be a better choice GPTQ or GGUF?
Definitely GGUF. Mixtral 4bit doesnt dit within 16GB Vram, so offloading layers would be necessary. I remember you could offload around 20 layers if I'm not mistaken. I do think the quantised variants, including the dolphin fine tune, are worth checking out.
@@MaartenGrootendorst Thanks!
what about exllamav2?
Well what about the title of the video? I still don't know wich one is right for me wich was the point of watching this video. All you did was explain what each method is, not wich one we should use :s
You have good information here in this video but still you missed the entire point of the title in my opinion.
well nevermind, I paused the video cause it was about to finish and go over to another on the playlist, and guess you do talk about it in the end, my bad. Thats what I get for not finishing the video before coming with stupid questions.
@@kiiikoooPT well i came here for similar reasons and although this is indeed very well explained, i still miss something i was hoping for (kinda like you): which quant works best for what mode? as i have read the other day that for example gptq is less suitable for rp than exl2. why? i dont know. thats why i came here. do the modes chat, cai and instruct have their preferred quantizations? if so, why?
im leaving without those answers but with further base knowledge, which is more important in the long run. but hey.. there is one answer i was interested in: why do some models assume both the character- and the user role and progress the rp-story themselves instead of letting me guidie it? because now i know thats sort of typical for the awq quant
from the litle I understand, the fact that a model prompt is diferent from model to model, has nothing to do with it being awq or gguf or whatever type off file, that is just about loading the model with diferent loaders. what you are talking about is another topic, you need to see videos about something like mistral vs mistral instruct. or diference betwen those. cause what you are saying has to do with how the model learned stuff, or how it was trained. the instruct models like the name says, are models that are based on instructions, so your prompt cannot be as a role play, cause it will give step by step anwsers instead of a simple conversation. What I really wanted to know is what kind of file is better for low end hardware, since my laptop has an nvidea graphic card with cuda cores, but is so hold that there is not even up to date drivers for it, so I can't install the stupid pytorch with cuda, only cpu mode, and I thought the type of file, gguf and so on, had diferent ways of loading the models, so I could manage to load it without the libraries everyone is using cause they have recent hardware. @@CitizUnReal