Quantization vs Pruning vs Distillation: Optimizing NNs for Inference

Four techniques to optimize the speed of your model's inference process:
0:38 - Quantization
5:59 - Pruning
9:48 - Knowledge Distillation
13:00 - Engineering Optimizations
References:
LLM Inference Optimization blog post: lilianweng.github.io/posts/20...
How to deploy your deep learning project on a budget: luckytoilet.wordpress.com/202...
Efficient deep learning survey paper: arxiv.org/abs/2106.08962
SparseDNN: arxiv.org/abs/2101.07948

[@thomasschmitt9669]

This was one of the best explanation videos I have ever seen! Well structured and right complexity grade to follow without getting a headache. 👌

[@bonob0123]

that was really nicely done. as a non-expert, I feel like I can now have a great general idea of what a quantized model is. thank you

[@DurgaNagababuMolleti]

Superb

[@lucaskeller656]

Great format, succinctness, and diagrams. Thank you!

[@muhannadobeidat]

Excellent video. Well spoken. Nice visualizations.

[@vineetkumarmishra2989]

wonderfully explained !!
Thanks for the video.

[@420_gunna]

This felt very nicely taught -- I loved that you pulled back a summary/review at the end of the video - great practice. Please continue, thank you!

[@unclecode]

Great content, well done. Please make a video for ONNX, and another one for Flash Attention. Appreciate.

[@huiwencheng4585]

Fantastic introduction and explanation !

[@jeremyuzan1169]

Great video

[@jokmenen_]

Awesome video!

[@heteromodal]

What a great video! Thank you!

[@user-bd7eq6vx1t]

your teaches so excellent.. we accepted many more videos from your side to understand for the fundamental NLP

[@kevon217]

^

[@user-qo7vr3ml4c]

Great summary, thank you.

[@kevon217]

Thanks for this!

[@hrsight]

nice video

[@MuhammadAli-dw7mv]

nicely done

[@yunlu4657]

Excellent video, learnt a lot! However, the definition of zero-point quantization is off. What you're showing in the video is the abs-max quantization instead.

[@EfficientNLP]

The example I showed is zero-point quantization because 0 in the original domain is mapped to 0 in the quantized domain (before transforming to unsigned). In abs-max (not covered in this video), the maximum in the original domain would be mapped to 127, and the minimum would be mapped to -128.

[@ricardokullock2535]

And if one was to quantize a distilled model? Is the outcome any good?

[@EfficientNLP]

Yes, these two techniques are often used together to improve efficiency.

[@andrea-mj9ce]

The explanation for distillation remains at the surface, it is not enough to understand it

[@EfficientNLP]

If you have any specific questions I’ll try to answer them!