Why Fine Tuning is Dead w/Emmanuel Ameisen

Arguments for why fine-tuning has become less useful over time, as well as some opinions as to where the field is going with Emmanuel Ameisen.
This is a talk from Mastering LLMs: A survey course on applied topics for Large Language Models.
More resources are available here:
bit.ly/applied-llms
00:00: Introduction and Background
01:23: Disclaimers and Opinions
01:53: Main Themes: Trends, Performance, and Difficulty
02:53: Trends in Machine Learning
03:16: Evolution of Machine Learning Practices
06:03: The Rise of Large Language Models (LLMs)
08:18: Embedding Models and Fine-Tuning
11:17: Benchmarking Prompts vs. Fine-Tuning
12:23: Fine-Tuning vs. RAG: A Comparative Analysis
25:03: Adding Knowledge to Models
33:14: Moving Targets: The Challenge of Fine-Tuning
38:10: Essential ML Practices: Data and Engineering
44:43: Trends in Model Prices and Context Sizes
47:22: Future Prospects of Fine-Tuning

[@alaad1009]

Excellent conversation!!!

[@darkmatter9583]

RAG,quantize data? favorite LLM? HELP

[@mrwhitecc]

I do not think he understand what happened to the model after fine tuning. Just give one example here, if you have a unique reasoning pattern that there is no chance public pretraining dataset can contain the correlated data , then the SFT is the only way that you can let the model simulate the "reasoning" ability that you want the model to behave , prompt engineering do not help at all , RAG either.

[@agenticmark]

you fine tune for BEHAVIOR, you use RAG for DATA.
it fine tuning is how the model interacts with the user, rag is how the model gets factual information. that does not equal prompt engineering....

[@agenticmark]

strange, prompt engineering over fine tuning? if you dont want control-ability sure... prompt engineering will disappear. fine tuning will not.
i train voice and chat models (fine tuning) and I have trained dozens of agent foundational models that play nintendo and atari games and a bunch of classifiers. training from scratch (foundational, pretraining) is very very costly. fine tuning is not.

[@agenticmark]

_very_ unscientific claim about the lines on that chart. try trading stocks with that mentality of guessing it will just keep going up!