Reinforcement Learning through Human Feedback - EXPLAINED! | RLHF

We talk about reinforcement learning through human feedback. ChatGPT among other applications makes use of this.
ABOUT ME
⭕ Subscribe: www.youtube.co...
📚 Medium Blog: / dataemporium
💻 Github: github.com/ajh...
👔 LinkedIn: / ajay-halthor-477974bb
PLAYLISTS FROM MY CHANNEL
⭕ Reinforcement Learning: • Reinforcement Learning...
Natural Language Processing: • Natural Language Proce...
⭕ Transformers from Scratch: • Natural Language Proce...
⭕ ChatGPT Playlist: • ChatGPT
⭕ Convolutional Neural Networks: • Convolution Neural Net...
⭕ The Math You Should Know : • The Math You Should Know
⭕ Probability Theory for Machine Learning: • Probability Theory for...
⭕ Coding Machine Learning: • Code Machine Learning
MATH COURSES (7 day free trial)
📕 Mathematics for Machine Learning: imp.i384100.ne...
📕 Calculus: imp.i384100.ne...
📕 Statistics for Data Science: imp.i384100.ne...
📕 Bayesian Statistics: imp.i384100.ne...
📕 Linear Algebra: imp.i384100.ne...
📕 Probability: imp.i384100.ne...
OTHER RELATED COURSES (7 day free trial)
📕 ⭐ Deep Learning Specialization: imp.i384100.ne...
📕 Python for Everybody: imp.i384100.ne...
📕 MLOps Course: imp.i384100.ne...
📕 Natural Language Processing (NLP): imp.i384100.ne...
📕 Machine Learning in Production: imp.i384100.ne...
📕 Data Science Specialization: imp.i384100.ne...
📕 Tensorflow: imp.i384100.ne...

[@RameshKumar-ng3nf]

Brilliant Bro 👌. Excellent explanation. I never understand RLHF reading so many books and notes. Your examples are GREAT & simple to understand 👌
I am new to your channel and subscribed.

[@neetpride5919]

Great video! I have a few questions:
1) Why do we need to manually train the reward model with human feedback if the point is to evaluate responses of another pretrained model? Can't we just cut out the reward model altogether, rate the responses directly using human feedback to generate a loss value for each response, then backpropagate on that? Does it require less human input to train the reward model than to train the GPT model directly?
2) When backpropagating the loss, do you need to do recurrent backpropagation for a number of steps that is the same as the length of the token output?
3) Does the loss value apply equally to every token that is output? Seems like this would overly punish some words e.g. if the question starts with "why" it's likely the response is going to start with "because" regardless of what comes after. Does RLHF only work with sentence embeddings rather than word embeddings?

[@0xabaki]

1) I think the point is to minimize the human feed back volume so humans just give enough responses to train a model for all future feedback. this way humans are not going to always have to give feedback, but instead will lay the basis, and probably come back to re-evaluate what the reward model is doing so it is still acting human
(2) and (3) seem more specific to the architecture of chatGPT and neither PPO nor RLHF. I would look into the other GPT specific videos he made

[@theartofwar1750]

At 6:58, you have an error: PPO is not used to build the reward model.

[@francisco444]

That's correct. The PPO algorithm is used to fine-tune the SFT model against the reward model scores, in order to prevent the model from "cheating" and generating outputs that maximize the reward score but are no longer normal human-like text.
PPO ensures the final RLHF model's outputs remain close to the original SFT model's outputs.

[@thangarajr-qw6wy]

(1) supervised fine-tuning (SFT), (2) reward model (RM) training, and (3) reinforcement learning via proximal policy optimization (PPO) on this reward model explain me

[@sangeethashowrya0318]

Sir ,please make a video on function approximation in RL

[@manigoyal4872]

Acts as a randomizing factor depending on whom you are getting feedback from

[@manigoyal4872]

what about the generation of rewards, will there be another model to check the relativity of the answer and the precision of the answer, cause we have a lot of data

[@0xabaki]

haha quiz time again:
0) when the person knows me well
1)D
2)B if proper human feedback
3)C

[@TheresaLopez-r7t]

Rodriguez Jennifer Miller Nancy Lewis Timothy

[@ayeshariaz3382]

where to det your slides?

[@ArielOmerez]

B

[@manigoyal4872]

Aren't we users are the humans in feedback loop for openai

[@akzytr]

Yeah, however openai has the final say on what feedback goes through

[@SysknShall]

Rodriguez Donna Miller Deborah Hernandez Frank

[@063harshsahu2]

looking like indian but accent like britisher, where u from bro ?

[@MichaelNeumann-n2v]

Brown Jennifer Jones Dorothy Lopez Shirley

[@ArielOmerez]

D

[@aswinselva03]

The video is informative and good. but stop saying quiz time in an annoying way

[@ArielOmerez]

C