Deep Q-Networks Explained!
Vložit
- čas přidán 27. 08. 2024
- Let's talk about deep q-learning, a popular reinforcement learning algorithm
ABOUT ME
⭕ Subscribe: www.youtube.co...
📚 Medium Blog: / dataemporium
💻 Github: github.com/ajh...
👔 LinkedIn: / ajay-halthor-477974bb
PLAYLISTS FROM MY CHANNEL
⭕ Reinforcement Learning: • Reinforcement Learning...
Natural Language Processing: • Natural Language Proce...
⭕ Transformers from Scratch: • Natural Language Proce...
⭕ ChatGPT Playlist: • ChatGPT
⭕ Convolutional Neural Networks: • Convolution Neural Net...
⭕ The Math You Should Know : • The Math You Should Know
⭕ Probability Theory for Machine Learning: • Probability Theory for...
⭕ Coding Machine Learning: • Code Machine Learning
MATH COURSES (7 day free trial)
📕 Mathematics for Machine Learning: imp.i384100.ne...
📕 Calculus: imp.i384100.ne...
📕 Statistics for Data Science: imp.i384100.ne...
📕 Bayesian Statistics: imp.i384100.ne...
📕 Linear Algebra: imp.i384100.ne...
📕 Probability: imp.i384100.ne...
OTHER RELATED COURSES (7 day free trial)
📕 ⭐ Deep Learning Specialization: imp.i384100.ne...
📕 Python for Everybody: imp.i384100.ne...
📕 MLOps Course: imp.i384100.ne...
📕 Natural Language Processing (NLP): imp.i384100.ne...
📕 Machine Learning in Production: imp.i384100.ne...
📕 Data Science Specialization: imp.i384100.ne...
📕 Tensorflow: imp.i384100.ne...
If you like this video and you think I deserve it, please consider giving this video a like. Subscribe for more!
I second this statement
quiz 1=A
q2=B
q3=C
Where does the target network come from, and if it's the ideal "conscience" why not just use that? If we already have the ideal network, why bother training a second one?
Good question. Maybe my rhetoric was not super clear here. Essentially, without that target network, the Q network would compute the loss by comparing to itself. In practice, this can lead to unstable values as it is chasing a moving target. Hence a slightly delayed network is introduced to stabilize training .
Note this target network- isn’t the final iteration of ideal conscience. It is rather an iteration in the direction of ideal conscience. I say “ideal conscience” in this context to illustrate that the loss is computed based on this target network value. But this target network also gets better over time
The target network should be called the "snapshot network". It's simply an older version of the Q-network, over which you improve.
Sorry Ajay, I'm not sure I'm getting it. What do you mean an idealised network (you say Frank's idealised conscience)? Where does it come from? Looks like you say that's the actual solution (idealised conscience) but what's it's origin?
Great Video!! Thank you for the explanation. My question is, why not use the current state in the target network, instead of the next state?
A scenario a computer could benefit from learning on it's own: I remember Google reporting research on a model that used RL and was able to find more efficient assembly code for a sorting algorithm
It was AlphaDev
It'll be really ideal if we can have the quize's answers presented in the video instead of answering by comments since it might be inaccurate and there will be a huge time loss during the waiting for reply.
The quiz was a cool idea for understanding though, really helps.
Great video, this was helpful for me. The only thing that I found pretty confusing was the target network explanation, which I saw you address in another comment. You described it as the ideal conscience which really made it seem like its the optimal q-network that we're comparing to (which would defeat the purpose of training if we had that). In fact since gets updated every few batches, its less ideal that the q-network.
This was a good video, but I would love to see a deeper dive into your transformer series, that was the best, but I am still missing clarity on some of the steps. Your explanations are the best and would love to see more.
I have re-watched your videos atleast 10 times and have many questions, we need more of your explanations. Keep it up.
Thanks! Yea I am trying to get core concept videos out first and will soon love to dive into a series where I implement this system too :)
@@CodeEmporium Hey! A decision transformer video would be really appreciated
Quiz 2:
B. It stores Q for future reference
where the target network comes from? Thnaks
Love your videos mehn. They’ve really helped me understand the concepts
don't think a DQN outputs actions, that would make it a policy gradients.
It uses MC to collect Q values and use it for supervised training right ?
Amazing video and explanation! I have a question, Can I use SGD instead of MSE?
Thanks! SGD is an optimizer (algorithm that describes HOW a model learns) while MSE is a loss function (a function that describes WHAT to minimize). They serve different purposes. But in general, you can replace loss functions with appropriate counter parts. They may not work exactly as described, but they can work in general
Awesome explanation, thanks. Except the quizzes.
Is the target network also randomly initialized? Is it initialized with the same parameters as the Q-network?
From what I gather, the Q-network is acting as our behavior policy, and the target network is acting as our target policy. The way you describe it here makes it seem like the target network is already learned, but that would defeat the purpose of the algorithm in the first place.
I did not understand where the target network comes from? And if it exists, why should a new one be trained?
Good question. From my understanding the answer is more practical than theoretical.
The target network ensures the Q network isn’t chasing a moving target. If the network was compared against itself for every iteration, training would not be stable. Hence another slightly delayed network is introduced to ensure this stability
If you'd like to see your channel perform better, you might consider that your audience is composed of intelligent adults.
QT-1: Option A (by definition)
Brillant explanation, well done
why is it called a Q network and not just a neural network?
Amazing explanation 🎉🎉🎉🎉
Thanks for the explanation.
this video is great
GOOD EXPLAINATION
2:40 A
That’s right! Nice!
εισαι κουκλος .
✨Quiiizz Timmmeeee✨
quiz time 3 ka answer hai C , sahI khe rha hu na codemporium bhai
q1=A
B
A
A! Yep that’s right for Quiz 1
On my life, I've got no idea what you're on about, at any stage of the video.
It feels like you're jumping between concepts and not explaining how they're linked.
I'd rather you say what a QN is, describe how it works, then give the Frank example
QAnon Network.
teach him how to use a pencil
I am a pencil
Q3.A
Very cringey but good video nonetheless 👍
SHUDDAP
HE'S GIVING A GOOD VIBE