That's an amazing resource! It's great to see you sharing such detailed information on a complex topic. Your effort to explain everything clearly will really help others understand and apply these concepts. Keep up the great work!
Great video, thanks for creating this. I have use DDP quite a lot but seeing the visualizations for communication overlap helped me build a very good mental model. Would love to see more content around distributed training - Deepspeed ZeRO, Megatron DP + TP + PP
Starting to watch my 3rd video on this channel, after transformer from scratch and quantization. Thank you for the great content and also for the code and notes to look back again. Thank you.
This is second video Ive watched from this channel after "quantization". And frankly wanted to express my gratitude towards your work as it is very easy to follow and the level of abstractions is tenable to understand concepts holistically.
well explained each and every detail, Great work Great Explanation👍 can you make this type of detailed video on distributed training through tensor parallelism? it would be very helpful. Thank you!
The video was very interesting and useful. Please make a similar video on DeepSpeed functionality. And in general, how to train large models (for example LLaMa SFT) on distributed systems (Multi-Server) when GPUs are located on different PCs.
In broadcast , if we are sending the copy of file from rank 0 and rank 4 node to other node. How is the total time still 10 second. Because still I am having same internet speed of 1MB/s. Could anyone explain? I am bit confused. Also what happens if I am having odd numbers of nodes
If time permits for you, Please make an video for entire GPU and TPU and how to them effectively and most of us donno . please create a playlist for pytorch for beginners and intermediates. Thanks for reading.
@@umarjamilai not sure if it’s in your plans, but if you are open to suggestions, I would love to watch a video on multimodal models. Again, awesome work!
Hi Umar, Great video and enjoyed thorughly but i have one question.why are we using the approach of sum(grad1+grad2+....+gradN), why cant we use Avg of Gradients.
Of course you can (but you don't have to) use the average of the gradients. Actually, people usually take the average of the gradients. The reason we use the average is because we want the loss to be (more of less) the same as the non-distributed model, so you can compare the plots of the two. I don't know if PyTorch internally automatically takes the average of the gradients, I'd have to check the documentation/source.
Wouldn't the accumulated gradient need to be divided by the total number of individual gradients summed (or the learning rate needs to be divided by this value) to make it equivalent?
Yes, if you want to treat the "cumulative gradient" as a big batch, then you'd usually divide it by the number of items to keep it equivalent to the single-item setup. But it's not mandatory: as a matter of fact, loss functions on PyTorch have a "reduction" parameter, which is usually set to "mean" (so dividing the loss by the number of items) but can also be set to "sum". One reason we usually calculate the "mean" loss is because we want to make comparisons between models with different hyperparameters (batch size), so the loss should not depend on the batch size. But remember that mathematically you don't have to
In my understanding, yes the loss is accumulated for one batch theoretically, and the gradients are computed based on this accumulated loss too. But in the parallel implementation, both the loss calculated in the feedforward process, and the gradients calculated in the back propagation process executed in a parallel way. Here @umarjamilai use a for loop to illustrate the de facto parallel mechanism.
That's an amazing resource! It's great to see you sharing such detailed information on a complex topic. Your effort to explain everything clearly will really help others understand and apply these concepts. Keep up the great work!
Great video, thanks for creating this. I have use DDP quite a lot but seeing the visualizations for communication overlap helped me build a very good mental model.
Would love to see more content around distributed training - Deepspeed ZeRO, Megatron DP + TP + PP
Dang. Never thought learning DDP would be this easy. Another great content from Umar. Looking forward for FSDP
Starting to watch my 3rd video on this channel, after transformer from scratch and quantization. Thank you for the great content and also for the code and notes to look back again. Thank you.
I really love your vidoes. you have a natural talent on simplifying logic and code. in same capacity as Andrej
This is second video Ive watched from this channel after "quantization". And frankly wanted to express my gratitude towards your work as it is very easy to follow and the level of abstractions is tenable to understand concepts holistically.
Super high quality lecture. You have a gift of teaching, man. Thank you!
this channel is hidden gem
absolutely amazing! You made these concepts so accessible!
Incredible content, Umar! Well done! 🎉
well explained each and every detail, Great work Great Explanation👍
can you make this type of detailed video on distributed training through tensor parallelism? it would be very helpful. Thank you!
Really impressive!
Amazing content! Thanks for your sharing
Another great video, Umar. Nice work
The video was very interesting and useful. Please make a similar video on DeepSpeed functionality. And in general, how to train large models (for example LLaMa SFT) on distributed systems (Multi-Server) when GPUs are located on different PCs.
very nice and informative video. Thanks
Thank you very much for your wonderful video. Can you teach a video on how to use the accelerate library with dpp?
Thankyou so much for this amazing video. It is really informative.
Amazing learning stuff ! Very Thanks !~ 🥰🥰🥰
You deserve many more likes and subscribers!
Awesome video. Please make tutorial on FSDP as well
In broadcast , if we are sending the copy of file from rank 0 and rank 4 node to other node. How is the total time still 10 second. Because still I am having same internet speed of 1MB/s.
Could anyone explain? I am bit confused.
Also what happens if I am having odd numbers of nodes
you teach soooooooo good
Thank you so much for this
Thanks!
谢谢你！我们在领英connect吧
Great video
Please create a video on model parallelism and FSDP.
If time permits for you, Please make an video for entire GPU and TPU and how to them effectively and most of us donno .
please create a playlist for pytorch for beginners and intermediates.
Thanks for reading.
Federated learning basics please.❤
You rock always 😂
I wish i could like it twice
You can share it on social media. That's the best way to thank me 😇
@@umarjamilai not sure if it’s in your plans, but if you are open to suggestions, I would love to watch a video on multimodal models. Again, awesome work!
Check my latest video!
fantastic
great video
Great!
Great intro video. Do you have any plans to also cover other parallelism: Model, Pipeline, Tensor, etc.
please do some thing related to audio large models like conformers,quartznet ,etc
Working with fsdp and megatron now and I really want to figure this out from scratch haha, it sounds fun but a big headache
I'm always confused with DP and DDP. Can you please tell me the difference between them? While both of them belong to data parallelism method.
DP only works on a single machine, while DDP can work on multiple machines. However, PyTorch now recommends using DDP also for single-machine setup.
@@umarjamilai thank you for your reply
could provide another videos with respect to model parallel and pipeline parallel ? thanks..
another banger
SUUUPERRRR
Hi Umar, Great video and enjoyed thorughly but i have one question.why are we using the approach of sum(grad1+grad2+....+gradN), why cant we use Avg of Gradients.
Of course you can (but you don't have to) use the average of the gradients. Actually, people usually take the average of the gradients. The reason we use the average is because we want the loss to be (more of less) the same as the non-distributed model, so you can compare the plots of the two. I don't know if PyTorch internally automatically takes the average of the gradients, I'd have to check the documentation/source.
@@umarjamilaithanks for the info.
How to do in Kubernetes? Please explain it.
Wouldn't the accumulated gradient need to be divided by the total number of individual gradients summed (or the learning rate needs to be divided by this value) to make it equivalent?
Yes, if you want to treat the "cumulative gradient" as a big batch, then you'd usually divide it by the number of items to keep it equivalent to the single-item setup. But it's not mandatory: as a matter of fact, loss functions on PyTorch have a "reduction" parameter, which is usually set to "mean" (so dividing the loss by the number of items) but can also be set to "sum".
One reason we usually calculate the "mean" loss is because we want to make comparisons between models with different hyperparameters (batch size), so the loss should not depend on the batch size.
But remember that mathematically you don't have to
shouldnt loss be accumulated ? loss += (y_pred - y_actual)^0.5
In my understanding, yes the loss is accumulated for one batch theoretically, and the gradients are computed based on this accumulated loss too. But in the parallel implementation, both the loss calculated in the feedforward process, and the gradients calculated in the back propagation process executed in a parallel way. Here @umarjamilai use a for loop to illustrate the de facto parallel mechanism.
do you have a discord channel?
Always great to watch your video, excellent work