OpenAI CLIP model explained

CLIP: Contrastive Language-Image Pre-training
In this video, I describe the CLIP model published by OpenAI. CLIP is based on Natural Language Supervision for pre-training. Natural Language Supervision is not a new, in fact there are two approaches for this, one approach tries to predict the exact caption for each image, whereas the other approach is based on contrastive loss, where instead of predicting the exact caption, they try to increase the similarity of correct pairs.

[@AI_For_Scientists]

Great video series on vit and derivatives, watched all of it. Thank you very much for sharing.

[@PyMLstudio]

Glad you enjoyed it.

[@SebastianRaschka]

Very nice video! I can also imagine that predicting the caption text exactly isn't only more difficult but it would also be more likely result in (more) overfitting if it is learned this way.
At 5:43, the pair-wise similarities, they are basically like cross-attention scores?

[@PyMLstudio]

Yes, in a way, it’s analogous to cross-attention, taking dot-product between the features from the text encoder and image encoder. This dot-product similarity is used as the final output of the model to determine if an image and a text caption are related or not.
Good question, thanks for the comment

[@fouziaanjums6475]

Please cover FasterViT model too...

[@PyMLstudio]

Absolutely, I’ll cover that , I have a few other topics lined up, then I’ll get to FasterViT
Thanks for the suggestion!

[@randomstuff39280]

thank you for explaining! very clear!
but I'm wondering how do you know WiT dataset is based on 50000 queries and 20000 pairs for each query? I can't find it in the paper.

[@PyMLstudio]

Thanks for the comment!
Please see Page 3, section 2.2: Creating a sufficiently large dataset
But it’s 500000 queries, balancing 20000 (Image, text) pairs per query