NLP Demystified 12: Capturing Word Meaning with Embeddings

Course playlist: • Natural Language Proce...
We'll learn a method to vectorize words such that words with similar meanings have closer vectors (aka "embeddings"). This was a breakthrough in NLP and boosted performance on a variety of NLP problems while addressing the shortcomings of previous approaches. We'll look at how to create these word embeddings and how to use them in our models.
Colab notebook: colab.research.google.com/git...
Timestamps
00:00:00 Word Vectors
00:00:37 One-Hot Encoding and its shortcomings
00:02:07 What embeddings are and why they're useful
00:05:12 Similar words share similar contexts
00:06:15 Word2Vec, a way to automatically create word embeddings
00:08:08 Skip-Gram With Negative Sampling (SGNS)
00:17:11 Three ways to use word vectors in models
00:18:48 DEMO: Training and using word vectors
00:41:29 The weaknesses of static word embeddings
This video is part of Natural Language Processing Demystified --a free, accessible course on NLP.
Visit www.nlpdemystified.org/ to learn more.

[@futuremojo]

CORRECTION: at 1:33, that should be "20,000-element vector" .
Timestamps
00:00:00 Word Vectors
00:00:37 One-Hot Encoding and its shortcomings
00:02:07 What embeddings are and why they're useful
00:05:12 Similar words share similar contexts
00:06:15 Word2Vec, a way to automatically create word embeddings
00:08:08 Skip-Gram With Negative Sampling (SGNS)
00:17:11 Three ways to use word vectors in models
00:18:48 DEMO: Training and using word vectors
00:41:29 The weaknesses of static word embeddings

[@moistnar]

So I attend a really, REALLY prestigious university in the US and I took a course on Neural Networks this last term--this video series has higher lecture quality than that. You are very good at teaching these concepts

[@futuremojo]

Thank you!

[@caiyu538]

A lot of CZcamsrs teach better than a lot of “professors”.

[@michaelm358]

I have watched 5 videos on this subject in the last 2 days, and browsed dozens. This one is OUTSTANDING!!! By far the best i have seen. Wow!
I will do the whole NLP course. Very grateful for Huge effort it took

[@futuremojo]

Thank you! I hope you get a lot out of it.

[@kevinoudelet]

Thank you !

[@AshishMishra-rk4df]

Great work 👍

[@loicbaconnier9150]

Awesome !

[@lochanaemandi6405]

In SGNS, when you are talking about matrices of context and target embeddings (10000 * 300), what do these matrices have/contain before the training has started (collection of one hot encodings or arbitrary numbers)? At 17:00, I also did not understand how only taking the target word embeddings would be sufficient to capture similarity between words.

[@caiyu538]

I saw openai also provide embedding tool. It seems that this make easier than the old library such as NLTK,spacy, making them outdated? It make these concepts as a black box for us. We do not need to know in detail if only to use it.

[@futuremojo]

Absolutely. LLM APIs (even open source ones), hide all the details and make it easy for anyone to build NLP applications. We explore these APIs in part two and see how things like sentiment analysis can be done with a single line now.

[@caiyu538]

@@futuremojo Great, your lectures uncovered these concepts hidden in the Blackbox.

[@caiyu538]

@@futuremojo I also look forward to these lectures. Thank your lectures to know so many hidden concepts.

[@dmytrokulaiev9083]

Are you planning to do courses on other machine learning topics, such as computer vision?

[@futuremojo]

I probably won't build another course. This one took a year. I would consider more frequent, short-form videos though. What would you find useful?

[@dmytrokulaiev9083]

@@futuremojo Perhaps some material on diffusion models

[@hungreee]

@@futuremojo yes definitely