Video

You're welcome!

You're welcome!

Hey @MrNadir, it is not reserved by the spaCy library!

Happy you found the video helpful!

Hi @tiny8999 there is no ready-made component for "paragraph segmentation" in spaCy, but you could try looking for two line breaks or a tab character in your document to identify the paragraph breaks?

Hey @haiyangai6057! A good question ... you could try providing your document to Stanza and disable all other components except tokenization and sentence segmentation. However, even if your documents are longer, you should be able to speed up the process by providing them to Stanza in a list for batched processing.

You can also pass a list of strings, of which each string consists of multiple sentences: ["This is the first text. It contains mjltiple sentences.", "And this is the second. Also multiple sentences."] The stanza tokenizer will perform sentence segmentation.

So yes, you can pass in a whole text document by reading it in as one single string.

Yes, as soon as we can sort out access to the learning environment for all users! The course should be open to everyone around the turn of 2023/2024!

Glad you like them!

I take that as a compliment. :-)

Thanks! Great to see more linguistics content on CZcams!

Hi @seanosuilleabhainemerald, yes, check out the details here: spacy.io/api/token#set_extension

Hi @brandongolub4454, unfortunately this kind of "morpheme-level" tokenization is not possible!

Glad it was helpful!

Thanks! Could you be a bit more specific? Does the code raise an error?

Glad you liked it!

Glad you found it useful!

One solution would be to create several example sentences such as the one you mentioned, before parsing them with spaCy! This gives you an idea of the linguistic patterns that should be captured by the syntactic Matcher.

Glad you liked it!

Hey! You can find some instructions here: huggingface.co/docs/hub/stanza If you have the model somewhere else than the default directory, use the "dir" parameter when initialising a Pipeline, e.g. nlp = stanza.Pipeline(lang='en', dir='path/to/your/directory')

Hi Rizki! This video is about techniques for working with multiple files - it does not matter what their type is!

Hey! No, annotations for named entities are stored into Doc objects, because they often span multiple Token objects. Always check if the model has actually located any named entities, e.g. import spacy nlp = spacy.load('en_core_web_sm') doc = nlp("Paris") print(doc.ents) If no named entities are detected, spaCy will return an empty tuple!

Glad you found it useful!

You're welcome! In this case, it's cosine distance!

Hey @Dormadum, remember that the videos are to be used in connection with the learning materials, which help to clarify the concepts: applied-language-technology.mooc.fi/

Hey! You can find all notebooks here: github.com/Applied-Language-Technology/notebooks

Using spaCy? See the video here for how to extract sentences from longer pieces of text: czcams.com/video/NknDZSRBT7Y/video.html

Maybe you can use regular expressions, but you cannot process Word files using Python.

No, you need to extract plain text from the PDF first.

@@AppliedLanguageTechnology how do I do that with python?

Google is your friend, there are plenty of alternatives.

when I search for this in google, they think I am looking por .pdf file book. If you have any link please share :(

Hi! I don't know how long your documents are, but passing them to a language model will give you one vector per token. You can access a vector representation for the entire Doc object through the attribute "vector", but please note that these representations are likely to become diluted if the Doc is very long. For a better alternative, you should look at algorithms such as Doc2Vec.

Hi Paola, yes, if you teach the model to predict those attributes during training - you can find an example in another repo of mine, which predicts labels for Span objects: github.com/thiippal/MoodCat

@@AppliedLanguageTechnology Awesome. Thank you very much for your guidance and for providing an example.

Hi Tuomo. I have started implementing the spancat, but I am getting to some issues with the labeling, which means maybe my Example object have some problem. I have set up a question in Stack Overflow. I am sharing it here in case you can give me some light. You are the more knowledgeable person I know about spaCy. Thank you so very much in advance.

Hi! Can you be a bit more specific? If you're looking to define a pattern that provides the best matches, then the answer is to study your data! There is no one-stop solution.

Hi Shane! Check out the materials here: applied-language-technology.mooc.fi/html/notebooks/part_iii/02_universal_dependencies.html#exploring-syntactic-dependencies-using-spacy Because the dependency relation points towards the head word (e.g. object), you can use the attributes of the Token object to retrieve the subtree.

Hi Paola! I assume you want to use scikit-learn for the logistic regression model? In that case, you need to think about the whole pipeline: use spaCy for tokenization (and perhaps removal of stopwords), which gives you the input to tf-idf vectorizer. Adding the tf-idf vectors to the spaCy Tokens would simply complicate things, as the final model will be in scikit-learn anyway!

Thanks!

Sorry for the belated reply (on holiday)! Sure: get the Token indices for matches, then get the indices for the sentences using the start and end attributes, and see if the Token indices fall within the range.

Actually, you can also check if a Token is contained within a Span (e.g. a sentence under Doc.sents) using the following expression: Token in Span, which will return a Boolean. You could loop over the Tokens returned by the DependencyMatcher and return the sentence at the index that has the value True.

Yep, because the dependency relation will point towards the head of a phrasal unit, you can use spaCy's Token-level attributes to fetch the dependents of that head: applied-language-technology.mooc.fi/html/notebooks/part_iii/02_universal_dependencies.html#exploring-syntactic-dependencies-using-spacy

Hi Ildar! Yes, but you will have to use the DependencyMatcher - I have a video here: czcams.com/video/gUbMI52bIMY/video.html

Hi Robby! I assume you have created a DependencyMatcher that contains the pattern that you want to match, and your data is in a pandas DataFrame? Create the DependencyMatcher object first, and then use the "apply" function with a lambda function to apply the matcher, e.g. df['matches'] = df['text'].apply(lambda x: matcher(x)) This code stores the result into a new colun named "matches".

Thank you! For the solution as well as the quick response. If I may ask another, since optional operators are not allowed in dep matcher, is there a work around to incorporate multiple patterns? The records I’m searching have the verb-noun as a dobj and other sentence structures it’s an amod. There are also cases where 0 or more compounds are needed. I was thinking I may have to run the first pattern then the other patterns without overwriting; I can research that option. But am curious if you are aware of another method. Again thank you for sharing your knowledge

Hi again, happy to hear that you solved the problem! As for the second question, I think the most optional solution is to define a single DependencyMatcher and then add multiple patterns to the same matcher. Also remember that the DependencyMatcher returns indices of Token objects - you could, for, example, retrieve the dependents of those Tokens before applying a normal Matcher to that Span. For more information on exploring the dependencies, see: applied-language-technology.mooc.fi/html/notebooks/part_iii/02_universal_dependencies.html#exploring-syntactic-dependencies-using-spacy

Hey! I've been thinking about creating some advanced content in the future, such as training spaCy models. What would you like to learn about interfacing HuggingFace with spaCy?

@@AppliedLanguageTechnology Hey!, I think it would be very interesting to see how to use a transformer model, and then for example fine-tune it with new entity types. Or for a second tutorial to train a transformer model from scratch (if that is even possible with spacy).

Glad you found it useful!

Thank you so much!

Good question! Keep in mind that these materials are intended for beginners, who may not have the skills to install the required libraries and may also find an IDE intimidating. For this audience, I think the Notebooks are a perfect choice, because they allow executing the code step-by-step. For actual development, I would naturally recommend getting familiar with an IDE such as Spyder or PyCharm.

@@AppliedLanguageTechnology Cool answer - make sense. Good for kids then BUT for adult ... it's more like an embarrassing for not knowing how to install/use the dev IDE. A self insulting attitude