Twitter Scraping Tutorial in Python
Vložit
- čas přidán 4. 07. 2024
- In this video, Explained how to do twitter scraping in Python with out any API. Its a simple tutorial shows how to download tweets from twitter with Hashtag, Term and User Basis. And Finally, in this tutorial shown how to get Twitter user Profile Information as well.
#twitterscraping #pythonscraping #scraping
0:00 - 0:02 - Intro
0:02 - 0:36 - Explaining Scraping
0:37 - 9:51 - Scraping Tweets from Twitter
9:51 - 11:36 - Scraping Profile Information from Twitter
Link to Code - drive.google.com/file/d/1a2bI...
Please subscribe to our channel for more related Videos and Python Machine Learning, Deep learning Projects.
Link to Machine Learning Projects in Python
• Machine Learning Projects
Link to Deep Learning Projects in Python
• Deep Learning Projects
Hi Folks, Some times Instances might not be available to fetch tweets. In that case, try after sometime.
Thanks. mode = 'user' works well. But mode = 'hashtag' has error fetching.
Thank you very much bhai! been scratching my head for week
Thanks you. You gave me clear idea to implement on hedge fund strategy.
Thanks for the video, how can i use this to scrape data for a specific time period? Say from 2019-2021 specifically?
Legend bro, I needed this for an end of year project, bless you 🙏
i need to any idea how to start or somethin i tried my own code but gets banned acc
by any chance was it an A level NEA?
excellent video, man! it's of great use since that old way of twitter scraping stopped working after elon musk implemented some changes to twitter API policy.
Thank you 🤩
Do you know any methods to scrape the comments from a certain tweet? I would be so grateful if you provide info on this, thanks.
Can this be used for stock sentiment analysis project???
Thanku... It's very rare package... So much useful for academic research projects
I am glad it’s helpful
Hello, the mode "hashtag" works fine but the mode user give me Instances are unreachable. Check your request and try again. Can you help me ?
Hi KothaEd,
Thanks for a great explanation!
is there any way to pull replies of a user with this package?
Hello Sir. Great content, very helpful and rare tutorial. It will help in my research. However, i will like to download tweets from a particular region in the USA, i mean geo tagged tweets, along with certain hashtags. Please, how do i achieve this using this package and the function
Broo... thank you soooo much!! It helped me a lot
Glad it helped!
Hey Thank you for this nice tutorial :) Is it possible to include the location from the different tweets in the file?
Hello.. Your video is great!❤️... I have a question... How can I analyse/classify comments of any post of twitter just by pasting the link of that post?... Can you help me??
Thank you sir, is it possible to scrape quote of the tweet？
Hi bro thanks for this amazing video. Can you please tell is it possible to get list of latest 10 hashtags running
thank you so much sir
This Package isn't Working anymore i.e ntscraper cannot create more instances
Do you have any other alternative??
THANK U SOO MUCH BROOO LOTS OF LOVE
Thank you 🤩
Muchas gracias por su video me ayudaste mucho
Thank you so much
thank you so much
Is there any way to fetch user location, just the country?
Hi, thx 4 the video. Question: How do I use the 'near' param? Bc I only need tweets from Peru
At current moment, there is functionality in Ntscraper python library to restrict tweets based on a location
Thanks for this Video Brother.
Thanks 🤩
Cool video! I want to ask that whether the number of tweets scraping by this method is limited or infinite? Because I need a lot of data to conduct my research. Thanks!
Hey, did you resolve this? :) I need the same
Hai KothaEd thank u for the vid, i need to scraping data from replies (text, retweet, like, & quote) on a someone thread, can u make a tutorial for that? i cannt find anywhere, its for my academic thesis sentiment analysis.Tthanks before, have a good day!
The results seem inconsistent; sometimes it returns the data but often returns no data even without any error messages. Is this normal?
Hi, i want to scrape Twitter data based on spesific keyword but to do SNA analysis i need column "in_reply_to_screen_name".
Can it scrape that column?
I wanna know too
Hi, this was great tutorial, thankyou.
Could you please do a video on how to create an api that can be used on front end? Also what is the best way to store the returned data in a database? Is it advisable to let the api handle that or should it be handled by front end code?
I'm new to development/coding.
Thanks.
It depends on requirements. How frequently data is read by downstream systems.
How can i get the comments of the tweet? The text of the comments.
Is it possible to fetch tweets from quote tweets?
You're doing a great job!
I'm so OVERWHELMED trying to complete my dream app. But your video saved me like a lifeline :D
My Question: I'm creating a Python program to find users similar to my bio, but I don't have an idea how I can retrieve.
So, any advice from you made it happen.
Try to get bio of all users and match relevant users based on similar words in bio
@@KothaEd okay, now it seems tangible.
Here's the breakdown:
- Fetch 50 random users.
- Extract their bio (words).
- Match their bios with the target user's bio using the same words.
- In the end, return 10 people with similar words in their bios.
@@KothaEd hello sir how can i scrape all pages of that user on twitter ?
Thanks for this tool. I was also wondering if it was possible to also retrieve any images or videos associated with a tweet.
Yes you can from tweet content
getting this if i run your code "No module named 'ntscraper'", why is that ?
Thanks bro! How can we get follower and following list ?
Hi, first of all thanks for the great video!
Q: My code is returning too many fetching errors, is there a better way to do it instead the line below? Why does this occur?
scraper = Nitter(0)
I'm student, working on a scientific research, and I need a good amount of data. Thanks!
The Nitter class object is going to initiate instances which are actually fetch tweets from twitter. Yes it causes some errors as instances might not be available at the moment. Try running the same code after sometime.
ntscrapperr is not working
Thank you for your help! i have a question: can you scrap tweet replies? thanks again
same question
Is scraping millions of tweets still possible because i would like to build an AI model using the text data. Please if it's possible how can I go about it?
is there a max amount of tweets we could pull using this model? thank you for such an informative video!
Have tried for 100 as maximum and it fetched 100 tweets for me
thanks a lot it is working fine for me, i really needed this for my research project.
can i scrape top tweets or tweets during a time period instead of latest tweets?
Thanks!!! If you need tweets from a specific period, just apply filter on tweet date field to get tweets from a specific period
@@KothaEd How exactly do you do that?
@@KothaEdi want also comments is it possible?
Appreciated your hard work. Is that really possible to fetch replies section on X profile and convert it into dataframe.csv? Kindly acknowledge.
Yea that’s what shown in video
Great!, can we also scrape instagram or facebook in a similar way?
can we get tweets from multiple users on News feed, or we can only get tweets from particular user account at a time?
Yes tweets can be scraped from multiple users in newsfeed.
@@KothaEd How please?
Why do I only get 60 tweets? after that number it says that the page is empty.
Thanks a lot
Happy to help
I want to extract tweets of #obesity of last two years for my research paper, so how to do that?
Thank you very much! Is it possible to select a specific period of time?
Just scrape all the tweets from Python Library. Later apply filter on tweet time to get tweets from a specific time.
@@KothaEd like applying it within code I don't get it. can you please explain?
@@AltafKhan-ka1zer After extracting tweets from a specific keyword, you would get result set as tweets. On the result set apply filter on tweet datetime column.
As far as i know elon musk doesn't give permission to extract data from twitter is this the new way of getting limitless data ?
sir the ntscraper not fetching the tweets anymore is there any issue in it or please refer any other library that can be successfully scrap the tweeter.
Sometimes instances might not be available. Please do after sometime.
hello, were you able to get it to work? it doesn't work for me either
It is coming up with 'empty page on'... Were you able to get the tweets?
how do you fetch tweets with certain keywords on it from a specific time frame? awesome vid btw!
Just scrape all the tweets from Python Library. Later apply filter on tweet time to get tweets from a specific time.
Can i get a tweet detail from tweet id ? if yes, then what should we need to write in a code? please help me on this for my final year proejct.
Same doubt
❤
Hello, how are you, is it still working? It gives me an error that does not allow me to connect to the instances and does not allow me to scrape any tweets? Might you help me ?
I just checked now. It’s working fine. Please try now
thx a lot bro, can this package scrape Chinese hashtags or terms? I've tried it on my own, it seems the URL in the Nitter doesn't match....
I tried to scrape Chinese Hashtags and Terms but Nitter is returning with Empty results.
so this means this package can not scrape chinese?@@KothaEd
thanks so much!! I want to scrape all tweets with a certain term, not by a specific user, how do I do that? please help!
You can do that with term
Thank you, so should I put nothing before the mode?@@KothaEd
As for my project, I need to find users by the intersection of hashtags like Python, writer, AI, and study. But idk how to get the same results.
Please reply as soon as possible if you know something.
Thanks.. I have two questions. 1) Are we only going to have access to the public pages?, 2) when I choose more than 5 tweets , I faced the error. How much data can be easily obtained ?
By default it fetches 5 tweets, it can scrape more than that without any issues
One question. Do we have to login first to out twitter to use this package?
Not required
i am getting module not found error
Is it still working, please? I've tried today, but there is an error:(
I seem to be having the problem of all instances are being unreachable, do we have a solution for this ?
thanks
Sometimes instances might not be available. Please do try after sometime
Hi, does this still work? I'm currently trying to use get_tweets and it works when I don't specify the mode, but if I add mode ='user' it says "Error fetching"
Thank you for the tutoriel !
I have "error fetching" too. Have you solved this issue?
I am able to fetch tweets with mode = 'user'. Try checking now
@@KothaEd I still have the same issue right now... but thanks !
Hi nathan, you have the basic package ? because I tried to not use web scrapping and it doesn’t work it says I have limited access while I am on API V2
@@alyaalsalamah2151 I don't know, I have the version 0.3.5, is it the good one ?
thank you sir
Did it work?
@@HIMANSHUMISHRA-yf5bm It didn't
how can i get location of tweets
Hello bro, cool project. I have thesis project to make crime map from geolocation twitter. Can i scrapping geolocation (longitude latitude) from twitter ???
Ntscraper python doesn’t have functionality to filter tweets based on geolocation
ahh i see, do you know any method to get geolocation??@@KothaEd
the same goes for me, I need to fetch the location coordinates of the tweets. Have you been able to figure out how?
Can I limit the country where the hastag comes from? What is the tweet limit?
I highly doubt restricting tweets based on country but you can restrict tweets limit by passing number argument
what ide are you running this code on ?
its Jupyter lab.
Thanks for sharing! Why the creation of Nitter object requires testing instances please?
Instances are the actual scrapers to get tweets from twitter
It doesn't work, is there any other alternative?
is there a way I can scrape only posts with more than 1000 likes? I can choose to scrape all and then filter but this would be super ineffecient as there will be alot of redundant tweets from bots. Thank you!
I’m working on it
can we add multiple users in a single code? like if I want to extract tweets from more than one users?
Yes you can. Add list of users as array and pass it
how scrap number of views and retweet ... thanks
Hi, are you purchasing the API for this to work?
No it’s completely free
Is it possible to scrape tweets with multiple terms?
Yes it is possible. Add all search terms in to an array and pass it in place of search term
@KothaEd, It only works for 2 search terms.
like - scraper.get_tweets(["python", "writer"])
i need to scrape followers list is there a way ?
This feature is not yet available
Hello bro, can i get long lat scrapping from twitter ?
Sorry long lat ??
longitude and latitude@@KothaEd
Another vedio for insta and Facebook account
Sure
how to keep waiting for a new tweet from a particular user and print in real-time?
No scraper will provide live feed. You have to check for new tweets by a specific time frame
thank u sir, can u show how to do same with restriction of specific country's tweet data ?
At current moment, there is functionality in Ntscraper python library to restrict tweets based on a location
Hi sir kindly tell how to scrap fast food chain like kfc and Macdonald's customer review data from Twitter and facebook using python for educational purpose.
I shall be so thankful to you.
Regards
Ok Sure
how can i fetch tweets only in urdu?
how can i get the trending hashtags by this ??
At the moment, trending hashtags feature is not available.
Is it possible to fetch all the tweets of user
Yes it is possible
Hello bro, plz help in scraping email id
it isn't posible to use nitter anymore
Is there anyway to scraper twitter profile followers?
The mentioned python library helps you to fetch profile data like No of followers and No of tweets. Are you looking for list of followers ?
@@KothaEd yes list of followers
IT showing - No module named 'tqdm' it seems. Kindly help
Help I have the same issues too @KothaEd PLEASEEEEE URGENTTT
Some times instances may not be available. pls try after sometime.
only user is working upto 10 tweets its working beyond that it is not working.. outdated.
Does this thing still work?
Yes it is still working.
It's saying high retries detected, I tried many times
It happens. Please do try it after sometime
ntscraper is not working lol
Nitter is a discontinued now what
can you provide that file?
Code link updated in Description. pls check
@@KothaEd I can't find it
Here it is - drive.google.com/file/d/1a2bI9_PcYF8oN_hoDxOh1JCx1j1Tu-v9/view?usp=sharing
I got some problem while using it. here it is:
New working instances:
17-Nov-23 23:21:20 - All instances are unreachable. Check your request and try again.
Sometimes instances might not be available. Try after sometime
same problem
same problem22
I have tested and it works;
same problem i need solution fast please help me
bad tutorial
Very helpful
Glad it helped!
Thanks a lot
Happy to help