How To Scrape (almost) ANY Website with Python
Vložit
- čas přidán 21. 07. 2024
- Grab IPRoyal Proxies and get 50% off with code JWR50 at iproyal.club/JWR50
Using browser automation isn't generally my go to for scraping but sometimes it gives us an easy option for grabbing data. Scaling is an issue however, but combining playwright with scrapy gives us a good solid robust scraping method to add to our repertoire.
JavaScript to Scroll to the bottom of the page:
"setInterval(function () {var scrollingElement = (document.scrollingElement || document.body);scrollingElement.scrollTop = scrollingElement.scrollHeight;}, 200);",
Scraper API www.scrapingbee.com/?fpr=jhnwr
Patreon: / johnwatsonrooney
Donations: www.paypal.com/donate/?hosted...
Proxies: iproyal.club/JWR50
Hosting: Digital Ocean: m.do.co/c/c7c90f161ff6
Gear I use: www.amazon.co.uk/shop/johnwat...
Disclaimer: These are affiliate links and as an Amazon Associate I earn from qualifying purchases
timestamps
00:00 Intro
01:49 Playwright & Selectolax
07:14 Playwright & Scrapy - Věda a technologie
Grab IPRoyal Proxies and get 50% off with code JWR50 at iproyal.club/JWR50
Can you make a video about how to use it properly?
yes will do
@@JohnWatsonRooney thank you
I signed up for IPRoyal after watching your videos but couldn't get the static proxies to work. It returned timeout error. I tried changing the network settings but could not solve the problem. Do you have any idea what could cause such issues? Thanks.
You are a great tutor, and I suggest a video discussing and comparing all of these tools, why and when we could use them what is the best compo
great work keep making tutorials
Brilliant! love your attitude, admire an out-of-the-box thinker! Keep up the good work buddy!
Thank you!
Another great video! Thanks for showing both methods. 💯
Thanks mate!
You are the webscrape master
As a newbe... Does anyone have some experience with a PUP - a command line tool for processing HTML? Is there any way to import it to the Playwright project the same way as the HTMLParser? Thanks.
Hey, how can we scrape PDFs that are embedded to be viewed by chrome pdf preview? I think they use javascript.
It is playwirght faster and lighter than selenium?
Because i have knowlede about Selenium but i dont know if its worth to start acquiring knowledge abour scrapy and playwrigh. ¿Should i start?
Also, should i use playwright o splash? If i want to scrape data from dinamic and authenticated pages what should i use?
I would love and appreciate any information or advice. Thank you! Sorry if my english is not perfect
How to run a script from playwright in jupyter notebook?
can you please do a video on your neovim configuration
Great video John
I'm wondering if there is a way to use playwright with scrapy's shell? For me
scrapy shell
just seems to open the browser at the url and then block the scrapy shell from opening.
Any way to get around PerimeterX?
Great video content about webscrape. Your doing amazing bro.
I'm literally just getting started with python and need a fast study done for my thesis so I decided to study word usage on reddit. Should go through with it? Idk if i need any special stuff :/ I don't even have python installed. Cheers
This is so timely for me @John, as I was literally building a scraper yesterday to scrape a website that used XHR. Top content! Additonally, would it be possible for you to share the java script "code" that was used in the PageMethod function?
Thanks for reminding me I've added it to the description now!
Excellent tutorial! Thank you
Glad you enjoyed it!
hiii, where is the github codes that you are using
I watched the section between 4:30 and 5:00 (roughly) so many times.
The off-by-one space there was extremely distracting as well as satisfying when fixed.
Cheers
Next button is hidden for me. I am stuck
Thanks for another great video!
This method seems so easy and wanted to try it myself but unfortunately, it seems that scrapy-playwright doesn't work on windows. Some sort of Linux emulation (WSL) is required.
Also thanks for the iproyal discount. I was looking for such a service and your discount comes just perfect, will use it after NYE party :)
PS: Everyone, a Happy new year!
Hey, the video is really really helpful. Thank you very much for it! You are the go to channel for me whenever I wish to research on any topic related to web-scraping. You're doing a great job man!
Also, in the end of the video you said that this is not your preferred method for scrapping infinite scroll dynamic websites. So which one is your preferred method, which is also scalable?
Thanks I’m glad you’re enjoying the content! The way I mentioned is by reverse engineering the sites backend api and making requests to it - I have a few videos on my channel that explain the basics of this idea!
@@JohnWatsonRooney Oh alright. Yeah I’ve gone through those videos of yours and they indeed made my task a lot easier. So thanks for that too! XD
Need you to make a nvim setup video because thats cool af
instead of playwright, can we use splash for any projects? which are recommended for web scraping?
You absolutely can - it’s not as easy to setup and use well in my opinion but fits well into a specific use case. However I don’t think it’s been updated for a while and I’ve had some people tell me it hasn’t been working for them recently. Give it a go and if it works for you then great
Hi John.. What options do you suggest if I have to save screenshot of webpage as jpeg or as html itself.. Is it possible to with Scrapy
Yes! Where I have the pagemethods - you can add in another “screenshot” that will do it
@@JohnWatsonRooney Thanks John
This is nice, but my problem with using playwright is that it the twisted reactor always leads to issues when I want to run my spiders using python scripts
you could try using Splash? It hasn't been updated in a few years but may still work. Or create your own scraping/render service separate and use that?
brother, i'm in a big problem. last 20 day's i'm trying to scrape one of site. but i failed all the time. I watched 100 of videos. but i failed. can you scrape a site for me. if possible plz reply to my comments. this is my final year project. you just scrape me some data. my final year defense in knocking my door. plz brother if possible reply my comments.
Thanks
What is the IDE and theme used for this?. Nice explanation
It’s neovim and catpuccin theme
Great video John, which Editor are you using ?
It’s neovim, with the basic ide config by chris@machine
@@JohnWatsonRooney Thank you bud, I've been looking for an alternative to PyCharm and vscode for a while now
Which is the best for JavaScript rendered websites?
Selenium or puppeteer or playwright?
I use playwright mostly now
@@JohnWatsonRooney but I'm selenium expert, Should I use playwright instead of selenium or stick with the selenium according to your expertise?
@@casual_gamer1413 selenium! as you say you already know it, they do the same thing for web scraping purposes
@@JohnWatsonRooney thank you❣️❤️
Off topic, but will you do a video on websockets?
Good idea I’ll add it to my list
John, Is it possible to scrapy an Android or iOS app
Is that turle neck from uniqlo? It looks 👌
hah thanks - its Indicode brand.. but i got it from tk max
Hey how to scrape data from an Android application.
not something I've done before, but i know that the app will have a backend server/api that it makes the requests too, you;'d need to find this and reverse engineer it. or it might be possible to emulate the app on a pc/through browser?
@@JohnWatsonRooney most apps have ssl pinning security so we can not intercept it, to bypass this we can use nox player and man in the middle proxy to intercept from nox
Do you think chatgpt will put many of us out of commision??
Not anytime soon no
yes
why you don't use selenium?
I used to! It’s great.. I just spent more time with playwright and find it a bit easier to use
@@JohnWatsonRooney I use selenium it's powerful but some time it's some modules not working properly and thay makes me angry 😅 and I think I have move to new solutions but then I reminded myself that I have use proxies. But I don't like use of proxy I don't know why but I scares from using proxy.
Is their any free proxies?
I would really be interested on how to find api‘s 😊
i have a few videos on it on my channel! basically using the network tab in your browser to look for requests when pages are loaded
@@JohnWatsonRooney YEAH BUT IT DOES NOT WORK ALL THE TIMES.
@@kanwaradnan4849 for api usage it’s important to look at thenpayload that is sent. Is this a data form or a json payload? Also look good at the headers, this will fix 9/10 of your issues. Still doesn’t work? You probably forgot to fake some cookies :)
@@jesseroeleveld5430 yeah i found it buy watching video on this channel
@@jesseroeleveld5430 the reason I told the above statement is because I created a react app and it doesn't show my api using old tricks
If you're looking for a challenge, then try to scrape 9anime, It will be an interesting challenge. And also great content to watch. 🍿🍿
My friend did it with just requests.
@@techlogger hmmm yes it's possible 🤔, but you have to find the API.
@@skshaheen7506 yes. Not only api.. you have to solve heavy Java obfuscation too. It's more complicated than normal scraping
@@techlogger so after some digging with fiddler I have found the api and also able to get the video embedded url but couldn't get it to stream, since I have little JavaScript knowledge and unable to use the devtool (because they blocked it) this is as far as i can go for now, will try it later 😌.. and also as you mentioned this site use Obfuscated JavaScript.
@@skshaheen7506 you can unblock devtools, stream links ( m3u8 links ) are mostly restricted.. you have to pass proper headers, payload or in some case a decryption key. And yes that website is heavily obfuscated. But it's doable.
Can't say I'm first 😂
are you using lunar vim?
No but it’s the basic ide by the lunarvim author, it’s great