How I Scrape 7k Products with Python (code along)

A short but complete project of scraping 7k+ products with Python.
johnwr.com
➡ COMMUNITY
/ discord
/ johnwatsonrooney
➡ PROXIES
www.scrapingbe...
proxyscrape.co...
➡ HOSTING
m.do.co/c/c7c9...
If you are new, welcome. I'm John, a self taught Python developer working in the web and data space. I specialize in data extraction and automation. If you like programming and web content as much as I do, you can subscribe for weekly content.
⚠ DISCLAIMER
Some/all of the links above are affiliate links. By clicking on these links I receive a small commission should you chose to purchase any services or items.

[@abdulrafay1951]

Please do share how to speed it up and other whatever stuff you do in real life. Thanks for sharing :D

[@tree3d22]

Fantastic video, I've always appreciated your expert approach and brief but clear explanations.

[@andi.herlan]

Hi John, thank you for your outstanding videos especially in web scraping topic. I still cannot support you as that much, but I always recommend your channel when someone ask me where to learn scraping in Python.

[@JohnWatsonRooney]

Thank you very much - just watching is support enough and very appreciated!

[@runnrnr]

NIce video! Added it to the selectolax repo as another example.

[@JohnWatsonRooney]

Awesome thanks!

[@cokemaya]

Great video as usual! what would be the best approach to filter down these urls if they already exist in our database?

[@JohnWatsonRooney]

Thanks. I would just add a function to check if they exist and if they do remove them from the urls list

[@RicmodUttara]

Next video is showing how easy this is to do in scrapy?

[@JohnWatsonRooney]

Yup was released yesterday

[@Aaron-is9ee]

Wouldn’t you want to change the headers too?

[@deepbrar1]

Hey john, I created a bot using selenium but i wanted to deploy it somewhere. I tried to deploy it on pythonanywhere but wasn't able to do that because for installing chrome for selenium i required root access.
I also tried with Playwright but pythonanywhere have a firewall so they blocked the ip when i ran the command "playwright install chromium"
So the question is where am i supposed to deploy them for free ??
I really want to make my project go live....

[@tree3d22]

Upload the selenium driver with your script

[@hariharang9221]

Switched to helix ?

[@JohnWatsonRooney]

Yup, a while back now

[@bakasenpaidesu]

೩ comment

[@theterminathan92]

second comment

[@aijokker]

Why people scrape products from some websites?

[@jw200]

Nobody says that, probably earning money somehow. Nobody tells about why tehy scrape.

[@streamingbroadcastwithsonal]

First Comment