Scrapy in 30 Minutes (start here.)

Join the Discord to discuss all things Python and Web with our growing community! / discord
This is the 5th video in the learn web scraping series, learning to use Python's premier scraping framework, Scrapy. We will redo the project from scratch and compare the code we have written to how it looks in Scrapy.
This is a series so make sure you subscribe to get the remaining episodes as they are released!
If you are new, welcome! I am John, a self taught Python (and Go, kinda..) developer working in the web and data space. I specialize in data extraction and JSON web API's both server and client. If you like programming and web content as much as I do, you can subscribe for weekly content.
:: Links ::
Recommender Scraper API www.scrapingbee.com/?fpr=jhnwr
My Patrons Really keep the channel alive, and get extra content / johnwatsonrooney (NEW free tier)
I Host almost all my stuff on Digital Ocean m.do.co/c/c7c90f161ff6
I rundown of the gear I use to create videos www.amazon.co.uk/shop/johnwat...
Proxies I recommend nodemaven.com/?a_aid=JohnWats...
:: Disclaimer ::
Some/all of the links above are affiliate links. By clicking on these links I receive a small commission should you chose to purchase any services or items.

[@vouky8747]

Hey John, thank you! You actually helped me with a project I was stuck on. Great video!

[@DietervanderWesthuizen]

Excellent video once again. Thanks a million John.

[@elu1]

What a wonderful series, gentleman! love it ! Now I think I am more knowledgeable to be able to follow along with other stuff from this channel. Thank you very much!

[@JohnWatsonRooney]

Glad you enjoyed it!

[@faustozambrano4901]

Great stuff... Thank you John

[@flaskwater44]

Great Tutorial!

[@TheBenchPressBoss]

Incredible scrapy video big JOhn thanks homie !

[@JohnWatsonRooney]

thanks!

[@talhanadeem1720]

Amazing Man. I'm surprised! yoo Thanks

[@tomlento6068]

great intro!

[@julianvargas228]

great video!!

[@oc3academy]

Great video. Could you please share your neovim configuration? Or what LSP are using?

[@surajsinghrajput5862]

Great tutorial John, thanks for sharing. I am trying to scrape a website that requires a login. Am able to do that by defining a function inside the class but not able to figure out how to scrawl from that point.

[@alan_tucker]

Another great video; keep up the great content.

[@JohnWatsonRooney]

Thanks mate appreciate it

[@ketankumar5689]

I am a newbie in scrapy. I am trying to access some info. in a job site(Monster) like jobtitle, company name, posting date etc.present in a job card through scrapy shell command but i am unable to do so and getting empty list even though i provide exact classname. What should i do or any video has been created to access such elements? Any help 🙏
P.S. - i tried and am able to access some elements in header and footer section but unable to access elements from cards which display info of each job.

[@valuetraveler2026]

How about deployment? Any tips on which is best for custom deployment? A video perhaps?

[@BeyonderW]

Can you do a scrapy tutorial with C# ?

[@daved8698]

old man I just saw your videos 3yrs ago then hansome

[@bakasenpaidesu]

OP ❤.
I thought scrapy will be overwhelming but it's great

[@JohnWatsonRooney]

Thanks 😅 also neovim config soon ;D

[@bakasenpaidesu]

@@JohnWatsonRooney I'll be waiting for that :)

[@phumudzomuvhango4193]

Thanks boss

[@mmcmobile4869]

🧡

[@loicleray]

Fuarrrk yeah. Thank you.

[@michaelmuolokwu5039]

Amazing video!
Are there any drawbacks in using crawlers instead of using normal spiders?

[@JohnWatsonRooney]

Thanks! No, they just have slightly different roles - use whichever suits your needs

[@mecrayavcin]

🥰Excellent video. Thanks
I'm confused about scraping Javascript based sites. Could you please make a fresh video about it.
Thanks very much love you.

[@user-bc5ye9vs7e]

open dev tools, go to network tab, press perserve logs. refresh page, click stuff on page and see which request has data u need. Or be a scrub and use browser 😛

[@SquirtleBaiano]

I had to use selenium to bypass this same problem easily, but I'm curious about it aswell

[@alebeatz_5179]

Whats system John use ?

[@JohnWatsonRooney]

Linux, i3wm and neovim

[@soul_maestro]

17:24 : "working just fine" while the price field isn't the price at all,
but it look like the tilte or a description - even the one hoovered over after stopping it a couple seconds later...

[@JohnWatsonRooney]

I had the wrong selector which I fixed later in the video

[@valuetraveler2026]

Where do good scraper engineers advertise their services?

[@prohacker5086]

CZcams

[@franciscooteiza]

ebutuoY

[@valuetraveler2026]

You mean here? I dont see any.@@prohacker5086

[@valuetraveler2026]

I found Scrapy to be much overkill but maybe they have improved it since?

[@JohnWatsonRooney]

depends on the use case but it shines for crawling and managing multi spiders in my opinion

[@valuetraveler2026]

yes I can imagine thats where it counts@@JohnWatsonRooney

[@daved8698]

ctrl +c ; ctrl +v

[@user-bc5ye9vs7e]

Scrapy, more like Crapy 🙈

[@rastaricky9584]

Way too much jumping around in this video…from shell, n I’m, to documentation…also the dialogue is everywhere…but thanks for taking the time to

[@sandunwijethunga6787]

Your the best John. thank you so much