Video

Hi Jhon! I have a question, following your hiden api videos and some others, i finally finished a project that creates datasets with walmart products based on whatever the user wants the dataset to be about. I did this project using their hidden api, creating datasets that can get pretty big (15000 products), but for every dataset, i have to make around 100 and 200 get requests in order to get all the products. Is this legal/ethical to put it in my curriculum or in a linkedin post as a personal project even though in the walmart website says that they do not allow web scraping?

amazing explanation. thanks

This is gold

Funcionaría junto con selenium?

John please make it longer nd scrap data in csv file nd please use undetected or captcha solver methods to scrap data please. I love your videos. John❤

I'm sure you know this by now, but the Polars read_csv method supports glob patterns, so the loop approach is unnecessary - you can simply pass _folder + "/*.csv"_ as the source parameter and the concatenation will be done automatically.

How to solve this error? ModuleNotFoundError Traceback (most recent call last) Cell In[1], line 1 ----> 1 from selenium import webdriver 3 url = 'www.youtube.com/@JohnWatsonRooney/videos' 5 driver = webdriver.Chrome() ModuleNotFoundError: No module named 'selenium'

Thaaank you so much 🙏🏽🙏🏽🙏🏽, I've really learned too much from your videos. What if the api is protected by Cloudflare and sometime it gives unauthorized, is there a solution?

Hey dude jsut stared my first internhsip, and this video has been immensly helpful! I really appreicate the effort put in and all the useful tips. Thanks!

What if we want to take a full page so we can give it to an LLM to parse? For example, what if we were parsing financial filings or contracts. We want chunks or pages to pass to an LLM to structure outputs. I think splitting the text on a tag and then joining the items together would be best, but maybe there is a better way.

Do you have any videos on how to use proxies in Python?

Good one

I hope you have a course

Great content, very useful now that I am learning about this subject. You earned a new sub here

Great point about separation of concerns. As you stated, the scraper should only be concerned with getting data and saving data. I am curious what other use cases would be compatible with scrapy’s pipelines. Would pipelines be a good place for things like “save to this OTHER database”, or “upload to S3”, or “ping this api”? Will be diving into this myself soon but curious about your thoughts here.

What’s wrong with scrapping them as string and change them in the end in your output file?

Hi Johne i have a question can we use scrapy with django? i mean make the webscraper as online tool

Dude thankyou so much❤

Your content is good but i think you should engage with your audience more instead of speaking like you are talking to yourself. You will see that you will get much more views. Take Gotham chess channel for example he is not a Grandmaster of chess but His channels have more views and subscriber than Hikaru and Magnus because of his communication skills.

Hi John, Is there any way to get around cloudflare protected api and get json data in python.

I have the issue with cookie consent prompt? How could i solve this? in every new context it will pop up....

can you make a request based auto product buyer instead and show us how to do that?

thanks for the video, amazing introduction to webscraping

Unfortunately, I don't know how to translate this to JavaScript, so sadly, it's not very helpful to me. 😢

Title is misleading . The excel files I see are monsters in terms of complexity . Your example is seriously simplistic . 😊

Is it one core per instance?

Any amazon automation

Unfortunately, what I understand from this is that it only works for browsers... The company I worked at had some sort of CRM, very old, very laggy at every button press. With tons of pages and small fields to fill up. I REALLY wish I could find a way to automate the data entry of that one... It would replace all backoffices data entry people of italy xD

Honestly, This channel is marvelous. It has helped me a lot. 'a lot' is even an understatement

Hey bro I cloned a website and now I am opening that website code in vs code editor but after doing the necessary editing only text is changing not the images. Like I am putting my image URL on the place of website image URL but after saving it and opening it with live server the preview is showing me the images of cloned website not mine and in inspect element it is showing the image code of cloned website not mine why. I am trying from 6 hours and nothing is works for me. Will you plz tell me how can I change the images and edit it. Is this any api who is sending data from the backend.

Great content john, new subscriber here

I also like non scraping content, keep it up

👌

By Stock, I undestand something else - would be great to see some python automation on stock market.

what python website you are using ?

would be good to do an example where there isn't a schema on the webpage as more websites are now not using that

for me nothing beats monokai

Great video as always!Have you tried crawlee for Python?

i had this error b'400 - Bad request'

I kept getting a not defined on the line where it says "beer_list = [ ]" which I noticed you didn't why would that have happened?

Can you help me write a python script that will make a ppt that describes all the steps and processes for the SDLC of a team of 100 IT professionals? Thanks

The Zorr of Web Scraping.. Thanks 😊 giving this to us

Thank you for this. Learnt so much. The try exception in the function helped a lot as well.

Amazing

Great stuff

Just found your channel. Can’t wait to take a deep-dive. I have been automating and scraping for 20+ years and I’m hoping joining a community will step up my game into the future. Looking forward to learning w/ you.

Man, this is gold. Thanks for sharing!

You know if you clicked Preview instead of response it would have formated the json without having to go to a website

Very nice job. Good luck.

installing scrapy-playwright using pipx on linux is causing issues, this is the error: Traceback (most recent call last): File "/home/user/.local/share/pipx/venvs/scrapy/lib/python3.12/site-packages/twisted/internet/defer.py", line 1999, in _inlineCallbacks result = context.run( File "/home/user/.local/share/pipx/venvs/scrapy/lib/python3.12/site-packages/twisted/python/failure.py", line 519, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) File "/home/user/.local/share/pipx/venvs/scrapy/lib/python3.12/site-packages/scrapy/core/downloader/middleware.py", line 54, in process_request return (yield download_func(request=request, spider=spider)) File "/home/user/.local/share/pipx/venvs/scrapy/lib/python3.12/site-packages/scrapy/utils/defer.py", line 81, in mustbe_deferred result = f(*args, **kw) File "/home/user/.local/share/pipx/venvs/scrapy/lib/python3.12/site-packages/scrapy/core/downloader/handlers/__init__.py", line 83, in download_request raise NotSupported( scrapy.exceptions.NotSupported: Unsupported URL scheme 'https': No module named 'scrapy_playwright'