Video

Hello! I am tasked with scraping the entire website for all the properties, that's half a million I believe. I'm a junior dev and I stumbled upon your video after having no luck for a few days, and this is the first time I have to scrape a website. First of all, is this even possible? If so, can it be done in pure vanilla JS? Or are external libraries needed for this many properties? Ideally, I need to write the script in vanilla JS without using any libraries because I need to deploy it as a Worker on Cloudflare that will fetch the data as JSON and then send it off somewhere else. I'm feeling like this is a lot of data to scrape. Thank you and have a nice day :)

grande!

Hey mate, when I try to paste it I get a warning that avoids me pasting the code into the console, this is the warning: Warning: Don’t paste code into the DevTools Console that you don’t understand or haven’t reviewed yourself. This could allow attackers to steal your identity or take control of your computer. Please type ‘allow pasting’ below and hit Enter to allow pasting. Is there any step I messed up?

I need bypass bot 🤔

possible to scrap high frequency data like sports odds?

How do you automated it?

do you have a codepen for the tiktok.js file?

I think if you made a guide of the 10 most popular requests you have gotten and why there is value for clients (maybe it's lead generation or it saves 100 hours of time), I would watch and share that video

Hey, I'm really interested in this. I'm trying to sign up on the website but the sign up links are just looping back to the top of the home page. Are you still working on this? I'm part of a few PUA groups and can get you a lot of sign ups if this works well

If you can share anything that can be used for free to get notifications, it would be really helpful

Can you share the script you use for notifications on your mail

Thank you, Have you succeeded in creating a solution that extracts both the images and all the reviews?

I deployed my express app with pupeteer on vercel. It cannot find the chromium, will using spartacuz work ? or this is a completely different thing for AWS only ?

hi , the link of the script is corrupted, can you share a github repo ? thanks

thank you sir very useful

Great tutorial, Do we have a CDK implementation of this, looking for something similar

Does this still work?

Hey Adrian Thats helpful. Thank you very much. Im just wondering if its possible to scrape all the posts in a subreddit (not just the initial 1000).

Great video! I was wondering if you have a bot that scrapes the pre foreclosure listings?

wow!! this is so awesome!

my fear its that google detects thats a bot. what do you think?

thank you @the-web-scraping-guy . it works very good .

I use the same method scrap chrome extension, but only scraped 1524 extension at most, why?

"Definitely with Big Miss there" lmaooo. Great vid!

out of curiosity, is this legal to get someone's data like that?

I've seen several people say that this doesn't work. My first attempt was the same. The csv file had no data in it. I then tried it on a different Facebook group and it worked. I don't know why it works on some but not others.

Hi Adrian, how does the chrome launch for you, it doesn't launch for me after I run the code. It might be a silly question, I just started learning node. Thanks in advance!!

I am trying to do but getting CSRF failed. Please suggest

what do you use to record your beautiful videos

Thank you so much on that. Finally get my puppeteer to work on Lambda

Thank you very much on this. Finally get the puppeteer to work on Lambda

Thank you so much, really helpful!

the best web scraper

I am trying to generate PDF from HTML Using puppeteer with Lambda but not working can you share any solution

Adrian, the video link seems to be private. Can you share it? Thank you!

Hola, muchas gracias! Great, great script. Could you adapt it to get the number of reactions for each group post? Please

hello thanks you for sharing. the columns email, post_text, postText, postAuthor, postAuthorId are empty

How much is this?

How to find sales data for a specific product through this method?

Can you still do this

cool tutorial

very useful both in headless:ture/false mode here is the code help to demo "puppeteer": "~22.6.3", "puppeteer-extra": "^3.3.6", "puppeteer-extra-plugin-user-preferences": "^2.4.1", puppeteerExtra.use(puppeteerExtraPluginUserPreferences({ userPrefs: { download:{ prompt_for_download:false, open_pdf_in_system_reader:true, }, plugins:{ always_open_pdf_externally:true } } })) let url = "somehost.pdf" const browser = await puppeteerExtra.launch({ headless: false, }) const page = await browser.newPage(); const downloadPath = path.resolve('./downloads') const client = await page.createCDPSession() await client.send('Page.setDownloadBehavior', { behavior: 'allow', downloadPath: downloadPath }); await page.goto(url);

straight to the point, keep it simple, love it

Any API I use to request depop via postman is a 403...

will watch it later

Your tabs are giving me anxiety! Have you found that there are similarities between county assessor sites, or are you starting from scratch for each instance?

i tried this but keep getting 403s even with headers any ideas?

How do you adapt this script so it logs the main post's copy + the first-level comments?

My account got blocked after I tried to run it. It worked up until it was blocked.