How to Scrape Websites with GPT-3.5 (Web Scraping with ChatGPT)
VloĆŸit
- Äas pĆidĂĄn 20. 07. 2024
- đ„ My FREE Cheat Sheets (ChatGPT, web scraping, data science): artificialcorner.substack.com...
đ Playground: beta.openai.com/playground
đ Subslikescript: subslikescript.com/movies
Web Scraping with ChatGPT. In this video, we're going to scrape Twitter and Amazon using ChatGPT. We'll use OpenAI Playground to generate Python code that scrapes any website out there.
Social Media
===========
đź Discord: / discord
đ LinkedIn: / thepycoach
đ„ Twitter: / thepycoach
âïž Medium: / frank-andrade
đș GitHub: github.com/ifrankandrade
My Courses
==========
đ„ Join My Automation Course in Python: www.udemy.com/course/automate...
đ„ Join My Python for Data Science Bootcamp: www.udemy.com/course/python-f...
đ„ 8-hour Web Scraping Course in Python: www.udemy.com/course/web-scra...
đ° Make money using your Programming & Data Science skills: codelessearnmore.teachable.co...
Support My Work
==============
đ” PayPal: www.paypal.com/donate/?hosted...
--------------------
Content:
0:00 Intro
0:34 The approach to use to scrape with ChatGPT Playground
4:21 Scraping Amazon with ChatGPT
9:51 Scraping Twitter with ChatGPT
If you'd like to know how I'd use ChatGPT to learn to code, check out my new video đczcams.com/video/Eh_KovOmQRQ/video.html
Can you make a video of getting ChatGPT to Google Dork or to write a code to do so for any website?
This is awesome, thank you! I have been trying to use this video to help me scrape certain aspects of flippa and apparently the commands I'm giving ChatGPT aren't correct. lol I want to scrape flippa every 5 minutes after the hour and I want it to scrape the site for site name, type, industry, monetization, age, net profit, and the site description. Any help would be highly appreciated.
Thank you!
Thanks for posting the video. Is there any way to do this for a site that requires a sign in? I have a sign in for a site and want to use a scraping tool to scrape all of the data from it.
The data is behind a click, i.e. you have to click the listing in the contacts database to see the email address.
Any idea how to do this?
what website did you past that in to get the test to print?
I have never uderstood how bots work. Like many people I have dabbled with code but haven't needed to go further. I didn't realise that the classes and IDs etc were to help others acess your ste data. It all makes so much more sense. Thanks.đ
Wow, this is really amazing, thank you for showing us how to tell ChatGPT to scrape the web page to get the content we need, this is a valuable lesson, than you so much đ
This is exactly what I needed. Thank you!
Due to the many questions and comments about ChatGPT and OpenAI Playground, here are some notes.
1. Yes, ChatGPT and OpenAI Playground are not exactly the same. That said, if you use the prompts I created in this video on ChatGPT, it'd generate code that scrapes the website just like with Playground.
2. Why did I use Playground instead of ChatGPT? Speed. I only wanted to get the code and not the bla bla bla you get with ChatGPT. I like the explanation ChatGPT gives, but, when recording the video it's a bit annoying.
3. Some people say Playground isn't free. At least, it was free for me at the moment I recorded the video. I never gave credit card information and it's still working fine for me today.
4. ChatGPT/Playground currently generates code for Selenium 3 and not for the last version (Selenium 4), so keep that in mind when using its generated code.
this is insane! TY for the videos, chat gpt3 is basically the next internet lol
Bcoz it doesn't know about world after 2021
@@abhijeetdhumal7385 yeah, I wonder why openAI team does this, if AI can't tell us the latest info, then it is no better than Google, you know đ€
The SOME REASON is it was trained on data on the internet, there is more data about the older version than the new one, have you tried asking it specifically to write for selenium 4?
@@maskman4821 so that u will pay subscription when they launch the latest dataset :P
Pretty useful and time saving. Thanks again.
Just 1 or 2 mins. It won't take more time to Google and copy from stack over flow.
you are SO good. Learn much from you
So basically you tell ChatGPT all the things that you'd tell any other web scraping tool as well. How genius! ChatGPT is now as intelligent as Scrapinghub was 10 years ago. đ
hich free tool would u recommend?
The projects shown in this video were once used to be to pass a whole examination or certification
Bro this is genius!!! Thank you
Best video of 2023 so far
thank you so much, that's really helpful!
thank you for the video, it was interesting use cases đ
It's a very simple and understandable educational video, thank you...
I have a few questions.
Is it possible to scan current ASINs (product identification numbers used by Amazon)?
Of course, I need to enter location information to fetch data from the Amazon website.
My other question is, how can I select the current profiles in my Chrome browser while using chromedriver?
Solid video!
Great tutuorial. Thank you. Question, i am trying to create a chrome extension to scrape products in every category that have no videos. Will this be possible using this same approach? TIA
Thank you for this great tutorial. I tried to scrape on tweeter by scrolling but i always get the same results
Twitter loads only so many tweets at once, so when you scroll too far, it will remove the objects from HTML on the top. I learned it when I was searching for something with ctrl+f in the browser.
Incroyable.
@@mamneo2 Excroblagaxyable
I love you so much I really love you months and months trying to figure it out finally I was able to complete a project
what project have you been working on months and months?
You can tell chatgpt to scrape for example items in items name just leave the selector empty then you can just fill out the selectors
good video, I think this is going to be the future of coding. Human don't need to do low-level coding, but giving high-level instruction.
That low-level code you speak of, is high-level code. GPT just adds a higher level :)
@@KCM25NJL The low-level code I mention, means the actual codes, like python, c or c++. The higher level code I mention means the natural language which explain the fundamental logic of the program. But I do get what you mean.
This is a high-level code. Languages such as Python or Ruby are amongst the least verbose. Create a code snippet once and use it later. What is the point writing essays in English? ;) Great for learning, though.
Hey brother can you tell me why i am getting only [ ] this in output whenever I try to scrap data of some websites? Does this happen due to they use javascript? Or it happens because javascript is not supported by beautiful soup? So is selenium best for all types of javascript, css and html websites? Also please make a video or provide your email i am facing issues in installation of chrome webdriver.
will this work on gambling websites that provide live stats? could you code it to constantly check the website for updated info? or would you need auth api for that.
Thank you for the tips
I'm stuck to scrape data related to this query on Google maps : Hotels near city name and getting the price of each hotel in detail view
what if I want to scrape the movie title, but also, the info inside each movie/url that leads to the info of the movie?
It loses a bit of usefulness when you have to go look at the source and figure out for the ai where and how to scrape. Is there no ai that can "search" the html structure for the content you want?
Can you make a video of getting ChatGPT to Google Dork or to write a code to do so for any website??
what if we have pagination or if the data is in api?
Thank you!
Got error at Playground OpenAI
You've reached your usage limit. See your usage dashboard and billing settings for more details.
You got to pay to play :(
Youâre using âtext-davinciâ GPT model but âcode-diviniciâ will probably work better for these kinds of things and will work more consistently
Thanks
Please there is any method to know keyword that use on some products on amazon with ChatGPT??
Can this work on a website that requires log in to access any data I want to scrape?
Thanks!
Can you do that product asin and product price like products finding for Amazon fba
I need to scrap video links from a website? can I do ?
Hello bro how to scrape a running website which will change all the time
How can web scraping all pages from spesific website ?
After getting the script of code. Where did you pasted it to run?
How would i scrape website for ad copy on there home page? đ€
What can you do once you scrap a list? Like can you show more possibilities?
you can save it!!!!!! & delete, recyclin bin - restore and again again, infinite possibilities instead of reading directly from the web!
But chatgpt says it cannot connect to internet, no real time access as its an ai model. Is this something other than chatgpt?
So how do you tell it to scrape the subsequent pages?
I just asked the same thing two months after your question. I guess he either missed your question or I just won't get an answer. Your question is all I really care about :)
I dint see a use case for it in the real world.
When you first need to dive into the source code to find the right divs, or scc, what is the point using ai. The code itself can be written in 5 mins manually and probably in 1 min with github copilot.
thanks bro i need this
how to integrate chatgpt to email, excel and messages?
Many thanks
Asante Sana
Can you make chatgpt learn something about the actual data and updates by giving it informations
Great video, could you do an example to scrape facebook pages and get post? I was trying this the other day and it seem there no way to scrape public data from fb
Iâve never tried to scrape fb. Whatâs the issue? There are websites like LinkedIn though that have strong anti scraper systems that make web scraper very challenging. As I mentioned in another comment, this AI tool gives you the code, then you have to come up with something like proxy rotation to deal with anti scraping systems
bit weird
Subscribed
I can't get it to scrape Amazon.
I see zero point in using Chat GPT as merely a direct, 1-to-1 translator from natural language to code. If I need to write the instructions at the same level of abstraction as the code, I might as well just write the code. That's largely why programming languages exist! They are specialized languages that, unlike natural languages, can precisely and succinctly express low-level technical requirements in a human readable form.
Es para ahorrar tiempo. Personalmente no he tocado HTML hace años, como electronico no me era necesario hasta el momento. Podria retomar los cursos, actualizarme, pero es otro tiempo adicional. Esto estrecha esos tiempos. Y de que maneras, refresca la memoria en los diversos lenguajes.
I guess it's just more about batching those instructions.
â@@kami2496 Hola Miguel, veo que has estado contestando varios comentarios en inglĂ©s, con respuestas en español đ
@@mamneo2 bueno, asumo que si hay algun problema, el traductor lo soluciona. No es una barrera necesariamente.
Thankssssssssssssssssssssssssssssssssssssssssssssssssssssss, for the first time I scraped a website by writing the exact code and also understood it completely, what was happening inside it
Hello, I used the exactly same code but it gave a syntax error.
I asked chatGPT the same question and it gave me a different code that worked.
What if I need to Log in first?
Everyone says the name as Amazon at the start but the tutorial is about a different website scraping. The real part is you can not scrape Amazon content easily and if you go by this same method it'll fail as the DOM element won't be available and no one is telling you this.
Great video! Is this also usefull for Instagram? They have a very strong Bot detector.
The tool simply gives you the code. The rest is up to you. In this particular case, youâd probably need to rotate proxies
@@ThePyCoach yeah thought so. Thanks for clarifying. Love your content. Keep it going!
Waittt is the ai actually going to that link and looking at the html of the page? No right? Then how is it knowing from the url how to scrape the website?
In the case of books.to.scrape, it simple takes a script already built from the internet. In case on Amazon/Twitter, we're simply giving instructions on how to build the scraper. The tool never gets to see the HTML (at least not in this tutorial)
Isn't what you're using (playground) running GPT-3, not chatGPT? You said yourself that chatGPT doesn't have Web access (which is correct). But GPT-3 does. You keep saying chatGPT in your video (and title), which doesn't make sense. Or is playground a special version of chatGPT (aka GPT-3.5) that has Web access?
what about scraping with facebook????
Much like the other comment by a Lance below?
This is NOT ChatGPT!
You're misleading your Viewers.
This is GPT-3 using the Davinci Text 3 option.
Maybe you're going for more Views by saying "ChatGPT" ?
But you are going to Confuse those that are Not too familiar with this technology.
However, having said All of the above? Just wanted to make sure everyone knows.
All in all....
This is an Excellent tutorial !!
I shouldn't have used the terms ChatGPT/Playground interchangeably. That said, the prompts I used generated the same code in both ChatGPT/Playground. I just decided to use the latter because it generated the code way faster than ChatGPT without wasting time explaining the code.
Playground is another AI, a much less capable one (davinci)⊠NOT ChatGPT.
Davinci is actually a bigger model thatâs more capable but you need to give it more instruction
Chatgpt is gpt3 3.5 fine tuned to have discourse, "playground" is all of their models which are all based off go t3.5 at the moment and fine tune to do specific things like codex writes code, davinci does embedding and generation of tex
Yes he keeps saying ChatGPT.. anyway his tutorial is very useful đč
No, it's not. The text davinci 003 is the latest model, which actually is the so called gpt3.5 and ChatGPT. đ€Šââïž
@@xWe2s Chat GPT is NOT the same da vinci you get in the Playground by ANY means.
what kind of use does this have?
Yo tenia en mente, aplicar un generador de resumenes. Imagina para un chat cualquiera, poder pedir generar un resumen de ## tiempo atras. Supongamos una conversacion que duro 30 minutos, para no leer todo, poder analizarla primero en forma de resumen, poder extraer las palabras claves, y determinar si resulta relevante revisar el contenido, o no... Asi omito el tiempo de leer comentarios de relleno. Felicitaciones, Saludos, repeticion de contenido, cosas fuera de contexto...
Pero el primer paso es extraer la informacion de una pagina, luego procesarla con tecnologias NLP [GPT3], y obtenemos resultados que nos ahorran mucho tiempo....
simple version of Github copilot
I think I would be more quicker with Copilot
What is copilot?
@@xWingzTV Microsofts AI tool specifically for programming. Been using it since the beginning
text davinci is not optimized for coding. use other model
Which model do you recommend?
@@pleabargain codex?
I didn't know that. Which one do you use for coding? đ€
@@ThePyCoach OpenAI Codex
@@deep2mixer chat gpt is actually better at coding than codex.
Iâm tryna make a program where I can give a python script a bunch of top 20-50 ad agency website list and then it gets each of those websites and then scrapes for all their email so I can contact them offering my video editing services
Up
How can we ask ChatGPT to do pagination and scrape data for multiple pages within the same website?
The pages are usually in a navigation bar ( tag). Just tell ChatGPT to locate an element that has the nav tag and add the class name of the element.
The ai will replace programmers bro, they say. Just ask on chat overflow how to write English question ;) I didn't intend to be mean, just found it funny ;)
:) now I have to Google scrape website so I can know what he is doingâŠContext gets me 60% there.
"loophole" ....
Chomedriver ainât working no more ?
You need to have the Chromedriver version that matches your Chrome Browser release version. So, as your Chrome browser updates, you'll need up update your Chromedriver.
Use Mozilla driver
Just tell ChatGPT what error you are getting, and it will tell you what you could do.
Script?
Amazon Ip bans for this
sorry but it's actually faster to write a scraper from scratch... try pasting the whole html and asking it to write a scraper by example instead...
Dude you can scrap with beautifulsoup, why you saying you canât scrap amazon with it lol
what the idea
I think typing code is much faster
Change heading as "How to scrape any website content using Python"
isn't playground a paid service?
That's cool... lets enable blatant copyright violations with AI. ;-)
This' no loophole
Cant scrape twitter no more
This is not ChatGPT, why do you keep saying that đą
Yeah, I shouldn't have used the terms interchangeably. That said, you'd generate the same code with both ChatGPT/Playground. I know because I tried both. I've just made the video with Playground because it was faster.
Youâre just pseudo coding at that point. Youâll need to know how the code will look to ask the question in the first place. I donât really see a point in doing this. What would be useful is dropping in html code and giving examples of what you want scraped
wtf it can do this too? XD
Why bother using gpt or anything if i know how it works i could just write the code myself and its will be more fun than talking to a robot
Wonât scrape LinkedIn correctly. Fuck I hate LinkedIn.
Bro scrap telegram
Warning!! The OpenAI Playground ISN'T FREE.
I think they give you free credit each month. I never put any credit card info and I've used it for free so far
You only get some credits when you register, for a month
Also "Playground chatGPT" is not chatGPT. It's another language model, that's called GPT-3. ChatGPT is a GPT-3 based model that can communicate in conversations.
Lol chat gpt is not connected to the internet đ
Jobs are at stake!!!!
@oxylabs we better watch out. ChatGPT is going to put everyone out of business. đ
whats more amazing is guess how much the Ai has grown in the last 4 weeks....now it knows what every hustler and hacker wants...feeds it back to: Google, Facebook, Insta, and the FBI and hey presto, secured loop. All for a fee sure, MS servers don't run themselves.