Video

Nice project, I worked on your code base for a while and used Groq mixtral instead, with multiple keys to pass limits, and Firecrawl is not automatic when it comes to pagination, you still need to add HTML code, which defeats the purpose, slow but ok for a free purpose. But I got around that I think. The next step is to use it in the front end. Zillow's API is only available for property developers, so scraping with manual inputs is the only way. However, working with the live API functionality would be the best way forward. Nice job!

Having my computer managed by an AI I can naturally communicate with is one of my biggest dreams for the short-term future.

I can't find a OPENAI model that works for me. I've tried gpt-3, gpt-3.5, gpt-3.5-turbo-1186, I always get a 404 does not exist or you don't have access to it. GPT says use davinci or curie. Any suggestions?

Hey, I am getting Invoke Code: Exception has been thrown by the target of an invocation. Error when ran this.

The AI agent is unable to bypass Cloudflare, even after trying Ollama.

Dear sir, can I use my local LLM models instead of OpenAI API?

Thanks bro

Hello Si Reda, all the best insh'Allah :)

Good technology to keep in good book!

can we use this for email and phone number extraction

What other options are beside Firecrawl? Thanks!

I like waht you did, but for no code people this is so hard because we dont know what we should install for windows etc.. really really nice video

Can you scrape multiple URLs at once? For example if you wanted to scrape all the zillow pages not just the first page with a few houses. @redamarzouk

This is a nice video and very useful. In my applications I'm looking for the 'system' to have ALL the customer pdF Invoices uploaded, or better yet, as an SalesOrder table in a database. this seems like a lot of work for just one customer and one email. Is there a way to create Agents that could filter out which customer order? Etc.

I'm curious - what do you do after structuring the data - do you store it in a vector DB? If so, do you store the Json as it is or something else? And can it actually be completely universal - by that i mean can it structure data by us not providing the fields on which it should strucutre the data. Can we make it in some way where upload a website and it understands the data and structures it according to it?

Im just getting "An error occurred: name 'phone_fields' is not defined"

nice! any idea on how to self host firecrawl? like with Docker? also, can it be coupled with n8n? how?

can Use LLMA 3/ Phi3 on local pc ?

awesome bro

nop it s not better that GPT

Nice video. Another helpful video on the same topic czcams.com/video/dSX5eoD4-u4/video.htmlsi=8iKzgqHG97Ivf8wK

Very helpful. How do you work around the output limit of 4096 tokens?

Helpful, Thank you

Does it work on mbp

I get Error code: 429 when running the code. -'You exceeded your current quota,...

Can you use this for self operating pc ? Thanks

Does it parse JavaScript, infinity scroll, button click navigations?

In the US, a “bedroom” is a room with a closet, a window, and a door that can be closed.

Neat overview. Curious about API costs associated with these demos. Try zooming into your code for viewers.

Wow! The AI was even clever enough to convert square meters into square feet, no need to write a conversion function!

Webscraping as it is right now is here to stay and AI will not replace it (it can just enhance it in certain scenarios). First of all the term "scraping" is tossed everywhere and being used vaguely. When you "scrape" all you do is move information from one place to another. For example getting a website's HTML into your computer's memory. Then comes "parsing", which is extracting different entities from that information. For example extracting product price and title, from the HTML we "scraped". These are separate actions, they are not interchangeable, one is not more important than the other, and one can't work without the other. Both actions come with their own challenges. What these kind of videos promise to fix is the "parsing" part of it. It doesn't matter how advanced AI gets, there is only ONE way to "scrape" information, and that is to make a connection to the place the information is stored(whether its HTTP request, browser navigation, RSS feed request, FTP download or a stream of data). It's just semi-automated in the background. Now that we have the fundamentals, let me clearly state this: For the vast majority(99%) of the cases "web scraping with AI" is a waste of time, money, resources and our environment. Time: its deceiving, as AI promises to extract information with a "simple prompt", you'll need to iterate over that prompt quite a few times in order to make a somewhat reliable data parsing solution. In that time you could have built a simple python script to extract the data required. More complicated scenarios will affect both the AI, and the traditional route. Money: You either use 3rd party services for LLM inference or you self-host an LLM. Both solutions in the long term will be in orders of magnitude more expensive than a traditional python script. Resources: A lot of people don't realize this but running an LLM for cases in which an LLM is not needed is extremely wasteful on resources. Ive ran scrapers on old computers, raspberry pi's and serverless functions, this is just a spec of dust of hardware requirements compared to running an LLM on an industrial grade computer with powerful GPU(s) Environment: As per the resources needed, this affects our environment greatly, as new and more powerful hardware needs to be invented, manufactured and ran. For the people that don't know, AI inference machines (whether self-hosted or 3rd party) are powerhouses, thus a lot of watt/hours wasted, fossil fuels burnt etc. Reliability: "Parsing" information with AI is quite unreliable, manly because of the nature of how LLMs work, but also because a lot more points of failure are introduced(information has to travel multiple times between services, LLM models change, you hit usage and/or budget limits, LLMs experience high loads and inference speed sucks or it fails all together, etc.) Finally: most of AI extraction is just marketing BS letting you believe that you'll achieve something that requires a human brain and workforce with just "a simple prompt". I've been doing web automation and data extraction for more than a decade for a living. Ive also started incorporating AI in some rare cases, where traditional methods just don't cut it. All that being said, for the last 1% of the cases that do make sense to use AI for data parsing, here's what I typically do (after the information is already scraped): 1. First I remove vast majority of the HTML. If you need an article from a website, its not going to be in the <script>, <style>, <head>, <footer> tags(you get the idea), so using a python library (I love lxml) I remove all these tags, along with their content. Since we are just looking for an article I will also remove ALL of the HTML attributes, like classes(big one), ids, and so on. After that I will remove all the parent/sibling cases where it looks like a useless staircase of tags. I've tried converting to markdown and parsing, Ive tried parsing with a screenshot, but this method is vastly superior due to important HTML elements still being present, and the general HTML knowledge of LLMs. This step will make each request at least 10 times cheaper, and will allow us to use models with lower context sizes. 2. I will then manually copy the article content that I need and will put it along with the above resulting string into a json object + prompts to extract an article form given HTML, I will do this at least 15 times. This is the step where training data is created. 3. Then I will fine tune a GPT3.5Turbo model with that json data. After 10ish minutes of fine-tuning and around $5-10, I have an "article extraction fine-tuned model", that will always outperform any agentic solution in all areas(price, speed, accuracy, reliability). Then I just feed the model a new(un-seen) piece of HTML that has passed step1(above) and it will reliably spew out an article for a fraction of a cent in a single step (no agents needed). I have a few of those running in production for clients(for different datapoints), and they do very good, but its important that a human goes over the results every now and again. Also if there is an edge case and the fine-tune did not perform well, you just iterate and feed it more training data, and it just works.

Dumpling ai is a startup doing The same! I’m swapping to this they are 50$ a month for 10,000 and 6 a min

It seems Zillow is blocking my access --> Press & Hold to confirm you are a human (and not a bot). I was able to run on trulia, but without my VPN.

tbarkallah 3lik a bro mashallah

Thanks for the helpful content.

You said that sometimes the model returning the response with different keynames, but if you pass the pydantic model to the OpenAI model as a function, you can expect invariable object with the keys that you need

What about Angie list 😢

What about captcha

Thank you Reda for sharing the knowledge! Very appreciated!

In my experience, function calling is way better at extracting consistent JSON than just prompting. Anyway, تبارك الله على ولد بلادي.

You could just ask GPT-4 one time to generate the extraction code or the tags to look for, per website, so that it doesn't need to always use AI for scraping, and you might get better results, and then in that code if it fails you fall back to regenerating it and cache it again.

Thank you. I have a case use, can I use the tool to make querys to a database, save the results as your tutorial shows and also print to PDF the result of every query?

While it is a practical solution, it still requires access to an OpenAI API or other LLM API, which can incur costs. BeautifulSoup and Selenium remain free alternatives. However, using LM Studio locally can provide the advantage of utilizing your own LLM, offering greater flexibility and control.

Nice idea. Now wake me up when there are no credits involved (completely free).

just use LLM, pass it the source code of the page, and generate the a-la-carte the scraping function LLM is the secret sauce

Bghit ghir nfhm chno dawr dial firecrawl fhadchi kamel ? Banli la mafih ta haja spéciale ga3 !!!

Awesome! thanks for sharing.

It's easy to do it with free python library. Reading HTML convert to markdown, even convert it for free to vector with transformer ect

50$ montly fee 🎉😂😅

firecrawl is not open-sourced!!!