Video není dostupné.
Omlouváme se.
Scraping Data with an ESP8266 or an ESP32
Vložit
- čas přidán 23. 01. 2020
- Have you ever wanted to get data for your project, but there is no public API for it or cases like the Instagram API where the setup process for the is not very convenient? Well in this video we are going to look at 2 different options for scraping data from a website for your ESP8266 or ESP32 projects.
We'll look at Non-Public APIs and scraping the data directly.
---------------------------------
Info Links:
---------------------------------
Arduino sample request: github.com/wit...
Previous Video on API: • Connecting to an API u...
Team Trees code: github.com/wit...
Crowdsupply NodeJS scraper: github.com/wit...
---------------------------------
My Discord:
---------------------------------
Come hang out and talk about this video or any project you are working on!
/ discord
---------------------------------
Github Sponsorship:
---------------------------------
I have set up a Github Sponsorship page, it's similar style thing to patreon, but Github are offering to match all sponsorships for the first year! (you sponsor $2, I receive $4)
github.com/spo...
Some people have said they would prefer a once off way of supporting the channel, here is a Paypal link if you would prefer that!
www.paypal.me/...
---------------------------------
My Tindie Store:
---------------------------------
I've started to sell stuff on Tindie, check it out!
www.tindie.com...
---------------------------------
Other Platforms:
---------------------------------
Twitter: @witnessmenow - / witnessmenow
Instagram - / brian_lough
Instructables: witnessmenow - www.instructab...
---------------------------------
Support the Channel:
---------------------------------
If you would like to support the channel, here are some ways you can.
1 - Like and comment on the videos - I really like hearing from you and apparently it helps!
2 - Share my videos - If you know somewhere that might be interested in my channel, please share it!
3 - Affiliate links - Where I get a small percentage of the purchase price but doesn't cost you anything extra.
Amazon.co.uk: amzn.to/2g7mJIO
Amazon.com : amzn.to/2ykDQ42
Amazon.de: amzn.to/2WMq7vF
Aliexpress: s.click.aliexpr...
LCSC (where I buy most of my parts for Tindie stuff): lcsc.com/?prom...
4 - My Amazon "store" - Really it's just a collection of stuff I use that I think are useful
www.amazon.co....
5 - Check out my Tindie Store
www.tindie.com...
---------------------------------
My Tools & Kit*:
---------------------------------
Ender 3 Pro 3d Printer - I got one of these recently and I love it!
- UK : amzn.to/31bebXT
- US : amzn.to/2IlSLid
Ts-100 Soldering Iron - I have this a couple of years and I love it, I thinking about getting a second!
- UK : amzn.to/2nlOF23
- US : amzn.to/2jDcyxl
Logitech C920 HD 1080p Pro webcam - I use one of these for my videos
- UK: amzn.to/2pnxk9Z
- US: amzn.to/2BsLMiF
Blue Snowball USB Microphone
- UK: amzn.to/2C0x4TH
- US: amzn.to/2DEA85J
Self Adjusting Wire Strippers
- UK: amzn.to/2C0urkR
- US: amzn.to/2BXQAh1
*= Affiliate Links
Great thumbnail!
Thanks! 😂
Agreed!
Good video as usual, Brian! I have the feeling this will be useful for many people!
Useful detail - last screen scraping I wrote was back in the early days of database driven web sites when APIs were as rare as hens teeth!
You can use a VPN service (including free ones) or Tor to get around usage limits. (But you most likely can't do it on an ESP8266 by itself.)
Legend You gained an instant Sub
Thanks Brian! Helpful video. I'd like to know more about Arduino Json ;)
Very cool! I'd like to see a video about using ArduinoJson !
Yep! Especially moving from V5 to V6 - I shelved that a few months ago so I could move on with other stuff
@@KeanM Totally! V6 might be better, but it broke everything ;)
@@UnexpectedMaker lol, i didn't read all comments, and made one with same results and questions, here above :D
Turd....lol!! That made me chuckle!!
I honestly tried....
Interesting. This might come in handy some day.
Very interesting thank you !!! :-)
ArduinoJson please.
I have my websites which is hosted on my own local network. I want to fetch data from here. Is any method for this.
Great video Brian, very informative. I do have a use case where the data sits behind a login screen, any ideas how to deal with authentication other than at the API level?
I wouldn't be an expert around this but I guess you would have to replicate any authentication that your browser is doing (does it send tokens or username and passwords in the headers)
I wouldn't exactly call these "non-public API's", because these will have to be public in order to be accessible without authentication from the internet. These are rather public machine API's such as a REST or similar, designed to integrate applications.
Sir can you webscrapping on esp32 on thonny micropython
Is there a non-deturded version of the video? ...and am I allowed to giggle when you say 'udder'? 😂 Joking aside, this is excellent step by step teaching, as always! GRMMA!
I'm too old of a dog to learn new tricks so you're stuck with me and my Irish accent 😂
@@BrianLough - sure it's worse I am though - Nordie living in Dún na nGall for the last number of years... 😂🙉
a video on arduinojson would be useful for sure... in particular to address all the issues that so many seem to have with v6... too much youtubers and githubers seem to solve projects compilation errors with just saying "just go back and stick at v5 if you have problems"... while maybe v6 would be more efficient in lots of ways, don't know... if ITS development is going on, why everybody seems to ignore that...
I'm as guilty as anyone for the v5 to V6 thing, nearly all my libraries still use v5. (CZcams API, telegram etc, although telegram has a branch with the change made)
V6 was in beta til last year (I think) and it was advised you used v5 while it was.
It's actually pretty straight forward to swap them over, but it's just a time thing. When I have free time (which is a rare enough concept with two kids) I usually have a laundry list of stuff Todo and non essential updates to existing libraries unfortunately is not the most pressing, so it usually goes to the bottom of the list
Brian we need a video on working with json and parsing stuffs
It's on my to-do list!
@@BrianLough 🔥
Do you have any tips for any parsing libraries or querying non formatted data, something that can split and format arbitrary data like delimited strings etc, with support for structures key value pairs etc perhaps
I haven't come across anything like that actually, sorry!
@@BrianLough I have written a bunch of stuff in php , for grabbing non api data and a token template mask and having it fill a key value array with what you expect as well as do datatyping translation , or error, was hoping for something in c++, I guess I will write one, I need to do some serial telnet stuff.
I'd recommend offloading this onto a separate server, implemented as a proxy - such as in Python or PHP. Standardise on what the IoT device expects, and that then makes it easier to handle future changes on the source being scraped. If you're comfortable with doing web stuff, then this is a good idea for all API calls to third party sources - helps with debugging and future proofing.
Ah, Brian covers this later on in the video. But he used NodeJS - Yuk! :)
I really wish I could have understood that...
Brian, why are your H's silent? How do you pronounce your last name?
I love your videos btw. Even if all your attempts at saying anything to do with a 3 is amusing :D.
30 trees or 30 3's or 30 tees? 😂🇨🇦👍
i love the irish accent and vocabulary !