Vložit
- čas přidán 9. 06. 2024
- Grafana, InfluxDB, and Node-Red on a Raspberry Pi form a dream team for visualization of IOT data. The setup is not easy, but this is what we will do together.
Today
- We will install everything necessary on a Raspberry Pi using Peter Scargill's newest script
- We will compare time-series databases like InfluxDB with SQL databases and see their advantages for IOT
- We will create an influxDB database and connect it to a Node-Red flow to read data of my weather station
- We will create a Grafana data source to get access to the InfluxDB database
- We will develop a dashboard with several panels showing the different weather data
- We will see the power of a time-series database combined with Grafana
- And in the end, you can download a ready-made SD card for your Raspberry
Links:
SD card image with Node-Red, InfluxDB, Grafana:
drive.google.com/open?id=1MnJ...
Node-REd example Flow: drive.google.com/open?id=1u2y...
Peter's Script: tech.scargill.net/the-script
Raspbian installation: hackernoon.com/raspberry-pi-h...
Grafana videos: / @earlyadopterme9292
Samsung SD cards: s.click.aliexpress.com/e/90y5IJW
Raspberry Pi 3+: bit.ly/2tNdULR
#NodeRed, #InfluxDB, #Grafana, #RaspberryPi
Supporting Material and Blog Page: www.sensorsiot.org
Github: www.github.com/sensorsiot
My Patreon Page: / andreasspiess
Alternative: flattr.com/@andreasspiess
My Bitcoin address: 19FSmqbBzb5zsYB1d8Bq4KbxVmezToDNTV
If you want to support the channel, please use the links below to start your shopping. No additional charges for you, but I get a commission (of your purchases the next 24 hours) to buy new stuff for the channel
For Banggood bit.ly/2jAQEf4
For AliExpress: bit.ly/2B0yTLL
For ebay.com: ebay.to/2DuYXBp
profile.php?...
/ spiessa
www.instructables.com/member/...
Please do not try to Email me or invite me on LinkedIn. These communication channels are reserved for my primary job - Věda a technologie
"We like programming, but we love result" Quote of the day. Thank you sir
You are welcome!
Even I can confirm, and I'm a developer.
this :]
Andreas - another perfectly formed video - in every aspect. We appear to run parallel lives as I was working on this for my open air pollution monitoring project. There must be more than 24 hours in the Swiss day as I can't for the life of me see how you manage to produce a) so much high quality material, b) over such a diverse range of subjects c) in such depth in such a short space of time and d) in a Swiss accent!!!).
I agree 100%. Andreas is amazing!! Thank you so much Sir for sharing knowledge with everyone and great commitment on making amazing videos !!
We are slow here. That is probably why our time runs a little slower... I hope the content still helps you a little for your pollution project!
@@AndreasSpiess Yes, it will. The issue is the simplest way of amassing JSON over https from many disparate sources. Not sure how to organise the secure apis.
Security is not my specialty, fortunately ;-)
Amazing! I had to stop the video on two occasions I was so overwhelmed by the sheer power of this suite of FREE software! Thanks Andreas!
That is the power of CZcams. You can use it at your pace ;-)
Extremely usefull video - I am very happy to be one of your Patrion-supporters! Keep it coming, Andreas!
Thank you for your support! It is always good to read that my videos are useful...
Mr Spiess, my deepest gratitude for your work and for sharing great content in a video like this. I became your Patreon today to support your work!
Thank you for supporting the channel. I appreciate it!
Super useful! Displaying data in a meaningful & useful way is always a great challenge between providing what is of interest to a range of user interests & storing what is needed to create this output in such a way that it is easily accessed & does not need a lot of storage. There were several great ideas within this both as shown with this software or for writing ones own software on different hardware. Thank you for sharing!
You are welcome. And I agree that retention strategies are not easy to define these days with very cheap disk space...
The timestamps of the example data are from 2019-02-06. Please choose an appropriate date range in Grafana if you want to display them
You are seriously amazing. Thank you so much for this video - I've spend days trying to get this to work, and with just a couple hours and following your video, I've got everything up and running.
Glad to read that!
This video just gave me the push to convert my smart energy meter and heating data into Grafana and Influxdb. Until now I was using Highcharts and the Dashboard of Node-RED which is so difficult to setup. Now I can add historical data. Great video, thanks Andreas!
I transferred historical data exporting to CSV and creating the needed input strings for influxDB with Excel.
This set of tools is very flexible, I am using it myself and I was amassed at how easy it was to add a new device and process it's data once the whole system is configured. In my case I didn't have to go through the setup process because I just used a ready made raspberry pi image from the BigClown IoT bundle.
I think it is not so easy if you have to do everything from the ground. So we both were happy we were able to stand on the shoulders of others ;-)
I have no words Andreas. Astonishing. Thanks.
You are welcome!
Nice to see you're making good use of "the script" Andreass.
I always like to stand on the shoulders of big men ;-) Makes me more productive! Like that, everybody can profit.
@@AndreasSpiess Hah. I do that all the time. My script enables software like Node-Red, written by people cleverer than me!
Thank you both for your wonderful work. I have a beautiful image on my Pi and it’s all working 😁🏴
Just to remind that "Retention Policies" in InfluxDB comes connected with the concept of "Continuous Queries". Maybe in a future video you can explain that. Great video as always. Thanks!
I mentioned the compression, but not in detail. I do not plan a video on that because it is very special. Maybe once part of another one. Who knows?
Well done you. This, #255, is my 'eureka' tutorial. Thank you for explaining why AS WELL AS how to. I've read the rest of the Internet and really should have started here. Sensors have been my professional 'thing' for decades and I can now see a better way to process and examine the data. Thank you.
Maybe you also watch my videos about IOTstack...
@@AndreasSpiess I did Andreas and for the first time doing useful things with this stack. More thanks. My stuff is at rogerfrost.com. Best wishes.
Amazing video for home use 😇 I have a similar setup running in my new flat, including Gas-Heater control and a combination of all sorts of different mcus with sensors and actuators based on various projects with a centralized mqtt/influx/nodered/grafana server (mostly own-software, tasmota...)
Keep up the amazing work - your channel is still one of the best (mostly because you don't have annoying intros and music or semi-funny jokes ;) )
Greetings from Austria
Thank you! Jokes are not easy in an international environment. Glad to read that you like them.
I like music, but in videos like that, I hate it myself. This is why I do not use it...
I did find another script from another person with red-node. It is handy for solar and batteries. I hated manually install parts of Peters script the .npm title. But Peter has done a great Job with this script with Mr Sharky. I love it. Peter is a great person thank goodness we have him around. The graphics I think was made by him too.
With his new version, you should not need any manual install...
Such a coincidence. I was up until 1:30 am this morning looking at almost the exact same thing.
Excellent content, I will give it a go today. Thanks. Martin.
Good luck. I hope it will work for you!
Thank you , always something to learn every week in an easy way
You are welcome!
Your channel is pure gold!!
Greetings from Chile!
Thank you very much!
this was an eye opener for me. always a good time learning from you
Glad to hear it!
Nice work Andreas! Thank you for putting this together, very informative.
My pleasure!
Fantastic job!
Just what I was needing for my project.
Thank you!
You are welcome!
Amzaing effort from you and Pete's , keep the videos coming Andreas 👍
Thank you!
Thank you for your great video. These are the projects that I can learn from a lot....
You are welcome!
Awesome guide to getting Grafana up and running, thank you :)
You are welcome!
Really helpful explanation of the benefits of time series databases for IoT, thank you 👍👍
Glad it was helpful!
I swear, you are a Swiss mind reader or something! I was sitting here, building my Node Red + Grafana + InfluxDB setup and your video got uploaded. O_O
Maybe you can save some time now ;-)
Thank You!
I love how well structured your videos are. Suits my taste as german engineer ;)
Thank you! Maybe you also watch the one about Docker. It simplifies the installation of those programs
@@AndreasSpiess Yes i did and everything is working nicely together. :)
What another great video, thank you Andreas! Your video inspired me for changing my implementation on how to use historical data.
I hope you will be successful!
@@AndreasSpiess The InfluxDB was really fast and easy! I installed influxDB in a chrooted Linux on an Android TV box where I am using it as a mqtt broker and node-red administrator. Now I am testing the first data recording from my air conditioning energy meter. Next step to install Grafana.
And all these thanks to you again :)
You're awesome, thanks for sharing this knowledge. I've been learning a lot from you.
You are welcome!
Great video, Andreas. Really enjoyed this one.
Thank you!
Everything new. I had watched twice to understand. I want this setup too for my home sensors. Thank you dear.
You are welcome!
Im now successfully using grafana influxdb with mosquitto (EspMqtt) using node red on my OrangePi Lite.
thanks a lot for this research and experimentation. we will definitely use this for our farm.
I hope your application will add some value :-)
Thank you. This is what I needed. I'll get to work right away...
Enjoy!
This is what i'm looking for. Thanks!!
:-)
Thank you so mucho for this. I succesfully followed this tutorial and now have an energy monitor and a water pressure sensor working and displaying data in grafana.
Excellent!
Very nice Andreas - I've just been playing with InfluxDB and Grafans with InfluxDB fed with SNMP data by telegraf, all in docker containers. This video has given me an idea to move that on and try to connect my weather station rather than just looking at my Internet traffic :) I'll need to try to understand my stations data stream format which should be fun.
Thanks for another great video and the idea...
There are a few other videos about weather station hacking and Pi and Docker (IOTstack)
I've been agonizing over my heating system lately (I have been using a homebrew controller for my parents house heating unit for over 10 years now). Now I can just copy your image and dump my logging data in there while using a esp32 to control all the relais and sensors. Thank you so much!
I hope you will be successful in transferring the data from your source into InfluxDB.
Thanks so much for providing the SD image. Thanks thanks thanks
You are welcome!
Really inspiring ideas, thank you! I'm thinking to move my solar panel yield visualization to an influxdb and grafana stack.
Good idea!
Fantastic project👍😀
Thanks for sharing😀👍
You are welcome!
Thank you very much sir! Always love your video
You are welcome!
Interesting and useful video!
It seems we are heading back to remote sensing? I like that.
Sensing and acting based on these data is IOT for me ;-)
Great work again Andreas!
Thank you!
Excellent video. Thanks.
Greetings from the galilee
Glad you liked it!
Another excellent video. Thanks!
You are welcome!
Useful and interesting as usual 😉
:-)
Many Thanks Andreas - Well Done and hope to see more
You are welcome!
Thanks, this was just what I needed
You are welcome!
Waw, excellent work ! Heavy stuff aswell !
The simple stuff is done by others ;-)
Great job!! Very useful video.
Thank you!
What a great video. Many thanks!!!!
You are welcome!
Dear Andreas, great Video! Quite complex things explain so amazing short. Stuff like this can only see very rarely. Love your comments why you choose this tools. Thank You! Greets Thomas
Thank you for your kind words!
I took the newest RasPi-4 & OS Buster, used your selections @2:53 and the latest version of peter's script. It took 17 minutes for a full installation. Thanks for this video
Very fast! Did you try the docker Installation?
no I did not. I'll try to create a surveillance for 24/7 running PCs (if message does not arrive periodically - alert)@@AndreasSpiess
Interesting project!
We created a whole project with lots of useful containers (IOTstack)...
Awesome as always.. Thank you...
:-)
Excellent demonstration of how little you need to collect and visualise data. I have a spare PINE64 which I'd like to try using these building blocks
You have to check which operating systems the script supports if you want to use it.
Wao!!!! Great idea!!!
:-)
Thanks for a great video! I use Influx and Grafana professionally at work to analyze metric data from a mobile application. Thankfully I have an entire team of people to manage maintaining those systems. This kind of inspires me to see if I can hack my solar system on my roof (it uses zigbee apparently) to get data into influx and match it with real time weather so I can analyze better how my solar is performing.
At least you would have great support for your project. Not as we "simple people" ;-)
Sehr geil, wie immer! Vielen Dank!
Bitte, gern geschehen!
Big thanks. I was using the Node Red Dashboard and it was awful, nevermind browser issues if you left it on the dashboard page too long. Pushing my data to InfluxDB and then viewing beautiful charts in Grafana from that DB information is awesome. Thanks,
I also like Grafana...
@@AndreasSpiess I would also add that I'm using Python on a (Pi Zero W) to push MQTT data to a remote server listening for MQTT and is hosting InfluxDB, Node Red, and Grafana. You just need to install Paho MQTT on the Pi (pypi.org/project/paho-mqtt/). Then in the script just import it (import paho.mqtt.client as mqtt), define the remote server (mqttBroker = 'foo.server.com'), set the client with client = mqtt.Client(), connect to the remote server, client.connect(mqttBroker), and then for every variable assign to your data feed, you then just publish it with something like this: client.publish('Temperature', temperature). Where temperature = bmp.temperature, a feed coming from a BMP390 sensor. I any case, following your instructions on Influx DB, Node Red (MQTT->InfluxDB), and Grafana has been a life saver. It sure beats trying to chart something from file data. :-)
Amazing work!
Thank you!
I'm a programming student and I was this kind of data retention policy saves a lot of work. Glad I don't have to reinvent the wheel.
We live in a good time with all this open source software...
Excellent, very good.
Thank you!
Congratulations and thank you again!
Thank you!
Awesome video!
Glad you enjoyed it
Many thanks for your video!
You are welcome!
Cool. Great stuff Andreas.
Its all the craze doing opensauce. I wonder what made u not choose non elk and or prometheus. I suppose grafana is top for these requirements. Miss lora and esp32 a bit... a lot to be said about long range iot.
Thank you!
Great explained sir.
Thank you!
Great, finally this seems to solve my IT problems for IoT. HomeAssistant is also good but seems to have an easy start with challenging future. (developing all the plugins etc. for simple additional custom features!)
So far I always used Node-Red and do not know Home Assistant.
@@AndreasSpiess For whom, who might be interested... Home Assistant (HA) has MQTT capability. An MQTT broker can direct messages to HA and HA will store them and display in a nice manner. Only by writing some making configuration files, one can trigger alerts, actions etc... e.g. I could set up a threshold for humidity sensor, after crossing this level, HA triggered a relay, which controls the fan. It is neat but extending it seems to be difficult.
Thank you. I've been struggling for years with mysql and an inflexible php graphing library.
This is maybe a little more flexible...
Grüezi Andreas, as always a great video with many explanation that everybody can follow and learn a lot. todays topic is very sexy, since data/process mining is the new lubricant of the world. as already told on TTN con I love your 'Mutterwitz'. cheers Michael (guy form Sulzer)
It was long overdue for me as I wanted to use it for my installation.
I hope you had a good trip home!
Excellent video Andreas. Clear instructions and to the point. We have used mqtt (activeMQ) +nodered+mysql and we were planning to use influx dB since its performance is more suited to IoT and time series data. Node red is really amazing and can help you to get to the results without the need of programming. Grafana (even if its free) its excellent visualisation tool. We have everything working under Ubuntu VPS, docker images of MQTT, node red, mysql, grafana etc and the performance is more than enough (more than 25k measurements in one minute transfered from the remote sensor via mqtt to the database without any issues)
it would be great if any of you wants to collaborate for any EU funded research based projects. (as mentioned most of us live parallel lives.. and do similar things)
25k points per minute are quite a lot! I did not know this is possible. Docker is a good concept. But with Peter's script, it is also quite easy to install. And we have full access to the RPI hardware...
you are my new god sir! thank you so much!
You are welcome!
Greate video... thanks a lot :-)
You are welcome!
Happy birthday Andreas, hope you have a good one.
Yes, I had a good one. My new bike for the Danube tour arrived right that day :-)
Cool, I use InfluxDB for my home, but never understood the retention policy, thank you.
I have three point about your setup:
- InfluxDB is part of the TICK stack, that has Chronograf as data visualization tool in place of Grafana, a different flavour of the same concept someone might prefer, and has Telegraf agent you can use to collect raspberry's own metrics to track its health, feeding them through the same InfluxDB and Grafana combination
- This raspberry use case tends to put some pressure on the SD because of the big quantity of data written and red, it is important to consider an external HDD and a backup strategy (obviously you already know this)
- All those "installations" and "SD images" are things of the past, you need to switch to the container approach, I am talking about Docker; that will give you ease of installation, configuration, resource balancing, sharing, testing different setup, replication of setup and tenths of other things you will discover
1. I made a video about SD cards and showed how you can enable an HDD. So far I had no problems with my SQLite database. The number of points written is not a lot.
2. As soon as Docker gives full HW access it will be a good solution also for the RPI. I had problems to access a simple USB port on my PC. Otherwise a good concept with a lot of potentials.
Great video again, Andreas and superb timing as I too have a project which requires this type of sensor recording/display. Have you used an IOT2020?
No, I usually do not use such "professional" components like the IOT2020
Number One!!! Thanks.
??
Thanks for IMG file. Awesome stuff.
:-)
Great video and very timely for me as I am looking to complement my old RRD based graphics site with Grafana and Infuxdb.
I have used Pete’s ( and Mr Shark’s) script along with his node-red big timer for years!
While I do enjoy node red, I am looking to lower my dependence on it. Thus hints of ways to direct mqtt data directly to influxdb are high on my curiosity list!!
A other viewer commented he is using python scripts instead of Node-red
github.com/influxdata/telegraf/tree/master/plugins/inputs/mqtt_consumer
you r amazing , thanks
You are welcome!
Thanks for this video
You are welcome!
Im waiting to try this when I get another rpi3. I want to have as many sensors i can stuff into a rpi3 with a display to let me know as much info as possible.
Nice video Andreas!
In my case, instead of manual instalation, since I started using Home Assistant a few days ago, I just added InfluxDb and Grafana pluggins in Home Assistant which was like 5 minutes effort (Node Red I already added before).
Anyway your instructions helped a lot for a fast start. And a sugestion - if you didn't already - check out Home Assistant. Nice platform + combined with Mosquitto, Node Red and now InfluxDb and Grafana ... make it very interesting.
I use some manually programmed microcontrolers (bathroom humidity and fan control), Nextion displays from your video over MQTT, and dwo days ago I reprogram own application for LED strip and Sonoff Basic with esphomeyaml. Addionally using Alexa over Node Red, that fits very niceli into Home Assistant. New graphs from bathroom humidity, temperature and fan control is today nice addon.
Point is that any additional knowledge nicely fits in picture. Constant evolution...
Other commenters also mentioned that it is easy with Home assistant. Maybe I have to try it one...
just putting up xmas lights but as soon as that is done i will be following this video and cant wait to use Mr. Scargills 'script'
Maybe you watch the „ultimate pi server“ video first.
@@AndreasSpiess LOL i was just wondering that !! good job the mrs LOVES your accent so i get away with watching geeky videos
Thank you for the video, otherwise I have a question is it possible to integrate Grafana in a php page or is it an open source that can be easily integrated into a web page in your site?
This is a new concept for me (and it's very interesting), I have used a more traditional sql approach based on the work by Csongor Varga "Working with time series data in Node Red with SQLite" on CZcams. Your viewers might find the comparison interesting.
SQL is always a possibility. It is stable and does its job.
Very interesting and easy to jump in.
Actually I was using a Python Jupiter Notebook for Receiving, parsing, sending payload to the MQTT broker. This solution is great, especially for future maintenance
I never thought about using Jupiter Notebook for something like that. Now I have to look into it. Tanks for the info.
Good stuff!
Thank you!
Hi!
Could you please upload the NR-flow? I am really interested in the details in the function-nodes. I am struggling with my own input from NR to InfluxDB. :)
muchas gracias
You are welcome!
Hi Andreas, you might want to consider hass.io which has all as point and click install packages for all you talked about today! Great also to create automation for your tv lighting etc.. all over a web gui so easy it will withstand the wife test.
Maybe I have to look at hass.io once...
Nice for my next step, 3 monts from now, i donate when i download the sd card, Have a Nice Day and Thank,S for so much Help!
You are welcome!
Thank You, now i have made a sensor for my garden, Very litle solarpanel, a Mini_Pro that sleeps all Day with a Ds3231 clock, wakeup Once a Day send by WiFi with a ESP8266 with turn On with mosfet and give data to mqtt. The System works and have with 18650 batery power enoff.
The py and MQTT make a Nice http:). So you can do a lot with a ESP8862 in this way. No power problem. I see On my computer How dry the garden is and maby give iT Some action.
Thank you for the hundreds of really useful, interesting, up to date tutorials and info, you're pure gold! I have a question regarding the pi node-red server side, i need to have it in a distant location and feed it with 12V batteries (car battery) and let it run node-red influx db and grafana. What would be a good way to convert 12V down to 5V for the pi? Would one of these cheap buck converters do? What do you suggest?
thank you in advance!
A buck converter with at least 3A should do it. There are many projects around if you search with Google
That is too complex for my level but is very nice to know that if needed, this is a good reference for learning.
It should be easy if you install the image of the SD card to start playing and learning...
Another great video. By coincidence I was looking at just these products yesterday. Thank you for such a detailed but concise summary.
I am left with one question though, since InfluxDB automatically timestamps data is there any way of moving already existing historical data into an InfluxDB without it all being given the current time?
AFAIK you can insert also timestamp fields. At least this is what I did when I copied data from one RPi to another.
@Andreas issues are now solved, as you know by latest comments on Pete's blog... i'm MrShark :)
Thanks! The video was already out then. I pin your comment that everyone can read it.
@@AndreasSpiess problems were caused by a double install of sqlite node, 1 in Pete's node selection, 1 after that with unsafe-perms, and the sqldbs node which is unmainained from 2 years, we removed it... and now nodes are installed 1 by 1, instead on a single batch, so if 1 fails, it's just that, it does not make the others fail all together... now seems all fine
@@AndreasSpiess minor issues solved this morning: switched to official grafana repository instead of the previous one, as now it exists and it's working properly (it was not so when i added grafana and influxdb to script, a year ago or so...)
@@squalazzo Influxdb node still requires manual installation - tried just a couple of hours ago
Lovely!. I've used InfluxDB + Grafana for almost a year, but for me the confusing part is Node-Red. I've instead used a InfluxDB libary for my ESP8266 codes, there is also for Python, etc.. and a small program "Telegraf" can send data from windows, macs, and linux machines.
I did not know a library exists. Thanks for the tip. In my case, it is more flexible to use Node-Red as an "intermediate". And the rest (telegraph, python) is maybe for other channels ;-)
You can easily send data our from Arduino and the ESP8266 as HTTP, ingest it in a Node-Red inout, then send it out from Node-Red in MQTT on port 1883 or 8883 depending if you need certs for TLS1.2. It can do a lot as an intermediary, as Andreas said.
I started out using node-red and MongoDB, because you used to be able to get free cloud-based MongoDB services, but that time-based data consolidation is really nice, so I'll probably switch to InfluxDB for my projects from now on.
MongoDB seems to use quite a lot of disk space because it is made for other purposes. There is a comparison on the influxDB page...
@@AndreasSpiess yeah, MongoDB seems to store a lot of metadata for every record, which could be useful for some things, but not so much for simple IoT projects.
I’m glad I was already on the path of least resistance w/ Grafana
Maybe you have a look at the IOTstack project. There you find everything easier...
Thanks Andreas, did you consider to wrap this config in Docker/Docker Compose instead SD ? It would be easier to upgrade
No, not for the moment. Maybe later...