Raspberry Pi IoT Server Tutorial: InfluxDB, MQTT, Grafana, Node-RED & Docker
Vložit
- čas přidán 14. 05. 2024
- This tutorial will create an internet of things server on your Raspberry Pi. This server will use Mosquitto MQTT, InfluxDB, Node-RED, Graphana in docker containers. We will use the helpful IoTStack tool which will configure our docker containers for us. This tutorial will allow you to collect any kind of sensor data you like, send it over your network to this raspberry pi server and visualise it on a Grafana dashboard.
This system consists of a series of sensor nodes that send sensor data in a JSON message over the network to a Raspberry Pi via MQTT. MQTT is a network protocol where a client pushes data to a broker, such as mosquitto, and the broker pushes that data to any client that has subscribed to that topic. A topic categorises the data that is sent. Any client that is subscribed to home slash sensor will be forwarded this MQTT message. NodeRED will subscribe to this topic, interpret the json message data and then send the data to a database to be stored and used. InfluxDB will store all the sensor data. Then a Graphana front end will query the database in order to display the data we want to see and do a whole host of other things.
The written version of this tutorial and the code used is available here:
learnembeddedsystems.co.uk/ea...
See other Raspberry Pi Tutorials here:
• Install Raspberry Pi O...
Timestamps
00:00 What this server will do
00:30 How will it do it?
02:16 Docker
03:59 IoTStack
06:46 InfluxDB Database
07:25 Node-RED Setup
10:15 Grafana Setup
11:58 Conclusion
You can find more embedded systems tutorials and projects on our website learnembeddedsystems.co.uk
-- Equipment we use regularly --
The following links are affiliate links where I may make a small percentage on qualifying sales through these links. Use the respective UK or US links listed.
Budget Soldering Iron: UK: amzn.to/3pQpvUu US: amzn.to/3bBawIJ
Breadboards: UK: amzn.to/3solZCe US: amzn.to/3dGsHiS
Jumper Cables: UK: amzn.to/37JfToi US: amzn.to/3r384S9
Camera: UK: amzn.to/3qKLwFr US: amzn.to/3pK0V7C
Lens: UK: amzn.to/380lQxp US: amzn.to/3knH3WL
Tripod: UK: amzn.to/37H0Nzw US: amzn.to/3byKHsV
All videos and tutorials on this channel and mentioned websites are for educational purposes only.
After what feels like 45000 tutorials, thank you - you finally gave me the insight needed to strip out unnecessary data and CHANGE the incoming data into a payload that influx was happy with. I now have a happy little chart of how bad the air is during the Canadian wildfire season.
This is ,by far, the best video I have seen for explaining an IOT server, thank you
Amazing guide! I watched it a couple of times and after a few hours I had my Grafana up and running, mind you I had never done anything MQTT before. For now I am just logging the room temp with an ESP32 and an LM35 temp sensor but I will be adding a lot of stuff soon and hopefully some solar production data in the future. Thanks a lot!
Maybe I just hadn't looked hard enough before but I think you're the first person I've seen explain grafana this well.
After struggling for a whole day trying to get mosquitto to work on my raspberry pi, following your video had me up and running within half an hour. This is a fantastic tutorial, thank you so much for the help ❤
Just a note, when setting up the MQTT server in Node-Red you can type in the name of the docker container as it is running in the same stack as Node-Red.
Great helpful video and website. One point, on the website it says to use 'docker-compose up' to check the status of the containers whereas you use 'docker-compose ps' in your video. I really appreciate the effort you are putting into your videos and supporting material.
Thank you! Your video is straightforward and to the point. Other videos were a complete waste of time. I'm glad I found your videos
Really good tutorial. I wish I watched this before setting up my IOT server
This is a really excellent tutorial. Every step is clearly explained in detail. Just one issue: it is generally not advised to use a leading forward slash '/' for the root an MQTT topic. But that's a minor quibble
Great tutorial to get started with IoT stack in R-Pi4. I am going to use this in a CM4 and use node red + mosquito talking to IO module downstream. Docker surely makes things flexible and easy.
Thank you for this excellent project & tutorial. It was my first step into the world of microcontrollers.
Absolutely brilliant.
Really well explained and informative video, subbed.
Really great video with written instructions. Thanks.
Thanks for this guide, I have referenced and linked to it because it helped me build a server on an M5Stack CM4 Stack.
Thank you for this video, i learned about portainer,grafana and influxdb in same video. thank u very much
Fantastic Video. Very crisp and to the point. Helped me learn the whole stack at one go.
Great to hear!
@@LearnEmbeddedSystems Is it possible to build this stack in the cloud? In other words can this container be ported to the cloud? A video on it would be helpful.
Incredible! Thanks for that. From Argentina.
Well done and I have a number of projects in mind that will use what was explained here.
Thank you! Very nice, well explained and easy to follow, bookmarking and saving your website tut too. I've got a couple renewable energy projects that I'm designing, and this should work perfectly for some monitoring hardware (tachometers on wind turbines and voltage readers on solar arrays). Also thinking about a few other metrics around the house I could definitely appreciate collecting :D.
Great video! Nice work. I would love to see the set up of the sensor node
You are inspiring. Thank you for sharing.
Very good tutorial!! Thank you I will experiment with this.
Amazing Video, cannot thank you enough. I tried it myself and here is feedback. The github code you shared is having issue with NodeRed, see error below -
I managed to install other containers by skipping nodered and later installed nodered as systemd.
It would be nice if you can make your github repo public in that way users can report issue and can pull the working code.
############
Start Stack:
docker-compose up -d --remove-orphans
WARNING: Some networks were defined but are not used by any service: nextcloud
Creating network "iotstack_default" with driver "bridge"
Building nodered
ERROR: Cannot locate specified Dockerfile: Dockerfile
Great tutorial... I am going to get the data from my netatmo devices and data from my "solar system" into this.
Very informative video, thanks!
great tutorial
Very helpful
Great video, thanks. I would be interested in a sensor node video as well.
Very informative. Definitely going to try this.
Let us know how you get on!
Great video! Thanks!
very useful iot server guide..
Well explained!
I've been debugging my IOTstack install for a week now -- I found that I had a corrupted SD Card. Best video I've seen so far getting right to the point, explaining what **should** happen. It would be nice if it included some more gotchas - I've been battling with my v2 B+ which is pretty slow running the IOTstack menu.
Maybe I could put an FAQ or common problems and how to solve them in the written article? Would that be a good compromise?
Storage problem is my biggest concern with Pi. Consider one with onboard eMMC or a M.2 NVMe. There are a few options out there using the CM4. Seeed minirouter, Waveshare IO boards- base B had a RTC, or the DFRobot mini router to name a few. The latter two require finding a CM4, which everyone says is on back order until Dec 2023.
@@LearnEmbeddedSystems that would be helpful, although I've just went the route of installing locally without docker containers and it is working
Running a database where data like temperatures etc. are constantly written with an SD card is only for people who enjoy reinstalling Raspis.
USB-SSD/HDD is mandatory for this application.
@@smaroukis I had the same problem, i have 5 corrupted sd cards on my desk now, I was about to try this videos method, I installed each item individually and the docker install script was killing sd cards
Absolutely awesome content 😊 + Your voice 🌟
Thank you!
Great video, also interested to see different sensor nodes that you can make and setup, thank you!
Thanks for your feedback! Will look at making a couple of videos on sensor nodes then :)
Very well done.
Very cool video on the Pi. For Arduino the ESP8266 NodeMCU is a cheaper option as it already have Wifi built-in and better power management. ESP32 would work too. Both very small form factor.
100% esp devices are better for 95% of iot projects and they are cheaper and smaller. Only problem is they are 3.3v devices so you might need a logic level shifter.
I use wemos mini clones plus bme280s for very compact temp/hum/pres sensors
love the video !! can you make another video about the hardware setup plz ? and how the arduino sent data to raspberry ?
Great tutorial, would be useful if Grafana could easily plot multiple values on one graph so that many sensors could be viewed easily
Hi, yes it can do this you just add a second query panel when you are setting up the graph (+ Query button) and it outputs to the same graph
@@janhatton4405 Thank you
Well Done. Thank you. Do you know if can I do that on an EC2 AWS instance?
This is great, going to try and connect the data from my Victron solar gear, Ill get back and let you know how I get on
It is very good idea to let i
Pi as IoT server as all IoT things can be made by myself.
Hi, extremely helpful video! Can one use the sensor data to control GPIO of another raspberry pi using MQTT?
love it!😄
Very useful. How can I do this on windows or mac? Are the instructions the same?
Thank You
That's an excellent tutorial !! Congratulations !! I would like to ask in case we have many sensors in /home/sensors. How can we get the unique id of each sensor and show in grafana? I've implemented your project with wemos D1 mini + DHT22 sensor + LCD 16x2 (parallel connection) and raspberry pi 4 (8Gb) as the server. The point is that viewing grafana i only see that data comes from sensor_data which is the db name from influxdb.
Superb video ❤
This is a very useful tutorial and a nice presentation!!
Just to be clear regarding viewing the containers using Portainer,
1) do I need to have an account in Portainer?
2) Is there any limitation on device numbers under the free account?
Nice video, thanks :)
Almost every single thing is very good explained in this video an exception of HOW CAN I MAKE THE INTERFACE COMMUNICATION BETWEEN MY ESP32 (WHICH IS READING THE DATA SENSOR) AND NODERED?
Nice intro!
Great tutorial. I have so many Pi 3s lying around, and wanna know if I can use them on this project? Now Im interested to see some videos on how tos with the sensor nodes.
Pi 3 should work just fine! ESP32 sensor Node video hopefully coming this weekend!
Thank you. Nice; quick and succinct.
One comment.Why skip over Portainer when - and this is from limited knowledge - I think you could have set everything up from within Portainer
Hi there, after running the linked IoT install script and script was processed, my Raspi boots up into blackscreen, no GUI available any longer. Luckily, I had a VNC server running, so I am able to connect via VNC viewer from my PC, although with a very poor resolution, that makes it nearly impossible to operate it this way. I have tried several reboots, with no success, obviously the script has "broken" my GUI, do you know how to make it available again ? I can access ssh via VNC. I'd like to avoid having to reformat my SD card again, Cheers mate !
can even upload these data to an app and control the actuaters from it only?
subscribed done, please make updated video for this some additional dashboard components
Great video....when I try to install thsi with a static IP adress set I can not connect to the server.
When must I set the static IP adress?
Hi, Thank you for this professional tutorial. I followed it and my ESP32 client can connect to the MQTT server jus if both the Raspberry pi and ESP32 on same wifi network. What am I missing?
I think MQTT is run upon a network, such as Wifi, bluetooth e.t.c.
Do you have any suggestion to decrease power usage to change it to make a reading every 1 mins instead of few seconds. And using only solar energy from a panel? To make sure its fully autonomous
You could look into lower-power mode on the device you are using, and use a watchdog timer or some other peripheral on the microcontroller to power it back on after X time to take another measurement.
Looking into this for a mesh network of environmental sensors including ground level pollution from vehicles. Thanks.
I am confused about Node-RED. Is it design-time-only tool? Or does it have to run also at runtime, after I deploy my solution?
Hello, excellent video! I'm Brazilian and I don't speak English (I'm using Google Translate lol) and I would like to know if I can use a conventional computer with the Raspberry Pi system installed in place of the Raspberry Pi board. Thank you in advance!I would also like to know if I can use the ESP8266 ESP-01 WiFi Module for the wireless connection, as I use an Arduino board without a built-in WiFi module.
Great video. But as I understand it, MQTT messages should not have a starting "/".
Amazing tutorial. May i have a question ?
Say i finish all the installation above, is it possible that my system can operate completely offline ? ( reading, workflow, visualization, modifying … ) Like i connect those 2 boards to the same network which doesnt have internet connection.
Yes, it shouldn't need an active internet connection
very useful video, I did this Iot Server, but with DHT22 temperature sensor with ESP32. Did anyone have the problem that at 7.30 am and pm for some reason grafana stop saving data? Any suggestion?
Based on my 3+y expirience InfluxDB wrong choice for time-series data. It can't be used on production setup. The better way - prometheus, or graphite with clickhouse backend.
Well this video was exepsional congrats! It worked fine for me, however for some reason I can't add the libraries of the functions on the pallet in node-red.
Mine did not work as when i want to start the stack it returns an error that three of the apps are obsolete. Just wondering if you had an issue like this or any way to resolve
Hello ! thank you for your video. On which raspberry pi did you install those softwares ? RPI 3 ? RPI 4 ? thank you !
I found an RPI3 struggled, RPI4 with SSD was better
Hi, thx for the nice tutorial but as some people said earlier, I think there is an issue with th IOTStack and Node-red.
Indeed while installing the stack the installation gets stuck and Node-red is not installed.
I found out the reason was because I have configured the Raspberry IP as Static ( like shown in your previous tutorial :) )
It seems that during installation an IP has to be assigned to the containers and without DHCP it gets blocked...
Can it be?
This exact thing happened to me, I also have a static IP set in /etc/dhcpcd.conf. Disabling the static IP resolved it. Nice find!
Please note: In the video, you state that you can check what containers are running with the command 'docker-compose ps' while on the related web page you show the command as 'docker-compose up'
docker-compose ps shows you status of docker containers, docker-compose up execute the docker-compose.yml file(that is something like a config dile for docker) you also use this command only when starting docker
Wow. You are doing what I have been reading so many tutorials to try to do. I am setting this up to remotely monitor a vacation property. I am looking at setting up a sensor that will measure the distance to the surface of water in a sump pit. I was wondering if you could provide some guidance on this. I was thinking of using a HC-SR04 to monitor the distance to the surface of the water. Wondering if you have any thoughts on this.
HC-SR04 is probably the best pre-built option - there aren't any real water level sensors out there that are bigger than a few inches. However, ultrasonic distance sensors are relatively sensitive, meaning a slight off-angle or shaking it could ruin the distance it computes. I'd recommend looking into creating your own water-level sensor, which would be the most-reliable long-term. I'm pretty sure one can be built with the right spare parts.
@@toastedregret1601 Thanks. I am thinking of mounting it to a piece of angle bar to make sure it is not moving. I have seen diy sensors that use conductance through screws and bolts. Just not sure how accurate that is going to be. I am currious how I would calibrate it outside of submersion. I also don't know that ground water with all the 'impurities' would have the same kind of conductance to get a feel for depth of the water level. Are you suggesting something different with an ultrasonic diy? Thanks.again.
@@mediaicon8774 I've used the ultrasonic distance sensors quite a bit for different projects, and they are pretty great to measure distances up to several feet, as long as you can get the transmission portion of the sensor mounted on something that would keep it from moving around. The sensor's face should ideally be as parallel as possible to the surface you're trying to get a reading on. Conductance is an alternative but the drawback is it may require calibration if exposed to liquids for a long period of time. It's probably worth trying an ultrasonic sensor and see if it suits your needs.
in theory this can be done on a headless pi zero from a pi because youre using the ip address to setup all the interfaces? (im a newb)
I have a question. This thing (API?) called "IOTstack" was made using docker?
Hello,
I want to set the parameters of an industrial drive using a node through RS485 serial communication.
In this project, I need to use the hex codes of an industrial drive and control the drive using the hex codes.
Can you help me with this project with node-red-node-serialport?
Thanks
Thank you for the generous super thanks! I really appreciate your support :)
If I'm using influxdb on web browser instead of installing it to my pc, will it not work?
Because when I'm connecting nodered with influxdb it's not binding with influxdb. I've used the host name for influxdb as my url and port as 8086.
So one topic is for each arduino with the sensor suite, so each room has a different topic? or is it a bunch of sensors sending data to one arduino and it is transmitting different topics per room?
Each arduino is an individual sensor node in the video. There are cheaper microcontroller alternatives with more IoT features (timers, power modes, WiFi, BLE), such as the ESP32. Arduino in this case was used as an example.
I've been trying to get a similar setup but using CasaOS instead of IOTStack (but resorting to IOTStack for hints about how to set up Mosquitto and Influxdb which are not in the CasaOS app store). This gives a nice web front end to the various web apps and easy remote file access. Was wondering if anyone knew of a web app for Mosquitto, something that would allow web page monitoring of message traffic (similar to MQTTX, but web) and maybe some configuration GUI avoiding all the command line and config file editing jugitsu, similar to what Portainer does for Docker. Likewise for InfluxDB. I am very confused about what URL to use for one container from within another. What are the (different) URLs listed in Portainer for each container for? I found using the specific URL for the Mosquitto container worked in some cases where the URL of the Pi did not.
Maybe wireshark to look at the datapackets being sent over the network? I know it's useful for UDP.
Good
Which raspberry bi model is used here ?
this video is supposing all devices are connected to the same network, right? Is there a way to implement same solution you provided, but with devices that are around 20-40 metres away from each other. either by somehow giving them same network or by working it out without same network.
I think VPS digital ocean is the solution
0.01 is this the circuit connection i need to make
NIce
it worked fine for a few hours and then started returning this error ( rc=-2Attempting MQTT connection...failed ) what might be the issue
Is there any code for Raspberry Pi Pico W? Or has anyone tried this setup with that board?
I kinda miss the hardware side info?
Has anyone been able to enable and enforce users for the mqtt container?
Where do you discuss the static IP? Should this be set in the router?
It should be set within your router, otherwise it will change the next time the raspberry pi power cycles. The video didn't mention that. Go to the IP address for your router, type the username and password (the very first time, it should be "admin" , "password"), find the raspberry pi in the list of devices on the router, view more info, and set "reserve" IP address.
(3:29) " fresh install of headless Raspbian OS ... (3:37) we have a video Linked In the cards above ... "
Anyone know where this link is?
I was wondering. Could anyone point me in a direction where I'm using a microbit instead of the Arduino? Why.. I don't have an arduno :(
I am a novice in IoT, and appreciate if you could explain how to do the sensor board part. Thanks!
The video uses an arduino as the microcontroller unit (MCU), but if you don't want the hassle of connecting a WiFi module to it and managing the communication between the two modules, you can get a microcontroller with wireless capability (bluetooth, WiFi) embedded directly into it. I'd reccomend using an ESP32.
can i use 32 bit raspberry pi os
Was so hopeful - but hit a big problem. When i run the Start Stack command, it dies quickly at the Grafana step "ERROR: no matching manifest for linux/arm/v7 in the manifest list entries" (FYI - i'm installing onto Rpi 4B) I googled that error, and found multiple questions about it for other packages, but i really don't understand enough to know how to fix it here. I'm stuck. Anybody willing to lead me out of this road block?
if the RPi is running as a server, how from an outside windows computer could check the database and dashboards? or do I need to enter to RPi to check that?
You can SSH into the raspberry pi similar to a virtual machine. There are tutorials online on how to do it. It's sometimes referred to as "headless mode". You can also take it a step further and access the raspberry pi from anywhere in the world by exposing its port on the router. That's a huge security risk though - you can also alternatively use an application like VNCViewer, set the raspberry pi address as static so it doesn't change on startup, and access the raspberry pi via VNCViewer from ideally anywhere.
how do you configure MOSQUITTO? I selected the IP of my RPI3 and I am not receiving anything :(
Why do it shows “docker-compose : not found” when i wants to start stack? Help mee
After installing the stack I can see all components up as shown 06:18, yet when trying to run the browser with port 9000,it says: "New Portainer installation
Your Portainer instance timed out for security purposes. To re-enable your Portainer instance, you will need to restart Portainer.". Restart the RPI didn't solve it. Any idea?
Solved with docker stop portainer-ce, then docker start portainer-ce