Big plans ahead! I want to harvest environmental data from my DIY Smart Heating system and feed it to Machine Learning. The plan is great, I have 7 Sonoff ZigBee Temperature sensors scattered around the house, weather API, room occupancy data, but no reasonable solution to store this. There are great tutorials that show you how to get started with InfluxDB & Grafana, but the guides stop as soon as all is connected. I will take this a step further show you how to harvest IoT data too.
Why InfluxDB & Grafana
InfluxDB is a time-based DB. It means that it optimises the data for time, and comes with several retentions policies to keep the more demanding databases small. It’s ideal for IoT environments, especially for Home Automation solutions.
You don’t need Grafana to store the data. Grafana excels at displaying data in a graphical way, letting you understand the data better or put it in a better context. InfluxDB integration comes in built-in, so it’s a match made in heaven if you want to understand your data better.
Both are very easy to install and compatible with Raspberry Pi, HA and NodeRED. Before you jump further, I would really encourage you to dump the SD card and switch to the boot from USB on Raspberry Pi. You could consider SSD compatible cases like Argon ONE M.2 (review) or DeskPi (review) or just go barebones with SunFounder NAS kit (review) or Geekworm X857 mSata Extension (review).
Installation
For the most part, the installation is easy and straightforward. You can start with either of them. I will show you how to connect the services, how to store the values efficiently, so you can present your data in the most insightful way.
Grafana
It’s always a good idea to update your Raspberry Pi before you install new software, so let’s start with that:
sudo apt-get update
sudo apt-get upgrade -yThen grab the key, and add the Grafana to the repository and install it (one line at a time)
wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -
echo "deb https://packages.grafana.com/oss/deb stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list
sudo apt-get update
sudo apt-get install -y grafana
OR
sudo apt-get install -y grafana=9.0.8 //see belowOn my recent installations, I came across an issue on 32bit Raspberry Pi OS (Bullseye) where no matter what I do, I cannot add InfluxDB (v 1.8.10) as a data source. After hours of trial and error I figured out that newer builds of Grafana than 9.0.8 were producing this error.
At this point, you need to switch to the 64bit version of Raspberry Pi OS if you want to work with a newer version of Grafana and InfluxDB 2.X (with Flux queries support)
Once installed, you want Grafana to start automatically as a service:
sudo /bin/systemctl enable grafana-server
sudo /bin/systemctl start grafana-serverAnd you are ready to log in at http://hostname:3000 or http://raspberry_IP:3000 – to access the dashboard using the default credentials:
user: admin
pass: adminInfluxDB
Installing InfluxDB is done in a similar way, but extra steps are needed to create and configure your database. Let’s grab the key and update the repo (execute one at a time)
curl -sL https://repos.influxdata.com/influxdb.key | sudo apt-key add -
echo "deb https://repos.influxdata.com/debian stretch stable" | sudo tee /etc/apt/sources.list.d/influxdb.list
sudo apt update
sudo apt-get install -y influxdbEnable it as a system service:
sudo systemctl unmask influxdb
sudo systemctl enable influxdb
sudo systemctl start influxdbIt’s all up and running so let’s create users and a database (homeautomation – no spaces):
influx
CREATE DATABASE homeautomationWe will configure the users inside the db:
USE homeautomation
CREATE USER <name> WITH PASSWORD '<password>' WITH ALL PRIVILEGES
exitInfluxDB & Grafana
Time to connect both services together. It’s a good practice to keep separate users for different services, so let’s start with setting up InfluxDB for work with NodeRED (nodered) and Grafana (grafana). Create 2 more users:
influx
USE homeautomation
CREATE USER nodered WITH PASSWORD '<password>' WITH ALL PRIVILEGES
grant all privileges on homeautomation to noderedand the same for Grafana
influx
USE homeautomation
CREATE USER grafana WITH PASSWORD '<password>' WITH ALL PRIVILEGES
grant all privileges on homeautomation to grafanaYou can check users with:
show users
exitIt’s time to log in to Grafana. I assume you have changed the default credentials (log in at the bottom left corner if you don’t see options). Navigate to Configuration – Data Sources and add a new source. You will notice InfluxDB integration.
Name the database to your liking, provide the URL (the default should be (http://localhost:8086) then authenticate your InfluxDB (InfluxDB Details) with grafana username and password created earlier. Set the query to GET. Provide the db name (homeautomation) and hit Save and test.
Embed charts in Grafana
I want to be able to share and embed my charts in NodeRED without logging in, to do so, I have to modify Grafana’s config file. Connect via SSH to your Raspberry Pi and open:
sudo nano /etc/grafana/grafana.iniand set these (you have to change the value and remove “;”:
#####security#########
# set to true if you want to allow browsers to r$
allow_embedding = true
#####Anonymous Auth######
# enable anonymous access
enabled = true
Save the file and restart Grafana – you will be able to embed charts in NodeRED Dashboard using URL and a Template node.
NodeRED
Open NodeRED and install node-red-contrib-influxdb from palette manager. It will help out with saving data to InfluxDB. I will cover how to store the data and why in the next paragraph. As we setting things up install node-red-dashboard if you don’t have it yet.
To display an embedded chart from Grafana, you will have to open the dashboard and select the option to share a chart from the chart’s menu. Share option will give tabs to embed the URL. If you cannot see this – make sure you enabled it in the grafana.ini.
Copied URL can be used in this template node (submit the URL as msg.payload or paste the URL there). You can change the size of the chart by changing width="700px" height="400px".
<html>
<iframe id="Grafana1" width="700px" height="400px" style = "overflow:hidden; margin:auto; border:0px solid green;"></iframe>
<script>
(function(scope) {
scope.$watch('msg', function(msg) {
var x = document.getElementById('Grafana1');
x.setAttribute('src', msg.payload);
});
})(scope);
</script>
</html>There is more to collecting data
Take your time and think carefully about how you want to store your data. It’s important as it will determine how you can access it and how can it be displayed and processed by Grafana. InfluxDB indexes by time and tags. It means that querying the DB using time ranges and associated tags is quicker than standard SQL databases.
The challenge is to organise your data before you submit it to the database. It will define how useful that data be for years to come. InfluxDB has the following schema:
measurement : "weather"; //name of the measurement
[{ temp: 19, //fields (key and value)
humidity: 56, //fields (key and value)
pressure: 998}, //fields (key and value)
{location: "outside"}, //tags (key and value)
{timestamp: 2019-03-16T01:01:26.234Z"} //timestamp It could be tricky to decide how would you store weather data. Should each measurement (humidity, temperature, pressure) become an individual measurement or should they be recorded as fields in a measurement called weather? And what’s up with tags?
Let me try to answer this with an example. I’m sure you figured by now that the code above is a point taken from weather API. You could be tempted to break this down into 3 individual measurements – one for temperature, one for humidity and one for pressure. Take a step back and think of what are you measuring and for what purpose?
Weather data is connected. Temperature, pressure and humidity create the weather outside which changes based on location and time. It makes perfect sense to track these 3 values at the same time and keep it as one measurement. Tags could be used to define the location of the measurement in case you want to monitor.
You can also take another approach and measure each metric separately especially if you want Grafana to perform calculations between values.
Another example could be measurements of the same IoT sensors in each room. You might be tempted to create a measurement for each room, but using tags could bring the advantage of using this data together. In an IoT scenario, a household would me my measurement, where individual measurements get a tag associated with the room the data is coming from.
Plan ahead and Grafana will thank you later. You will see this when we will start to plot this data in graphs.
Store InfluxDB data in NodeRED
NodeRED has an influx out node to save the data in your database. Use the settings and the user-created in InfluxDB to connect the node to your NodeRED. For IoT precision of the DB should be set to seconds.
To send a data point to InfluxDB with NodeRED you have to format it in the following way:
msg.measurement = "weather"; //measurement
msg.payload = [{
temp: 19, //fields
humidity: 45,
pressure: 998},
{location: "outside"} //tags
];
return msg;Note that timestamp is automatically submitted when the update is taking place. You can submit an array of updates in a single call – something you should consider if you are going to spam the DB often with values. My updates will happen every 30-60 seconds so I’m not worried.
Sample: (using weather API to save weather info in InfluxDB – I’m going to use that in further examples – configuration available in the Setup of the function node)
Plotting charts in Grafana
Give it an hour or so before playing with the dashboard. You might be tempted to check it out as soon as you have 2-3 points, but the truth is, the more points available the more options Grafana gives you. I’m going to use Weather API to plot the graphs.
Grafana is split into dashboards. You can think of a dashboard as a single webpage with layouts of graphs and charts. These can display simple values or plot incredibly useful and complex mathematical calculations based on manipulating data points. This is why organising your data structure is important.
To plot a chart select a measurement first. This will load the fields and tags for you. Each query will display a single data structure. I’m calling it a structure as the data points can be modified. To chart a simple temperature history from the weather API – in the example add a dashboard, and then add a panel:
In the Query section – use the pencil option to change the input mode and paste this:
SELECT "temp" FROM "weather" WHERE ("location" = 'outside') AND $timeFilter
Then you can switch to edit mode to see what happened. We have selected a measurement “weather” and loaded all fields and tags. I picked the temperature fields and tag outside. If I had weather sets for other locations, I could pick the dataset I want.
Then I selected the mean value of the datapoint submitted and displayed it as time series without grouping. My updates happen every 5 min so that’s the resolution of my chart. You should see these fields in the visual edit mode. In the side panel, apart from visual options you can define the type of data displayed – select temperature and marvel at your chart.
It’s time to take a closer look at the options available. You will notice that you can change a lot of values to generate new graphs. Try this to show how the temperature had changed based on mean values measured every hour:
SELECT mean("temp") FROM "weather" WHERE ("location" = 'outside') AND $timeFilter GROUP BY time(1h)
The same set of data, completely new chart!
Now if you had a measurement from another location submitted to the measurement “weather” you could display both temperatures in the same chart, or even only draw a line displaying a difference between temperatures in 2 points.
SELECT count("temp") FROM "weather" WHERE ("location" = 'outside') AND $timeFilter GROUP BY time(1h)
Set the values to this, and you will get the information about how many calls to API the script is making every hour. Any dips in the chart will show issues with obtaining the information.
This is just a beginning since there are dozens of calculations and different options to display the data. This is just to get you started and let you think about how your data structure should be composed to visualise the information you need.
Final thoughts
The road to mastering Grafana and InfluxDB is long and I’m far from understanding everything available there. This post is to just get you started and put you on the right track as many other tutorials don’t go past the installing stage. Plan the data structure ahead. Think of how the data will serve you and write it in the manner that meets your need and Grafana will be there to help you visualise it. Plus it looks cool, while embedding keeps all the information you need in a single place. Any questions? Let me know in this Reddit thread.
