Home Raspberry Pi Tutorials A comprehensive guide to Grafana & InfluxDB

A comprehensive guide to Grafana & InfluxDB

Embedding IoT charts in NodeRED running on Raspberry Pi

Big plans ahead! I want to harvest environmental data from my DIY Smart Heating system and feed it to Machine Learning. The plan is great, I have 7 Sonoff ZigBee Temperature sensors scattered around the house, weather API, room occupancy data, but no reasonable solution to store this. There are great tutorials that show you how to get started with InfluxDB & Grafana, but the guides stop as soon as all is connected. I will take this a step further show you how to harvest IoT data too.

Why InfluxDB & Grafana

InfluxDB is a time-based DB. It means that it optimises the data for time, and comes with several retentions policy to keep the more demanding databases small. It’s ideal for IoT environment, especially for Home Automation solutions.

You don’t need Grafana to store the data. Grafana excels at displaying data in a graphical way, letting you understand the data better or put it in a better context. InfluxDB integration comes in built-in, so it’s a match made in heaven if you want to understand your data better.

Both are very easy to install and compatible with Raspberry Pi, HA and NodeRED. Before you jump further, I would really encourage you to dump SD card and switch to the boot from USB on Raspberry Pi. You could consider SSD compatible cases like Argon ONE M.2 (review) or DeskPi (review) or just go barebones with SunFounder NAS kit (review) or Geekworm X857 mSata Extension (review).

Installation

For the most part, the installation is easy and straight forward. You can start with either of them. I will show you how to connect the services, how to store the values efficiently, so you can present your data in the most insightful way.

Grafana

It’s always a good idea to update your Raspberry Pi before you install new software, so let’s start with that:

sudo apt-get update
sudo apt-get upgrade -y

Then grab the key, and add the Grafana to the reopsitory and install it (one line at the time)

echo "deb https://packages.grafana.com/oss/deb stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list
sudo apt-get update
sudo apt-get install -y grafana

Once installed, you want Grafana to start automatically as service:

sudo /bin/systemctl enable grafana-server
sudo /bin/systemctl start grafana-server

And you are ready to log in at http://hostname:3000 or http://raspberry_IP:3000 – to access the dashboard use the default credentials:

user: admin
pass: admin

InfluxDB

Installing InfluxDB is done in a similar way, but extra steps are needed to create and configure your database. Let’s grab the key and update the repo (execute one at the time)

curl -sL https://repos.influxdata.com/influxdb.key | sudo apt-key add -
echo "deb https://repos.influxdata.com/debian stretch stable" | sudo tee /etc/apt/sources.list.d/influxdb.list
sudo apt update
sudo apt-get install -y influxdb

Enable it as a system service:

sudo systemctl unmask influxdb 
sudo systemctl enable influxdb
sudo systemctl start influxdb

It’s all up and running so let’s create users and database (homeautomation – no spaces):

influx
CREATE DATABASE homeautomation

We will configure the users inside the db:

USE homeautomation
CREATE USER <name> WITH PASSWORD <password> WITH ALL PRIVILEGES
exit

InfluxDB & Grafana

Time to connect both services together. It’s a good practice to keep separate users for different services, so let’s start with setting up InfluxDB for work with NodeRED (nodered) and Grafana (grafana). Create 2 more users:

influx
USE homeautomation
CREATE USER nodered WITH PASSWORD <password> WITH ALL PRIVILEGES
grant all privileges on homeautomation to nodered

and the same for Grafana

influx
USE homeautomation
CREATE USER grafana WITH PASSWORD <password> WITH ALL PRIVILEGES
grant all privileges on homeautomation to grafana

You can check users with:

show users
exit

It’s time to log in to Grafana. I assume you have changed the default credentials (log in at the bottom left corner if you don’t see options). Navigate to Configuration Data Sources and add a new source. You will notice InfluxDB integration.

Name the database to your liking, provide the URL (the default should be (http://localhost:8086) then authenticate your InfluxDB (InfluxDB Details) with grafana username and password created earlier. Set the query to GET. Provide the db name (homeautomation) and hit Save and test.

Embed charts in Grafana

I want to be able to share and embed my charts in NodeRED without logging in, to do so, I have to modify Grafana’s config file. Connect via SSH to your Raspberry Pi and open:

sudo nano /etc/grafana/grafana.ini

and set these (you have to change the value and remove “;”:

#####security#########

# set to true if you want to allow browsers to r$
allow_embedding = true

#####Anonymous Auth######

# enable anonymous access
enabled = true

Save the file and restart Grafana – you will be able to embed charts in NodeRED Dasboard using URL and a Template node.

NodeRED

Open NodeRED and install node-red-contrib-influxdb from palette manager. It will help out with saving data to InfluxDB. I will cover how to store the data and why in the next paragraph. As we setting things up, install node-red-dashboard if you don’t have it yet.

To display embedded chart from Grafana, you will have to open the dashboard, and select the option to share a chart from the chart’s menu. Share option will give tabs to embed the URL. If you cannot see this – make sure you enabled it in the grafana.ini.

Copied URL can be used in this template node (submit the URL as msg.payload or paste the URL there). You can change the size of the chart by changing width="700px" height="400px".

<html>
<iframe id="Grafana1" width="700px" height="400px" style = "overflow:hidden; margin:auto; border:0px solid green;"></iframe>
    <script>
    (function(scope) {
        scope.$watch('msg', function(msg) {
            var x = document.getElementById('Grafana1');
            x.setAttribute('src', msg.payload); 
            });
    })(scope);
    </script>
</html>

There is more to collecting data

Take your time and think carefully of how you want to store your data. It’s important as it will determine how you can access it and how can it be displayed and processed by Grafana. InfluxDB indexes by time and tags. It means that querying the DB using time ranges and associated tags is quicker than standard SQL databases.

The challenge is to organise your data before you submit it to the database. It will define how useful that data be for years to come. InfluxDB has the following schema:

measurement : "weather";                            //name of the measurement
[{    temp: 19,                                     //fields (key and value)
      humidity: 56,                                 //fields (key and value)
      pressure: 998},                               //fields (key and value)
{location: "outside"},                              //tags (key and value)
{timestamp: 2019-03-16T01:01:26.234Z"}              //timestamp            

It could be tricky to decide how would you store weather data. Should each measurement (humidity, temperature, pressure) become individual measurement or should they be recorded as fields in a measurement called weather? And what’s up with tags?

Let me try to answer this with an example. I’m sure you figured by now that the code above is a point taken from weather API. You could be tempted to break this down to 3 individual measurements – one for temperature, one for humidity and one for pressure. Take a step back and think of what are you measuring and for what purpose?

Weather data is connected. Temperature, pressure and humidity create the weather outside which changes based on location and time. It makes perfect sense to track these 3 values at the same time and keep it as one measurement. Tags could be used to define a location of the measurement in case you want to monitor.

You can also take another approach and measure each metric separately especially if you want Grafana to perform calculations between values.

Another example could be measurements of the same IoT sensors in each room. You might be tempted to create a measurement for each room, but using tags could bring the advantage of using this data together. In IoT scenario, a household would me my measurement, where individual measurements get a tag associated with the room the data is coming from.

Plan ahead and Grafana will thank you later. You will see this when we will start to plot this data in graphs.

Store InfluxDB data in NodeRED

NodeRED has a influx out node to save the data your database. Use the settings and the user created in InfluxDB to connect the node to your NodeRED. For IoT precision of the DB should be set to seconds.

To send a data point to InfluxDB with NodeRED you have to format it in a following way:

msg.measurement =  "weather";           //measurement
msg.payload = [{
            temp: 19,                   //fields
            humidity: 45,
            pressure: 998},
            
    {location: "outside"}               //tags
];
return msg;

Note that timestamp is automatically submitted when the update is taking place. You can submit an array of updates in a single call – something you should consider if you are going to spam the DB often with values. My updates will happen every 30-60 seconds so I’m not worried.

Sample: (using weather API to save weather info in InfluxDB – I’m going to use that in further examples – configuration available in the Setup of the function node)

Example flow: Weather API -> InfluxDB
[{"id":"8da5ebc3.3ce6e8","type":"group","z":"8dd9238a.5d62","name":"Weather API","style":{"stroke":"#ffC000","fill":"#ffcf3f","fill-opacity":"0.12","label":true},"nodes":["c93b0f9a.91d24","943e96a3.21be98","f83a30a6.523d3","de9ac35b.69b59"],"x":54,"y":199,"w":832,"h":82},{"id":"c93b0f9a.91d24","type":"inject","z":"8dd9238a.5d62","g":"8da5ebc3.3ce6e8","name":"Every 10 min","props":[{"p":"payload"}],"repeat":"600","crontab":"","once":true,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":180,"y":240,"wires":[["de9ac35b.69b59"]]},{"id":"943e96a3.21be98","type":"http request","z":"8dd9238a.5d62","g":"8da5ebc3.3ce6e8","name":"","method":"GET","ret":"obj","paytoqs":"ignore","url":"","tls":"","persist":false,"proxy":"","authType":"","x":570,"y":240,"wires":[["f83a30a6.523d3"]]},{"id":"f83a30a6.523d3","type":"change","z":"8dd9238a.5d62","g":"8da5ebc3.3ce6e8","name":"","rules":[{"t":"set","p":"weather","pt":"flow","to":"payload","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":780,"y":240,"wires":[[]]},{"id":"de9ac35b.69b59","type":"function","z":"8dd9238a.5d62","g":"8da5ebc3.3ce6e8","name":"Configure API URL","func":"var lat = global.get('Latitude');\nvar lon = global.get('Longitude');\nvar API = global.get('weatherAPI');\n\n// get weather\nmsg.url= \"api.openweathermap.org/data/2.5/weather?lat=\"+lat+\"&lon=\"+lon+\"&appid=\"+API+\"&units=metric\"\nreturn msg;","outputs":1,"noerr":0,"initialize":"/*\nglobal.set('Latitude', \"XXXXX\");\nglobal.set('Longitude', \"XXXXX\");\nglobal.set('weatherAPI', \"XXXXX\");\n*/","finalize":"","x":370,"y":240,"wires":[["943e96a3.21be98"]]}]

Plotting charts in Grafana

Give it an hour or so before playing with the dashboard. You might be tempted to check it out as soon as you have 2-3 points, but the truth is, the more points available the more options Grafana gives you. I’m going to use Weather API to plot the graphs.

Grafana is split into dashboard. You can think of dashboard as a single webpage with layouts of graphs and charts. These can display simple values, or plot incredibly useful and complex mathematical calculations based on manipulating data points. This is why organising your data structure is important.

To plot a chart select a measurement first. This will load the fields and tags for you. Each query will display a single data structure. I’m calling it a structure as the datapoints can be modified. To chart a simple temperature history from the weather API – in the example add a dashboard, and then add a panel:

In the Query section – use pencil option to change the input mode and paste this:

SELECT "temp" FROM "weather" WHERE ("location" = 'outside') AND $timeFilter

Then you can switch to edit mode to see what happened. We have selected a measurement “weather” and loaded all fields and tags. I picked the temperature fields and tag outside. If I had weather sets for other locations, I could pick the dataset I want.

Then I selected the mean value of the datapoint submitted and displayed it as time series without grouping. My updates happen every 5 min so that’s the resolution of my chart. You should see these fields in the visual edit mode. In the side panel, apart from visual options you can define the type of the data displayed – select temperature and marvel at your chart.

It’s time to take a closer look at the options available. You will notice that you can change a lot of values to generate new graphs. Try this to show how the temperature had changed based on mean values measured every hour:

SELECT mean("temp") FROM "weather" WHERE ("location" = 'outside') AND $timeFilter GROUP BY time(1h)

The same set of data, completely new chart!

Now if you had a measurement from another location submitted to the measurement “weather” you could display both temperatures in the same chart, or even only draw a line displaying a difference between temperature in 2 points.

SELECT count("temp") FROM "weather" WHERE ("location" = 'outside') AND $timeFilter GROUP BY time(1h)

Set the values to this, and you will get the information about how many calls to API the script is making every hour. Any dips in the chart will show issues with obtaining the information.

This is just a beginning, since there are dozens of calculations and different options to display the data. This is just to get you started and let you think about how your data structure should be composed to visualise the information you need.

Final thoughts

The road to mastering Grafana and InfluxDB is long and I’m far from understanding everything available there. This post is to just get you started and put you on the right track as many other tutorials don’t go past the installing stage. Plan the data structure ahead. Think of how the data will serve you and write it in the manner that meets your need and Grafana will be there to help you visualise it. Plus it looks cool, while embedding keeps all the information you need in a single place. Any questions? Let me know in this Reddit thread.

PayPal

Nothing says "Thank you" better than keeping my coffee jar topped up!

Patreon

Support me on Patreon and get an early access to tutorial files and videos.

image/svg+xml

Bitcoin (BTC)

Use this QR to keep me caffeinated with BTC: 1FwFqqh71mUTENcRe9q4s9AWFgoc8BA9ZU

M5Paper

Programable, ESP32 based awesome dev platform with 4.7 e-ink display by M5Stack

More HATs

Argon One M.2

Enclose Raspberry Pi 4 inside this great case with custom I/O, cooling and GPIO and M.2 SSD support

More cases on

Best Raspberry Pi Projects

How to use Raspberry PI as WOL (wake on lan) server

0
While you could wake up your PC from a mobile directly, having a dedicated server capable of doing so is the best solution. The reason is simple. You can hook up as many devices as you wish with a single endpoint. This is why Raspberry Pi is perfect for this.

How fast Raspberry Pi NAS is?

0
Let's see how fast Raspberry Pi NAS really is?

Argon18: Argon ONE SSD modification

0
Argon One case just got better - now you can boot it from USB without ruining the design thanks to Argon 18: Argon One SSD modification

The most pointless upgrade to my NAS – Raspberry Pi 3B+

0
This upgrade is almost pointless!

HOW TO...

A comprehensive guide to Grafana & InfluxDB

0
How to use Grafana and InfluxDB on Raspberry Pi for IoT sensors in home automation

How to boot Raspberry Pi 4 from USB

0
How to set up and boot Raspberry Pi 4 from USB drive - headless guide.

Raspberry Pi Zero, Raspberry Pi 3A+ WiFi without keyboard or cables

0
Getting Raspberry Pi boards connected in seconds

Everything you need to start using JSON

0
It's JSON not Jason, just saying!

How to use Join App in NodeRED

0
The day finally comes where Join app is almost everywhere!