If you made your first flow already, you know that data is passed between each node. If you want to see this in a simplified example, please check the First Flow part first. In this series, I will show you what happens to the data between flows and how to set, change and manipulate between nodes and flows. It’s all about NodeRED and data processing.
NodeRED for beginners: 4. Data Processing
Before we start, it’s crucial that you read the introduction to JSON, as I’m not going to get into details about how JSON data is organised. It’s a 5 min read so do that unless you feel fairly comfortable with JSON.
The data between two nodes is sent as a JSON object called “message”. This message contains 2 main properties: topic & payload. While you can get away without using msg.topic for a long time, you will quickly see that manipulating the msg.payload is the daily bread of working with NodeRED. A lot of nodes, will either assign the topic themselves (or keep it unchanged) or let you type that information in.
Usually, when data is received by a node, the payload and topic data is checked and processed. The debug node is a perfect tool to monitor the progress of your data. You will be using it a lot before you learn how to do things.
Debug Node
The node will output the message to the debug window. The node comes in two modes. It will either output the entire message object or a specified property of the message object.
Inject Node Config:
Payload: "I LOVE NOTENOUGHTECH"
Topic: "Nice Topic"
By default, the node is set to output msg.payload (note the debug window will also show you the topic, so you don’t have to modify the node to get that). The second mode will show you the entire JSON message object including metadata.
Data transfer between nodes
Nodes can access, act upon or change the topic and payload data. Please pay attention to how information is affected by each node, otherwise, you can lose or override the data stored in the payload.
Nodes like switch node, read the content of the message (in this case switch node checks if the payload isn’t null) and passes the unchanged payload to the next node. The payload (or any part of the msg object) is evaluated and sent to the correct output.
In the second scenario, I have used a change node to move the topic value into the payload. You can see that the topic has been removed, and the new payload reads as “Nice Topic”. This new data is sent to the next node, the previous values are lost forever.
In the 3rd scenario, the node appends the information to the payload. The template node takes the data stored in the payload and processes it by adding extra text in front of the payload. The new data is sent to the next node. Note that topic is unaffected by this change.
Custom properties
You will quickly notice a need for storing the information aside to make sure that is not affected by other nodes and can be used later. To do so, we have to create a new custom property for the message object *(create a new key and value). We can do this in multiple ways. The two most common are the change node and the function node.
I used the change node to set a new property: msg.custom to “You CAN’T touch this” which is going to be added to our message. Looks what happens when I run it through the same template node as before:
Note, that I needed to change the debug node output to the message object to show you the unaffected custom properties of the node. Only payload has been modified. The msg.custom is unchanged.
Let’s do this again with the function node. I’m going to use the following code to set the properties:
var x = "You CAN'T touch this"; msg.custom = x; return msg;
The result is the same. The function node is more flexible but requires you to know a little bit of JavaSript. You will be able to assign multiple values and obtain the data from multiple sources.
Storing data across the nodes & flows
So far, all the information saved within the message object has been available between connected nodes only. Sooner or later you will need this info to be shared between nodes that are not connected, or between the flows all together.
To do this in NodeRED, you need to set variables. As with many other languages, variables come with different scopes. The scope of the variable is how widely it is shared with other parts of the NodeRED. There are 3 types of variables (objects):
- context (store information available within a single node)
- flow (store information available within the same flow)
- global (store information available to all flows)
All data stored in context variables is lost if the node redeployed. Flow and Global objects are lost if the server is restarted learn more about preserving your variables here. You should always pick the smallest scope possible for your scenario. It might be tempting to add the global variables everywhere, but you will decrease the NodeRED efficiency.
Context Object
context.set('YourVariable', value); // to store a variable (YourVariable)
var x = context.get('YourVariable'); //to retrieve a variable (YourVariable)
Let’s test it out on a simple counter. I’m going to count the number of times I have sent the data to the system. As this object is stored in a single node, I will have to enclose it all in a function node:
var count=context.get('counter') || 0; // create "counter" and set it to 0 if undefined
count +=1; // iterate +1
msg.payload= msg.payload+" "+count; // set a new payload & the counter
context.set('counter',count); // save the count in the "counter"
return msg;
Each time a new message is injected to the NodeRED the context object will save a new value. This can be recalled at the end of the script. The value is not shared with other nodes. To pass the value elsewhere I would have to write it out as a custom property.
Flow Object
This variable is shared with all nodes within the flow. It might be one of the most popular ways of sharing the data across the nodes. I usually keep my projects within the scope of a single flow.
To save the values as flow object, NodeRED uses the same syntax as before:
flow.set('YourVariable', value); // to store a variable (YourVariable)
var x = flow.get('YourVariable'); //to retrieve a variable (YourVariable)
To test it out, I will create two flows one to write a message, and the other one to read it.
The first inject node sends the “important message” as payload and function node assigns it as the flow object.
flow.set('storeit',msg.payload);
return msg;
In the second flow, I’m triggering the function node to retrieve the flow object and post it back as payload to the debug node:
var x = flow.get('storeit');
msg.payload = x;
return msg;
As you can see the debug node posts the same payload as the 1st flow.
Global Object
The global object is available to all flows. NodeRED uses the same syntax to set and retrieve the information stored in that variable, across all flows.
global.set('YourVariable', value); // to store a variable (YourVariable)
var x = global.get('YourVariable'); //to retrieve a variable (YourVariable)
I’m using the previous example, but moving the first inject node and function node to another flow. The message is saved in one flow and retrieved in another using store command in the function node:
global.set('storeit',msg.payload);
return msg;
and the retrieve:
var x = global.get('storeit');
msg.payload = x;
return msg;
Writing to file
Saving values to variables is easy but does not protect you from data loss. To store the chunks of data permanently, you can use the file nodes. As of version v0.19, you can also configure NodeRED to store the variables automatically. This way any data stored on your SD card will survive the power loss or server shut down and can be reloaded later.
I have injected a message with the topic “save this file” and payload “This is important” and saved it as a .txt file. The same path has been used in the read file node to get the data back. Please note that only payload has been saved to file. The topic has been lost.
There are options to configure how the data will be saved (append, overwrite etc) so pick the method that suits your needs best.
Conclusion
This should explain in details how data is processed in NodeRED. You will be able to process the information correctly. In the next tutorial, I will focus on connectivity. I’m going to explain how to get the data in and out of the node using MQTT, HTTP, IFTTT, TASKER and JOIN. Stay tuned, subscribe and support me if you can!
