MQTT in Node-RED [HowTo]

Node-RED has built-in support to act as an MQTT client, which can be used with TTN’s MQTT Data API.

:warning: For new developments, one should not use the The Things Network’s node-red-contrib-ttn, which is no longer supported.

Setting up the MQTT client

The following may seem a lot of steps, but Node-RED will guide one through the initial setup. For the current version V2 as used on the TTN community network:

  1. Drop an “mqtt in” node on your flow:

    mqtt in

  2. Double-click it:

  3. Topic: e.g., +/devices/+/up for the uplinks of all devices in your application

  4. Output: “a parsed JSON object” (if you’re using an old version of Node-RED then use its default)

  5. Server: select “Add new mqtt-broker…” and click the pencil icon:

  6. Name: e.g. your application id from TTN Console (the default name would be the server name, but the configuration is only valid for a single application; seeing only the server name may be confusing if you set up flows for multiple TTN applications)

  7. Server: e.g., eu.thethings.network (in your application in TTN Console, the suffix from, e.g., ttn-handler-eu defines the prefix needed in the server URL)

  8. Port: 8883 (the default 1883 is for non-TLS)

  9. Select “Enable secure (SSL/TLS) connection”

  10. TLS Configuration: select “Add new tls-config…” and click the pencil icon:

  11. Just accept the defaults, so click “Add”

  12. Back in the “mqtt-broker config” node, select the Security tab:

  13. Username: your application id from TTN Console

    TTN Console Application ID

  14. Password: an access key from TTN Console with at least access rights to “messages”, something like ttn-account-v2.m2xb7...

    For this, you could use the “default key”, or first generate a key just for Node-RED, through your application’s Settings, Access Keys (or clicking “manage keys” from the application’s Overview):

  15. Click “Add”

  16. Back in the “mqtt in” node: click “Done”

  17. Done!

That’s all as for the MQTT config with standard Node-RED nodes. When adding another “mqtt in” node for a different topic, the broker you just added will already be preselected. Same goes when adding an “mqtt out” node to schedule downlinks. (This creates a single connection with the MQTT broker for the TTN application, regardless the number of “mqtt in” and “mqtt out” nodes.)

JSON text vs a JavaScript object in msg.payload

For older versions of Node-RED that do not support “Output: a parsed JSON object”, you’ll want to convert the JSON text into a full blown JavaScript object before any further handling. Like:

// Convert the MQTT text message into a JavaScript object
msg.payload = JSON.parse(msg.payload);

Note that Node-RED also uses the name payload within its own messages that flow around. So, in Node-RED any msg.payload is not related to the LoRaWAN payload.

Decoding uplinks

When using Node-RED, one can also get rid of any Decoder in the payload formats in TTN Console (which, in rare occasions, may randomly fail or introduce latency). To get the LoRaWAN application payload bytes and port:

// When using a Decoder in the application's Payload Formats in
// TTN Console, then msg.payload.payload_fields is also available.
const bytes = Buffer.from(msg.payload.payload_raw, 'base64');
const port = msg.payload.port;

msg.payload.payload_fields = {
  // Unsigned 16 bits integers, MSB
  battery: (bytes[0] << 8 | bytes[1]) / 1000,
  light: bytes[2] << 8 | bytes[3],

  // Sign-extend 16 bits to 32 bits to support negative values: shift
  // the MSB 16 bits too far to the left, followed by a sign-propagating
  // right shift of 16 bits to effectively shift the MSB 8 bits.
  temperature: (bytes[4] << 24 >> 16 | bytes[5]) / 100
};

return msg;

The byte shifting above could be copied from a TTN Console Decoder as-is. And just like in TTN Console JavaScript will interpret its operands as 32-bits signed integers.

But Node-RED also provides the built-in Buffer utility methods, which are not available in the payload formatters in TTN Console:

msg.payload.payload_fields = {
  // Unsigned 16 bits integers, MSB
  battery: bytes.readUInt16BE(0) / 1000,
  light: bytes.readUInt16BE(2),
  // Signed 16 bits integer, MSB
  temperature: bytes.readInt16BE(4) / 100 
};

Scheduling downlinks

To schedule a downlink right after receiving an uplink, simply use msg.payload.app_id and msg.payload.dev_id as provided in the uplink:

// AppID and DevID from the current uplink
const appId = msg.payload.app_id;
const devId = msg.payload.dev_id;

Next, create a downlink payload of one or more bytes, Base64-encode it, and create a message to be published through MQTT:

// Color value from binary RGB string; '110' = 6 = yellow
const color = parseInt(msg.rgb, 2);

// Single-byte payload for the downlink
const bytes = [color];

msg.topic = `${appId}/devices/${devId}/down`;

msg.payload = {
  dev_id: devId,
  port: 1,
  confirmed: false,
  schedule: 'replace',
  payload_raw: Buffer.from(bytes).toString('base64')
};

return msg;

The output of the above can then be passed to an “mqtt out” node, like the one named “mqtt publish” in the screenshot further below.

Beware that the scheduled downlink may not be transmitted until the next uplink is received.

To schedule a downlink at any time (rather than directly after handling an uplink), one may want to use the AppID and DevID from an earlier activation or uplink. In the example further below, that is saved for both uplinks and events, using:

// When scheduling a downlink, we need the AppID and DevID
const levels = msg.topic.split('/');
flow.set('appId', levels[0]);
flow.set('devId', levels[2]);

The values stored in the context can then be used like:

// Get AppID and DevID as saved from last uplink
const [appId, devId] = flow.get(['appId', 'devId']);

Logging errors

For logging, one may want to ensure the topic and time are always included:

// A file node only saves msg.payload, but +/+/+/events/down/scheduled
// holds no details about application, device nor timestamp in the
// MQTT payload, but only: 
//   {"message": {"port": 1, "payload_raw": "BA=="}}
// Same goes for TTN-published errors:
//   {"error": "Unable to decode payload fields: Internal error: ..."}}
// So, ensure the topic and time are also saved into the log file:
msg.payload = {
  topic: msg.topic,
  timestamp: new Date().toISOString(),
  payload: msg.payload,
  error: msg.error
}

return msg;

Example flow

Further below is a Node-RED export of an example that handles a few cases for a The Things Node device, and only logs what’s happening. That looks like this:

Of course, one can also create multiple, more specific, MQTT subscriptions for the events. In this example, the “switch on topic” node delegates a message from the wildcard +/+/+/events/# to (only) one of its outputs, based on the actual topic, using regular expressions:

Above, the $ in, e.g., /errors$ means that the topic must end with /errors to match.

Alternative using a function node, also extracting AppID and DevID

flow-ttnode-20200823.json (16.6 KB)

A function node from an earlier version of this forum post achieved the same result, but also extracted the AppID and DevID. Of course, and extra node can still be added to extract those if needed. When using a function node, one needs to manually set the number of outputs of that node:

// Get details from the MQTT topic, e.g.:
//
// - <AppID>/devices/<DevID>/up
// - <AppID>/devices/<DevID>/up/<field>
// - <AppID>/devices/<DevID>/events/up/errors
// - <AppID>/devices/<DevID>/events/create
// - <AppID>/devices/<DevID>/events/update
// - <AppID>/devices/<DevID>/events/delete
// - <AppID>/devices/<DevID>/events/activations
// - <AppID>/devices/<DevID>/events/activations/errors
// - <AppID>/devices/<DevID>/events/down/scheduled
// - <AppID>/devices/<DevID>/events/down/sent
// - <AppID>/devices/<DevID>/events/down/acks
// - <AppID>/devices/<DevID>/events/down/errors
//
// Note that /up/<field> is not supported for EU868; unclear if
// subscribing to a wildcard yields both publication of the full
// payload, plus an additional publication for each field.
//
// Of course, one could also create multiple MQTT subscriptions,
// to be more specific.

const levels = msg.topic.split('/');
msg.appId = levels[0];
msg.devId = levels[2];

// Quick & dirty mapping of topic to events we care about.
// Make sure to change the number of outputs to match this.
return [
  // When subscribing to # rather than +/+/+/events, we could add:
  // levels[3] === 'up' ? msg : null,
  levels[4] === 'activations' ? msg : null,
  levels[5] === 'scheduled' ? msg : null,
  levels[5] === 'sent' ? msg : null,
  levels[5] === 'acks' ? msg : null,
  levels.slice(-1)[0] === 'errors' ? msg : null
];

After importing the following flow, you’ll need to set the username and password for the MQTT Broker.

flow-ttnode-20200824.json (17.4 KB)

13 Likes

Thank you, @arjanvanb for this valuable guide. I started Node-Red some days ago and was happy to find “TTN uplink” ( Node-red-contrib-ttn) on my installation on a NAS. As I moved to my Raspberry Pi 2 B Model I was faced to the note that TTN uplink is deprecated. It was not easy to find, that this was only for the Raspberry Pi.

Nevertheless you are right. It is much more flexible to go with “MQTT in” as you have access to EVERY information and not the sensor payload only.

One thing I like to add is, that in case there are more than one sensor in ones application I colud address them by replacing

+/devices/+/up

with e.g.

+/devices/dragino_senso_lht65_1/up

Changes in “Setting up the MQTT client” for V3 in the following steps:

  1. Topic is: v3/+/devices/+/up

You can find your server, port, username and password on the same site: Application → Integrations ->MQTT

  1. Server e.g. eu1.cloud.thethings.network

  2. Username: “your-application-id” + “@ttn” => “your-application-id@ttn”
    (You can find it also here: Application → Integrations ->MQTT)

  3. Password: the API key from (Application → Integrations ->MQTT). You may havte to generate one first.

3 Likes

Is there an arduino sketch to go with this?

In addition also see: Major Changes In The Things Stack

MQTT V3 does not give you downlink sent topics if they are done via V2 gateway (I have no V3 gateway setup to test), downlink queued and ack topics seem to work ok

In fact not seeing any ‘sent downlink’ in V3 live data view, scheduled and ack are shown, my guess is the msg from V2 re sent downlinks are not making it back

The above worked for me on v3. Note - use MQTT 3.1.1 and QoS 0 as per the note here:
https://www.thethingsindustries.com/docs/integrations/mqtt/

Also, I had to generate a new API key using on this page in TTN v3 console: Applications > My MQTT connector > MQTT. My previous API keys didn’t work even though the access rights were the same. Not sure if this was just a copy-paste error.

1 Like

hello
i did read the instruction, as i did understand it is not to get the Gateway Information but only the device connected?!
have a nice day
vinc

***** update, this has now been fixed in one of the many updates to V3, downlinks are showing up in Node-Red via the documented MQTT connection

Hey, can someone explain to me the "+/+/+/events/# ", because I’m having a hard time understand what is that’s stand for or understand what its subscribe to? does we have to change like “+/devices/+/up” or just have to let it as it is?

You can read Events

“v3/+/devices/+/up”
This is the topic for all your devices in a application uplinks in V3.

Or you can de specific for a device – v3/+/devices/End device ID/up –

Hello Everybody,

I tried to run the above example, but the messages don’t hit the TTN application.
I can see the “subscribe application” in my TTN log every time i redeploy the NR flow, but the downlink messages don’t show up there.

I’ve also added v3/ as the new topic prefix.
How can I debug this?

What is your complete topic? You use for the downlink.

Hi Johan,

I’m using the above example with
v3/APPID/devices/DEVICEID/down
in the mqtt publish node (using the real IDs).

I’ve also tried the topic with a /push at the end.

image

2 Likes

Hi Johan,

That was my topic, but I discovered, that the point was the formatter which has to include the downlink payload in an array like

msg.payload = {
  "downlinks": [{ ... }]
}

I was able to send messages from NR to the TTN console for about ten minutes, but then it stopped.
Does the FairUse policy apply to the MQTT traffic too, or just to the nodes connected via LoRa?

MQTT traffic to TTN should be downlinks to LoRaWAN devices. Those devices have a limit of 10 downlinks a day.
Messages posted to TTN will not be forwarded to the application as uplinks.
If you need a generic MQTT server you should look at other services in the internet, TTN is for LoRaWAN, not a generic MQTT service.

1 Like

Hello Jac,

that answered my question.
Thanks a lot!

Well you are generating downlinks that are LoRaWan traffic, so yes it applies to MQTT traffic to TTN