Easiest way to get data from TNN via a javascript API (trying unsuccessfully with Node RED

[heaversm]

Hi - I’m a competent javascript developer but a NOOB with IOT - I’m trying to get a couple of values from a sensor from TNN to a web visualization I built, and my first attempt was to try and follow the tutorials for Node-RED / IFTTT integrations - the idea being hopefully once it’s in IFTTT I have a few more options with tools I’m familiar with for retrieving the data via a JS API endpoint.

First off - is this a reasonable approach? I’m guessing there’s probably a way to get it straight from Node RED but I haven’t found it yet.

Second, in my attempt to set this up, I’m getting a “Forward Uplink Data Message” which successfully shows the data from my sensor. In particular the value I’m after is data.uplink_message.decoded_payload.ble

I added a webhook integration in TNN of the format: JSON, URL: http://127.0.0.1:1880 (the address that Node-RED exposes when I run it on my machine), and for the enabled event types I’ve selected uplink message raphael-paxcounter

I followed the diagram for the node setup in node-red, with an http node with url raphael-paxcounter , a function that looks like

msg.url = "https://maker.ifttt.com/trigger/raphael-paxcounter/with/key/[my_key]"
msg.method = "POST";
msg.payload = {
    'ble' : msg.payload.uplink_message.decoded_payload["ble"],
    'wifi' : msg.payload.uplink_message.decoded_payload["wifi"]
}
return msg;

and followed the rest of the tutorial for the other nodes, wired up without changes.

The data logs to TNN properly, and I know the payload I want to receive.

I start node red which exposes http://127.0.0.1:1880/

My particular flow in node red has a URL of http://localhost:1880/#flow/5086e3421505115e

The startup output in terminal for node-red is attached, as is the error I’m getting in the things network for my integration:

Fail to send webhook
"name": "as.webhook.fail",
"cause": {
      "namespace": "pkg/errors",
      "name": "net_operation",
      "message_format": "net operation failed",
      "attributes": {
        "address": "127.0.0.1:1880",
        "net": "tcp",
        "op": "dial",
        "timeout": false
      },
      "correlation_id": "49c1eac9147c4628aa0b0e4f1b9c09b8",
      "code": 14

any idea how to get unstuck? Or any resources that show an alternative tutorial where data can be set up to be accessed via the web?

Thanks!

This is what the data looks like coming into TNN:

{
  "name": "as.up.data.forward",
  "time": "2022-06-26T00:49:10.704078328Z",
  "identifiers": [
    {
      "device_ids": {
        "device_id": "eui-70b3d57ed0051779",
        "application_ids": {
          "application_id": "paxcounter-raphael"
        },
        "dev_eui": "70B3D57ED0051779",
        "join_eui": "0000000000000000",
        "dev_addr": "260CE2C8"
      }
    }
  ],
  "data": {
    "@type": "type.googleapis.com/ttn.lorawan.v3.ApplicationUp",
    "end_device_ids": {
      "device_id": "eui-70b3d57ed0051779",
      "application_ids": {
        "application_id": "paxcounter-raphael"
      },
      "dev_eui": "70B3D57ED0051779",
      "join_eui": "0000000000000000",
      "dev_addr": "260CE2C8"
    },
    "correlation_ids": [
      "as:up:01G6EQQRDN9TQ4TGWN0YTCTS8G",
      "gs:conn:01G69BA3NCXQTHA6AXHJZ7JVQX",
      "gs:up:host:01G69BA3SZRKDNXRBDE3401T3T",
      "gs:uplink:01G6EQQR6VW24P6CBHNVXYNDC0",
      "ns:uplink:01G6EQQR6XW5XHCQMW6S30T1C7",
      "rpc:/ttn.lorawan.v3.GsNs/HandleUplink:01G6EQQR6WRTARCBFNXEDA641S",
      "rpc:/ttn.lorawan.v3.NsAs/HandleUplink:01G6EQQRDFHY52ECFQKNA0G8W2"
    ],
    "received_at": "2022-06-26T00:49:10.582007349Z",
    "uplink_message": {
      "session_key_id": "AYGdX/BwY710zXsC1fv1ZA==",
      "f_port": 9,
      "f_cnt": 41,
      "frm_payload": "/w==",
      "decoded_payload": {
        "timesync_seqno": 255
      },
      "rx_metadata": [
        {
          "gateway_ids": {
            "gateway_id": "ptldorah1",
            "eui": "50544C444F524148"
          },
          "time": "2022-06-26T00:49:10.338742Z",
          "timestamp": 323289340,
          "rssi": -116,
          "channel_rssi": -116,
          "snr": -12.5,
          "uplink_token": "ChcKFQoJcHRsZG9yYWgxEghQVExET1JBSBD8gZSaARoMCIba3pUGEJ2KsK0BIOCwr6y0iCoqDAiG2t6VBhDwlcOhAQ==",
          "channel_index": 3
        }
      ],
      "settings": {
        "data_rate": {
          "lora": {
            "bandwidth": 125000,
            "spreading_factor": 10
          }
        },
        "coding_rate": "4/5",
        "frequency": "904500000",
        "timestamp": 323289340,
        "time": "2022-06-26T00:49:10.338742Z"
      },
      "received_at": "2022-06-26T00:49:10.365057053Z",
      "consumed_airtime": "0.288768s",
      "version_ids": {
        "brand_id": "opensource",
        "model_id": "esp32-paxcounter",
        "hardware_version": "_unknown_hw_version_",
        "firmware_version": "2.4.0",
        "band_id": "US_902_928"
      },
      "network_ids": {
        "net_id": "000013",
        "tenant_id": "ttn",
        "cluster_id": "nam1",
        "cluster_address": "nam1.cloud.thethings.network"
      }
    }
  },
  "correlation_ids": [
    "as:up:01G6EQQRDN9TQ4TGWN0YTCTS8G",
    "gs:conn:01G69BA3NCXQTHA6AXHJZ7JVQX",
    "gs:up:host:01G69BA3SZRKDNXRBDE3401T3T",
    "gs:uplink:01G6EQQR6VW24P6CBHNVXYNDC0",
    "ns:uplink:01G6EQQR6XW5XHCQMW6S30T1C7",
    "rpc:/ttn.lorawan.v3.GsNs/HandleUplink:01G6EQQR6WRTARCBFNXEDA641S",
    "rpc:/ttn.lorawan.v3.NsAs/HandleUplink:01G6EQQRDFHY52ECFQKNA0G8W2"
  ],
  "origin": "ip-10-101-13-137.us-west-1.compute.internal",
  "context": {
    "tenant-id": "CgN0dG4="
  },
  "visibility": {
    "rights": [
      "RIGHT_APPLICATION_TRAFFIC_READ"
    ]
  },
  "unique_id": "01G6EQQRHG8JYJ5AD753QYZQ8E"
}

I also attached the error I get on the webhooks integration screen:

“this webhook has been deactivated due to several unsuccessful forwarding attempts”

[kersing (Jac Kersing)]

@heaversm

I start node red which exposes http://127.0.0.1:1880/

That means node red is only available on your local machine and not remote which TTN requires.
To use a web hook you need a webserver which listens at port 443 on an IP address exposed to the internet. So either somewhere in the cloud or you need to use DDNS and open a port on your internet router (dangerous unless you know very well what you are doing, it can be abused by hackers to gain access to your computer)

An easier way to get the data to your local node-red is to use mqtt subscribe. For that you don’t have to worry about exposing your local system to the internet as it ‘pulls’ the information to the system.

[heaversm]

Ah - thanks @kersing. Maybe I’m overcomplicating things. Would it be easier to just get this information over serial to javascript? How would I set that up? Or if MQTT-subscribe is that right path, is there a good guide for that?

[kersing (Jac Kersing)]

@heaversm

Would it be easier to just get this information over serial to javascript?

Serial from TTN? That would be something… Or are you referring to something else?

@heaversm

MQTT-subscribe is that right path, is there a good guide for that?

Have you checked the documentation? You can use any MQTT client, no doubt google will find some for you.

[heaversm]

no, just serial via the sensor - the code I am using was written specifically for ttn, but I’m wondering if I can somehow manipulate it to go straight to something I write in JS - it says it supports SPI, so I can try to figure that out.

As far as MQTT - again, I don’t know much about that side of things, but it looks like there are just some simple MQTT libraries for node from which I can extract the values I need, like mqtt.js.

At any rate - you’ve been a big help. I can actually now see the data payload in node-red via MQTT, so that’s a start!

[descartes (Nick McCloud)]

@heaversm

no, just serial via the sensor - snip - if I can somehow manipulate it to go straight to something I write in JS - it says it supports SPI, so I can try to figure that out.

The SPI interface is the main processor to the radio module - if you want to get some serial output directly from the ESP32 you can ditch the radio and the entire of the LoRaWAN software (LMIC) and just send it out one of the serial interfaces.

Which would technically be off topic for this forum, but I’ll be fascinated to see how long this serial cable will be. Why did you even start with the TTGO/TTN solution?

[Johan_Scheepers]

Node-Red can connect direct to TTN via MQTT, Just search and read a bit on the forum, lots of information.

https://www.thethingsindustries.com/docs/integrations/node-red/

The first part is a bit old but read the entire thread.

MQTT in Node-RED [HowTo]

Node-RED has built-in support to act as an MQTT client, which can be used with TTN’s MQTT Data API. For new developments, one should not use the The Things Network’s node-red-contrib-ttn, which is no longer supported. Setting up the MQTT client The following may seem a lot of steps, but Node-RED will guide one through the initial setup. For the current version V2 as used on the TTN community network: Drop an “mqtt in” node on your flow: [mqtt in] Double-click it: [mqtt in] …

MQTT, Node-RED for large or small decoders [HowTo]

I have some Abeeway Smart barges and micro trackers, the decoder for them are more the 4k so you can’t utilize the decoder on the “The Things Stack Community Edition”, here I am going to use Node-Red to decode the raw data for me. In the same way you can use the decoder for both small and large packet decoders. Create your application in TTN, add you device to the application. [image] In the integration add MQTT , copy your “Public TLS address”, “Username” and “Generate new API key” [ima…

[heaversm]

@Johan_Scheepers Yes - I have node red working now with TNN - the last thing I need to figure out is how to get my app to listen for the messages node red collects from TNN - I am attempting to do that via websockets in node red. At any rate, that problem probably sits outside this forum, so unless there’s an alternative path that avoids all the routing through node red, I can probably close this post.

[heaversm]

The board uses LoRAWAN for bluetooth sniffing to estimate how many people are in a given area.

[descartes (Nick McCloud)]

@heaversm

The board uses LoRAWAN for bluetooth sniffing

It does not. It uses the ESP32 chipset for bluetooth sniffing for people counting.

It just happens that particular code base uses LoRaWAN to send the results as the TTGO LoRa boards are relatively common to source.

You could use the onboard WiFi. Or a serial port. But in both instances, my comments stand.

@heaversm

the last thing I need to figure out is how to get my app to listen for the messages node red collects from TNN - I am attempting to do that via websockets in node red. At any rate, that problem probably sits outside this forum

That’s fine for this forum - people use Node-Red & MQTT regularly, mostly to put the data straight in to a database, but if you expand the range of use, that’s good for the community.

However we don’t know what the destination is for your data and how JavaScript figures beyond it being your preferred language. Is the final destination a database? If so, then Node and the MQTT.js library would make more sense for you rather than have data go through MQTT to Node-Red to WebSocket to some JS …