Connection from TTN node to Luftdaten API

Besides my LoRa stuff I also got a wifi based Luftdaten sensor. This often suffers from connectivity issues so I want to connect it using LoRa. But it turns out to be far more difficult then I expected. I know some people did some attempts in the past but what I can find is quite shattered and I can’t get the picture as a whole.

What I would expect: Modify the luftdaten sensor and add a LoRa node -> to TTN backend -> decode payload and send it to a HTTP integration using API credentials-> results are on Luftdaten.

But:

  • The limited number of examples I find does not seem to use any authentication.Yet calling the API (http://api.luftdaten.info/v1/push-sensor-data/) did result in a few ‘unauthorized’ messages though, but I can’t recreate it so it might have been a functional thing
  • Some examples use an intermediate service (https://github.com/verschwoerhaus/ttn-ulm-muecke). I don’t understand why and I wonder if this is related to the fact that I don’t manage to get it working
  • a simple piece of C# code (below) to test it resulted in an error “{“detail”:“Unsupported media type “text/plain; charset=utf-8” in request.”}” although I did set the content type to application/json instead of plain text.

Any clues how to proceed? Does anyone has an example how the TTN application payload decoding and integration would look like? The reason I find it in particular strange is that using LoRa for this would make much sense. And both TTN and Luftdaten are pretty common platforms… So am I on the wrong track and did I miss a reason that makes another approach much better?


            string sensor = "16741";
            string sensorValue = "";
            string softwareVersion = "123";

            HttpClient client = new HttpClient();

            client.BaseAddress = url;

            client.DefaultRequestHeaders.Accept.Add(new System.Net.Http.Headers.MediaTypeWithQualityHeaderValue("application/json"));
            
            var result = client.PostAsync(url.ToString(), new StringContent("{\"sensor\": 16741, \"sampling_rate\": null, \"timestamp\": \"2020-01-02 09:29:41\", \"sensordatavalues\": [{\"id\":12628208575, \"value\":\"1.00\", \"value_type\":\"temperature\" },{\"value\":\"98.40\", \"id\":12628208576, \"value_type\":\"humidity\" }], \"software_version\": \"123\" }")).Result;

            string json_res = await result.Content.ReadAsStringAsync();

            Console.WriteLine(json_res);
2 Likes

@bertrik did you manage to get it running?

Yes, there are a couple of nodes for which I am forwarding the data to luftdaten, in Maastricht.

The chain looks like this:

  • the node sends data in a (slightly weird) binary format over LoRaWAN to TTN. Keys are ABP.
  • at TTN the payload is decoded and fields are parsed into JSON
  • my own Java program receives the message from TTN over MQTT and converts it to two luftdaten API calls (one for particulate matter, another for temperature humidity)

Code is at: https://github.com/bertrik/LoraLuftdatenForwarder

I have no experience at all with the HTTP integrations of TTN.

Also note:

  • before you can send data, you need to register your node first, you can do this at http://my.luftdaten.info . The convention for TTN connected nodes is that your luftdaten node name looks like TTN-00878E9A62XXYYZZ, where the part after TTN is the unique hardware serial number of the node. Not registering before explains the error codes you got. My code uses the TTN “hardware_serial” field for that (prefixed with “TTN-”).
  • The API is a bit weird in the sense that you need to set some HTTP headers with the POST request: “X-Sensor” with the node name “TTN-…” (as above) and “X-Pin” with a number indicating what kind of data you send (e.g. 1 for dust measurement data)
  • There is no authentication of the node towards the network, so in principle anyone can “spoof” data
  • The luftdaten API URL changed from api.luftdaten.info to api.sensor.community

Great! Many thanks. It seems very useful. I’ll have a closer look this evening, but do you mind adding the code you use for your payload decoding?

My node was already registered because it already works with wifi. Is this TTN prefix technically mandatory?

I think you can reuse the “esp-” name.
That one should work because it’s already registered.

I didn’t write the decoder, but it looks as shown below. This decoder can be found in the console under applications / payload formats / decoder.

function Decoder(bytes, port) {
  // Decode an uplink message from a buffer
  // (array) of bytes to an object of fields.
  var decoded = {};

  var SDS_ID = (bytes[1] << 8) | bytes[0];
  var T = (bytes[2] << 8) | bytes[3];
  var RH = (bytes[4] << 8) | bytes[5];
  var P = (bytes[6] << 8) | bytes[7];
  var PM10_Avg = (bytes[8] << 8) | bytes[9];
  var PM25_Avg = (bytes[10] << 8) | bytes[11];
  
  return {
    SDS_ID: SDS_ID,
    T: T/100,
    RH: RH/100,
    P: (P - 100),
    PM10_Avg: (PM10_Avg - 1000) / 10,
    PM25_Avg: (PM25_Avg - 1000) / 10,
  };
}

Ok, and this “at TTN the payload is decoded and fields are parsed into JSON” happens automatically when the decoded payload is converted to a MQTT message?

Yes, the fields created by the decoder appear in the “payload_fields” field in the JSON received over MQTT.

Example MQTT message:

{
  "app_id": "testerwin",
  "dev_id": "1001",
  "hardware_serial": "00878E9A6221870B",
  "port": 1,
  "counter": 48396,
  "payload_raw": "ZUkDKRb3BGwEngSD",
  "payload_fields": {
    "P": 1032,
    "PM10_Avg": 18.2,
    "PM25_Avg": 15.5,
    "RH": 58.79,
    "SDS_ID": 18789,
    "T": 8.09
  },
  "metadata": {
    "time": "2019-12-29T15:16:13.264793951Z",
    "frequency": 867.3,
    "modulation": "LORA",
    "data_rate": "SF12BW125",
    "airtime": 1482752000,
    "coding_rate": "4/5",
    "gateways": [
      {
        "gtw_id": "eui-7276ff000b032609",
        "timestamp": 271952500,
        "time": "2019-12-29T15:16:13.210224Z",
        "channel": 4,
        "rssi": -121,
        "snr": -14.5,
        "rf_chain": 0,
        "latitude": 51.44633,
        "longitude": 5.48514,
        "altitude": 69
      },
      {
        "gtw_id": "ohnix2",
        "gtw_trusted": true,
        "timestamp": 390941652,
        "time": "2019-12-29T15:16:17Z",
        "channel": 4,
        "rssi": -107,
        "snr": -14,
        "rf_chain": 0,
        "latitude": 50.842007,
        "longitude": 5.7229476,
        "altitude": 6,
        "location_source": "registry"
      },
      {
        "gtw_id": "eui-b827ebfffef6cc6b",
        "timestamp": 1018653940,
        "time": "2019-12-29T15:16:13.238966Z",
        "channel": 4,
        "rssi": -119,
        "snr": -2.5,
        "rf_chain": 0,
        "latitude": 50.85338,
        "longitude": 5.70069,
        "altitude": 60
      }
    ],
    "latitude": 50.83832,
    "longitude": 5.700839,
    "location_source": "registry"
  }
}

Great, MQTT has always been a blind spot for me since I primarily focused on the node firmware. I didn’t really realize MQTT was JSON but now I get it.

It is indeed JSON for TTN. (But an MQTT payload could be anything; it’s just binary data really, where only a few bytes are reserved for the protocol and the rest is up to the application/API.) For TTN, the JSON messages for the MQTT Data API can be found in its API reference.

@TijnOnlijn

I found a solution to move the TTN data from my airquality sensor (a sensebox) to luftdaten.info.

It boils down to use a node-red server where a function node parses the json object for the x-pins and sensordatavalues as the luftdaten api requires.

The results in our Leuven area can be found here and look for L-38035 and L-38036.
In this experimental phase the lora data transfer is currently done with ABP, later on with OTAA as it should security wise.

@arjanvanb
A great many thanks Arjan for the pull request on github to the author of the sensebox code. It works fine now after Felix from the uni in Münster changed his Arduino code for his SAMD board and also the appropriate payload format for TTN (that was new for him). And indeed, he admitted, the ‘% 255’ was a mistake in its shifting.
The results can be seen for now on https://opensensemap.org/explore/5e10d6b0b0c088001bf22393 (with one little mistake though on the lux values).
I will publish it soon in the right topic: Payload format for the sensebox

Interesting to see @wdebbaut. I’m looking forward to see your nodered solution.

@bertrik wrote an interesting OTAA solution for a luftdaten sensor. It might be cool to merge the core of his firmware into a format that fits the sensebox too. Might save you some trial and error while solving the same problem with the same solution is usually a good idea.

1 Like

I wrote prototype code for an SDS011 particulate matter sensor, it uses the mcci-catena arduino-lmic stack for LoRaWAN communication using an ESP32+sx1276. The basic idea is that it uses OTAA, instead of hardcoding the keys in the sketch. This LMIC stack allows you to retrieve the application and network keys after an OTAA exchange, so the next time the node powers on, it can resume communication with those keys instead of doing the OTAA procedure again. I uses the unique ESP32 id as the LoRaWAN device EUI.
This way, the firmware is universal in the sense that all nodes can run the same firmware and are distinguished only by their unique EUI. The code is at:
https://revspace.nl/LoraWanDustSensor

Also, I wrote a TTN-to-LuftDaten forwarder, that accepts data from TTN over MQTT and sends it to luftdaten in the format that they expect in their API. That code is at:

1 Like

You can find some links on the luftdaten FAQ page, Forum Sensor.Community

Es gibt die Möglichkeit, die mit anderer Hardware erfassten Daten an unsere API zu schicken. Dafür existieren bereits einige fertige Skripte und Schnittstellen.

Raspberry PI:
Python Skript von Corny auf Github.com

TheThingsNetwork, LoRaWAN:
muecke-server (Python) der TTN-Gruppe in Ulm auf Github
TTN-Forwarder der Civic Labs Belgium auf Github

Hello,
I have just written a tutorial about reporting SDS011 measurements to Luftdaten (and Cayenne myDevice) via LoRaWAN (Arduino + ATIM LoRaWAN shield)
https://www.thethingsnetwork.org/labs/story/reporting-pm25-pm10-micro-particle-dust-sensor-by-using-arduino-and-lorawan-atim-acw-duino-shield/

1 Like

Thank you, but I did put it aside since it looked somewhat complex. I was hoping to get it running without an additional server, but it seems that either that or some nodered transformations are required… So I’ll have a look at that.

You may want to have a look to Google scripts. This may be your workaround to avoid the own server. I tried to build a script for paxcounter that you may modify to your needs: