Why decoded_payload is not availbale

pierrot10 · May 16, 2022, 6:48am

Hello there,

When a node send data to my server through TTN V3, as an integration, I use a web hook
In the field ‘Base URL’, I saved the path to my PHP Script of my web server. The PHP script read the post data and save it into a database. It’s work since some years

Today, I got a surprise. I added a rain gauge sensor to one my station. It’s sunny and my station send the measure, the measure has been save to my database and I could read a new line in my graph.

After, a measure has been sent, I put 50ml of water in my rain gauge, I wait one hour and I observed that no measures has been sent. (none of all sensors connected to the station)

My PHP script log the data received by TTN, I checked the log file and I was surprised to not see decoded_playload

{
    "end_device_ids": {
        "device_id": "st-21",
        "application_ids": {
            "application_id": "name"
        },
        "dev_eui": "00000000XXXXXXXX",
        "join_eui": "00000000XXXXXXXX",
        "dev_addr": "XXXXXXXX"
    },
    "correlation_ids": [
        "as:up:01G33VD27M4M7N59X7XXXXXXXX",
        "gs:conn:01G2H1B3K1P2DHJEN1XXXXXXXX",
        "gs:up:host:01G2H1B3K53ZWW7AR6XXXXXXXX",
        "gs:uplink:01G33VD215CAAR0XKVXXXXXXXX",
        "ns:uplink:01G33VD216RVA8CK3XXXXXXXXX",
        "rpc:/ttn.lorawan.v3.GsNs/HandleUplink:01G33VD216S7V3KSK8XXXXXXXX",
        "rpc:/ttn.lorawan.v3.NsAs/HandleUplink:01G33VD27KVJ5Y53EVXXXXXXXX"
    ],
    "received_at": "2022-05-15T12:34:30.260503051Z",
    "uplink_message": {
        "session_key_id": "AYDHf5qY7PBXteXXXXXXXX==",
        "f_cnt": 1,
        "rx_metadata": [
            {
                "gateway_ids": {
                    "gateway_id": "ic800a",
                    "eui": "B827EBFFXXXXXXXX"
                },
                "time": "2022-05-15T12:34:30.033273Z",
                "timestamp": 4243657139,
                "rssi": -47,
                "channel_rssi": -47,
                "snr": 8,
                "location": {
                    "latitude": 46.XXXXXXXX313159,
                    "longitude": 6.XXXXXXXX7700196,
                    "altitude": 700,
                    "source": "SOURCE_REGISTRY"
                },
                "uplink_token": "XXXXXXXXaWM4MDBhEgi4J+v//kaBbhCzo8TnDxoLCNbmg5QGEJ/11BkguMa07cDIjgEqXXXXXXXXXXXXXXXX"
            }
        ],
        "settings": {
            "data_rate": {
                "lora": {
                    "bandwidth": 125000,
                    "spreading_factor": 7
                }
            },
            "coding_rate": "4/5",
            "frequency": "868100000",
            "timestamp": 4243657139,
            "time": "2022-05-15T12:34:30.033273Z"
        },
        "received_at": "2022-05-15T12:34:30.054707053Z",
        "consumed_airtime": "0.046336s",
        "network_ids": {
            "net_id": "000013",
            "tenant_id": "ttn",
            "cluster_id": "eu1",
            "cluster_address": "eu1.cloud.thethings.network"
        }
    }
}

I tried it again. I add a few water in my rain gauge and I checked the Live data of my TTN V3 application. I can see the all measures are sent and receive at TTN V3. The ga has a value (ga is the parameter for the rain gauge) Great

I tried again with no watter in my rain gauge and the data has been saved this time. Compared my two file log, and the second time, I could see the ‘decoded_payload’. The ‘uplink_message’ contains more information

{
    "end_device_ids": {
        "device_id": "st-21",
        "application_ids": {
            "application_id": "name"
        },
        "dev_eui": "00000000XXXXXXXX",
        "join_eui": "00000000XXXXXXXX",
        "dev_addr": "XXXXXXXX"
    },
    "correlation_ids": [
        "as:up:01G33QZBYSXPE7JK2FXXXXXXXX",
        "gs:conn:01G2H1B3K1P2DHJEN1XXXXXXXX",
        "gs:up:host:01G2H1B3K53ZWW7AR6XXXXXXXX",
        "gs:uplink:01G33QZBRAT7D5PFPVXXXXXXXX",
        "ns:uplink:01G33QZBRBH1RNVG5XXXXXXXX",
        "rpc:/ttn.lorawan.v3.GsNs/HandleUplink:01G33QZBRBEQNAACX1XXXXXXXX",
        "rpc:/ttn.lorawan.v3.NsAs/HandleUplink:01G33QZBYSDAH1QJTEXXXXXXXX"
    ],
    "received_at": "2022-05-15T11:34:35.738040564Z",
    "uplink_message": {
        "session_key_id": "AYDHf5qXXXXXXXXXXXXXXXXX",
        "f_port": 1,
        "frm_payload": "YTIzYjkzYzU5ajBrNDIwbTB0XXXXXXXX",
        "decoded_payload": {
            "an": "0",
            "ar": null,
            "b1": null,
            "b2": null,
            "b3": null,
            "b4": null,
            "ba": "420",
            "da": null,
            "ga": "0",
            "hu": "59",
            "it": null,
            "lu": null,
            "pr": "93",
            "sb": null,
            "sl": null,
            "su": "159",
            "te": "23",
            "ts": null,
            "w1": null,
            "w2": null,
            "w3": null,
            "wd": "45",
            "wr": null
        },
        "rx_metadata": [
            {
                "gateway_ids": {
                    "gateway_id": "ic800a",
                    "eui": "B827EBFFFE46816E"
                },
                "time": "2022-05-15T11:34:35.502070Z",
                "timestamp": 649139603,
                "rssi": -65,
                "channel_rssi": -65,
                "snr": 7.5,
                "location": {
                    "latitude": 46.XXXXXXXX313159,
                    "longitude": 6.XXXXXXXX7700196,
                    "altitude": 700,
                    "source": "SOURCE_REGISTRY"
                },
                "uplink_token": "XXXXXXXXaWM4MDBhEgi4J+v//kaBbhCTq8S1AhoMCMvKg5QGELvvkf0BILiM8Z3y340BKgwIXXXXXXXX",
                "channel_index": 2
            }
        ],
        "settings": {
            "data_rate": {
                "lora": {
                    "bandwidth": 125000,
                    "spreading_factor": 7
                }
            },
            "coding_rate": "4/5",
            "frequency": "868500000",
            "timestamp": 649139603,
            "time": "2022-05-15T11:34:35.502070Z"
        },
        "received_at": "2022-05-15T11:34:35.531803582Z",
        "consumed_airtime": "0.082176s",
        "network_ids": {
            "net_id": "000013",
            "tenant_id": "ttn",
            "cluster_id": "eu1",
            "cluster_address": "eu1.cloud.thethings.network"
        }
    }
}

I just tried again with 50ml of water in my rain gauge and the rain gauge value has been sent

So as I far I understand,

4 or 5 time, the ‘decode_payload’ was not sent but mainly after I switched on my station and it send the first value.
I observed another station which has the same problem. You can see here when the two measures are with a value of 0.1

What can be the reason why TTN do not send the ‘decoded_payload’ section?

My configuration has always been working
Do you need more information?

Many thanks

kersing · May 16, 2022, 9:47am

If the decoder script takes too long to decode the data it will be aborted and the decoded_payload will be missing. This also happens if the load on the TTN servers is too high.
If you want to make sure you have decoded information you will need to decode it at your own server. There is no guarantee TTN decodes the data.

pierrot10 · May 17, 2022, 7:45pm

Dear Kersing,

Thanks for your reply. What do you mean by “he decoder script take too long”.
Are you speaking about the Payload → Formatters → Uplink?

I have a custome javascript formatter

function Decoder(bytes, port) {
  var str=String.fromCharCode.apply(null,bytes);
  var astr = str.split(",");
  //console.log(astr);

  //var regex = /([a-z]+)(\d+)/g;
  var regex = /([a-z]+)(\-?\d+)/g;

  //console.log(regex.exec(astr[0]));
  
  //var resultats = [];
	var paire;
 
  var decoded = {}
  var te,pr,hu,lu,w1,w2,w3,da,ts,ga,ba,ar,an,it,wr,b1,b2,b3,b4,su,wd,sb,sl;
  while (null !== (paire = regex.exec(astr[0]))) {
		//console.log(paire[1]);
		switch(paire[1]){
		  case 'a':
		    //console.log("te",paire[2]);
		    te=paire[2];
		    break;
		  case 'b':
		    //console.log("pr",paire[2]);
		    pr=paire[2];
		    break;
		  case 'c':
		    //console.log("hu",paire[2]);
		    hu=paire[2];
		    break;
		  case 'd':
		    //console.log("lu",paire[2]);
		    lu=paire[2];
		    break;
		  case 'e':
		    //console.log("w1",paire[2]);
		    w1=paire[2];
		    break;
		  case 'f':
		    //console.log("w2",paire[2]);
		    w2=paire[2];
		    break;
		  case 'g':
		    //console.log("w3",paire[2]);
		    w3=paire[2];
		    break;
		  case 'h':
		    //console.log("da",paire[2]);
		    da=paire[2];
		    break;
		  case 'i':
		    //console.log("ts",paire[2]);
		    ts=paire[2];
		    break;
		  case 'j':
		    //console.log("ga",paire[2]);
		    ga=paire[2];
		    break;
		  case 'k':
		    //console.log("ba",paire[2]);
		    ba=paire[2];
		    break;
		  case 'l':
		   // console.log("ar",paire[2]);
		    ar=paire[2];
		    break;
		  case 'm':
		   // console.log("an",paire[2]);
		    an=paire[2];
		    break;
		  case 'n':
		    //console.log("IR temp",paire[2]);
		    it=paire[2];
		    break;
		  case 'o':
		    //console.log("wr",paire[2]);
		    wr=paire[2];
		    break;
		  case 'p':
		    //console.log("bud 1",paire[2]);
		    b1=paire[2];
		    break;
		  case 'q':
		    //console.log("bud 2",paire[2]);
		    b2=paire[2];
		    break;
		  case 'r':
		    //console.log("bud 3",paire[2]);
		    b3=paire[2];
		    break;
		  case 's':
		    //console.log("bud4",paire[2]);
		    b4=paire[2];
		    break;
		  case 't':
		    //console.log("sun radiation",paire[2]);
		    su=paire[2];
		    break;
		  case 'u':
		    //console.log("wind direction",paire[2]);
		    wd=paire[2];
		    break;
		  case 'v':
		    //console.log("SF-110 bud",paire[2]);
		    sb=paire[2];
		    break;
		  case 'w':
		    //console.log("SF-110 leaf",paire[2]);
		    sl=paire[2];
		    break;
		  
		}
		//resultats.push(paire);
	}
	
  return {
    te:te,
    pr:pr,
    hu:hu,
    lu:lu,
    w1:w1,
    w2:w2,
    w3:w3,
    da:da,
    ts:ts,
    ga:ga,
    ba:ba,
    ar:ar,
    an:an,
    it:it,
    wr:wr,
    b1:b1,
    b2:b2,
    b3:b3,
    b4:b4,
    su:su,
    wd:wd,
    sb:sb,
    sl:sl,
  }
  
}

Will it decrease the time to decode?

If I understand your writting, I should delete that and decose from my PHP script? Isn’t?

PS : Actually I have that problem only with 2 station over 10 stations

kersing · May 17, 2022, 9:11pm

Simply that the decoder script uses too much time to produce a result. That can be caused by a large load on the TTN infrastructure (too many decoders running) but also by a script that is not optimized to produce a result fast or gets stuck in a loop.

If you want to be sure you always have a decoded payload you will need to do the deciding in PHP, if you do it helps to delete the unnecessary script from TTN but that is not required.

What is different for those two? More sensors resulting in more values?

pierrot10 · May 18, 2022, 6:08am

Dear Kersing

What is different for those two? More sensors resulting in more values?

8 bud stations send two different measure of temperature an the value of battery (3 measures). Mainly, the problem is with that station.
2 meteo stations send the wind speed and direction, pressure, humidity, temperature, rain auge, battery and sun radiation (8 measures). The two station I eperiment that issue is with one meteo station and one bud station. The other bud station does not make that problem (for now ).

All of my stations use that that decoder
(payload formatters → uplink - custom javascript formatter)

function Decoder(bytes, port) {
  var str=String.fromCharCode.apply(null,bytes);
  var astr = str.split(",");
  //console.log(astr);

  //var regex = /([a-z]+)(\d+)/g;
  var regex = /([a-z]+)(\-?\d+)/g;

  //console.log(regex.exec(astr[0]));
  
  //var resultats = [];
var paire;
 
  var decoded = {}
  var te,pr,hu,lu,w1,w2,w3,da,ts,ga,ba,ar,an,it,wr,b1,b2,b3,b4,su,wd,sb,sl;
  while (null !== (paire = regex.exec(astr[0]))) {
		//console.log(paire[1]);
		switch(paire[1]){
		  case 'a':
		    //console.log("te",paire[2]);
		    te=paire[2];
		    break;
		  case 'b':
		    //console.log("pr",paire[2]);
		    pr=paire[2];
		    break;
		  case 'c':
		    //console.log("hu",paire[2]);
		    hu=paire[2];
		    break;
		  case 'd':
		    //console.log("lu",paire[2]);
		    lu=paire[2];
		    break;
		  case 'e':
		    //console.log("w1",paire[2]);
		    w1=paire[2];
		    break;
		  case 'f':
		    //console.log("w2",paire[2]);
		    w2=paire[2];
		    break;
		  case 'g':
		    //console.log("w3",paire[2]);
		    w3=paire[2];
		    break;
		  case 'h':
		    //console.log("da",paire[2]);
		    da=paire[2];
		    break;
		  case 'i':
		    //console.log("ts",paire[2]);
		    ts=paire[2];
		    break;
		  case 'j':
		    //console.log("ga",paire[2]);
		    ga=paire[2];
		    break;
		  case 'k':
		    //console.log("ba",paire[2]);
		    ba=paire[2];
		    break;
		  case 'l':
		   // console.log("ar",paire[2]);
		    ar=paire[2];
		    break;
		  case 'm':
		   // console.log("an",paire[2]);
		    an=paire[2];
		    break;
		  case 'n':
		    //console.log("IR temp",paire[2]);
		    it=paire[2];
		    break;
		  case 'o':
		    //console.log("wr",paire[2]);
		    wr=paire[2];
		    break;
		  case 'p':
		    //console.log("bud 1",paire[2]);
		    b1=paire[2];
		    break;
		  case 'q':
		    //console.log("bud 2",paire[2]);
		    b2=paire[2];
		    break;
		  case 'r':
		    //console.log("bud 3",paire[2]);
		    b3=paire[2];
		    break;
		  case 's':
		    //console.log("bud4",paire[2]);
		    b4=paire[2];
		    break;
		  case 't':
		    //console.log("sun radiation",paire[2]);
		    su=paire[2];
		    break;
		  case 'u':
		    //console.log("wind direction",paire[2]);
		    wd=paire[2];
		    break;
		  case 'v':
		    //console.log("SF-110 bud",paire[2]);
		    sb=paire[2];
		    break;
		  case 'w':
		    //console.log("SF-110 leaf",paire[2]);
		    sl=paire[2];
		    break;
		  
		}
		//resultats.push(paire);
	}
	
  return {
    te:te,
    pr:pr,
    hu:hu,
    lu:lu,
    w1:w1,
    w2:w2,
    w3:w3,
    da:da,
    ts:ts,
    ga:ga,
    ba:ba,
    ar:ar,
    an:an,
    it:it,
    wr:wr,
    b1:b1,
    b2:b2,
    b3:b3,
    b4:b4,
    su:su,
    wd:wd,
    sb:sb,
    sl:sl,
  }
}

It’s not very clear for me how I can avoid that issue

Simply that the decoder script uses too much time to produce a result. That can be caused by a large load on the TTN infrastructure (too many decoders running)

Can I do something?

but also by a script that is not optimized to produce a result fast or gets stuck in a loop.
Can I really optimize my decoder script???
It’s look no so complex, isn’t?

pierrot10 · June 19, 2022, 8:45pm

I still have that problem and I realize something interresting.

Into my database, I have a collections table. A collection is a set of measure. For example, my station have 4-5 sensors. My station take measure will and send the measures. All of those measure are a collection, a collection of measures.

In my ‘collections’ table I save some information that I collect from TTN

Hardware serial
the port
counter
m_time
m_freq
rssi
channel
snr
sf
*etc…

I obersvered that the ‘port’ colomn is always at 1 excepted when there is no measure.
In other word, if my payload is not available, the port is egal to 0

Does it make sense?
Someone can explain me what is the port and why is egal to 0? What does it means?

"uplink_message": {
      "session_key_id": "AYF8EZvQkR7wbL8pDlvorA==",
      "f_port": 0,
      "f_cnt": 8,
....
}

When I receive the data from TTN, before saving the data to ma database, should I filter the port and do not save the data if the port is egal to 0?

kersing · June 19, 2022, 9:31pm

Port 0 is reserved for uplinks (and downlinks) without data and just network information, you can safely ignore those packets. Checking the port of an uplink is valid (as in the value(s) your node used) is a good strategy.