ThingSpeak Integration: not all fields are being updated

Hi all!
sorry, but I’m relatively new to LoRa/TTN and also javascript…
I’m getting some data from a meter and trying to see them in thingspeak. I can see some fields in thingspeak channel but not all the fields. Specifically, fields1, 2 and 3 doesn’t update but the rest are ok.

Payload examples:
0010E423010C43707620436F75C8436DCC70705A
0010E423020C40D6D08040BF4280408B8A80D34B
0010E423030CBF5A6ED6BF5CB1EEBF7FD34889AC
0010E423040C43CFF84C43CDB6FC43CEB8987CC7
0010E423050CBF6A23D645822E0E45126E429E99
0010E423060C44B7204544ADD6AB44A3C1440A50
0010E423070C448536DB447DB880438325A708B7

Here is my decoder function, any idea will be appreciated. Thanks!

function Decoder(bytes, port)
{
  var meterID = bytes[0] << 24 | bytes[1] << 16 | bytes[2] << 8 | bytes[3];
  var step = bytes[4];
  var nbytes = bytes[5];
  var bits1 = bytes[6]<<24  | bytes[7]<<16  | bytes[8]<<8  | bytes[9];
  var bits2 = bytes[10]<<24 | bytes[11]<<16 | bytes[12]<<8 | bytes[13];
  var bits3 = bytes[14]<<24 | bytes[15]<<16 | bytes[16]<<8 | bytes[17];
  var crc =  bytes[18] << 8 | bytes[19];
  var bits; // generica

  function bytestofloat(bytes, bits){
    var sign = ((bits >>> 31) == 0) ? 1.0 : -1.0;
    var e = ((bits >>> 23) & 0xff);
    var m = (e == 0) ? (bits & 0x7fffff) << 1 : (bits & 0x7fffff) | 0x800000;
    var x = sign * m * Math.pow(2, e - 150);
    return x;
  }
  
  if (step == 1){
   var V1 = bytestofloat(bytes,bits1);
   var V2 = bytestofloat(bytes,bits2);
   var V3 = bytestofloat(bytes,bits3);
  }
  if (step == 2){
   var I1 = bytestofloat(bytes,bits1);
   var I2 = bytestofloat(bytes,bits2);
   var I3 = bytestofloat(bytes,bits3);
  }
  if (step == 3){
   var PF1 = bytestofloat(bytes,bits1);
   var PF2 = bytestofloat(bytes,bits2);
   var PF3 = bytestofloat(bytes,bits3);
  }
  if (step == 4){
   var V12 = bytestofloat(bytes,bits1);
   var V23 = bytestofloat(bytes,bits2);
   var V31 = bytestofloat(bytes,bits3);
  }  
  if (step == 5){
   var PF = bytestofloat(bytes,bits1);
   var kWh = bytestofloat(bytes,bits2);
   var kVArh = bytestofloat(bytes,bits3);
  }  
  if (step == 6){
   var ikWh1 = bytestofloat(bytes,bits1);
   var ikWh2 = bytestofloat(bytes,bits2);
   var ikWh3 = bytestofloat(bytes,bits3);
  } 
  if (step == 7){
   var ekWh1 = bytestofloat(bytes,bits1);
   var ekWh2 = bytestofloat(bytes,bits2);
   var ekWh3 = bytestofloat(bytes,bits3);
  } 
  return{
    field1:ikWh1,
    field2:ikWh2,
    field3:ikWh3,
    field4:PF,
    field5:kWh,
    field6:V1,
    field7:V2,
    field8:V3
  };
}

What does “doesn’t update” mean? Are you sure you’re not getting the very same payload from the device, every time step == 6?

Testing with your 0010E423060C44B7204544ADD6AB44A3C1440A50 nicely decodes to:

{
  "field1": 1465.0084228515625,
  "field2": 1390.7083740234375,
  "field3": 1310.03955078125
}

Of course, I don’t know what the expected values are. But when indeed expecting floating point numbers (MSB) then the decoder seems fine.

For future readers who might like to use your Decoder: can you add some details about the meter? (Brand, type, …)

(Some asides: for the meterID see also 32 Bit overflow in Payload Format decoder, as I’m sure you’re not expecting that to ever be decoded into a negative value, though you’re not outputting that anyhow. The accuracy of the float values is not truly that many decimals; you can use toFixed to limit the decimals. Also, nbytes seems to suggest the payload might be shorter or longer as well, and bytestofloat does not use its bytes parameter at all. And finally: it seems the meter includes a CRC, which I feel is not needed as LoRa already adds a CRC, and the LoRaWAN message already includes a MIC. But I assume that’s out of your control.)

1 Like

Hi Arjan,
thanks for your support.
With “doesn’t update” I mean data is not being refreshing in thingspeak. I can’t see data in fields1, 2, 3 but in fields 4, 5, 6, 7, 8.
Regarding the payloads, I periodically receive 7 payloads with step varying from 01 to 07. If I test each payload in “ttn-payload formats” everything works fine, in fact in ttn/devices/data I can see all decoded payloads but not in thingspeak.
Expected values are ok.
The meter is SDM630 from Eastron Electronic.
“Asides”: ok to all, thanks!
Here are some pics, both ttn and thingspeak screens:
imagen
imagen

Your TTN Console screenshot cannot be the result of the decoder you’ve shown earlier. It shows string values, enclosed in quotes like field1: "1720.32", rather than true numbers like field1: 1720.32:

string values
If meanwhile you added toFixed then ensure to convert to a number, like explained in my note about the unary plus-operator in my earlier link.

Given the new (different) values in the payload for step == 06 it seems the meter is sending updated values just fine. (And of course, if the values would be the same all the time, then you’d see a flat line, not a single dot.) And it seems the TTN Console decoder is working fine too—except for it currently returning string values, but apparently that works for the other fields just fine?

So, I’d guess the problem is in the configuration on ThingSpeak. What happened around the time 09:56:53 as shown in the screenshot? Maybe simply remove those 3 charts and recreate them? Maybe there’s some rate-limiting at ThingSpeak, meaning that you’re sending values too often?

HI Arjan,
sorry you’re right, I was testing with toFixed. I’ve just kept the code as the first I posted and here is the output:
TTN Console
I also think that the problem could be in thingspeak, but channel settings are so simple and I’m afraid I can’t change almost nothing.
However, if I change the variable assigned to each field in ttn, for example assigning variables in fields1,2,3 to fields6,7,8 and viceversa, now the problem is within fields 6,7,8 and I can see f1,2,3 in thingspeak, so the problem could be related to the variables themselves?
I don’t now what happends at 09:56:53. There are sometimes that data arrives to all thingspeak fields, but this happends just very little, maybe 1 of 100? data transmissions.

Thanks again.

You would still have the same order in which the data comes in from the sensor, so when handling step == 6 you’ve already sent 5 readings to ThingSpeak in just 24 seconds, and the reading for step == 6 is the 6th value in 30 seconds.

Does ThingSpeak document anything on rate-limiting?

If there’s a limit, then I’d assume that even the “empty” fields like for step == 2 and so on, might count against such rate limit. Maybe using return null rather than returning an empty object will make those empty items not being forwarded to ThingSpeak, but I really don’t know. That would then read something like:

  if (step == 2) {
    return null;
  }

You might also be able to shield the device or disable the gateway(s) for the first readings (like shield it for 15 seconds after you see the data for step == 1 coming in, to make a few transmissions not being received).

Also, increasing the SF might make the device use a longer interval between each step. But that’s for testing only; you should not use a high SF if not needed (and you should never use a fixed value of SF11 or SF12); see Limitations: data rate, packet size, 30 seconds uplink and 10 messages downlink per day Fair Access Policy [guidelines].

If testing by shielding the device, disabling the gateway, or changing the SF proves that some rate limiting is active, but if returning null does not help: maybe a Validator can be used to suppress forwarding empty results to ThingSpeak. However, in V3 support for that will be dropped, so this would be temporary then.

Something like:

// UNTESTED; not sure if this actually stops integrations or MQTT
// from being triggered. Also, support will be dropped in V3.
function Validator(converted, port) {
  // Suppress further handling if the number of fields is zero
  return Object.keys(converted).length > 0;
}

If some rate limiting is active (and applies to both the integration and MQTT), and the above doesn’t help either, then you’d need an intermediate server to collect the results for step 1 through 7, and only forward all collected data to ThingSpeak after receiving step 7 (or after some timeout).

Of course, it would be best if the device would not need a whopping 7 transmissions to get its data through, and would not transmit data that is not of your interest to start with… :frowning:

I’ve tested with returning null and a validator function and results are the same. I have no control over the SF of the “node” :(.
I also suspect that the problem could be related to thingspeak limitations so I think it’s time to change to other integration or try paying a thingspeak account…
Thank you!

If they didn’t document limits, then you could still validate that assumption by temporarily shielding the device or disabling the gateway, to get fewer LoRaWAN uplinks.

A quick search for SDM630 LoRaWAN manual shows me the following.

SDM630 LoRaWAN settings

What documentation do you have? Where did you get the details about the payload?

Again, it would be best if you can configure the device to send much less data. The manufacturer might be able to tell if that’s an option?

Yes, I need to test it.

Yes, I read this document, but my device interface looks different, maybe version issues or whatever. I got the protocol info directly from the manufacturer.