Understanding the Structure of Device Payload

TTN Network and Services The Things Stack v3
1

Hello all,

Apologies if this has been asked already, but I am having some trouble understanding how to write a JavaScript payload formatter. From the examples shown in the TTN docs, I’m unsure of how the writers of the code knew which bytes corresponded to what information and how to manipulate them to translate the payload to human-readable information. I also looked at the following three references in hopes of some insight, but was unable to find anything that would point me in the right direction:

I’m developing with a SAM R34 Xplained Pro evaluation module and one thought that crossed my mind was that the SAM has a device-specific format for its payload, but I was unable to find any information in the MLS API guide or the datasheet.

How do we know what data is represented by certain bytes in a given payload, and how those bytes can be manipulated to yield some human-readable information? An example payload may be “08D95736FF36338801333380FB79BE791B02334313803C234B43C849”.

Thank you in advance for all help!

2

The application on your node determines creates the payload. If you are working with sources you should be able to find the code where the data to be transmitted is put into a buffer. If you are developing your own code you are the one responsible for putting the data in the buffer so you should know what bytes contain what data.

For bought devices usually the documentation contains a hint.

3

For the SAMR34 Xplained Pro, this:

Whilst there are some demo apps, the whole point of a dev board is for you to write your own app which, as part of the fundamentals, means you write the byte sequence of the payload.

4

Hi kersing,

Thank you for your reply. I’m developing my own application but I’m relying heavily on the LoRaWAN API that Microchip provides and I don’t recall seeing any API functions that made it clear how the byte sequence is built before being sent.

Perhaps I missed something, I will look more closely.

Thank you!

5

You need to look at the code that fills the byte array for transmission. That array is passed to the API which handles the transmission. I don’t recall any API calls that fill the byte array.

6

Hi descartes,

Thank you for your reply.

I used the LoRaWAN mote demo application as the basis for my own program, but as far as I can see there’s not really any explicit code that makes it clearer how the byte sequence is being built.

I’ll look more closley at the documentation and see if I can find anything.

Thank you!

7

Hi kersing,

I’m using an snprintf() call to create a C-string that I am passing as the payload array, but the payload at my Console doesn’t seem to reflect the ASCII character codes for those individual characters of the string, hence why I thought there was something extra to the payload that I was missing.

This may be a silly question, but is there a different kind of character encoding scheme that’s being used here instead of ASCII?

Thank you in advance!

8

Have you tried to decode the hex string back to ASCII?
Use https://www.binaryhexconverter.com/hex-to-ascii-text-converter and be surprised by the result.

BTW. sending ASCII text is extremely bad practice. The example code from microchip should not use it either. You should use https://www.thethingsnetwork.org/docs/devices/bytes/ (ignore the warning about V2) to find an efficient encoding.

9

Hi kersing,

The first payload formatter I created was a conversion from hex to ASCII which produced garbage output that wasn’t remotely close to what I had sent. I corroborated that with the link you provided.

Thank you for the heads up on the bad practice. I’m new to this and was rushed into prototyping so I wasn’t able to take the time to carefully study. I’ll keep this in mind for the future. I’ll also take look at that encoding link to see if it might reveal something I’m missing.

Thank you, I appreciate it!

10

There aren’t any because it’s just a byte array that you define and fill and the pass over to the API for transmission.

Not explicit as in it has a huge neon sign arrow pointing at it, but clear enough in the run of the code.

Again, not something really in the docs, mostly because there is an assumed level of prior experience for what is a relatively straightforward part of an IoT device firmware.

If this is a commercial project that you have been hired or contracted to do, you need to find a way to take some time out to get a better overview of both firmware & LoRaWAN.

You could do worst then just printing out the basic app source code & going through it with a highlighter pen.

Or looking at something like the Arduino LMIC code which manages to fit it on to a couple of screen fulls.

As well as consume the LoRaWAN section of the Learn pages (link at the top of the page) and the Devices & Gateways section where you will find a page about working with bytes.

Very very few people have succeeded with LoRaWAN without the body of knowledge provided in those sections.

11

Hi descartes,

Thanks for your reply.

There aren’t any because it’s just a byte array that you define and fill and the pass over to the API for transmission.

This is what I’m understanding from your and kersing’s replies, but I’m just quite puzzled how even sending a 0 or 1-bit is producing this long payload sequence that I’m not sure how to even begin to decode. Perhaps it shouldn’t be behaving this way and there’s something else afoot here that I’ll have to investigate.

If this is a commercial project that you have been hired or contracted to do, you need to find a way to take some time out to get a better overview of both firmware & LoRaWAN.

I wish I could make this point clearer to my overlords, but I have to make do with the time I have.

Thank you for pointing me in the right direction, I appreciate it!

12

Point them at the forum?

13

Start by showing your encoder.

And where did you find this ‘payload’

14

Hi Johan,

Thank you for your reply.

I’m on the v3 stack and my encoder is as follows:

function decodeUplink(input) {
  return {
    data: {
      decodedPayload: String.fromCharCode(...input.bytes)
    },
    warnings: [],
    errors: []
  };
}

I’m not a JavaScript maestro by any means, but this seems to be a simple approach for converting hex-represented bytes to characters. If I am doing something blatantly wrong, please correct me. This outputs the following:

{
  "decodedPayload": "\bÙW6ÿ63ˆ\u000133€ûy¾y\u001b\u00023C\u0013€<#KCÈI"
}

As for the “08D95736FF36338801333380FB79BE791B02334313803C234B43C849”, this is what shows up in my Console as a result of my device sending a char array (C-style string) that originally reads “Occupied”.

Thank you in advance for your help!

15

That is a decoder.

Encoder is a bit of the code that resides on your device/end node, that bit of code that makes up your payload you are going to send.

Where di you find this on your console?

16

Hi Johan,

That is a decoder.

I apologize, I misread your reply. I’m not sure if this qualifies as an “encoder” per se, but this is a rough idea of the code I am running on my end device using Microchip’s LoRaWAN Stack API:

char sensor_payload[15]; 
LorawanSendReq_t lorawan_send_req; 

snprintf(sensor_payload, sizeof(sensor_payload), "%s", 
    sensor_get_num_detected_objects() != 0 ? "Occupied" : "Available"); 

uint8_t data_len = strlen(sensor_payload); 
lorawan_send_req.buffer       = &sensor_payload; 
lorawan_send_req.bufferLength = data_len - 1; 
lorawan_send_req.confirmed    = LORAWAN_UNCNF; 
lorawan_send_req.port         = 1; 
status = LORAWAN_Send(&lorawan_send_req);

As for where I find the raw payload in my Console, I first navigate to Console → Applications → my-application-id → End Devices → my-end-device-id → Live Data. Then, when I receive an uplink from my end device, a new row appears in the Live Data tab that reads “Forward Uplink Data Message” followed by my device’s address and the payload that I reference.

Thank you in advance for your help!

17

Please use something like 0 for available and 1 (or 255) for occupied. That way you just need to transmit one byte in stead of 8 or 9. Less battery consumption and more importantly less airtime, leaving the airwaves available for another sensor.

18

It seems you didn’t take the huge hint I gave you about the Learn section or the explicit hint that Jac gave you to the page on dealing with bytes.

You really can’t hack your way through LoRaWAN without understanding the fundamentals - like the way the payload is Base64 encoded so you can’t just take the console entry and try and treat that as ASCII bytes. But what you see in the gateway console - do you know what the data is there? And if you encounter DevNonce issues, typical of using MLS 1.0.5 with LW v1.0.4, how do you clear those down so you can carry on with development?

Everyone answering here are volunteers so there is a limit to the time they can provide. If you were a maker taking baby steps that would be fine, we’d provide a steer and, most importantly, look to see you are consuming the materials you are directed to.

As this is a commercial project, you have clearly demonstrated that it needs some input from someone who knows what they are doing to get you over the hump of your first LoRaWAN project - and then maybe for the next few after. Or you take out a month, assume the role of an apprentice and start with LoRaWAN 101, building from there.

As for MLS, there aren’t many people on here that use it - it’s mostly Arduino LMIC with a bit of LoRaMac-node. But the fundamentals still apply.

1 Like
19

Hi kersing,

Thank you for your suggestion. My original plan has always been to send a 0 or 1, but I was sending this particular message to become more familiar with how payloads work and how to decode the bytes.

I will definitely implement that whenever this particular prototype reaches the final stages.

Thank you!

20

Hi descartes,

Thank you for your reply.

It seems you didn’t take the huge hint I gave you about the Learn section or the explicit hint that Jac gave you to the page on dealing with bytes.

I do have other projects I need to work on here and I don’t think stating that I am not utilizing the valuable resources that you and kersing have provided after less than a day is fair. I was replying to other replies was receiving in the meantime and gauging what other subjects I should be looking into to improve my knowledge on this topic.

You really can’t hack your way through LoRaWAN without understanding the fundamentals - like the way the payload is Base64 encoded so you can’t just take the console entry and try and treat that as ASCII bytes. But what you see in the gateway console - do you know what the data is there?

Learning more about this is the reason I posted to the forum.

And if you encounter DevNonce issues, typical of using MLS 1.0.5 with LW v1.0.4, how do you clear those down so you can carry on with development?

I used a CLI command to reset join nonces.

Everyone answering here are volunteers so there is a limit to the time they can provide. If you were a maker taking baby steps that would be fine, we’d provide a steer and, most importantly, look to see you are consuming the materials you are directed to.

I understand this and I have indicated that I appreciate all the time and guidance that has been given to me. I don’t mean to appear unappreciative or unwilling to learn on my own, and I apologize if this is the impression I have given.

As this is a commercial project, you have clearly demonstrated that it needs some input from someone who knows what they are doing to get you over the hump of your first LoRaWAN project - and then maybe for the next few after. Or you take out a month, assume the role of an apprentice and start with LoRaWAN 101, building from there.

As much as I agree with you, I am powerless to do anything about this. I’m a new-grad working alone on the firmware of this project and LoRaWAN is unfamiliar to me. The Learn section and these forums are all the resources I have.

I do appreciate your time and help. I will close this thread and take more time to learn the fundamentals.