# Payload Formats \[HowTo\]

**URL:** <https://www.thethingsnetwork.org/forum/t/payload-formats-howto/3441>\
**Category:** Application Development\
**Created:** [September 10, 2016, 2:47pm UTC](https://www.thethingsnetwork.org/forum/t/payload-formats-howto/3441 "2016-09-10T14:47:46Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![mark-stanley](https://www.thethingsnetwork.org/forum/user_avatar/www.thethingsnetwork.org/mark-stanley/32/44641_2.png) [@mark-stanley](https://www.thethingsnetwork.org/forum/u/mark-stanley)\
**Post date:** [September 10, 2016, 2:47pm UTC](https://www.thethingsnetwork.org/forum/t/payload-formats-howto/3441/1 "2016-09-10T14:47:46Z")

</div>

I have seen [some examples on payload functions](https://www.thethingsnetwork.org/forum/t/best-practices-when-sending-gps-location-data/1242/14).

Is there any guide on writing payload functions - what language are you using - what functions are available?

---

<div class="post-metadata">

**Author:** ![mogul](https://www.thethingsnetwork.org/forum/user_avatar/www.thethingsnetwork.org/mogul/32/32037_2.png) [@mogul](https://www.thethingsnetwork.org/forum/u/mogul)\
**Post date:** [December 30, 2016, 7:56am UTC](https://www.thethingsnetwork.org/forum/t/payload-formats-howto/3441/2 "2016-12-30T07:56:52Z")

</div>

It looks like javascript, but either it is not or some of the convenient libraries are missing. Would have loved to use ArrayBuffer to convert from bytes to float. See [https://developer.mozilla.org/en-US/docs/Web/JavaScript/Typed\_arrays](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Typed_arrays)

---

<div class="post-metadata">

**Author:** ![arjanvanb](https://www.thethingsnetwork.org/forum/user_avatar/www.thethingsnetwork.org/arjanvanb/32/46626_2.png) [@arjanvanb](https://www.thethingsnetwork.org/forum/u/arjanvanb)\
**Post date:** [January 26, 2017, 10:14am UTC](https://www.thethingsnetwork.org/forum/t/payload-formats-howto/3441/3 "2017-01-26T10:14:48Z")

</div>

It’s JavaScript (ECMAScript 5) [indeed](https://github.com/TheThingsNetwork/ttn/blob/master/core/handler/application/application.go). For uplinks:

- Decoder is a JavaScript function that accepts the payload as byte array and returns an object containing the decoded values. It [gets passed](https://www.thethingsnetwork.org/forum/t/payload-functions/5277/2):
  - First parameter, typically called `bytes`: the binary payload as an array of numbers. Each number is a byte, 0 to 255 in decimal notation, which is the same as 0x00 to 0xFF in hexadecimal notitation.
  - Second parameter, typically called `port`: the [port number](https://www.lora-alliance.org/portals/0/specs/LoRaWAN%20Specification%201R0.pdf#page=20) from 1 to 223.

- Converter is a JavaScript function that accepts the object as returned by Decoder and returns an object containing the converted values.
- Validator is a JavaScript function that validates the data as returned by Converter and returns a boolean value indicating the validity of the data.
- ⚠ Converter and Validator [will be dropped in V3](https://www.thethingsnetwork.org/forum/t/convert-custom-payload-to-lpp-payload-in-ttn-console/12171/9).

For downlinks:

- Encoder is a JavaScript function that encodes the data queued for a Downlink messages and returns the converted values as an array of bytes.

### Metadata

Only the application payload and the port can be accessed, not any of the metadata such as the gateway(s) that received an uplink. See [Access Metadata in decoder](https://www.thethingsnetwork.org/forum/t/access-metadata-in-decoder/19118) for alternatives.

To add the approximate time an uplink was received, one could use the server time:

```javascript
var now = new Date().toISOString();

```

### Development

A good JavaScript resource is MDN Web Docs (formerly Mozilla Developer Network), and “mdn” is a great keyword for search engines: [mdn array](https://duckduckgo.com/?q=mdn+array).

Also, you can use auto-completion in the developer tools of your browser. Just start by defining your payload with something like:

```
var bytes = [0xFF, 0x31, 0xF9, 0x12, 0x00, 0x31]

```

Or, to easily copy/paste a hexadecimal string such as `0xff31f9120031`:

```
var bytes = 'ff31f9120031'.match(/(..)/g).map(b => parseInt(b, 16))

```

Next, typing `bytes.` will show you the supported functions and all:

 ![image](https://www.thethingsnetwork.org/forum/uploads/default/original/3X/2/1/21174236a4739080293158fe18ae8afe96b4aad4.png "Chrome developer tools")

### Debugging

Anything you send to `console.log` is shown in TTN Console during testing:

```
console.log(bytes);

```

You should always test your format using the Payload input field in TTN Console, and [the error events in `+/devices/+/events/#` in the MQTT API](https://www.thethingsnetwork.org/docs/applications/mqtt/api.html#error-events) can also be used to track errors at runtime, like:

> Unable to decode payload fields: Internal error: Decoder threw error: ReferenceError: ‘myVariable’ is not defined

> Unable to decode payload fields: Internal error: Interrupted javascript execution for Decoder after 153.496916ms

> Payload Function output cannot be marshaled to JSON: json: unsupported value: NaN

Given the “Error” filter in the application/device’s Data page in TTN Console one might expect such errors to be shown there too, but: no. When not seeing payload fields in TTN Console, then paste the application payload bytes into the Decoder’s test input, or use MQTT to see any errors.

### Gotchas

- The [payload formats may fail randomly](https://www.thethingsnetwork.org/forum/t/payload-available-but-sometimes-not-decoded-to-payload-fields-in-mqtt/34255), so consider decoding in your application instead. (And consider subscribing to [the MQTT error events](https://www.thethingsnetwork.org/docs/applications/mqtt/api.html#error-events), regardless where you’re decoding.)

- A nice read: [Here is what you need to know about JavaScript’s Number type](https://medium.com/angular-in-depth/javascripts-number-type-8d59199db1b6).

- When doing bitwise operations, [JavaScript will interpret its operands as 32-bits signed integers](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Bitwise_Operators#Signed_32-bit_integers). (Other programming languages are unlikely to always assume 32-bits signed integers for bitwise operators.)

- Don’t confuse binary with “hexadecimal”: [How to send payload in Hex-format? - #4 by arjanvanb](https://www.thethingsnetwork.org/forum/t/how-to-send-payload-in-hex-format/8534/4)

- Leading zeroes in number literals [denote octal values](https://www.thethingsnetwork.org/forum/t/decoding-smart-waste-bin-detector-payload/23291/12).

- While testing in TTN Console, `null` values might not be shown. But integrations might actually get such `null` values.

- The Data Storage Integration [silently fails for attributes with `null` values](https://www.thethingsnetwork.org/forum/t/data-storage-not-showing-decoded-data-and-only-stores-data-two-times-a-day/39316/9) and gets confused when the decoder outputs [attributes that it uses itself, such as `device_id`](https://www.thethingsnetwork.org/forum/t/data-storage-device-id-query-not-working/40516/11).

- The Data Storage Integration [stores nested objects as text](https://www.thethingsnetwork.org/forum/t/storage-integration-not-showing-payload-fields-when-integers-or-floats/32220/3).

- An erroneous Converter might suppress output: ["Fields" not being populated by decoder? - #2 by evs01](https://www.thethingsnetwork.org/forum/t/fields-not-being-populated-by-decoder/36384/2)

- An erroneous Encoder may make a downlink being [scheduled but not transmitted](https://www.thethingsnetwork.org/forum/t/downlink-shows-as-scheduled-but-is-no-longer-sent-to-my-device-or-gateway/39641/3). Subscribe to the MQTT error events to see the error.

### Alternatives

- Decode [the Base64-encoded `payload_raw`](https://www.thethingsnetwork.org/forum/t/what-to-do-with-the-raw-payload-via-mqtt/4889) in your actual application (like an application that uses [the MQTT Data API](https://www.thethingsnetwork.org/docs/applications/mqtt/api.html)).

### Examples

- [Working with Bytes](https://www.thethingsnetwork.org/docs/devices/bytes.html) in the official documentation, also explaining bitwise operators such as `<<`, `|` and `&`.

- “Decode the Payload in the Console” in [The Things Uno Workshop](https://github.com/TheThingsNetwork/workshops/blob/master/The%20Things%20Network/README.md#decode-the-payload-in-the-console).

- Handling (negative) 3 byte values in [Best practices when sending GPS location data](https://www.thethingsnetwork.org/forum/t/best-practices-when-sending-gps-location-data/1242/21), using [sign extension](https://en.wikipedia.org/wiki/Sign_extension) for both [MSB and LSB](https://en.wikipedia.org/wiki/Most_significant_bit#Most_Significant_Bit_First_vs_Least_Significant_Bit_First).

- A step-by-step example of byte shifting and sign extension in both binary and hexadecimal representation, in [Payload Format / Decode for TM-901 / N1C2 sensor from KCS TraceME - #4 by arjanvanb](https://www.thethingsnetwork.org/forum/t/payload-format-decode-for-tm-901-n1c2-sensor-from-kcs-traceme/36313/4).

- Sending (negative) temperature and humidity values, optionally extended with battery level, using the port number to determine what type of packet is being decoded in [Getting Badgerboard to work with TTN](https://www.thethingsnetwork.org/forum/t/getting-badgerboard-to-work-with-ttn/5224).

- Decoding _unsigned_ 32 bits numbers, getting a value from only a few bits from a byte, outputting arrays with nested data, pretty-printing hexadecimal output, and handling different message types in a single decoder, in [Abeeway Microtracker Decoder - #2 by arjanvanb](https://www.thethingsnetwork.org/forum/t/abeeway-microtracker-decoder-help/22473/2).

- Working with a variable number of measurements, and variable types of measurements, indicated using bit masks, in [Nemeus NIS-UL – Ultrasonic sensor - #13 by arjanvanb](https://www.thethingsnetwork.org/forum/t/nemeus-nis-ul-ultrasonic-sensor/14770/13).

- Converting a true IEEE-754 floating point in [Decode float sent by Lopy as Node - #2 by arjanvanb](https://www.thethingsnetwork.org/forum/t/decode-float-sent-by-lopy-as-node/8757/2).

- Using the third-party [lora-serialization library](https://github.com/thesolarnomad/lora-serialization) for encoding and decoding, a bit on accidentally mixing MSB and LSB, a bit more on `(bytes[0] & 0x80 ? 0xFFFF<<16 : 0)` and `bytes[0]<<24>>16` used for sign extension, and some more info on using port numbers, in [Decrypting messages for dummies](https://www.thethingsnetwork.org/forum/t/decrypting-messages-for-dummies/4894).

- Decoding [nibbles](https://en.wikipedia.org/wiki/Nibble) (4 bits) of a “packed [binary-coded decimal](https://en.wikipedia.org/wiki/Binary-coded_decimal)” (BCD), like interpreting the hexadecimal bytes 0x0230 as decimal 230 (rather than, [as more usual, 560](https://www.rapidtables.com/convert/number/hex-to-decimal.html?x=0230)), in [Decoding Smart Waste Bin Detector DF702 payload](https://www.thethingsnetwork.org/forum/t/decoding-smart-waste-bin-detector-payload/23291/10).

- Likewise in [How to decode Elvaco CMi4110 standard M-Bus payload? - #22 by arjanvanb](https://www.thethingsnetwork.org/forum/t/how-to-decode-elvaco-cmi4110-standard-m-bus-payload/31747/22), which also shows iterating a variable-length payload based on metadata in that very payload.

- Getting 3 12-bits counters, each encoded in 1.5 bytes (3 nibbles), from 4.5 bytes in [Agoraopinion Feedback Button Payload decoder - #5 by arjanvanb](https://www.thethingsnetwork.org/forum/t/agoraopinion-feedback-button-payload-decoder/35670/5).

- Decoding 7 bytes into two 28 bits numbers (really; 3.5 bytes per number) in [Help a Poor Guy with this Payload](https://www.thethingsnetwork.org/forum/t/help-a-poor-guy-with-this-payload/5014/8).

- Getting two integers from a null-terminated string like `"111 014"` in [Simple payload conversion](https://www.thethingsnetwork.org/forum/t/simple-payload-conversion/5273/2).

- Working with `struct` data on the node’s side, and decoding that in TTN Console, in [How can I correctly cast my value in byte before sending - #3 by arjanvanb](https://www.thethingsnetwork.org/forum/t/how-can-i-corretly-cast-my-value-in-byte-before-sending/14766/3).

- A Decoder for the Laird RS1xx, combining a byte for the signed integer part with another byte for the unsigned decimal part, in [Laird RS1xx payload format](https://www.thethingsnetwork.org/forum/t/laird-rs1xx-payload-format/19313).

- A Decoder and Encoder for the Dragino LT-33222, also explaining why in the Encoder one needs `& 0xFF` in `bytes[2] = object.field>>8 & 0xFF`, in [LT-33222-L LoRa I/O Controller - Payload decoder and downlink config](https://www.thethingsnetwork.org/forum/t/lt-33222-l-lora-i-o-controller-payload-decoder-and-downlink-config/19177).

- Generic decoders on the [Elsys website](https://www.elsys.se/en/elsys-payload/).

- Decoders for their products on the [Decentlab GitHub](https://github.com/decentlab/decentlab-decoders) (but beware those may output a field `device_id` [which confuses the Data Storage Integration](https://www.thethingsnetwork.org/forum/t/data-storage-device-id-query-not-working/40516/11)).

- Decoders for, e.g., TEKTELIC Agriculture and KONA Sensor, Bosch Parking Lot Sensor, Smart Building Sensors, MCF88 LW12CO2, Netvox R312A Emergency Button and R718N37 3-phase current monitor, on the [Sensational Systems GitHub](https://github.com/SensationalSystems).