Approaches for Effcient Payload Encoding

Continuing the discussion from Is there a list with public available sensors:

The design and configuration appears odd to me, I’d rather prefer that something deals with one and only one aspect. So a plain library and support tools. ConCaVa tries to solve too many things for my taste :wink:

What I’d be looking for are mechanisms for efficient encoding of information on both the sender and the reciever side.
Protobuf may be an option (Google does not support JS but there is an implementation here GitHub - protobufjs/protobuf.js: Protocol Buffers for JavaScript (& TypeScript). - I am looking for JavaScript as I want to be able do the encoding straight in TTN.

Also ASN.1 unaligned PER is super small.

The challenge is to be able to create a message on the node without manually assembling it.
Protobuf and ASN.1 can do this although I have never tried ASN.1 generated code on small arduinos.

Not long ago, @rozzie suggested using NanoPB (lightweight protobuf implementation) for a similar situation.

1 Like

Yes - I have found nanopb to be an incredibly convenient option for my app, where there are a highly-variable combination of “required” and “optional” fields. The encoding is really efficient and it is trivial to program.

Some people find it easier to understand if they see an example, so here’s a snippet from one of my projects:

{
    // Allocate space on the stack to store the message data
    teletype_Telecast message = teletype_Telecast_init_zero;

    // Create a pb stream within that buffer
    pb_ostream_t stream = pb_ostream_from_buffer(buffer, sizeof(buffer));

    // Set fields marked as required in .proto
    message.DeviceType = teletype_Telecast_deviceType_BGEIGIE_NANO;
    strncpy(message.DeviceID, DeviceID, sizeof(message.DeviceID));
    strncpy(message.CapturedAt, DateTimeISO, sizeof(message.CapturedAt));
    message.Unit = teletype_Telecast_unit_CPM;
    message.Value = RadiationCPM;

    // Set fields marked as optional in .proto
    if (GPSValid) {
        message.Latitude = GPSLatitude;
        message.Longitude = GPSLongitude;
        message.Altitude = GPSAltitude;
        message.has_Latitude = 
            message.has_Longitude = 
            message.has_Altitude = true;
    }

    // Encode and transmit the message
    status = pb_encode(&stream, teletype_Telecast_fields, &message);
    if (!status)
        DEBUG_PRINTF("pb_encode: %s\n", PB_GET_ERROR(&stream));
    else
        send_to_ttn(buffer, stream.bytes_written);

}
2 Likes

Just in case you didn’t know: TTN supports ConCaVa-like “payload functions”, which can even give one very readable JSON output directly in the MQTT messages.

See “Decode the Payload in the Console” in The Things Uno Workshop, to get from, for example:

03 0B 07 BE

…to:

{
  "celcius": 19.82,
  "light": 779
}

…by simply defining the following for your application’s decoder function:

function Decoder(bytes, port) {
  var light = bytes[0]<<8 | bytes[1];
  // Sign-extend to 32 bits to support negative values, by shifting 24 bits 
  // (too far) to the left, followed by a sign-propagating right shift:
  var temperature = bytes[2]<<24>>16 | bytes[3];  
  return {
    light: light,
    celcius: temperature / 100
  };
}

(And see also Best practices to limit application payloads.)

Hi, I send from my Arduino a temperature (float on Arduino) as 4 bytes:

temp: 20.12
bytes send : 0xD0, 0xF6, 0xA0, 0x41

Any idea how to convert this ‘back’ to 20.12 using payload functions?

I tried the example from above but that doesn’t handle negative values well.

Very nice library by @joscha, as mentioned elsewhere, with good documentation: