Convert custom payload to lpp payload in TTN console

Absolutely, the drawback of this is the need of another clouded server running MQTT client and some JS scripts (node red or native).
I’m fan of “less streams, less servers” => less things to check and less problems :wink:

But I will do that, got an NodeRed Instance running on my server
Thanks for the tip.

@johan
I tried your solution, unfortunately I’m not happy how cayenne handle that. While everyone is using user/pass in the MQTT broker and then ThingsID in the topic, cayenne NEEDS broker clientID set as ThingsID in the broker (and also in the topic).
This means if you have 100 devices, you need 100 different broker connexion (each with it’s own clientID). despite the fact than configuring 100 brokers in node red will be a nightmare, even if it could works, I can’t (don’t want sorry) use as many connexion as things, painfull
I know it’s not TTN fault but I’m back to my first step. I think adding the possibility to return a byte buffer from the converter will be really simple for everyone here to convert any payload.
But, I know simple for us does not mean simple for you.
If you could just think about the idea it would be very kind.

1 Like

I see. It’s not the first IoT platform with that requirement. That MQTT is really designed for end device to cloud, not cloud to cloud.

What you can do is posting with TTN’s JSON format, i.e. the one you see in MQTT and HTTP integration, to: https://lora.mydevices.com/v1/networks/ttn/uplink. Cayenne looks at the payload_raw, in which you can put anything that you like. But, it has to be CayenneLPP.

Now, I would say this is the hacky solution and there’s no guarantee this API will stay available this way.

Exactly what I wanted to do, I thought if TTN can send LPP I should be able to do also just needed the undocumented interface
Thanks for sharing, since I’m doing from node red should be easy just to transform payload_raw between TTN mqtt and cayenne HTTPS post :wink:
Hope there is no auth or source ip filtering for this one
Thanks

There is indeed no auth nor IP filtering, and that is indeed what’s is wrong with this interface :wink:

1 Like

@johan So

won’t be implemented? Currently I don’t have any Node RED running, so I can do what @Charles plans to do… :confused:

In V2 we don’t allow altering payload_raw. It will, technically, be supported in V3 but not sure if it will be GA in the public Consoles

@johan I tried your solution with no luck, may be I’m missing something

I convert the received TTN payload_raw to cayenne LPP payload_raw => OK
image

Then I send the full msg object (converted to JSON) as new msg.payload (so http request node has all original message in it’s msg.payload)

The recode function (after decoding incoming payload) is

var buf = Buffer.alloc(4+4+11);

// Channel 1, Analog In (Battery)
buf.writeUInt16BE(0x0102,  0);
buf.writeUInt16BE(batt*100, 2);

// Channel 2, Temp sensor
buf.writeInt16BE(0x0267, 4);
buf.writeInt16BE(temp*10, 6);

// Channel 3, GPS
buf.writeUInt16BE(0x0388,  8);
buf.writeIntBE(lat*10000, 10,3);
buf.writeIntBE(lon*10000, 13,3);
buf.writeIntBE(  0*10000, 16,3);

// Save new payload format to current msg payload_raw
msg.payload_raw = buf.toString('hex');

var retmsg = {};
// Get full current message as JSON
retmsg.payload = JSON.stringify(msg);

return retmsg;

Looks fine, output of recode (going to http request to url you mentioned) looks like OK
image

and output of http request (cayenne) says ok with HTTP 200
image

But I have noting on cayenne dashboard, and my node is created of course
image

I tried just sending payload_raw and hardware_serial but in this case, cayenne return 500 error so my 1st output seems good.

We’re so close, that’s it’s a pity it does not work. Do you know what field cayenne needs? Did I missed something?

Thanks for your help

1 Like

I’ve not got time atm - but I would try to emulate the HTTP using postman first (just quicker to test really) This could help?

At first sight the payload_raw is hex and it should be base64

I got your code to work as an “integration” between a TTN node device to Cayenne - well done.
In my case, I started with a TTN node device running the “Cayenne_lpp” sketch with the integration “Cayenne” setup and running.

The first thing I did, was remove the “Cayenne” integration in the TTN console.
Next I set up a node-red as you have shown.
The only thing I did differently. was change a few things in your “Recode” function
i.e. assign the correct channels to data and use this line to encode to base64
msg.payload_raw = buf.toString('Base64');
After this, I could see the values change in Cayenne
N.B. Even if the data code isn’t mapped correctly to channels, I still saw that the RSSI and SNR change in Cayenne - so that proves that it is receiving data


*edit - I also got postman to work using as follows: (the trick is not to include Payload)

POST           https://lora.mydevices.com/v1/networks/ttn/uplink
Authorization  No Auth       
Headers        Content-Type          application/json
Body           the json after payload e.g.   {"app_id":"xxx.....  ...."temperature_5":23.1},"_msgid":"cc993c47.19999"}

For any node, this is the JSON, you need to emulate - no secret needed and you have to publicise the DEVEUI. You also “Base 64” the payload_raw from the byte array (that was set up as per the “Cayenne lpp” format)
N.B. It doesn’t seem to matter what the “metadata” values are, as long as they are there

{"app_id":"YOUR_APPLICATION_ID","dev_id":"YOUR_DEV_ID","hardware_serial":"YOUR_8_BYTE_DEVEUI","payload_raw":"BAIAKAAAAAAAAAAAAAAAAAAAAA==","metadata":{"gateways":[{"gtw_id":"TEST_1","rssi":-73,"snr":6.75}]}}


*edit - added curl command (windows)

curl -v -X POST --data "{\"app_id\":\"YOUR_APP_ID\",\"dev_id\":\"YOUR_DEV_ID\",\"hardware_serial\":\"1122334455667788\",\"payload_raw\":\"BAIAKAAAAAAAAAAAAAAAAAAAAA==\",\"metadata\":{\"gateways\":[{\"gtw_id\":\"TEST_1\",\"rssi\":-73,\"snr\":6.75}]}}" https://lora.mydevices.com/v1/networks/ttn/uplink -H "Content-Type: application/json"

That was the trick, noticed some strange things .

  • If I’m not on the device tree selected widged (in my case ls18002) on cayenne browser, then nothing appears on dashboard, looks like 1st time device integration, you need to be on the device on cayenne dashboard
  • as said @CurlyWurly the 1st widget that came, was only RSSI and SNR
  • i need to check why gps coord are wrong
  • after that all went fine, great

I used node base64 and json but @CurlyWurly just showed that base64 can be native I’ll integrate this in Recode function. I’m doing some check and post new Recode source code
image

@johan you saved my day, @CurlyWurly thanks also, I just saw your post after got it working :wink:

1 Like

@johan,
I’m still having problem with GPS, I looked around everywhere (in CayenneLPP.ccp and and Cayenne doc) and CayenneLPP.cpp source code is coded exactly how the documentation is stating
image

So I’ve done the same with my GPS position says (node red logs)

ls18014 lat=5.760811666666667 lon=45.60061666666667

So I multiply per 10000 and then construct LPP format channel 08 GPS 0x88 and display LPP payload buffer (as hex so see) in node console

fixed lat=57608.11666666667 lon=456006.1666666667 Payload => 098800e10806f546000000

and final payload send (with other sensors)

Final payload 0102018a02670036098800e10806f546000000

Doing reverse calculation of hex show me

lat = 0x00e108 = 57608  so / 10000 = 5.7608
lon = 0x06f546 = 456006 so / 10000 = 45.6006

So all looks good, any idea why cayenne show my position in africa (and not the same as lat=lon=0 so not the pb) ?
image

I’ve noticed that the map in Cayenne can be confusing - If someone can’t see the GPS point, these are the things I found I have to check

  • Click the back arrow on the date field, and then click to go forward to the present day
  • If you don’t see your GPS point, zoom out until you can
  • Check if the LAT and LONG are correct (not swapped and have the correct sign!) e.g. lat 5.7608, long 45.6006 points to Africa and not France :slight_smile:
  • Be aware that the map wants to display from a mid point of your GPS plots and the GPS location of the Gateway (so it seems).

This code works for me in the node-red function (points to Chester in UK North-West).

var buf = Buffer.alloc(4+11);
var batt = 2.4;
var lat = 53.1934;
var lon = -2.8931;
var alt = 145;
var i   = 0; 
var h;

// Channel 4, Analog In (Battery)
    buf[i++] = 0x04;
    buf[i++] = 0x02;
    buf[i++] = (batt*100 >> 8) & 0xff;
    buf[i++] = (batt*100) & 0xff;

// Channel 8, GPS
    buf[i++] = 0x08;
    buf[i++] = 0x88;

//Lat
    h = lat * 10000;
    buf[i++] = (h >> 16) & 0xff;
    buf[i++] = (h >> 8) & 0xff;
    buf[i++] = h & 0xff;

//Lon
    h = lon * 10000;
    buf[i++] = (h >> 16) & 0xff;
    buf[i++] = (h >> 8) & 0xff;
    buf[i++] = h & 0xff;

//alt
    buf[i++]= (alt >> 16) & 0xff;
    buf[i++] = (alt >> 8) & 0xff;
    buf[i++] = alt & 0xff;

@CurlyWurly what stupid am I, of course I reversed lat and lon, btw I also checked signed value and I was decoding it with readUInt32BE instead of readInt32BE

So now the final working code is here, I also placed my decoding routine as example

flow
image

http request config, just set to POST and URL
image

recode node

// Return message
var retmsg = {};

// Get raw payload buffer (binary)
var b = msg.payload_raw; 

// Decode an uplink message from a buffer
lon = b.readInt32BE(5)/600000;
lat = b.readInt32BE(9)/600000;
accel = { x:((b.readInt8(16)<<8)*2)/32768,
          y:((b.readInt8(17)<<8)*2)/32768, 
          z:((b.readInt8(18)<<8)*2)/32768 };
batt = (b.readUInt16LE(27)* 0.006406) ;
temp = (b[25]<<8 | b[26]) & 0x0FFF ;
if (temp > 0x7FF ) { 
    temp = -(temp&0x7FF) ;  
}
temp = temp*0.0625;

// Size here is LPP, depends on what you send
var buf = Buffer.alloc(4+4+8);
// Channel 1, Analog In (Battery)
buf.writeUInt16BE(0x0102,  0);
buf.writeUInt16BE(batt*100, 2);
// Channel 2, Temp sensor
buf.writeInt16BE(0x0267, 4);
buf.writeInt16BE(temp*10, 6);
// Channel 3, Accel
buf.writeInt16BE(0x0371, 8);
buf.writeInt16BE(accel.x*1000, 10);
buf.writeInt16BE(accel.y*1000, 12);
buf.writeInt16BE(accel.z*1000, 14);

// Channel 9, GPS (if fixed)
if (lat!==0 || lon!==0) {
    lon *= 10000;
    lat *= 10000;
    var bufgps = Buffer.alloc(11);
    bufgps.writeUInt16BE(0x0988, 0);
    bufgps.writeIntBE(lat, 2, 3);
    bufgps.writeIntBE(lon, 5, 3);
    bufgps.writeIntBE( 0 , 8, 3);
    buf = Buffer.concat([buf, bufgps]);
}

// Save converted payload format to current msg payload_raw in base64
msg.payload_raw = buf.toString('Base64');
delete msg._msgid; // Not needed

// Get FULL current message (not only payload) as JSON, 
retmsg.payload = JSON.stringify(msg);

return retmsg;

Big thanks to @johan and @CurlyWurly, now we can reencode for cayenneLPP even only for specific devices in the same app, excellent :wink:

4 Likes

That’s not solved for me as you need to setup a whole node RED instance to perform that, and indeed you need to have some hardware running for that…

Re “How to convert custom payload in TTN console”, Johan answered your original question before
"In V2 we don’t allow altering payload_raw. It will, technically, be supported in V3 but not sure if it will be GA in the public Consoles"

If you have a need to know about cloud solution options, then that is another thread title really :wink:

I changed the topic title, correct it’s not done in the console but my first need are done, it works, not the easiest way I agree, but it works. You do not need node red instance, you can do it with any SDK Api and using nodejs should be easy since majority of the code is already written, but yes you need a computer/server for that, they are plenty of autonomous linux box such as PI 3/Zero that can do the job fine for some bucks or using a corporate existing server is also an option.

Oh, just thought I was the thread initiator and it’s you, sorry about that, you can revert back topic title.

I reverted the change in the title (again), as the reference to TTN Console and the answers of Johan are quite useful for future reference, I feel. (Of course, all the workarounds are great too.)

1 Like

Thanks @arjanvanb