HTTP Integration: pass only payload_fields

Hello,

I’ve been trying to use the “HTTP integration,” but cannot seem to find a way to only pass the payload_fields to another website (thingspeak is the goal, posting to https://api.thingspeak.com/update.json).
Of course I can put a “proxy” in between to filter out the payload_fields and pass them trough, but that feels like overkill and requires me to have a server running 24x7 for just that purpose.
Am I overlooking something or is it simple not possible right now?

Best regards,
Dirk-Jan

I’m quite sure it simply cannot do that. The documentation is quite specific about the payload that is posted to the URL:

The integration will post data in the following format: […]

See also:

Thanks for the response. I had read it in the documentation. The documentations home apparently is https://github.com/TheThingsNetwork/docs, but that repo does not hold the actual underlying code.
Any idea which repository holds the integration code for a feature request?

I made a small esp8266 program, fixing this annoyance. What it does is to catch the JSON from the HTTP integration of The Things Network, filter out the payload_fields and pass that through to Thingspeak.

Yes, I ended up with running a “server”, but as it is powerd from usb, it is not too bad. The code can be found on https://github.com/steviethecat/ttn-tp-proxy-wemos

If you don’t mind passing your data through on online service, you can use https://putsreq.com

And write 3 lines of code:

var parsedBody = JSON.parse(request.body);
request.body = parsedBody.payload_fields;
request.forwardTo = 'YOURURI';

For investigating responses coming from an HTTP integration (or the putsreq forward), use: https://rbaskets.in/

4 Likes

Using a third-party server just to translate one JSON message from ttn to a thingspeak http request seems such overhead for such a simple task!

If users are allowed to use javascript payload functions to create their own JSON messages to send to any specified URL, why not allow creating thingspeak-compatible messages?

Or do we need to build a seperate integration service for that?
Putsreq.com seems down now so I cannot test that.
I used a simple php script on a hosting server to bridge ttn to thingspeak but it would be cleaner to skip the middle man. Can a payload function do http calls? Then that would be a (dirty) solution.

1 Like

What are you referring to here?

No.

It would be nice to have a tool to compose our own JSON format based on current fields you can use, I think that would ease integration with 3rd parties. An example would be SigFox backend callbacks where you can compose the messages to be sent based on variables (rssi,geoloc,…)

Sorry, I mixed up two things. The payload functions are part of the ttn console and a nice way to construct JSON objects. The URL config I mentioned is part of the HTTP integration tool and it would be nice if there is a way to change the output format of the HTTP integration tool so that it can post directly to ThingsSpeak. But maybe that feature should be part of another Integration tool (a Thingspeak Integration tool maybe).

This would be a most useful feature for my company’s project. I would like to request that an optional field is added to the payload decoder’s returned object that instructs the HTTP integration to use only the payload decoder’s returned object as the content of the HTTP POST submitted to the specified endpoint.

For example, the decoder may return:

  {
    "payload_override" : true,
    "signatures": [     "SIG_K1_K7eWqZmnneUHhnhcR9r6LwNnXYSEqpv6HcE124nNfSL5eZADyTDGfXTprxH79cMgh74GAX5Fo9mYiiJQxCMp2cN8hSept7"
    ],
    "compression": "none",
    "packed_context_free_data": "",
    "packed_trx": "822f5e5d94f08b818c41000000000100a6823403ea3055000000572d3ccdcd0100000857219de8ad00000000a8ed32322400000857219de8ad00001057219de8ad102700000000000004454f53000000000374783100"
  },

This exact JSON message is put into the payload minus the “payload_override” field.

My company is building a decentralized and distributed IoT data platform that uses IPFS as the datalake and blockchain as the access control system. LoRaWAN and TTN is one of the data ingestion methods we’re adding support for. Our customers, and your users, would be interested, we think, in directly accessing the blockchain network avoiding middlemen.

In the meantime, we’ll develop connectors using Azure and AWS as needed.

In case anyone is wondering, no, the decoder is not generating the signatures, this is done on the sensors and is a key feature of our platform. The decoder would need only to do a base58 encoding of a portion of the payload sent by the devices and to build the JSON message in the required format.

Thanks for your consideration!

@arjanvanb

How would your endpoint recognize retries for confirmed uplinks, without a frame counter and without the is_retry attribute that is otherwise added for that?

Aside, I’m not part of the core team, so I cannot help here. Also, I would not expect any changes in the current V2 that’s running the community network. For V3, see https://github.com/TheThingsNetwork/lorawan-stack

Instead of implementing this, maybe I’d rather have some optional template mechanism (allowing for simple logic, think Handlebars, Mustache) and also allow for tokens in the URL, in the HTTP Integration instead. This would have access to much more data, such as the DevEUI, metadata and gateway data, and even user-defined device attributes.

But I can understand that the team might feel there’s no end to that: next person will ask for some form-urlencoded, throttling, retries, multiple endpoints, double-sided SSL, SOAP headers, … :wink:

The encoded signature is a whopping 101 characters, 94 without the prefix. So if that is a Base58 representation of binary data, then the source for that alone is about 69 bytes? And how are you creating packed_trx? In other words: it seems that the payloads are quite long…?

How would your endpoint recognize retries for confirmed uplinks, without a frame counter and without the is_retry attribute that is otherwise added for that?

We are leveraging our platform’s (blockchain) built-in de-dup capability, meaning a transaction (message) from the sensor gets applied just once.

I can see where the maintainers would be reluctant to make changes at this stage.

The encoded signature is a whopping 101 characters, 94 without the prefix. So if that is a Base58 representation of binary data, then the source for that alone is about 69 bytes? And how are you creating packed_trx ? In other words: it seems that the payloads are quite long…?

Yes the sensor payloads are indeed relatively large, but still capable of being handled by LoRaWAN. The sensors deliver a payload consisting of a binary representation of the transaction data (incl. sensor readings) and EC signature, on the order of 100 bytes. In practical terms we’re looking at SF7 or SF8 at best and fairly infrequent uplinks to work within airtime restrictions.

Beware that this exceeds the max length of some providers, and of some spreading factors.