What UDP packets are needed to register a gateway 'live' on TTN?


(Ivovanling) #1

Hello all,

Can somebody help us with the following? We are trying to connect a LoRa gateway to TTN. The gateway itself is an RFI Engineering L-gateway (http://www.rfi-engineering.com/index.php/iot-and-localisation/) running Semtech's packet forwarder. When the packet forwarder on the gateway is pointed straight to TTN I can see on the TTN console that the gateway comes 'live' and also that LoRa packets are being received. So the gateway itself is able to communicate with the TTN back office without problems.

However; the network setup we have is such that we have to 'loop' the packet forwarder's UDP packets through a centralised 'solver' first. The 'solver' calculates GPS coordinates from the timestamps in the LoRa 'rxpk' packets based on TDoA. This network setup is needed because we need the raw timestamps, coming from all gateways receiving the LoRa sensor, in order for TDoA localisation to work.

After reception of the UDP packets coming for the gateways, the 'solver' re-assembles the UDP packet again and forwards this to TTN. In principle this should look to TTN like the UDP packets came from the gateway. However, now the TTN back office does not register the gateways coming 'live'. So there could be a difference between the UDP packets coming from the gateway and the re-assembled UDP packets coming from our 'solver'. Or we are missing a 'handshake' between the gateway and the TTN back office. But at this point in time we don't know...

As an example, this is the UDP packet straight from the L-gateway:

12:15:46.058641 IP (tos 0x0, ttl 54, id 13074, offset 0, flags [DF], proto UDP (17), length 179)
82-148-204-238.bbserv.nl.58059 > ip230.ip-91-134-50.eu.1700: [udp sum ok] UDP, length 151
0x0000: 4500 00b3 3312 4000 3611 6339 5294 ccee E...3.@.6.c9R...
0x0010: 5b86 32e6 e2cb 06a4 009f 06d6 01b4 ca00 [.2.............
0x0020: a522 17e3 d419 0828 7b22 7374 6174 223a .".....({"stat":
0x0030: 7b22 7469 6d65 223a 2232 3031 372d 3035 {"time":"2017-05
0x0040: 2d31 3620 3130 3a31 363a 3032 2047 4d54 -16.10:16:02.GMT
0x0050: 222c 226c 6174 6922 3a35 322e 3338 3335 ","lati":52.3835
0x0060: 362c 226c 6f6e 6722 3a35 2e32 3133 3031 6,"long":5.21301
0x0070: 2c22 616c 7469 223a 362c 2272 786e 6222 ,"alti":6,"rxnb"
0x0080: 3a35 2c22 7278 6f6b 223a 342c 2272 7866 :5,"rxok":4,"rxf
0x0090: 7722 3a34 2c22 6163 6b72 223a 302e 302c w":4,"ackr":0.0,
0x00a0: 2264 776e 6222 3a30 2c22 7478 6e62 223a "dwnb":0,"txnb":
0x00b0: 307d 7d 0}}

And this is the re-assembled UDP packet coming from the 'solver':
12:15:46.063101 IP (tos 0x0, ttl 64, id 8423, offset 0, flags [DF], proto UDP (17), length 222)
ns3013446.ip-94-23-54.eu.43927 > 52.169.76.203.1700: [udp sum ok] UDP, length 194
0x0000: 4500 00de 20e7 4000 4011 02bb 5e17 36e2 E.....@.@...^.6.
0x0010: 34a9 4ccb ab97 06a4 00ca d0ca 0103 d100 4.L.............
0x0020: a522 17e3 d419 0828 7b22 7374 6174 2220 .".....({"stat".
0x0030: 3a20 7b22 7278 6677 2220 3a20 342e 302c :.{"rxfw".:.4.0,
0x0040: 2022 6c6f 6e67 2220 3a20 352e 3231 3330 ."long".:.5.2130
0x0050: 312c 2022 6c61 7469 2220 3a20 3532 2e33 1,."lati".:.52.3
0x0060: 3833 3536 2c20 2274 786e 6222 203a 2030 8356,."txnb".:.0
0x0070: 2e30 2c20 2264 776e 6222 203a 2030 2e30 .0,."dwnb".:.0.0
0x0080: 2c20 2261 6c74 6922 203a 2036 2e30 2c20 ,."alti".:.6.0,.
0x0090: 2272 786f 6b22 203a 2034 2e30 2c20 2274 "rxok".:.4.0,."t
0x00a0: 696d 6522 203a 2022 3230 3137 2d30 352d ime".:."2017-05-
0x00b0: 3136 2031 303a 3136 3a30 3220 474d 5422 16.10:16:02.GMT"
0x00c0:

We are forwarding 'stat' packets and 'rxpk' packets from the gateways.

Can somebody comment on this? Do we need to send additional packets besides 'stat' and 'rxpk' in order for the TTN back office to 'see' the gateway? Is there a certain order in which the JSON elements are handled? We expect there is not, but in the original 'stat' packet the first element is 'time' and in the re-assembled 'stat' packet the order of the JSON elements is different. Here the first element is 'rxfw'. Also the 'solver' inserted a space between the elements. See above.

Any help is appreciated, thanks.

Ivo van Ling


#2

A normal gateway will also poll the back-end for data to send. You do not mention that data stream in your message, is your gateway polling TTN?

Have you considered using poly/mp forwarder on the gateway to send the data to two back-ends? Once to TTN as well as to your own back-end? If you choose to do this, make sure you are NOT accepting data to be transmitted from your own back-end, only one back-end can be in control of data transmissions at a time. (MP forwarder allows you to disable the downstream for a server)


(Ivovanling) #3

Thanks for your comment. Yes we considered running the poly/mp forwarder. However, we run an adapted packet forwarder on the gateway in order to deliver accurate timestamps for TDoA. So we would have to adapt the poly/mp forwarder to do that too. Not saying it is impossible, but it was not the route we took.

The 'solver' is not polling TTN at this time.


#4

If you don't mind sharing the code I could include it in the mp forwarder. (If you can't share the code and want it included in a custom release anyway contact me by PM)


(Thomas Telkamp) #5

Hi Ivo,

Maybe you can setup Orne Brocaar's bridge, and forward the UDP to that:

https://github.com/brocaar/lora-gateway-bridge

It will show you the error (maybe JSON formatting?). TTN is using an adapted version of that bridge.


(Ivovanling) #6

So we have been doing quite some experimenting sending a 'stat' package to the TTN backend.

It looks like the TTN server expects the JSON to be in a particular order for it to understand the message. To test this I constructed a very simple 'stat' message to 'reset' the 'Last seen' counter on the gateway page. I used this:

echo -e '\x01\x67\xc6\x00\xb8\x27\xeb\xff\xff\x1f\x54\xa8{"stat":{"time":"2017-06-08 09:40:42 GMT","lati":52.34223,"long":5.29685,"alti":66,"rxnb":0,"rxok":0,"rxfw":0,"ackr":0.0,"dwnb":0,"txnb":0}}' > /dev/udp/52.169.76.203/1700

Which constructs a UDP message with the right header and a 'stat' packet in JSON. This resets the 'Last seen' counter on the TTN server for my gateway.

However, when I change the order of the JSON elements it fails to work. For example:

echo -e '\x01\x67\xc7\x00\xb8\x27\xeb\xff\xff\x1f\x54\xa8{"stat":{"rxfw":0,"lati":52.34223,"long":5.29685,”txnb":0,"dwnb":0,"alti":66,"rxok":0,"time":"2017-06-08 09:40:42 GMT","rxnb":0,"ackr":0.0}}' > /dev/udp/52.169.76.203/1700

Is in principle the same message, with a different random token. This fails to reset the 'Last seen' counter for my gateway.

Can anybody from TTN confirm or deny this?

In the meantime we are changing the order of the JSON elements we send to TTN to a message that we hope works.


(Lachmeijer) #7

I have setup Orne Brocaar's bridge for debugging this and I tested with some different devices that work fine in a normal setup (Lorank8 + TTN config).
Using the RFI gateway and the bridge with all standard settings I see the complete packets arriving in Orne Brocaar's Lora application server as well.


(Eric Gourlaouen) #8

Hi,

From what comes out of your example JSONs, it seems as if the beginning quote is not exact for the txnb property. Before txnb, you're not using a standard Unicode quotation mark (", U+0022) but a right double quotation mark (”, unicode U+201D). It is possible that our JSON parsers can't parse this special character ; online JSON formatters (such as JSONLint) don't seem to be able to parse them.
I don't know if that's the core of your issue here, but for this particular packet, that should be the reason why it's not parsed on our bridge. The order of the JSON properties shouldn't be an issue.


(Ivovanling) #9

@egourlao

You are right, there is a (”, unicode U+201D) character before the txnb property. Well spotted...

I can confirm that replacing this with the standard Unicode quotation mark actually makes the second example also work now.