RAK2245 gateway with Balena "[main] failed to start the concentrator"

The reason is the RAK2245 has level shifters on the SPI port to allow it to work with a RasPi at 3.3 volts or on a 96Board system at 1.8 volts. The Level Shifters limit the maximum clock frequency of the SPI port to 2MHz…

2 Likes

Unfortunately my success was short lived. After running for a few hours the gateway stops working.

Sometimes GPS errors:

main  03:51:07  INFO: [TTN] server "thethings.meshed.com.au" connected
 main  03:51:07  INFO: [main] Starting the concentrator
 main  03:51:10  INFO: [main] concentrator started, radio packets can now be received.
 main  03:51:10  INFO: [up] Thread activated for all servers.
 main  03:51:10  INFO: JIT thread activated.
 main  03:51:10  INFO: Disabling GPS mode for concentrator's counter...
 main  03:51:10  INFO: Validation thread activated.
 main  03:51:10  WARNING: [gps] GPS out of sync, keeping previous time reference
 main  03:51:10  WARNING: [gps] GPS out of sync, keeping previous time reference
 main  03:51:10  INFO: host/sx1301 time offset=(1565149868s:91504µs) - drift=-639080336µs
 main  03:51:10  INFO: Enabling GPS mode for concentrator's counter.

Sometimes calibration firmware errors:

 main  03:47:56  INFO: [main] Starting the concentrator
 main  ERROR: Version of calibration firmware not expected, actual:0 expected:2
 main  03:47:56  ERROR: [main] failed to start the concentrator

Sometimes:

ERROR: Failed to load fw 0 

Receiving RF packets, but more than I would expect, and often with CRC_FAIL

 main  # RF packets received by concentrator: 72
 main  # CRC_OK: 100.00%, CRC_FAIL: 0.00%, NO_CRC: 0.00%

 main  # RF packets received by concentrator: 83
 main  # CRC_OK: 0.00%, CRC_FAIL: 100.00%, NO_CRC: 0.00%

If I restart the service, same problem, but rebooting the whole device brings it back for a few hours.

Correction, rebooting the device sometimes brings it back. It did this morning, but not now. I’ve turned it off for a while, heatsink is too hot to touch.

Booted up again after it had cooled. Still unable to connect.

WARNING: [gps] could not get GPS time from GPS

Have switched back to my RakWireless raspbian image which has come online fine.

Make sure you are really driving the reset GPIO - actually monitor it with a scope or something. And make sure it isn’t being left floating.

Some Linuxes have an odd habit where if you try to re-export an already exported GPIO, it ends up getting unexported again, and then you have no reset pin…

Also consider lowering the SPI speed even further.

No real idea on the GPS, do you actually need it? If not configuring to ignore it could simplify the number of things which can go wrong.

heatsink is too hot to touch.

Consider adding a fan. I’m unimpressed by the cooling design of many of these products which don’t really seem to have a plan for how the heat will exit the overall enclosure.

Two things:

  1. Make sure that the lora_gateway library IS working at 2MHz.
  2. Clean-up the board.

Our Gateway was having exactly the same issue. Doing a close inspection we found flux residues on the Raspberry-Pi HAT pins. There was enough residues to generate issues.

After a very good cleanup with contact cleaner it worked perfectly. :slight_smile:

Personally we think that design sacrifices were made on the RAK2245 HAT. I mean, there are high-speed voltage level converters, and the traces for SPI are really long and thin the board almost asks for interference and/or crosstalk. Again for us the problem is not the RAK2245 module but the HAT adapter for Raspberry Pi.

So far so good for 1Mhz and GPS disabled. 12 hours uptime so far. Heatsink is quite warm, but not as hot as before.
Will see if it keeps running overnight.

What are other suggestions for better quality hardware that will do the same sort of job?

So far so good for 1Mhz and GPS disabled. 12 hours uptime so far. Heatsink is quite warm, but not as hot as before.

SPI clock speed would change reliability (given the badly chosen level shifter), not really thermal issues. Only the tiniest fraction of the logic is clocked at the SPI clock. The actual clock speed of the digital radio remains unchanged - that’s still going to turn quite a lot of power into heat any time it is running.

Will see if it keeps running overnight.

While failure is not a good thing, things should be setup to restart on failure, so typically you’d lose at most a couple of packets as it does so, which is nothing in terms of what routinely happens with LoRa on the air at moderate ranges. And that should really only be a single loss for a given node.

What are other suggestions for better quality hardware that will do the same sort of job?

RAK831 and RAK833 do not have the mistaken level translator. 833 is smaller, 831 is easier to connect to.

Just as a note: I have tried the RAK Pilot Pro - the same as yours but with LTE - using the resin/balena setup. After a long struggle to get LTE to work I gave up and also ran RAK’s firmware so that I can reverse engineer it. Again after some struggle it became clear Balena won’t support this custom LTE hat very well. So I never got around to properly test the LoRa part. But it is still something I want to do so that I have remote management of this gateway.

Yeah remote management is what I’m trying to achieve as well.

Unfortunately it’s stopped again. Different issue this time though. Gateway shows up fine and current in TTN console, but the devices haven’t connected for 2 hours.
Starting to get some CRC_FAIL’s in the logs:

2.08.19 22:46:05 (+1200) main ##### 2019-08-12 10:46:05 GMT #####
12.08.19 22:46:05 (+1200) main ### [UPSTREAM] ###
12.08.19 22:46:05 (+1200) main # RF packets received by concentrator: 2
12.08.19 22:46:05 (+1200) main # CRC_OK: 50.00%, CRC_FAIL: 50.00%, NO_CRC: 0.00%
12.08.19 22:46:05 (+1200) main # RF packets forwarded: 1 (23 bytes)
12.08.19 22:46:05 (+1200) main # PUSH_DATA datagrams sent: 0 (0 bytes)
12.08.19 22:46:05 (+1200) main # PUSH_DATA acknowledged: 0.00%
12.08.19 22:46:05 (+1200) main ### [DOWNSTREAM] ###
12.08.19 22:46:05 (+1200) main # PULL_DATA sent: 0 (0.00% acknowledged)
12.08.19 22:46:05 (+1200) main # PULL_RESP(onse) datagrams received: 0 (0 bytes)
12.08.19 22:46:05 (+1200) main # RF packets sent to concentrator: 0 (0 bytes)
12.08.19 22:46:05 (+1200) main # TX errors: 0
12.08.19 22:46:05 (+1200) main ### BEACON IS DISABLED!
12.08.19 22:46:05 (+1200) main ### [JIT] ###
12.08.19 22:46:05 (+1200) main # INFO: JIT queue contains 0 packets.
12.08.19 22:46:05 (+1200) main # INFO: JIT queue contains 0 beacons.
12.08.19 22:46:05 (+1200) main ### [GPS] ###
12.08.19 22:46:05 (+1200) main # No time keeping possible due to fake gps.
12.08.19 22:46:05 (+1200) main # Manual GPS coordinates: latitude -45.91360, longitude 170.46727, altitude 0 m
12.08.19 22:46:05 (+1200) main ### [PERFORMANCE] ###
12.08.19 22:46:05 (+1200) main # Upstream radio packet quality: 50.00%.
12.08.19 22:46:05 (+1200) main ### [ CONNECTIONS ] ###
12.08.19 22:46:05 (+1200) main # thethings.meshed.com.au: Connected
12.08.19 22:46:05 (+1200) main # Semtech status report send.
12.08.19 22:46:05 (+1200) main ##### END #####
12.08.19 22:46:05 (+1200) main 10:46:05 INFO: [TTN] thethings.meshed.com.au RTT 48
12.08.19 22:46:05 (+1200) main 10:46:05 INFO: [TTN] send status success for thethings.meshed.com.au

I’m seeing data showing up under the Gateway Traffic from my 2 clients, but it’s not getting through to the device console traffic.
Is that airtime correct?

There has been an outage which resulted in data not being delivered to applications. May-be that is what you are seeing?

I’m seeing data showing up under the Gateway Traffic from my 2 clients, but it’s not getting through to the device console traffic.

Raw packets failing to become application packets typically have nothing to do with the gateway, rather they are either a node sending a packet with invalid contents (wrong keys, replayed fcount, etc) or infrastructure problems

No obvious reason to suspect it is not. Besides this is calculated server-side and has no real role in the success of the process, since RX windows are relative to the end of the uplink packet which is what the LoRa chip in the gateway actually reports. The airtime calcs are really only a view of how active a node is.

I think you might be right. With the original RAK software it’s also not working, both node devices have cut out at exactly the same time (19 hours ago, around 2100 hours NZ time)

Any idea when it’ll be back up or where I can find more info? https://status.thethings.network/ doesn’t show anything. Slack shows some people back online.
I’ve switched back from RAK to Balena install, and Gateway is showing up fine in TTN console, but both devices are still not getting anything through.

I seem to be getting huge numbers of packets hitting the gateway now. But no real data.

13.08.19 16:46:00 (+1200)  main  ##### 2019-08-13 04:46:00 GMT #####
13.08.19 16:46:00 (+1200)  main  ### [UPSTREAM] ###
13.08.19 16:46:00 (+1200)  main  # RF packets received by concentrator: 0
13.08.19 16:46:00 (+1200)  main  # CRC_OK: 0.00%, CRC_FAIL: 0.00%, NO_CRC: 0.00%
13.08.19 16:46:00 (+1200)  main  # RF packets forwarded: 0 (0 bytes)
13.08.19 16:46:00 (+1200)  main  # PUSH_DATA datagrams sent: 0 (0 bytes)
13.08.19 16:46:00 (+1200)  main  # PUSH_DATA acknowledged: 0.00%
13.08.19 16:46:00 (+1200)  main  ### [DOWNSTREAM] ###
13.08.19 16:46:00 (+1200)  main  # PULL_DATA sent: 0 (0.00% acknowledged)
13.08.19 16:46:00 (+1200)  main  # PULL_RESP(onse) datagrams received: 0 (0 bytes)
13.08.19 16:46:00 (+1200)  main  # RF packets sent to concentrator: 0 (0 bytes)
13.08.19 16:46:00 (+1200)  main  # TX errors: 0
13.08.19 16:46:00 (+1200)  main  ### BEACON IS DISABLED! 
13.08.19 16:46:00 (+1200)  main  ### [JIT] ###
13.08.19 16:46:00 (+1200)  main  # INFO: JIT queue contains 0 packets.
13.08.19 16:46:00 (+1200)  main  # INFO: JIT queue contains 0 beacons.
13.08.19 16:46:00 (+1200)  main  ### [GPS] ###
13.08.19 16:46:00 (+1200)  main  # No time keeping possible due to fake gps.
13.08.19 16:46:00 (+1200)  main  # Manual GPS coordinates: latitude -45.91360, longitude 170.46727, altitude 0 m
13.08.19 16:46:00 (+1200)  main  ### [PERFORMANCE] ###
13.08.19 16:46:00 (+1200)  main  # Upstream radio packet quality: 0.00%.
13.08.19 16:46:00 (+1200)  main  ### [ CONNECTIONS ] ###
13.08.19 16:46:00 (+1200)  main  # thethings.meshed.com.au: Connected
13.08.19 16:46:00 (+1200)  main  # Semtech status report send. 
13.08.19 16:46:00 (+1200)  main  ##### END #####
13.08.19 16:46:00 (+1200)  main  04:46:00  INFO: [TTN] thethings.meshed.com.au RTT 49
13.08.19 16:46:00 (+1200)  main  04:46:00  INFO: [TTN] send status success for thethings.meshed.com.au
13.08.19 16:46:30 (+1200)  main  
13.08.19 16:46:30 (+1200)  main  ##### 2019-08-13 04:46:30 GMT #####
13.08.19 16:46:30 (+1200)  main  ### [UPSTREAM] ###
13.08.19 16:46:30 (+1200)  main  # RF packets received by concentrator: 0
13.08.19 16:46:30 (+1200)  main  # CRC_OK: 0.00%, CRC_FAIL: 0.00%, NO_CRC: 0.00%
13.08.19 16:46:30 (+1200)  main  # RF packets forwarded: 0 (0 bytes)
13.08.19 16:46:30 (+1200)  main  # PUSH_DATA datagrams sent: 0 (0 bytes)
13.08.19 16:46:30 (+1200)  main  # PUSH_DATA acknowledged: 0.00%
13.08.19 16:46:30 (+1200)  main  ### [DOWNSTREAM] ###
13.08.19 16:46:30 (+1200)  main  # PULL_DATA sent: 0 (0.00% acknowledged)
13.08.19 16:46:30 (+1200)  main  # PULL_RESP(onse) datagrams received: 0 (0 bytes)
13.08.19 16:46:30 (+1200)  main  # RF packets sent to concentrator: 0 (0 bytes)
13.08.19 16:46:30 (+1200)  main  # TX errors: 0
13.08.19 16:46:30 (+1200)  main  ### BEACON IS DISABLED! 
13.08.19 16:46:30 (+1200)  main  ### [JIT] ###
13.08.19 16:46:30 (+1200)  main  # INFO: JIT queue contains 0 packets.
13.08.19 16:46:30 (+1200)  main  # INFO: JIT queue contains 0 beacons.
13.08.19 16:46:30 (+1200)  main  ### [GPS] ###
13.08.19 16:46:30 (+1200)  main  # No time keeping possible due to fake gps.
13.08.19 16:46:30 (+1200)  main  # Manual GPS coordinates: latitude -45.91360, longitude 170.46727, altitude 0 m
13.08.19 16:46:30 (+1200)  main  ### [PERFORMANCE] ###
13.08.19 16:46:30 (+1200)  main  # Upstream radio packet quality: 0.00%.
13.08.19 16:46:30 (+1200)  main  ### [ CONNECTIONS ] ###
13.08.19 16:46:30 (+1200)  main  # thethings.meshed.com.au: Connected
13.08.19 16:46:30 (+1200)  main  # Semtech status report send. 
13.08.19 16:46:30 (+1200)  main  ##### END #####
13.08.19 16:46:30 (+1200)  main  04:46:30  INFO: [TTN] thethings.meshed.com.au RTT 50
13.08.19 16:46:30 (+1200)  main  04:46:30  INFO: [TTN] send status success for thethings.meshed.com.au
13.08.19 16:46:52 (+1200)  main  04:46:52  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:52 (+1200)  main  04:46:52  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:52 (+1200)  main  04:46:52  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:52 (+1200)  main  04:46:52  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:52 (+1200)  main  04:46:52  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:52 (+1200)  main  04:46:52  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:52 (+1200)  main  04:46:52  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:52 (+1200)  main  04:46:52  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:52 (+1200)  main  04:46:52  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:53 (+1200)  main  04:46:53  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:53 (+1200)  main  04:46:53  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:53 (+1200)  main  04:46:53  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:53 (+1200)  main  04:46:53  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:53 (+1200)  main  04:46:53  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:53 (+1200)  main  04:46:53  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:53 (+1200)  main  04:46:53  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:53 (+1200)  main  04:46:53  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:53 (+1200)  main  04:46:53  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:53 (+1200)  main  04:46:53  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:53 (+1200)  main  04:46:53  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:53 (+1200)  main  04:46:53  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:53 (+1200)  main  04:46:53  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:53 (+1200)  main  04:46:53  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:53 (+1200)  main  04:46:53  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:53 (+1200)  main  04:46:53  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:53 (+1200)  main  04:46:53  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:53 (+1200)  main  04:46:53  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:53 (+1200)  main  04:46:53  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:53 (+1200)  main  04:46:53  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:54 (+1200)  main  04:46:54  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:54 (+1200)  main  04:46:54  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:54 (+1200)  main  04:46:54  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:54 (+1200)  main  04:46:54  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:54 (+1200)  main  04:46:54  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:54 (+1200)  main  04:46:54  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:54 (+1200)  main  04:46:54  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:54 (+1200)  main  04:46:54  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:54 (+1200)  main  04:46:54  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:54 (+1200)  main  04:46:54  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:54 (+1200)  main  04:46:54  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:54 (+1200)  main  04:46:54  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:54 (+1200)  main  04:46:54  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:54 (+1200)  main  04:46:54  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:54 (+1200)  main  04:46:54  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:54 (+1200)  main  04:46:54  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:54 (+1200)  main  04:46:54  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:54 (+1200)  main  04:46:54  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:54 (+1200)  main  04:46:54  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:54 (+1200)  main  04:46:54  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:55 (+1200)  main  04:46:55  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:55 (+1200)  main  04:46:55  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:55 (+1200)  main  04:46:55  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:55 (+1200)  main  04:46:55  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:55 (+1200)  main  04:46:55  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:55 (+1200)  main  04:46:55  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:55 (+1200)  main  04:46:55  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:55 (+1200)  main  04:46:55  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:55 (+1200)  main  04:46:55  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:55 (+1200)  main  04:46:55  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:55 (+1200)  main  04:46:55  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:55 (+1200)  main  04:46:55  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:55 (+1200)  main  04:46:55  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:55 (+1200)  main  04:46:55  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:55 (+1200)  main  04:46:55  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:55 (+1200)  main  04:46:55  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:55 (+1200)  main  04:46:55  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:55 (+1200)  main  04:46:55  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:55 (+1200)  main  04:46:55  INFO: Disabling GPS mode for concentrator's counter...
13.08.19 16:46:55 (+1200)  main  04:46:55  INFO: host/sx1301 time offset=(1565669873s:136373µs) - drift=60000605µs
13.08.19 16:46:55 (+1200)  main  04:46:55  INFO: Enabling GPS mode for concentrator's counter.
13.08.19 16:46:55 (+1200)  main  
13.08.19 16:46:55 (+1200)  main  04:46:55  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:55 (+1200)  main  04:46:55  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:56 (+1200)  main  04:46:56  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:56 (+1200)  main  04:46:56  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:56 (+1200)  main  04:46:56  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:56 (+1200)  main  04:46:56  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:56 (+1200)  main  04:46:56  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:56 (+1200)  main  04:46:56  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:56 (+1200)  main  04:46:56  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:56 (+1200)  main  04:46:56  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:56 (+1200)  main  04:46:56  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:56 (+1200)  main  04:46:56  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:56 (+1200)  main  04:46:56  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:56 (+1200)  main  04:46:56  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:56 (+1200)  main  04:46:56  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:56 (+1200)  main  04:46:56  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:56 (+1200)  main  04:46:56  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:56 (+1200)  main  04:46:56  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:56 (+1200)  main  04:46:56  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:56 (+1200)  main  04:46:56  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:56 (+1200)  main  04:46:56  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:57 (+1200)  main  04:46:57  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:57 (+1200)  main  04:46:57  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:57 (+1200)  main  04:46:57  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:57 (+1200)  main  04:46:57  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:57 (+1200)  main  04:46:57  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:57 (+1200)  main  04:46:57  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:57 (+1200)  main  04:46:57  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:57 (+1200)  main  04:46:57  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:57 (+1200)  main  04:46:57  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:57 (+1200)  main  04:46:57  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:57 (+1200)  main  04:46:57  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:57 (+1200)  main  04:46:57  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:57 (+1200)  main  04:46:57  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:57 (+1200)  main  04:46:57  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:57 (+1200)  main  04:46:57  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:57 (+1200)  main  04:46:57  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:57 (+1200)  main  04:46:57  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:57 (+1200)  main  04:46:57  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:57 (+1200)  main  04:46:57  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:57 (+1200)  main  04:46:57  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:58 (+1200)  main  04:46:58  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:58 (+1200)  main  04:46:58  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:58 (+1200)  main  04:46:58  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:58 (+1200)  main  04:46:58  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:58 (+1200)  main  04:46:58  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:58 (+1200)  main  04:46:58  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:58 (+1200)  main  04:46:58  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:58 (+1200)  main  04:46:58  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:58 (+1200)  main  04:46:58  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:58 (+1200)  main  04:46:58  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:58 (+1200)  main  04:46:58  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:58 (+1200)  main  04:46:58  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:58 (+1200)  main  04:46:58  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:58 (+1200)  main  04:46:58  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:58 (+1200)  main  04:46:58  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:58 (+1200)  main  04:46:58  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:58 (+1200)  main  04:46:58  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:58 (+1200)  main  04:46:58  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:58 (+1200)  main  04:46:58  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:58 (+1200)  main  04:46:58  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:59 (+1200)  main  04:46:59  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:59 (+1200)  main  04:46:59  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:59 (+1200)  main  04:46:59  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:59 (+1200)  main  04:46:59  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:59 (+1200)  main  04:46:59  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:59 (+1200)  main  04:46:59  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:59 (+1200)  main  04:46:59  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:59 (+1200)  main  04:46:59  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:59 (+1200)  main  04:46:59  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:59 (+1200)  main  04:46:59  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:59 (+1200)  main  04:46:59  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:59 (+1200)  main  04:46:59  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:59 (+1200)  main  04:46:59  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:59 (+1200)  main  04:46:59  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:59 (+1200)  main  04:46:59  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:59 (+1200)  main  04:46:59  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:59 (+1200)  main  04:46:59  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:59 (+1200)  main  04:46:59  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:59 (+1200)  main  04:46:59  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:46:59 (+1200)  main  04:46:59  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:47:00 (+1200)  main  04:47:00  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:47:00 (+1200)  main  04:47:00  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:47:00 (+1200)  main  04:47:00  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:47:00 (+1200)  main  04:47:00  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:47:00 (+1200)  main  04:47:00  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:47:00 (+1200)  main  04:47:00  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:47:00 (+1200)  main  04:47:00  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:47:00 (+1200)  main  04:47:00  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:47:00 (+1200)  main  04:47:00  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:47:00 (+1200)  main  04:47:00  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:47:00 (+1200)  main  04:47:00  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:47:00 (+1200)  main  04:47:00  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:47:00 (+1200)  main  04:47:00  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:47:00 (+1200)  main  04:47:00  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:47:00 (+1200)  main  04:47:00  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:47:00 (+1200)  main  04:47:00  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:47:00 (+1200)  main  04:47:00  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:47:00 (+1200)  main  04:47:00  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
13.08.19 16:47:00 (+1200)  main  
13.08.19 16:47:00 (+1200)  main  ##### 2019-08-13 04:47:00 GMT #####
13.08.19 16:47:00 (+1200)  main  ### [UPSTREAM] ###
13.08.19 16:47:00 (+1200)  main  # RF packets received by concentrator: 1559
13.08.19 16:47:00 (+1200)  main  # CRC_OK: 12.44%, CRC_FAIL: 87.56%, NO_CRC: 0.00%
13.08.19 16:47:00 (+1200)  main  # RF packets forwarded: 194 (4462 bytes)
13.08.19 16:47:00 (+1200)  main  # PUSH_DATA datagrams sent: 0 (0 bytes)
13.08.19 16:47:00 (+1200)  main  # PUSH_DATA acknowledged: 0.00%

You might as well get your own gateway to give you the raw data and decode it yourself, there’s little reason to depend on external services when the data is flowing through your own gateway.

Best case you get data even when the servers are unreliable; worst you discover something specifically wrong with your packets that prevents them from being properly decoded, and then fix that.

All operational issues are discussed on Slack #ops channel. If you are (still) experiencing issues you should mention them there.
However, are your gateway, gateway registration and application all pointing to the same region? (your gateway is using meshed so everything else should as well) If not you should change things to make sure everything is using meshed. TTN repeatedly communicated there are known issues with inter region communications that will not be fixed in V2 of the back-end.

The low-level library is getting valid (CRC is valid) packets from the hardware. These are forwarded to TTN. Does TTN show anything on the gteway traffic page?

The idea behind the community network is that you share your gateway with other users and other users share their gateways with you. Recommending people to implement their own ‘network’ is not in the spirit of TTN and takes resources away from the community network.

there’s little reason to depend on external services when the data is flowing through your own gateway

The idea behind the community network is that you share your gateway with other users and other users share their gateways with you. Recommending people to implement their own ‘network’ is not in the spirit of TTN and takes resources away from the community network.

That’s not actually what I recommended; I suggested extracting and decoding your own data directly, without relying on TTN - I did not mention disconnecting the gateway from TTN.

However, as long as the architecture doesn’t make it easy to do this in parallel, flakiness in the TTN infrastructure does indeed point people who have real needs towards entirely private setups.

Long range, it makes a lot of sense to handle (at least in the sense of having backup decoding for) your own data locally, while still exchanging data with the servers to support other’s sensors in your coverage area and your sensors in others’ coverage areas.

For the decoding side this is fairly straightforward, what is missing is a way to arbitrate a gateway’s single transmitter between distinct network servers - ie, your gateway might be in the best position to respond to someone else’s TTN node, but if it were busy transmitting to a node TTN didn’t know about, then someone else’s gateway that was 2nd best for received signal would actually be the best choice for that particular transmission.

But ideally, a node should be designed to keep feeding data even if they don’t receive any downlink for a day or two, so a decode-only scheme (which causes no transmitter usage contention) is a viable fallback compared to having complete outages.

Both poly_pkt_fwd and mp_pkt_fwd allow forwarding data to multiple destinations. Having two sources for downlinks is an issue as it might cause duty cycle violations even if the transmissions do not overlap. (The network keeps track of the duty cycle, not the gateway, that might be something worth building into the gateway software some day. Anyone with an urgent need and willing to sponsor development?)

Good point kersing, I had different gateway and application regions. All running smoothly again now, at least through the RAK image.

Tried running Balena image again, but it worked for a while, then started getting what I can only assume was bogus packets. 12,896 RF packets seems too high to me. Especially when it’s 58 per second.

I think I might give up and just use the RAK image.

14.08.19 19:19:47 (+1200)  main  ##### 2019-08-14 07:19:47 GMT #####
14.08.19 19:19:47 (+1200)  main  ### [UPSTREAM] ###
14.08.19 19:19:47 (+1200)  main  # RF packets received by concentrator: 12896
14.08.19 19:19:47 (+1200)  main  # CRC_OK: 100.00%, CRC_FAIL: 0.00%, NO_CRC: 0.00%
14.08.19 19:19:47 (+1200)  main  # RF packets forwarded: 12896 (296608 bytes)
14.08.19 19:19:47 (+1200)  main  # PUSH_DATA datagrams sent: 0 (0 bytes)
14.08.19 19:19:47 (+1200)  main  # PUSH_DATA acknowledged: 0.00%
14.08.19 19:19:47 (+1200)  main  ### [DOWNSTREAM] ###
14.08.19 19:19:47 (+1200)  main  # PULL_DATA sent: 0 (0.00% acknowledged)
14.08.19 19:19:47 (+1200)  main  # PULL_RESP(onse) datagrams received: 0 (0 bytes)
14.08.19 19:19:47 (+1200)  main  # RF packets sent to concentrator: 0 (0 bytes)
14.08.19 19:19:47 (+1200)  main  # TX errors: 0
14.08.19 19:19:47 (+1200)  main  ### BEACON IS DISABLED! 
14.08.19 19:19:47 (+1200)  main  ### [JIT] ###
14.08.19 19:19:47 (+1200)  main  # INFO: JIT queue contains 0 packets.
14.08.19 19:19:47 (+1200)  main  # INFO: JIT queue contains 0 beacons.
14.08.19 19:19:47 (+1200)  main  ### [GPS] ###
14.08.19 19:19:47 (+1200)  main  # No time keeping possible due to fake gps.
14.08.19 19:19:47 (+1200)  main  # Manual GPS coordinates: latitude -45.91360, longitude 170.46727, altitude 0 m
14.08.19 19:19:47 (+1200)  main  ### [PERFORMANCE] ###
14.08.19 19:19:47 (+1200)  main  # Upstream radio packet quality: 100.00%.
14.08.19 19:19:47 (+1200)  main  ### [ CONNECTIONS ] ###
14.08.19 19:19:47 (+1200)  main  # thethings.meshed.com.au: Connected
14.08.19 19:19:47 (+1200)  main  # Semtech status report send. 
14.08.19 19:19:47 (+1200)  main  ##### END #####
14.08.19 19:19:47 (+1200)  main  07:19:47  INFO: [TTN] thethings.meshed.com.au RTT 99
14.08.19 19:19:47 (+1200)  main  07:19:47  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
14.08.19 19:19:47 (+1200)  main  07:19:47  INFO: [TTN] send status success for thethings.meshed.com.au
14.08.19 19:19:47 (+1200)  main  07:19:47  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
14.08.19 19:19:47 (+1200)  main  07:19:47  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
14.08.19 19:19:47 (+1200)  main  07:19:47  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
14.08.19 19:19:47 (+1200)  main  07:19:47  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
14.08.19 19:19:47 (+1200)  main  07:19:47  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
14.08.19 19:19:47 (+1200)  main  07:19:47  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
14.08.19 19:19:47 (+1200)  main  07:19:47  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
14.08.19 19:19:47 (+1200)  main  07:19:47  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
14.08.19 19:19:47 (+1200)  main  07:19:47  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
14.08.19 19:19:47 (+1200)  main  07:19:47  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
14.08.19 19:19:47 (+1200)  main  07:19:47  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
14.08.19 19:19:48 (+1200)  main  07:19:48  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
14.08.19 19:19:48 (+1200)  main  07:19:48  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
14.08.19 19:19:48 (+1200)  main  07:19:48  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
14.08.19 19:19:48 (+1200)  main  07:19:48  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
14.08.19 19:19:48 (+1200)  main  07:19:48  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
14.08.19 19:19:48 (+1200)  main  07:19:48  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
14.08.19 19:19:48 (+1200)  main  07:19:48  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
14.08.19 19:19:48 (+1200)  main  07:19:48  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
14.08.19 19:19:48 (+1200)  main  07:19:48  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
14.08.19 19:19:48 (+1200)  main  07:19:48  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
14.08.19 19:19:48 (+1200)  main  07:19:48  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
14.08.19 19:19:48 (+1200)  main  07:19:48  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
14.08.19 19:19:48 (+1200)  main  07:19:48  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
14.08.19 19:19:48 (+1200)  main  07:19:48  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
14.08.19 19:19:48 (+1200)  main  07:19:48  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
14.08.19 19:19:48 (+1200)  main  07:19:48  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
14.08.19 19:19:48 (+1200)  main  07:19:48  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
14.08.19 19:19:48 (+1200)  main  07:19:48  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
14.08.19 19:19:48 (+1200)  main  07:19:48  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"
14.08.19 19:19:49 (+1200)  main  07:19:49  INFO: [up] TTN lora packet send to server "thethings.meshed.com.au"

The contrast between the 12896 packets allegedly received, all with correct CRC, and the 0 datagrams sent seems rather odd.

These stats should actually be resetting every time they print - so if you keep seeing messages like this, something is imagining traffic.

I’d probably be wanting to build the packet forwarder from source and add some debug code to it - your SPI clock might still be too fast, though one would think you’d be getting CRC failures or even total communication loss leading to packet forwarder crash and restart.

I have no experience with Balena, but there should not be any great mystery in getting an ordinary packet forwarder source base to work on this hardware, provided that you use a low SPI clock.