Hello,
I have a ic880a Gateway which I was able to connect successfully to TheThingsNetwork. A couple of weeks ago it went offline the first time and after rebooting the RaspberryPi it came up again. But since then I need to restart the gateway nearly daily (sudo systemctl restart ttn-gateway.service). I cannot find any useful information in the logs. How to troubleshoot the problem?
Thanks!
Posting the 20+ or so entries before it stops would be useful - someone may spot something you haven’t. Preferably for more than one occasion as well.
Has anything else changed?
One common issue with Pi’s is the power supply - if the gateway HAT is doing something extra, maybe the power is sagging?
I digged deeper into the logs and found out that there is a dhcp renew at the time when the gateway goes offline. I configured a fixed IP and will monitor if this resolves the issue…
Logs:
Nov 24 06:39:36 ttn-gateway ttn-gateway[5058]: INFO: [up] PUSH_ACK for server router.eu.thethings.network received in 50 ms
Nov 24 06:39:36 ttn-gateway ttn-gateway[5058]: INFO: [down] for server router.eu.thethings.network PULL_ACK received in 51 ms
Nov 24 06:39:36 ttn-gateway ttn-gateway[5058]: INFO: [down] for server router.eu.thethings.network PULL_ACK received in 50 ms
Nov 24 06:39:36 ttn-gateway ttn-gateway[5058]: INFO: [down] for server router.eu.thethings.network PULL_ACK received in 50 ms
Nov 24 06:39:36 ttn-gateway ttn-gateway[5058]: ##### 2020-11-24 05:39:36 GMT #####
Nov 24 06:39:36 ttn-gateway ttn-gateway[5058]: ### [UPSTREAM] ###
Nov 24 06:39:36 ttn-gateway ttn-gateway[5058]: # RF packets received by concentrator: 2
Nov 24 06:39:36 ttn-gateway ttn-gateway[5058]: # CRC_OK: 0.00%, CRC_FAIL: 100.00%, NO_CRC: 0.00%
Nov 24 06:39:36 ttn-gateway ttn-gateway[5058]: # RF packets forwarded: 0 (0 bytes)
Nov 24 06:39:36 ttn-gateway ttn-gateway[5058]: # PUSH_DATA datagrams sent: 1 (206 bytes)
Nov 24 06:39:36 ttn-gateway ttn-gateway[5058]: # PUSH_DATA acknowledged: 100.00%
Nov 24 06:39:36 ttn-gateway ttn-gateway[5058]: ### [DOWNSTREAM] ###
Nov 24 06:39:36 ttn-gateway ttn-gateway[5058]: # PULL_DATA sent: 3 (100.00% acknowledged)
Nov 24 06:39:36 ttn-gateway ttn-gateway[5058]: # PULL_RESP(onse) datagrams received: 0 (0 bytes)
Nov 24 06:39:36 ttn-gateway ttn-gateway[5058]: # RF packets sent to concentrator: 0 (0 bytes)
Nov 24 06:39:36 ttn-gateway ttn-gateway[5058]: # TX errors: 0
Nov 24 06:39:36 ttn-gateway ttn-gateway[5058]: ### [GPS] ###
Nov 24 06:40:26 ttn-gateway rngd[314]: stats: bits received from HRNG source: 11200064
Nov 24 06:40:26 ttn-gateway rngd[314]: stats: bits sent to kernel pool: 11154336
Nov 24 06:40:26 ttn-gateway rngd[314]: stats: entropy added to kernel pool: 11154336
Nov 24 06:40:26 ttn-gateway rngd[314]: stats: FIPS 140-2 successes: 560
Nov 24 06:40:26 ttn-gateway rngd[314]: stats: FIPS 140-2 failures: 0
Nov 24 06:40:26 ttn-gateway rngd[314]: stats: FIPS 140-2(2001-10-10) Monobit: 0
Nov 24 06:40:26 ttn-gateway rngd[314]: stats: FIPS 140-2(2001-10-10) Poker: 0
Nov 24 06:40:26 ttn-gateway rngd[314]: stats: FIPS 140-2(2001-10-10) Runs: 0
Nov 24 06:40:26 ttn-gateway rngd[314]: stats: FIPS 140-2(2001-10-10) Long run: 0
Nov 24 06:40:26 ttn-gateway rngd[314]: stats: FIPS 140-2(2001-10-10) Continuous run: 0
Nov 24 06:40:26 ttn-gateway rngd[314]: stats: HRNG source speed: (min=431.515; avg=762.495; max=773.392)Kibits/s
Nov 24 06:40:26 ttn-gateway rngd[314]: stats: FIPS tests speed: (min=6.276; avg=7.108; max=7.149)Mibits/s
Nov 24 06:40:26 ttn-gateway rngd[314]: stats: Lowest ready-buffers level: 2
Nov 24 06:40:26 ttn-gateway rngd[314]: stats: Entropy starvations: 0
Nov 24 06:40:26 ttn-gateway rngd[314]: stats: Time spent starving for entropy: (min=0; avg=0.000; max=0)us
Nov 24 06:41:12 ttn-gateway dhcpcd[421]: eth0: NAK: Invalid Request Message from 10.0.0.138
Nov 24 06:41:12 ttn-gateway dhcpcd[421]: eth0: message: Invalid Request Message
Nov 24 06:41:12 ttn-gateway avahi-daemon[313]: Withdrawing address record for 10.0.0.9 on eth0.
Nov 24 06:41:12 ttn-gateway avahi-daemon[313]: Leaving mDNS multicast group on interface eth0.IPv4 with address 10.0.0.9.
Nov 24 06:41:12 ttn-gateway avahi-daemon[313]: Interface eth0.IPv4 no longer relevant for mDNS.
Nov 24 06:41:12 ttn-gateway dhcpcd[421]: eth0: deleting route to 10.0.0.0/24
Nov 24 06:41:12 ttn-gateway dhcpcd[421]: eth0: deleting default route via 10.0.0.138
Nov 24 06:41:13 ttn-gateway dhcpcd[421]: eth0: soliciting a DHCP lease
Nov 24 06:41:13 ttn-gateway dhcpcd[421]: eth0: offered 10.0.0.1 from 10.0.0.138
Nov 24 06:41:13 ttn-gateway dhcpcd[421]: eth0: probing address 10.0.0.1/24
Nov 24 06:41:18 ttn-gateway dhcpcd[421]: eth0: leased 10.0.0.1 for 86400 seconds
Nov 24 06:41:18 ttn-gateway avahi-daemon[313]: Joining mDNS multicast group on interface eth0.IPv4 with address 10.0.0.1.
Nov 24 06:41:18 ttn-gateway avahi-daemon[313]: New relevant interface eth0.IPv4 for mDNS.
Nov 24 06:41:18 ttn-gateway dhcpcd[421]: eth0: adding route to 10.0.0.0/24
Nov 24 06:41:18 ttn-gateway avahi-daemon[313]: Registering new address record for 10.0.0.1 on eth0.IPv4.
Nov 24 06:41:18 ttn-gateway dhcpcd[421]: eth0: adding default route via 10.0.0.138
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: # Invalid gps time reference (age: 1606196376 sec)
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: # Manual GPS coordinates: latitude 42.29708, longitude 10.92648, altitude 990 m
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: ##### END #####
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: INFO: [up] PUSH_ACK for server router.eu.thethings.network received in 51 ms
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: INFO: [down] for server router.eu.thethings.network PULL_ACK received in 50 ms
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: INFO: [down] for server router.eu.thethings.network PULL_ACK received in 51 ms
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: INFO: [down] for server router.eu.thethings.network PULL_ACK received in 51 ms
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: ##### 2020-11-24 05:40:06 GMT #####
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: ### [UPSTREAM] ###
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: # RF packets received by concentrator: 0
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: # CRC_OK: 0.00%, CRC_FAIL: 0.00%, NO_CRC: 0.00%
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: # RF packets forwarded: 0 (0 bytes)
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: # PUSH_DATA datagrams sent: 1 (206 bytes)
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: # PUSH_DATA acknowledged: 100.00%
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: ### [DOWNSTREAM] ###
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: # PULL_DATA sent: 3 (100.00% acknowledged)
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: # PULL_RESP(onse) datagrams received: 0 (0 bytes)
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: # RF packets sent to concentrator: 0 (0 bytes)
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: # TX errors: 0
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: ### [GPS] ###
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: # Invalid gps time reference (age: 1606196406 sec)
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: # Manual GPS coordinates: latitude 42.29708, longitude 10.92648, altitude 990 m
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: ##### END #####
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: INFO: [up] PUSH_ACK for server router.eu.thethings.network received in 50 ms
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: INFO: [down] for server router.eu.thethings.network PULL_ACK received in 51 ms
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: INFO: [down] for server router.eu.thethings.network PULL_ACK received in 51 ms
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: INFO: [down] for server router.eu.thethings.network PULL_ACK received in 50 ms
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: ##### 2020-11-24 05:40:36 GMT #####
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: ### [UPSTREAM] ###
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: # RF packets received by concentrator: 0
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: # CRC_OK: 0.00%, CRC_FAIL: 0.00%, NO_CRC: 0.00%
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: # RF packets forwarded: 0 (0 bytes)
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: # PUSH_DATA datagrams sent: 1 (206 bytes)
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: # PUSH_DATA acknowledged: 100.00%
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: ### [DOWNSTREAM] ###
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: # PULL_DATA sent: 3 (100.00% acknowledged)
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: # PULL_RESP(onse) datagrams received: 0 (0 bytes)
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: # RF packets sent to concentrator: 0 (0 bytes)
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: # TX errors: 0
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: ### [GPS] ###
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: # Invalid gps time reference (age: 1606196436 sec)
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: # Manual GPS coordinates: latitude 42.29708, longitude 10.92648, altitude 990 m
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: ##### END #####
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: INFO: [up] PUSH_ACK for server router.eu.thethings.network received in 51 ms
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: INFO: [down] for server router.eu.thethings.network PULL_ACK received in 50 ms
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: INFO: [down] for server router.eu.thethings.network PULL_ACK received in 50 ms
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: INFO: [down] for server router.eu.thethings.network PULL_ACK received in 51 ms
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: ##### 2020-11-24 05:41:06 GMT #####
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: ### [UPSTREAM] ###
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: # RF packets received by concentrator: 0
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: # CRC_OK: 0.00%, CRC_FAIL: 0.00%, NO_CRC: 0.00%
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: # RF packets forwarded: 0 (0 bytes)
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: # PUSH_DATA datagrams sent: 1 (206 bytes)
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: # PUSH_DATA acknowledged: 100.00%
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: ### [DOWNSTREAM] ###
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: # PULL_DATA sent: 3 (100.00% acknowledged)
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: # PULL_RESP(onse) datagrams received: 0 (0 bytes)
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: # RF packets sent to concentrator: 0 (0 bytes)
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: # TX errors: 0
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: ### [GPS] ###
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: # Invalid gps time reference (age: 1606196466 sec)
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: # Manual GPS coordinates: latitude 42.29708, longitude 10.92648, altitude 990 m
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: ##### END #####
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: INFO: [up] PUSH_ACK for server router.eu.thethings.network received in 51 ms
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: ##### 2020-11-24 05:41:36 GMT #####
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: ### [UPSTREAM] ###
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: # RF packets received by concentrator: 0
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: # CRC_OK: 0.00%, CRC_FAIL: 0.00%, NO_CRC: 0.00%
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: # RF packets forwarded: 0 (0 bytes)
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: # PUSH_DATA datagrams sent: 1 (206 bytes)
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: # PUSH_DATA acknowledged: 100.00%
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: ### [DOWNSTREAM] ###
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: # PULL_DATA sent: 3 (0.00% acknowledged)
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: # PULL_RESP(onse) datagrams received: 0 (0 bytes)
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: # RF packets sent to concentrator: 0 (0 bytes)
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: # TX errors: 0
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: ### [GPS] ###
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: # Invalid gps time reference (age: 1606196496 sec)
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: # Manual GPS coordinates: latitude 42.29708, longitude 10.92648, altitude 990 m
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: ##### END #####
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: ##### 2020-11-24 05:42:06 GMT #####
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: ### [UPSTREAM] ###
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: # RF packets received by concentrator: 0
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: # CRC_OK: 0.00%, CRC_FAIL: 0.00%, NO_CRC: 0.00%
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: # RF packets forwarded: 0 (0 bytes)
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: # PUSH_DATA datagrams sent: 1 (206 bytes)
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: # PUSH_DATA acknowledged: 0.00%
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: ### [DOWNSTREAM] ###
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: # PULL_DATA sent: 3 (0.00% acknowledged)
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: # PULL_RESP(onse) datagrams received: 0 (0 bytes)
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: # RF packets sent to concentrator: 0 (0 bytes)
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: # TX errors: 0
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: ### [GPS] ###
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: # Invalid gps time reference (age: 1606196526 sec)
Nov 24 06:42:06 ttn-gateway ttn-gateway[5058]: # Manual GPS coordinates: latitude 42.29708, longitude 10.92648, altitude 990 m
Nov 24 06:44:02 ttn-gateway dhcpcd[421]: eth0: Router Advertisement from fe80::1
Nov 24 06:45:36 ttn-gateway ttn-gateway[5058]: ##### END #####
Since setting a static IP address there was no more outage. I am wondering why an IP change needs a reboot?