[SOLVED] Gateway disconnected

Hello all,
I have a gateway build with iC880A + RaspberryPI 2 + Gonzalo backplane
The software is https://github.com/ttn-zh/ic880a-gateway
It work for about two years withouth problems.
Actually it’s lat seen from 7 hours.
I don’t touch nothing, the raspberry work well, ssh ok, gateway software run, no network problems, internet work and curl download from domain names.
Gateway id is eui-b827ebfffe25f356
Some idea?
Thanks.

Alberto

You’ll need to look at the log files on the gateway. Often output is in /var/log/syslog or similar

I run the service from command line ad this is the output:

root@ttn-panu:/home/ttn# systemctl stop ttn-gateway.service
root@ttn-panu:/home/ttn# cd /opt/ttn-gateway/bin/
root@ttn-panu:/opt/ttn-gateway/bin# ./start.sh
*** Poly Packet Forwarder for Lora Gateway ***
Version: 2.1.0
*** Lora concentrator HAL library version info ***
Version: 3.1.0; Options: native;
***
INFO: Little endian host
INFO: found global configuration file global_conf.json, parsing it
INFO: global_conf.json does contain a JSON object named SX1301_conf, parsing SX1301 parameters
INFO: lorawan_public 1, clksrc 1
INFO: Configuring TX LUT with 16 indexes
INFO: radio 0 enabled (type SX1257), center frequency 867500000, RSSI offset -166.000000, tx enabled 1
INFO: radio 1 enabled (type SX1257), center frequency 868500000, RSSI offset -166.000000, tx enabled 0
INFO: Lora multi-SF channel 0>  radio 1, IF -400000 Hz, 125 kHz bw, SF 7 to 12
INFO: Lora multi-SF channel 1>  radio 1, IF -200000 Hz, 125 kHz bw, SF 7 to 12
INFO: Lora multi-SF channel 2>  radio 1, IF 0 Hz, 125 kHz bw, SF 7 to 12
INFO: Lora multi-SF channel 3>  radio 0, IF -400000 Hz, 125 kHz bw, SF 7 to 12
INFO: Lora multi-SF channel 4>  radio 0, IF -200000 Hz, 125 kHz bw, SF 7 to 12
INFO: Lora multi-SF channel 5>  radio 0, IF 0 Hz, 125 kHz bw, SF 7 to 12
INFO: Lora multi-SF channel 6>  radio 0, IF 200000 Hz, 125 kHz bw, SF 7 to 12
INFO: Lora multi-SF channel 7>  radio 0, IF 400000 Hz, 125 kHz bw, SF 7 to 12
INFO: Lora std channel> radio 1, IF -200000 Hz, 250000 Hz bw, SF 7
INFO: FSK channel> radio 1, IF 300000 Hz, 125000 Hz bw, 50000 bps datarate
INFO: global_conf.json does contain a JSON object named gateway_conf, parsing gateway parameters
INFO: gateway MAC address is configured to AA555A0000000000
INFO: Found 2 servers in array.
INFO: Server 0 configured to "127.0.0.1", with port up "1680" and port down "1681"
INFO: Server 1 configured to "iot.semtech.com", with port up "1680" and port down "1680"
INFO: Found 5 system calls in array.
INFO: System command 0: "df -m"
INFO: System command 1: "free -h"
INFO: System command 2: "uptime"
INFO: System command 3: "who -a"
INFO: System command 4: "uname -a"
INFO: monitor hostname or IP address is configured to "127.0.0.1"
INFO: monitor port is configured to "2008"
INFO: ghost hostname or IP address is configured to "127.0.0.1"
INFO: ghost port is configured to "1918"
INFO: downstream keep-alive interval is configured to 10 seconds
INFO: statistics display interval is configured to 30 seconds
INFO: upstream PUSH_DATA time-out is configured to 100 ms
INFO: packets received with a valid CRC will be forwarded
INFO: packets received with a CRC error will NOT be forwarded
INFO: packets received with no CRC will NOT be forwarded
INFO: GPS serial port path is configured to "/dev/ttyAMA0"
INFO: SSH path is configured to "/usr/bin/ssh"
INFO: SSH port is configured to 22
INFO: HTTP port is configured to 80
INFO: NGROK path is configured to "/usr/bin/ngrok"
INFO: Reference latitude is configured to 10.000000 deg
INFO: Reference longitude is configured to 20.000000 deg
INFO: Reference altitude is configured to -1 meters
INFO: GPS is enabled
INFO: Using fake GPS coordinates instead of real.
INFO: Upstream data is enabled
INFO: Downstream data is enabled
INFO: Ghoststream data is disabled
INFO: Radiostream data is enabled
INFO: Statusstream data is enabled
INFO: Beacon is disabled
INFO: Monitor is disabled
INFO: Platform configured to "IMST + Rpi"
INFO: Contact email configured to "operator@gateway.tst"
INFO: Description configured to "Update me"
INFO: found local configuration file local_conf.json, parsing it
INFO: redefined parameters will overwrite global parameters
INFO: local_conf.json does not contain a JSON object named SX1301_conf
INFO: local_conf.json does contain a JSON object named gateway_conf, parsing gateway parameters
INFO: gateway MAC address is configured to B827EBFFFE25F356
INFO: Found 1 servers in array.
INFO: Server 0 configured to "router.eu.thethings.network", with port up "1700" and port down "1700"
INFO: packets received with a valid CRC will be forwarded
INFO: packets received with a CRC error will NOT be forwarded
INFO: packets received with no CRC will NOT be forwarded
INFO: Reference latitude is configured to 45.425785 deg
INFO: Reference longitude is configured to 9.201454 deg
INFO: Reference altitude is configured to 0 meters
INFO: GPS is enabled
INFO: Using fake GPS coordinates instead of real.
INFO: Upstream data is enabled
INFO: Downstream data is enabled
INFO: Ghoststream data is disabled
INFO: Radiostream data is enabled
INFO: Statusstream data is enabled
INFO: Beacon is disabled
INFO: Monitor is disabled
INFO: Contact email configured to "alberto@panu.it"
INFO: Description configured to "love from Milano"
INFO: Successfully contacted server router.eu.thethings.network
INFO: [main] Starting the concentrator
INFO: [main] concentrator started, radio packets can now be received.
INFO: [down] Thread activated for all server router.eu.thethings.network
INFO: [up] Thread activated for all servers.
INFO: [down] for server router.eu.thethings.network PULL_ACK received in 35 ms
INFO: [up] PUSH_ACK for server router.eu.thethings.network received in 37 ms
INFO: [down] for server router.eu.thethings.network PULL_ACK received in 36 ms
INFO: [down] for server router.eu.thethings.network PULL_ACK received in 34 ms

##### 2019-07-25 14:38:16 GMT #####
### [UPSTREAM] ###
# RF packets received by concentrator: 1
# CRC_OK: 100.00%, CRC_FAIL: 0.00%, NO_CRC: 0.00%
# RF packets forwarded: 1 (49 bytes)
# PUSH_DATA datagrams sent: 1 (280 bytes)
# PUSH_DATA acknowledged: 100.00%
### [DOWNSTREAM] ###
# PULL_DATA sent: 3 (100.00% acknowledged)
# PULL_RESP(onse) datagrams received: 0 (0 bytes)
# RF packets sent to concentrator: 0 (0 bytes)
# TX errors: 0
### [GPS] ###
# Invalid gps time reference (age: 1564065496 sec)
# Manual GPS coordinates: latitude 45.42578, longitude 9.20145, altitude 0 m
##### END #####
INFO: [up] PUSH_ACK for server router.eu.thethings.network received in 38 ms
INFO: [down] for server router.eu.thethings.network PULL_ACK received in 72 ms
INFO: [down] for server router.eu.thethings.network PULL_ACK received in 35 ms
INFO: [down] for server router.eu.thethings.network PULL_ACK received in 34 ms
INFO: [up] PUSH_ACK for server router.eu.thethings.network received in 35 ms

##### 2019-07-25 14:38:46 GMT #####
### [UPSTREAM] ###
# RF packets received by concentrator: 3
# CRC_OK: 33.33%, CRC_FAIL: 66.67%, NO_CRC: 0.00%
# RF packets forwarded: 1 (23 bytes)
# PUSH_DATA datagrams sent: 2 (467 bytes)
# PUSH_DATA acknowledged: 100.00%
### [DOWNSTREAM] ###
# PULL_DATA sent: 3 (100.00% acknowledged)
# PULL_RESP(onse) datagrams received: 0 (0 bytes)
# RF packets sent to concentrator: 0 (0 bytes)
# TX errors: 0
### [GPS] ###
# Invalid gps time reference (age: 1564065526 sec)
# Manual GPS coordinates: latitude 45.42578, longitude 9.20145, altitude 0 m
##### END #####
INFO: [up] PUSH_ACK for server router.eu.thethings.network received in 43 ms
INFO: [down] for server router.eu.thethings.network PULL_ACK received in 34 ms
INFO: [up] PUSH_ACK for server router.eu.thethings.network received in 35 ms
INFO: [down] for server router.eu.thethings.network PULL_ACK received in 34 ms
INFO: [up] PUSH_ACK for server router.eu.thethings.network received in 35 ms
^C
##### 2019-07-25 14:39:01 GMT #####
### [UPSTREAM] ###
# RF packets received by concentrator: 3
# CRC_OK: 66.67%, CRC_FAIL: 33.33%, NO_CRC: 0.00%
# RF packets forwarded: 2 (72 bytes)
# PUSH_DATA datagrams sent: 3 (749 bytes)
# PUSH_DATA acknowledged: 100.00%
### [DOWNSTREAM] ###
# PULL_DATA sent: 2 (100.00% acknowledged)
# PULL_RESP(onse) datagrams received: 0 (0 bytes)
# RF packets sent to concentrator: 0 (0 bytes)
# TX errors: 0
### [GPS] ###
# Invalid gps time reference (age: 1564065541 sec)
# Manual GPS coordinates: latitude 45.42578, longitude 9.20145, altitude 0 m
##### END #####

INFO: End of upstream thread

INFO: End of downstream thread for server  0.
INFO: concentrator stopped successfully
INFO: Exiting packet forwarder program

Thanks.

100 lines of syslog:

root@ttn-panu:/var/log# cat syslog | fgrep ttn-gateway | tail -100
Jul 25 16:43:00 ttn-panu ttn-gateway[1274]: INFO: Description configured to "Update me"
Jul 25 16:43:00 ttn-panu ttn-gateway[1274]: INFO: found local configuration file local_conf.json, parsing it
Jul 25 16:43:00 ttn-panu ttn-gateway[1274]: INFO: redefined parameters will overwrite global parameters
Jul 25 16:43:00 ttn-panu ttn-gateway[1274]: INFO: local_conf.json does not contain a JSON object named SX1301_conf
Jul 25 16:43:00 ttn-panu ttn-gateway[1274]: INFO: local_conf.json does contain a JSON object named gateway_conf, parsing gateway parameters
Jul 25 16:43:00 ttn-panu ttn-gateway[1274]: INFO: gateway MAC address is configured to B827EBFFFE25F356
Jul 25 16:43:00 ttn-panu ttn-gateway[1274]: INFO: Found 1 servers in array.
Jul 25 16:43:00 ttn-panu ttn-gateway[1274]: INFO: Server 0 configured to "router.eu.thethings.network", with port up "1700" and port down "1700"
Jul 25 16:43:00 ttn-panu ttn-gateway[1274]: INFO: packets received with a valid CRC will be forwarded
Jul 25 16:43:00 ttn-panu ttn-gateway[1274]: INFO: packets received with a CRC error will NOT be forwarded
Jul 25 16:43:00 ttn-panu ttn-gateway[1274]: INFO: packets received with no CRC will NOT be forwarded
Jul 25 16:43:00 ttn-panu ttn-gateway[1274]: INFO: Reference latitude is configured to 45.425785 deg
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: INFO: Reference longitude is configured to 9.201454 deg
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: INFO: Reference altitude is configured to 0 meters
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: INFO: GPS is enabled
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: INFO: Using fake GPS coordinates instead of real.
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: INFO: Upstream data is enabled
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: INFO: Downstream data is enabled
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: INFO: Ghoststream data is disabled
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: INFO: Radiostream data is enabled
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: INFO: Statusstream data is enabled
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: INFO: Beacon is disabled
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: INFO: Monitor is disabled
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: INFO: Contact email configured to "alberto@panu.it"
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: INFO: Description configured to "love from Milano"
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: INFO: Successfully contacted server router.eu.thethings.network
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: INFO: [main] Starting the concentrator
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: INFO: [main] concentrator started, radio packets can now be received.
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: INFO: [down] Thread activated for all server router.eu.thethings.network
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: INFO: [up] Thread activated for all servers.
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: INFO: [down] for server router.eu.thethings.network PULL_ACK received in 35 ms
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: INFO: [down] for server router.eu.thethings.network PULL_ACK received in 34 ms
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: INFO: [down] for server router.eu.thethings.network PULL_ACK received in 34 ms
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: ##### 2019-07-25 14:43:33 GMT #####
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: ### [UPSTREAM] ###
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: # RF packets received by concentrator: 1
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: # CRC_OK: 0.00%, CRC_FAIL: 100.00%, NO_CRC: 0.00%
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: # RF packets forwarded: 0 (0 bytes)
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: # PUSH_DATA datagrams sent: 0 (0 bytes)
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: # PUSH_DATA acknowledged: 0.00%
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: ### [DOWNSTREAM] ###
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: # PULL_DATA sent: 3 (100.00% acknowledged)
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: # PULL_RESP(onse) datagrams received: 0 (0 bytes)
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: # RF packets sent to concentrator: 0 (0 bytes)
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: # TX errors: 0
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: ### [GPS] ###
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: # Invalid gps time reference (age: 1564065813 sec)
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: # Manual GPS coordinates: latitude 45.42578, longitude 9.20145, altitude 0 m
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: ##### END #####
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: INFO: [up] PUSH_ACK for server router.eu.thethings.network received in 36 ms
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: INFO: [down] for server router.eu.thethings.network PULL_ACK received in 34 ms
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: INFO: [down] for server router.eu.thethings.network PULL_ACK received in 34 ms
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: INFO: [up] PUSH_ACK for server router.eu.thethings.network received in 35 ms
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: INFO: [down] for server router.eu.thethings.network PULL_ACK received in 34 ms
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: INFO: [up] PUSH_ACK for server router.eu.thethings.network received in 35 ms
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: ##### 2019-07-25 14:44:03 GMT #####
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: ### [UPSTREAM] ###
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: # RF packets received by concentrator: 2
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: # CRC_OK: 100.00%, CRC_FAIL: 0.00%, NO_CRC: 0.00%
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: # RF packets forwarded: 2 (72 bytes)
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: # PUSH_DATA datagrams sent: 3 (745 bytes)
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: # PUSH_DATA acknowledged: 100.00%
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: ### [DOWNSTREAM] ###
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: # PULL_DATA sent: 3 (100.00% acknowledged)
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: # PULL_RESP(onse) datagrams received: 0 (0 bytes)
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: # RF packets sent to concentrator: 0 (0 bytes)
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: # TX errors: 0
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: ### [GPS] ###
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: # Invalid gps time reference (age: 1564065843 sec)
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: # Manual GPS coordinates: latitude 45.42578, longitude 9.20145, altitude 0 m
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: ##### END #####
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: INFO: [up] PUSH_ACK for server router.eu.thethings.network received in 35 ms
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: INFO: [down] for server router.eu.thethings.network PULL_ACK received in 34 ms
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: INFO: [down] for server router.eu.thethings.network PULL_ACK received in 34 ms
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: INFO: [up] PUSH_ACK for server router.eu.thethings.network received in 35 ms
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: INFO: [down] for server router.eu.thethings.network PULL_ACK received in 34 ms
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: ##### 2019-07-25 14:44:33 GMT #####
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: ### [UPSTREAM] ###
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: # RF packets received by concentrator: 3
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: # CRC_OK: 33.33%, CRC_FAIL: 66.67%, NO_CRC: 0.00%
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: # RF packets forwarded: 1 (19 bytes)
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: # PUSH_DATA datagrams sent: 2 (463 bytes)
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: # PUSH_DATA acknowledged: 100.00%
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: ### [DOWNSTREAM] ###
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: # PULL_DATA sent: 3 (100.00% acknowledged)
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: # PULL_RESP(onse) datagrams received: 0 (0 bytes)
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: # RF packets sent to concentrator: 0 (0 bytes)
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: # TX errors: 0
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: ### [GPS] ###
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: # Invalid gps time reference (age: 1564065873 sec)
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: # Manual GPS coordinates: latitude 45.42578, longitude 9.20145, altitude 0 m
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: ##### END #####
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: INFO: [up] PUSH_ACK for server router.eu.thethings.network received in 56 ms
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: INFO: [down] for server router.eu.thethings.network PULL_ACK received in 34 ms
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: INFO: [down] for server router.eu.thethings.network PULL_ACK received in 34 ms
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: INFO: [down] for server router.eu.thethings.network PULL_ACK received in 34 ms
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: ##### 2019-07-25 14:45:03 GMT #####
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: ### [UPSTREAM] ###
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: # RF packets received by concentrator: 0
Jul 25 16:45:03 ttn-panu ttn-gateway[1274]: # CRC_OK: 0.00%, CRC_FAIL: 0.00%, NO_CRC: 0.00%

This shows that your gateway is in round-trip communication with at least the ingest side of TTN server infrastructure.

Could be a temporary failure in the TTN infrastructure?

Hi,

i have the same problem with my 2 gateways (RPi3+iC880a), packet forwarding works fine, but gateway status in console is “not connected”.
Last seen for one of them is 7 hrs, and the other is 4 hrs.

Packets are visible in application data tab, but not in gateway data tab.

Probably TTN serverside problem…

yes it probably is … it’s working , data is passed to applications, but only the console is not updated

Now the status is: connected.
Thanks all.

Alberto

Hello all,
today I have the same problem.
I checked the gateway from ssh and all is OK.
But the gateway is disconnected on the ttn console from 3 hours.
Somehone has an idea?
Thanks.

Alberto

What exactly did you learn from the gateway via SSH that leads you to be certain things are ok?

Do you see push and pull acks being received?

Do you see LoRa rxpk reports?

Yes:

##### 2019-12-12 08:57:01 GMT #####
### [UPSTREAM] ###
# RF packets received by concentrator: 2
# CRC_OK: 50.00%, CRC_FAIL: 50.00%, NO_CRC: 0.00%
# RF packets forwarded: 1 (45 bytes)
# PUSH_DATA datagrams sent: 2 (495 bytes)
# PUSH_DATA acknowledged: 0.00%
### [DOWNSTREAM] ###
# PULL_DATA sent: 3 (66.67% acknowledged)
# PULL_RESP(onse) datagrams received: 0 (0 bytes)
# RF packets sent to concentrator: 0 (0 bytes)
# TX errors: 0
### [GPS] ###
# Invalid gps time reference (age: 1576141021 sec)
# Manual GPS coordinates: latitude 45.42578, longitude 9.20145, altitude 0 m
##### END #####
INFO: [down] for server router.eu.thethings.network PULL_ACK received in 32 ms
INFO: [down] for server router.eu.thethings.network PULL_ACK received in 33 ms

##### 2019-12-12 08:57:31 GMT #####
### [UPSTREAM] ###
# RF packets received by concentrator: 1
# CRC_OK: 100.00%, CRC_FAIL: 0.00%, NO_CRC: 0.00%
# RF packets forwarded: 1 (49 bytes)
# PUSH_DATA datagrams sent: 2 (503 bytes)
# PUSH_DATA acknowledged: 0.00%
### [DOWNSTREAM] ###
# PULL_DATA sent: 3 (66.67% acknowledged)
# PULL_RESP(onse) datagrams received: 0 (0 bytes)
# RF packets sent to concentrator: 0 (0 bytes)
# TX errors: 0
### [GPS] ###
# Invalid gps time reference (age: 1576141051 sec)
# Manual GPS coordinates: latitude 45.42578, longitude 9.20145, altitude 0 m
##### END #####
INFO: [down] for server router.eu.thethings.network PULL_ACK received in 33 ms

same here:
Status = not connected, but

tail -n 10 /var/log/syslog
Dec 12 11:01:20 loragate lora_pkt_fwd[2871]: REPORT~ # RF packets sent to concentrator: 0 (0 bytes)
Dec 12 11:01:20 loragate lora_pkt_fwd[2871]: REPORT~ # TX errors: 0
Dec 12 11:01:20 loragate lora_pkt_fwd[2871]: REPORT~ # BEACON queued: 0
Dec 12 11:01:20 loragate lora_pkt_fwd[2871]: REPORT~ # BEACON sent so far: 0
Dec 12 11:01:20 loragate lora_pkt_fwd[2871]: REPORT~ # BEACON rejected: 0
Dec 12 11:01:20 loragate lora_pkt_fwd[2871]: REPORT~ ### [PPS] ###
Dec 12 11:01:20 loragate lora_pkt_fwd[2871]: REPORT~ # SX1301 time (PPS): 482427125
Dec 12 11:01:20 loragate lora_pkt_fwd[2871]: src/lora_pkt_fwd.c:1433:main(): ### [GPS] ###
Dec 12 11:01:20 loragate lora_pkt_fwd[2871]: src/lora_pkt_fwd.c:1449:main(): # GPS sync is disabled
Dec 12 11:01:20 loragate lora_pkt_fwd[2871]: src/lora_pkt_fwd.c:1451:main(): ##### END #####

and

sudo systemctl status lorapktfwd
● lorapktfwd.service - packet forwarder
   Loaded: loaded (/lib/systemd/system/lorapktfwd.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2019-12-12 10:52:47 CET; 11min ago
  Process: 2859 ExecStartPre=/usr/bin/reset_lgw.sh start (code=exited, status=0/SUCCESS)
 Main PID: 2871 (lora_pkt_fwd)
    Tasks: 5 (limit: 2200)
   Memory: 972.0K
   CGroup: /system.slice/lorapktfwd.service
           └─2871 /usr/bin/lora_pkt_fwd
Dez 12 11:03:50 loragate lora_pkt_fwd[2871]: REPORT~ # BEACON sent so far: 0
Dez 12 11:03:50 loragate lora_pkt_fwd[2871]: REPORT~ # BEACON rejected: 0
Dez 12 11:03:50 loragate lora_pkt_fwd[2871]: REPORT~ ### [PPS] ###
Dez 12 11:03:50 loragate lora_pkt_fwd[2871]: REPORT~ # SX1301 time (PPS): 602427408
Dez 12 11:03:50 loragate lora_pkt_fwd[2871]: src/lora_pkt_fwd.c:1433:main(): ### [GPS] ###
Dez 12 11:03:50 loragate lora_pkt_fwd[2871]: src/lora_pkt_fwd.c:1449:main(): # GPS sync is disabled
Dez 12 11:03:50 loragate lora_pkt_fwd[2871]: src/lora_pkt_fwd.c:1451:main(): ##### END #####
Dez 12 11:03:50 loragate lora_pkt_fwd[2871]: RXTX~ {"stat":{"time":"2019-12-12 10:03:20 GMT","rxnb":0,"rxok":0,"rxfw":0,"ackr":0.0,"dwnb":0,"txnb":0}}
Dez 12 11:03:50 loragate lora_pkt_fwd[2871]: REPORT~ ################## Report at: 2019-12-12 10:03:50 GMT ##################
Dez 12 11:03:50 loragate lora_pkt_fwd[2871]: REPORT~ ### [UPSTREAM] ###

both seems OK.

(Dragino PG1301)

I just saw the same on my Sweden-GW : Disconnected “last seen 17 hr ago”
Raspi w. IC880A.

My Denmark-GW is connected , uses the “same HW” & “Build”.

Both units are talking fine w the EU TTN router

**** Begin Sweden *********
Dec 12 11:39:39 sv-lorawangw-01 ttn-gateway[276]: ##### 2019-12-12 10:39:08 GMT #####
Dec 12 11:39:39 sv-lorawangw-01 ttn-gateway[276]: ### [UPSTREAM] ###
Dec 12 11:39:39 sv-lorawangw-01 ttn-gateway[276]: # RF packets received by concentrator: 0
Dec 12 11:39:39 sv-lorawangw-01 ttn-gateway[276]: # CRC_OK: 0.00%, CRC_FAIL: 0.00%, NO_CRC: 0.00%
Dec 12 11:39:39 sv-lorawangw-01 ttn-gateway[276]: # RF packets forwarded: 0 (0 bytes)
Dec 12 11:39:39 sv-lorawangw-01 ttn-gateway[276]: # PUSH_DATA datagrams sent: 1 (233 bytes)
Dec 12 11:39:39 sv-lorawangw-01 ttn-gateway[276]: # PUSH_DATA acknowledged: 100.00%
Dec 12 11:39:39 sv-lorawangw-01 ttn-gateway[276]: ### [DOWNSTREAM] ###
Dec 12 11:39:39 sv-lorawangw-01 ttn-gateway[276]: # PULL_DATA sent: 3 (100.00% acknowledged)
Dec 12 11:39:39 sv-lorawangw-01 ttn-gateway[276]: # PULL_RESP(onse) datagrams received: 0 (0 bytes)
Dec 12 11:39:39 sv-lorawangw-01 ttn-gateway[276]: # RF packets sent to concentrator: 0 (0 bytes)
Dec 12 11:39:39 sv-lorawangw-01 ttn-gateway[276]: # TX errors: 0
Dec 12 11:39:39 sv-lorawangw-01 ttn-gateway[276]: ### [GPS] ###
Dec 12 11:39:39 sv-lorawangw-01 ttn-gateway[276]: # Invalid gps time reference (age: 1576147148 sec)
Dec 12 11:39:39 sv-lorawangw-01 ttn-gateway[276]: # Manual GPS coordinates: latitude 56.52237, longitude 13.32014, altitude 19 m
Dec 12 11:39:39 sv-lorawangw-01 ttn-gateway[276]: ##### END #####
Dec 12 11:39:39 sv-lorawangw-01 ttn-gateway[276]: INFO: [up] PUSH_ACK for server router.eu.thethings.network received in 32 ms
Dec 12 11:39:39 sv-lorawangw-01 ttn-gateway[276]: INFO: [down] for server router.eu.thethings.network PULL_ACK received in 31 ms
Dec 12 11:39:39 sv-lorawangw-01 ttn-gateway[276]: INFO: [down] for server router.eu.thethings.network PULL_ACK received in 31 ms
Dec 12 11:39:39 sv-lorawangw-01 ttn-gateway[276]: INFO: [down] for server router.eu.thethings.network PULL_ACK received in 31 ms
**** End Sweden *********

**** Begin DK *********
Dec 12 11:19:11 kv-lorawangw-01 ttn-gateway[270]: ##### 2019-12-12 10:19:10 GMT #####
Dec 12 11:19:11 kv-lorawangw-01 ttn-gateway[270]: ### [UPSTREAM] ###
Dec 12 11:19:11 kv-lorawangw-01 ttn-gateway[270]: # RF packets received by concentrator: 0
Dec 12 11:19:11 kv-lorawangw-01 ttn-gateway[270]: # CRC_OK: 0.00%, CRC_FAIL: 0.00%, NO_CRC: 0.00%
Dec 12 11:19:11 kv-lorawangw-01 ttn-gateway[270]: # RF packets forwarded: 0 (0 bytes)
Dec 12 11:19:11 kv-lorawangw-01 ttn-gateway[270]: # PUSH_DATA datagrams sent: 1 (223 bytes)
Dec 12 11:19:11 kv-lorawangw-01 ttn-gateway[270]: # PUSH_DATA acknowledged: 0.00%
Dec 12 11:22:10 kv-lorawangw-01 ttn-gateway[270]: ### [DOWNSTREAM] ###
Dec 12 11:22:10 kv-lorawangw-01 ttn-gateway[270]: # PULL_DATA sent: 3 (33.33% acknowledged)
Dec 12 11:22:10 kv-lorawangw-01 ttn-gateway[270]: # PULL_RESP(onse) datagrams received: 0 (0 bytes)
Dec 12 11:22:10 kv-lorawangw-01 ttn-gateway[270]: # RF packets sent to concentrator: 0 (0 bytes)
Dec 12 11:22:10 kv-lorawangw-01 ttn-gateway[270]: # TX errors: 0
Dec 12 11:22:10 kv-lorawangw-01 ttn-gateway[270]: ### [GPS] ###
Dec 12 11:22:10 kv-lorawangw-01 ttn-gateway[270]: # Invalid gps time reference (age: 1576145950 sec)
Dec 12 11:22:10 kv-lorawangw-01 ttn-gateway[270]: # Manual GPS coordinates: latitude 55.71132, longitude 12.48321, altitude 17 m
Dec 12 11:22:10 kv-lorawangw-01 ttn-gateway[270]: ##### END #####
Dec 12 11:22:10 kv-lorawangw-01 ttn-gateway[270]: INFO: [up] PUSH_ACK for server router.eu.thethings.network received in 29 ms
Dec 12 11:22:10 kv-lorawangw-01 ttn-gateway[270]: INFO: [down] for server router.eu.thethings.network PULL_ACK received in 27 ms
Dec 12 11:22:10 kv-lorawangw-01 ttn-gateway[270]: INFO: [down] for server router.eu.thethings.network PULL_ACK received in 27 ms
Dec 12 11:22:10 kv-lorawangw-01 ttn-gateway[270]: INFO: [down] for server router.eu.thethings.network PULL_ACK received in 28 ms

**** End DK *********

I suppose it’s @TTN infrastructure related ,

/Bingo

My Sweden gateway just came Online :slight_smile:

I did nothing

The back end map (I looked at Global and UK) showed it ‘dropped’ ~20-25% of all gateways yesterday through this morning _ I see recovered and back on the map today so likley a console integration problem (again). Likley if functional the gateways would have continued to pass routed traffic, certainly that has been the case in the past, unless it was a UDP load issue again

I had been monitoring a few times a day looking to see if we hit 10K active GW’s - I saw it peak around 9987 (new record?) through the day then fell back to <7500!

mine’s back online, too.

It happened the same to mine yesterday evening around 6 PM. Back online this morning around 10 PM.

Do you see push and pull acks being received?

Actually, you are showing missing ACKs

RF packets forwarded: 1 (45 bytes)
PUSH_DATA datagrams sent: 2 (495 bytes)
PUSH_DATA acknowledged: 0.00%

It’s been a while since I’ve run that protocol long haul over the Internet, and it’s UDP anyway so perfect success should not be expected. But a lot of the logs posted in this thread are showing very high failure rates on getting protocol replies from the servers.

This probably does not identify the actual cause of failure within the infrastructure symptoms, but it does look like an indication that the gateways actually are not in ordinary communication with the infrastructure.

Actually all ok:

##### 2019-12-12 20:04:13 GMT #####
### [UPSTREAM] ###
# RF packets received by concentrator: 4
# CRC_OK: 100.00%, CRC_FAIL: 0.00%, NO_CRC: 0.00%
# RF packets forwarded: 4 (196 bytes)
# PUSH_DATA datagrams sent: 5 (1340 bytes)
# PUSH_DATA acknowledged: 100.00%
### [DOWNSTREAM] ###
# PULL_DATA sent: 3 (33.33% acknowledged)
# PULL_RESP(onse) datagrams received: 0 (0 bytes)
# RF packets sent to concentrator: 0 (0 bytes)
# TX errors: 0
### [GPS] ###
# Invalid gps time reference (age: 1576181053 sec)
# Manual GPS coordinates: latitude 45.42578, longitude 9.20145, altitude 0 m
##### END #####

and gateway returned connected on console…