My TTOG dropped off my LAN at 06:40:01 this morning. My router couldn’t see the gateway, I couldn’t access via either SSH or web console and it wasn’t responding to a ping. The green ‘port ok’ light was illuminated on the PPoE injector.
I rebooted the gateway by unplugging the injector which brought the gateway back up, but obviously any unexpected downtime is worrying.
Syslog (/var/log/syslog) shows the following:
Jan 15 06:38:10 OutdoorAP ntpd_intres[677]: host name not found: 2.ubuntu.pool.ntp.org
Jan 15 06:38:10 OutdoorAP ntpd_intres[677]: host name not found: 3.ubuntu.pool.ntp.org
Jan 15 06:38:10 OutdoorAP ntpd_intres[677]: host name not found: ntp.ubuntu.com
Jan 15 06:38:16 OutdoorAP dhclient: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 10 (xid=0x70b4e676)
Jan 15 06:38:26 OutdoorAP dhclient: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 21 (xid=0x70b4e676)
Jan 15 06:38:47 OutdoorAP dhclient: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 18 (xid=0x70b4e676)
Jan 15 06:39:04 OutdoorAP kernel: [ 129.291309] usb 1-1.2: USB disconnect, device number 5
Jan 15 06:39:04 OutdoorAP kernel: [ 129.291597] option1 ttyUSB0: GSM modem (1-port) converter now disconnected from ttyUSB0
Jan 15 06:39:04 OutdoorAP kernel: [ 129.291672] option 1-1.2:1.0: device disconnected
Jan 15 06:39:04 OutdoorAP kernel: [ 129.291838] option: option_instat_callback : error -2
Jan 15 06:39:04 OutdoorAP kernel: [ 129.297260] option1 ttyUSB1: GSM modem (1-port) converter now disconnected from ttyUSB1
Jan 15 06:39:04 OutdoorAP kernel: [ 129.297370] option 1-1.2:1.1: device disconnected
Jan 15 06:39:04 OutdoorAP kernel: [ 129.297570] option: option_instat_callback : error -2
Jan 15 06:39:04 OutdoorAP kernel: [ 129.297881] option1 ttyUSB2: GSM modem (1-port) converter now disconnected from ttyUSB2
Jan 15 06:39:04 OutdoorAP kernel: [ 129.297966] option 1-1.2:1.2: device disconnected
Jan 15 06:39:04 OutdoorAP kernel: [ 129.298145] option: option_instat_callback : error -2
Jan 15 06:39:04 OutdoorAP kernel: [ 129.298356] option1 ttyUSB3: GSM modem (1-port) converter now disconnected from ttyUSB3
Jan 15 06:39:04 OutdoorAP kernel: [ 129.298423] option 1-1.2:1.3: device disconnected
Jan 15 06:39:05 OutdoorAP dhclient: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 14 (xid=0x70b4e676)
Jan 15 06:39:15 OutdoorAP cron[721]: (CRON) INFO (pidfile fd = 3)
Jan 15 06:39:16 OutdoorAP cron[756]: (CRON) STARTUP (fork ok)
Jan 15 06:39:16 OutdoorAP cron[756]: (CRON) INFO (Running @reboot jobs)
Jan 15 06:39:18 OutdoorAP kernel: [ 143.096640] usb 1-1.2: new high speed USB device number 6 using usb20_host
Jan 15 06:39:18 OutdoorAP /etc/mysql/debian-start[934]: Upgrading MySQL tables if necessary.
Jan 15 06:39:18 OutdoorAP kernel: [ 143.206439] usb 1-1.2: New USB device found, idVendor=2c7c, idProduct=0125
Jan 15 06:39:18 OutdoorAP kernel: [ 143.206467] usb 1-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
Jan 15 06:39:18 OutdoorAP kernel: [ 143.206493] usb 1-1.2: Product: Android
Jan 15 06:39:18 OutdoorAP kernel: [ 143.206508] usb 1-1.2: Manufacturer: Android
Jan 15 06:39:18 OutdoorAP kernel: [ 143.208072] option 1-1.2:1.0: GSM modem (1-port) converter detected
Jan 15 06:39:18 OutdoorAP kernel: [ 143.208568] usb 1-1.2: GSM modem (1-port) converter now attached to ttyUSB0
Jan 15 06:39:18 OutdoorAP kernel: [ 143.266251] option 1-1.2:1.1: GSM modem (1-port) converter detected
Jan 15 06:39:18 OutdoorAP kernel: [ 143.266660] usb 1-1.2: GSM modem (1-port) converter now attached to ttyUSB1
Jan 15 06:39:18 OutdoorAP kernel: [ 143.356284] option 1-1.2:1.2: GSM modem (1-port) converter detected
Jan 15 06:39:18 OutdoorAP kernel: [ 143.356883] usb 1-1.2: GSM modem (1-port) converter now attached to ttyUSB2
Jan 15 06:39:18 OutdoorAP /etc/mysql/debian-start[938]: /usr/bin/mysql_upgrade: the '--basedir' option is always ignored
Jan 15 06:39:18 OutdoorAP /etc/mysql/debian-start[938]: Looking for 'mysql' as: /usr/bin/mysql
Jan 15 06:39:18 OutdoorAP /etc/mysql/debian-start[938]: Looking for 'mysqlcheck' as: /usr/bin/mysqlcheck
Jan 15 06:39:18 OutdoorAP /etc/mysql/debian-start[938]: FATAL ERROR: Upgrade failed
Jan 15 06:39:18 OutdoorAP /etc/mysql/debian-start[953]: Checking for insecure root accounts.
Jan 15 06:39:18 OutdoorAP kernel: [ 143.436229] option 1-1.2:1.3: GSM modem (1-port) converter detected
Jan 15 06:39:18 OutdoorAP kernel: [ 143.436509] usb 1-1.2: GSM modem (1-port) converter now attached to ttyUSB3
Jan 15 06:39:19 OutdoorAP dhclient: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 11 (xid=0x70b4e676)
Jan 15 06:39:27 OutdoorAP kernel: [ 152.389014] init: cron main process (756) killed by TERM signal
Jan 15 06:39:27 OutdoorAP cron[1021]: (CRON) INFO (pidfile fd = 3)
Jan 15 06:39:27 OutdoorAP cron[1022]: (CRON) STARTUP (fork ok)
Jan 15 06:39:27 OutdoorAP cron[1022]: (CRON) INFO (Skipping @reboot jobs -- not system startup)
Jan 15 06:39:28 OutdoorAP kernel: [ 152.931538] tun: Universal TUN/TAP device driver, 1.6
Jan 15 06:39:28 OutdoorAP kernel: [ 152.931567] tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
Jan 15 06:39:30 OutdoorAP dhclient: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 9 (xid=0x70b4e676)
Jan 15 06:39:30 OutdoorAP kernel: [ 154.987587] init: cron main process (1022) killed by TERM signal
Jan 15 06:39:30 OutdoorAP cron[1257]: (CRON) INFO (pidfile fd = 3)
Jan 15 06:39:30 OutdoorAP cron[1258]: (CRON) STARTUP (fork ok)
Jan 15 06:39:30 OutdoorAP cron[1258]: (CRON) INFO (Skipping @reboot jobs -- not system startup)
Jan 15 06:39:39 OutdoorAP dhclient: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 20 (xid=0x70b4e676)
Jan 15 06:39:59 OutdoorAP dhclient: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 19 (xid=0x70b4e676)
Jan 15 06:40:01 OutdoorAP kernel: [ 185.804544] netlink: 20 bytes leftover after parsing attributes.
Jan 15 06:40:01 OutdoorAP kernel: [ 185.804577] netlink: 20 bytes leftover after parsing attributes.
Jan 15 06:40:01 OutdoorAP kernel: [ 185.804752] netlink: 20 bytes leftover after parsing attributes.
Jan 15 06:40:01 OutdoorAP kernel: [ 185.805069] netlink: 20 bytes leftover after parsing attributes.
Jan 15 06:40:01 OutdoorAP dhclient: receive_packet failed on eth0: Network is down
Jan 15 06:40:01 OutdoorAP CRON[3593]: (root) CMD (sleep 30; /etc/init.d/gtk_spilora.service0 start)
Jan 15 06:40:01 OutdoorAP CRON[3592]: (root) CMD (/usr/bin/service ntp stop;/usr/sbin/ntpd -gq;/usr/bin/service ntp start)
Jan 15 06:40:01 OutdoorAP CRON[3601]: (root) CMD (/usr/bin/monitor_dualwan.sh #monitor_dualwan service)
Jan 15 06:40:01 OutdoorAP CRON[3588]: (root) CMD (/app/prodtest/check_dns.sh)
Jan 15 06:40:01 OutdoorAP CRON[3602]: (root) CMD (sleep 30; /app/lora_pkg/lora_wdg_check.sh)
Jan 15 06:40:01 OutdoorAP CRON[3607]: (root) CMD (/usr/bin/monitor_wan_for_lora.sh)
Jan 15 06:40:01 OutdoorAP CRON[3543]: (CRON) info (No MTA installed, discarding output)
Jan 15 06:40:01 OutdoorAP ntpd[674]: ntpd exiting on signal 15
Jan 15 06:40:01 OutdoorAP rsyslogd: [origin software="rsyslogd" swVersion="7.4.4" x-pid="370" x-info="http://www.rsyslog.com"] exiting on signal 15.
Jan 15 10:59:17 OutdoorAP rsyslogd: [origin software="rsyslogd" swVersion="7.4.4" x-pid="369" x-info="http://www.rsyslog.com"] start
Jan 15 10:59:17 OutdoorAP rsyslogd: rsyslogd's groupid changed to 104
Jan 15 10:59:17 OutdoorAP rsyslogd: rsyslogd's userid changed to 101
Jan 15 10:59:17 OutdoorAP rsyslogd-2039: Could no open output pipe '/dev/xconsole': No such file or directory [try http://www.rsyslog.com/e/2039 ]
Jan 15 10:59:17 OutdoorAP kernel: [ 0.000000] Initializing cgroup subsys cpu
Jan 15 10:59:17 OutdoorAP kernel: [ 0.000000] Linux version 3.0.36+ (root@browan-PC) (gcc version 4.9.1 20140529 (prerelease) (crosstool-NG linaro-1.13.1-4.9-2014.06 - Linaro GCC 4.9-2014.06) ) #6 SMP PREEMPT Mon Mar 2 13:40:04 CST 2020
Jan 15 10:59:17 OutdoorAP kernel: [ 0.000000] CPU: ARMv7 Processor [413fc090] revision 0 (ARMv7), cr=10c5387d
Jan 15 10:59:17 OutdoorAP kernel: [ 0.000000] CPU: VIPT nonaliasing data cache, VIPT aliasing instruction cache
It appears that the issue may have been provoked by a CRON job. Do the above logs give any indications as to what went wrong and how I can prevent this issue from reoccurring the in the future?
Thanks again for everyone’s help.