Latest MultiTech packet forwarder stops sending packets


(Ssozonoff) #1

So I just had a strange situation where the packet forwarder stopped sending/processing packets!

MT firmware 1.4.1 with the latest released packet forwarder.

The packet forwarder was still running according to the linux process list but nothing was happening and no recent activity in the packet log. TTN console showed an ever increasing "last seen" value for the gateway. Internet connectivity checked and is fine.

Following a stop/start of the packet forwarder everything resumed as normal.

Very strange


(Ssozonoff) #2

Happened again this morning.... and again a simple restart of the packet forwarded and we were up and running again.


#3

Could you stop the packet forwarder using:
/etc/init.d/ttn-pkt-forwarder stop
and then from your terminal run:
/opt/lora/mp_pkt_fwd -c /var/config/lora
This will start the process without output redirection making it easier to view the last lines of output without a buffer. Drawback is you have to keep the terminal session open or the forwarder will stop working.

It would help if you could post the last few lines of output you observe when the issue occurs.


(Rustie0125) #4

Hi kersing

Im getting the same issue, will try your suggestion and post lines


(Rustie0125) #5
Here is the lines. ttn says it has not seen gateway in 3 minutes 


# TX errors: 0
### BEACON IS DISABLED!
### [JIT] ###
# INFO: JIT queue contains 0 packets.
# INFO: JIT queue contains 0 beacons.
### GPS IS DISABLED!
### [PERFORMANCE] ###
# Upstream radio packet quality: 0.00%.
# Semtech status report send.
##### END #####
INFO: [status] TTN send status success
INFO: [up] TTN lora packet send to server "bridge.eu.thethings.network"

##### 2017-05-12 11:38:51 GMT #####
### [UPSTREAM] ###
# RF packets received by concentrator: 1
# CRC_OK: 100.00%, CRC_FAIL: 0.00%, NO_CRC: 0.00%
# RF packets forwarded: 1 (15 bytes)
# PUSH_DATA datagrams sent: 0 (0 bytes)
# PUSH_DATA acknowledged: 0.00%
### [DOWNSTREAM] ###
# PULL_DATA sent: 0 (0.00% acknowledged)
# PULL_RESP(onse) datagrams received: 0 (0 bytes)
# RF packets sent to concentrator: 0 (0 bytes)
# TX errors: 0
### BEACON IS DISABLED!
### [JIT] ###
# INFO: JIT queue contains 0 packets.
# INFO: JIT queue contains 0 beacons.
### GPS IS DISABLED!
### [PERFORMANCE] ###
# Upstream radio packet quality: 100.00%.
# Semtech status report send.
##### END #####
INFO: [status] TTN send status success

(Rustie0125) #6

O and how do i fix it again to run the gateway without the terminal being open ?


#7

To run the gateway without the terminal run /etc/init.d/ttn-pkt-forwarder start

Was the time listed in the message within the 3 minutes?


(Rustie0125) #8

mmm not sure what you asking, the messages also stops when the gateway stops. looking at the terminal the packet forwarding log also stops. can send it restart command and everything works again for another few minutes. also not a set amount of time. last session lasted 10 min before stopping.


#9

Can you run 'ps ax' and post the result? Also run 'dmesg' and post post the last 20 lines of output?


(Ssozonoff) #10

Since the issue is unpredictable (at least from my experience so far) maybe one option is to run the process in a screen session. I will see if I can get screen installed on the Conduit.


(Rustie0125) #11
root@mtcdt:~# ps ax
  PID TTY      STAT   TIME COMMAND
    1 ?        Ss     0:05 init [5]
    2 ?        S      0:00 [kthreadd]
    3 ?        S      0:00 [ksoftirqd/0]
    5 ?        S<     0:00 [kworker/0:0H]
    6 ?        S      0:00 [kworker/u2:0]
    7 ?        S<     0:00 [khelper]
    8 ?        S      0:00 [kdevtmpfs]
    9 ?        S<     0:00 [writeback]
   10 ?        S<     0:00 [bioset]
   11 ?        S<     0:00 [crypto]
   12 ?        S<     0:00 [kblockd]
   13 ?        S      0:00 [khubd]
   23 ?        S<     0:00 [rpciod]
   25 ?        S      0:00 [khungtaskd]
   26 ?        S      0:00 [kswapd0]
   27 ?        S      0:00 [fsnotify_mark]
   28 ?        S<     0:00 [nfsiod]
   29 ?        S<     0:00 [cifsiod]
   62 ?        S<     0:00 [deferwq]
   74 ?        SN     0:00 [jffs2_gcd_mtd6]
   84 ?        Ss     0:00 /lib/udev/udevd -d
  153 ?        S      0:00 /lib/udev/udevd -d
  190 ?        S      0:00 [kworker/u2:2]
  250 ?        SN     0:11 [jffs2_gcd_mtd8]
  379 ?        Ss     0:00 udhcpc -R -n -p /var/run/udhcpc.eth0.pid -i eth0
  387 ?        Ss     0:00 /usr/bin/dbus-daemon --system
  394 ?        Ss     0:00 /usr/sbin/sshd
  404 ?        S      0:00 /sbin/syslogd -n -O /var/log/messages -s 512 -b 4
  406 ?        S      0:00 /sbin/klogd -n
  453 ?        S      0:00 /lib/udev/udevd -d
  457 ?        S      0:32 /usr/sbin/mosquitto -d -c /etc/mosquitto/mosquitto.co
  460 ?        Ss     0:00 /usr/sbin/crond
  498 ?        Sl    64:23 /opt/lora/mp_pkt_fwd -c /var/config/lora -l /var/log/
  504 ?        S      0:00 /bin/bash /usr/sbin/reset-handler
  511 ttyS0    Ss+    0:00 /sbin/getty 115200 ttyS0
  512 ttyGS0   Ss+    0:00 /sbin/getty 115200 ttyGS0
  742 ?        S      0:00 [kworker/0:2]
  749 ?        S      0:00 [kworker/0:0]
  753 ?        S      0:00 [kworker/0:1]
  757 ?        Ss     0:00 sshd: root@pts/0
  760 pts/0    Ss     0:00 -sh
  761 pts/0    R+     0:00 ps ax



root@mtcdt:~# dmesg
TCP: reno registered
UDP hash table entries: 256 (order: 0, 4096 bytes)
UDP-Lite hash table entries: 256 (order: 0, 4096 bytes)
NET: Registered protocol family 1
RPC: Registered named UNIX socket transport module.
RPC: Registered udp transport module.
RPC: Registered tcp transport module.
RPC: Registered tcp NFSv4.1 backchannel transport module.
NFS: Registering the id_resolver key type
Key type id_resolver registered
Key type id_legacy registered
jffs2: version 2.2. (NAND) (SUMMARY)  © 2001-2006 Red Hat, Inc.
ROMFS MTD (C) 2007 Red Hat, Inc.
msgmni has been set to 496
Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252)
io scheduler noop registered (default)
io scheduler deadline registered
io scheduler cfq registered
fffff200.serial: ttyS0 at MMIO 0xfffff200 (irq = 16, base_baud = 8333333) is a ATMEL_SERIAL
console [ttyS0] enabled
f8040000.serial: ttyS1 at MMIO 0xf8040000 (irq = 26, base_baud = 8333333) is a ATMEL_SERIAL
brd: module loaded
loop: module loaded
at24_get_setup: found a match for eeprom at 56
at24 0-0056: 512 byte 24c04 EEPROM, writable, 16 bytes/write
sam9x5: read 512 bytes from id eeprom
at24_get_setup: found a match for eeprom at 50
at24 1-0050: 512 byte 24c04 EEPROM, writable, 16 bytes/write
sam9x5: ap1 eeprom read failed: -110
at24_get_setup: found a match for eeprom at 52
at24 1-0052: 512 byte 24c04 EEPROM, writable, 16 bytes/write
sam9x5: read 512 bytes from ap2 eeprom
atmel_nand 40000000.nand: Using dma0chan0 for DMA transfers.
ONFI param page 0 valid
ONFI flash detected
NAND device: Manufacturer ID: 0x2c, Chip ID: 0xaa (Micron MT29F2G08ABBEAHC), 256MiB, page size: 2048, OOB size: 64
atmel_nand 40000000.nand: ONFI params, minimum required ECC: 4 bits in 512 bytes
atmel_nand 40000000.nand: Initialize PMECC params, cap: 4, sector: 512
Scanning device for bad blocks
9 ofpart partitions found on MTD device atmel_nand
Creating 9 MTD partitions on "atmel_nand":
0x000000000000-0x000010000000 : "NANDFlash"
0x000000000000-0x000000040000 : "at91bootstrap"
0x000000040000-0x0000000c0000 : "u-Boot"
0x0000000c0000-0x000000160000 : "u-Boot Config"
0x000000160000-0x000000200000 : "u-Boot Redundant Config"
0x000000200000-0x000000800000 : "uImage"
0x000000800000-0x000001000000 : "Config"
0x000001000000-0x000001800000 : "OEM Config"
0x000001800000-0x000010000000 : "Rootfs"
libphy: MACB_mii_bus: probed
macb f802c000.ethernet eth0: Cadence MACB at 0xf802c000 irq 23 (00:08:00:4a:01:ae)
macb f802c000.ethernet eth0: attached PHY driver [Generic PHY] (mii_bus:phy_addr=f802c000.etherne:00, irq=-1)
PPP generic driver version 2.4.2
PPP BSD Compression module registered
PPP Deflate Compression module registered
ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
ehci-atmel: EHCI Atmel driver
atmel-ehci 700000.ehci: EHCI Host Controller
atmel-ehci 700000.ehci: new USB bus registered, assigned bus number 1
atmel-ehci 700000.ehci: irq 31, io mem 0x00700000
atmel-ehci 700000.ehci: USB 2.0 started, EHCI 1.00
usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
usb usb1: Product: EHCI Host Controller
usb usb1: Manufacturer: Linux 3.12.27 ehci_hcd
usb usb1: SerialNumber: 700000.ehci
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 3 ports detected
ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
at91_ohci 600000.ohci: AT91 OHCI
at91_ohci 600000.ohci: new USB bus registered, assigned bus number 2
at91_ohci 600000.ohci: irq 31, io mem 0x00600000
usb usb2: New USB device found, idVendor=1d6b, idProduct=0001
usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
usb usb2: Product: AT91 OHCI
usb usb2: Manufacturer: Linux 3.12.27 ohci_hcd
usb usb2: SerialNumber: at91
hub 2-0:1.0: USB hub found
hub 2-0:1.0: 3 ports detected
rtc (null): invalid alarm value: 1900-1-1 0:0:0
at91_rtc fffffeb0.rtc: rtc core: registered fffffeb0.rtc as rtc0
at91_rtc fffffeb0.rtc: AT91 Real Time Clock driver.
i2c /dev entries driver
at91sam9_wdt: enabled (heartbeat=15 sec, nowayout=0)
Netfilter messages via NETLINK v0.30.
nf_conntrack version 0.5.0 (3968 buckets, 15872 max)
ipip: IPv4 over IPv4 tunneling driver
gre: GRE over IPv4 demultiplexor driver
ip_gre: GRE over IPv4 tunneling driver
IPv4 over IPSec tunneling driver
ip_tables: (C) 2000-2006 Netfilter Core Team
TCP: cubic registered
Initializing XFRM netlink socket
NET: Registered protocol family 10
mip6: Mobile IPv6
sit: IPv6 over IPv4 tunneling driver
NET: Registered protocol family 17
NET: Registered protocol family 15
l2tp_core: L2TP core driver, V2.0
Key type dns_resolver registered
at91_rtc fffffeb0.rtc: setting system clock to 2017-05-14 04:35:08 UTC (1494736508)
usb 1-2: new high-speed USB device number 2 using atmel-ehci
usb 1-2: New USB device found, idVendor=0424, idProduct=2534
usb 1-2: New USB device strings: Mfr=0, Product=0, SerialNumber=0
hub 1-2:1.0: USB hub found
hub 1-2:1.0: 4 ports detected
usb 2-3: new full-speed USB device number 2 using at91_ohci
usb 2-3: New USB device found, idVendor=04e2, idProduct=1412
usb 2-3: New USB device strings: Mfr=0, Product=0, SerialNumber=0
usb 1-2.2: new high-speed USB device number 3 using atmel-ehci
usb 1-2.2: New USB device found, idVendor=0403, idProduct=6014
usb 1-2.2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
usb 1-2.2: Product: Single RS232-HS
usb 1-2.2: Manufacturer: FTDI
jffs2: notice: (1) jffs2_build_xattr_subsystem: complete building xattr subsystem, 0 of xdatum (0 unchecked, 0 orphan) and 0 of xref (0 dead, 0 orphan) found.
VFS: Mounted root (jffs2 filesystem) readonly on device 31:8.
devtmpfs: mounted
Freeing unused kernel memory: 168K (c053d000 - c0567000)
jffs2: notice: (73) jffs2_build_xattr_subsystem: complete building xattr subsystem, 0 of xdatum (0 unchecked, 0 orphan) and 0 of xref (0 dead, 0 orphan) found.
jffs2: notice: (73) jffs2_build_xattr_subsystem: complete building xattr subsystem, 0 of xdatum (0 unchecked, 0 orphan) and 0 of xref (0 dead, 0 orphan) found.
udevd[84]: starting version 182
tmp102 0-0048: initialized
atmel_usba_udc 500000.gadget: MMIO registers at 0xf803c000 mapped at d0922000
atmel_usba_udc 500000.gadget: FIFO at 0x00500000 mapped at e1100000
atmel_spi f0000000.spi: version: 0x212
atmel_spi f0000000.spi: Using dma0chan1 (tx) and dma0chan2 (rx) for DMA transfers
atmel_spi f0000000.spi: Atmel SPI Controller at 0xf0000000 (irq 28)
atmel_spi f0000000.spi: master is unqueued, this is deprecated
atmel_spi f0004000.spi: version: 0x212
atmel_spi f0004000.spi: Using dma1chan0 (tx) and dma1chan1 (rx) for DMA transfers
atmel_spi f0004000.spi: Atmel SPI Controller at 0xf0004000 (irq 29)
atmel_spi f0004000.spi: master is unqueued, this is deprecated
pps_core: LinuxPPS API ver. 1 registered
pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
usbcore: registered new interface driver cdc_acm
cdc_acm: USB Abstract Control Model driver for USB modems and ISDN adapters
usbcore: registered new interface driver usbserial
usbcore: registered new interface driver usbserial_generic
usbserial: USB Serial support registered for generic
vizzini 2-3:1.0: This device cannot do calls on its own. It is not a modem.
vizzini 2-3:1.0: ttyXRUSB0: XR21v14x usb uart device
pps pps0: new PPS source pps.7.-1
pps pps0: Registered IRQ 32 as PPS source
atmel_mci f0008000.mmc: version: 0x504
atmel_mci f0008000.mmc: using dma0chan3 for DMA transfers
atmel_mci f0008000.mmc: Atmel MCI controller at 0xf0008000 irq 22, 1 slots
vizzini 2-3:1.2: This device cannot do calls on its own. It is not a modem.
vizzini 2-3:1.2: ttyXRUSB1: XR21v14x usb uart device
usbcore: registered new interface driver vizzini
vizzini: Exar USB UART Driver for XR21V141x
usbcore: registered new interface driver ftdi_sio
usbserial: USB Serial support registered for FTDI USB Serial Device
ftdi_sio 1-2.2:1.0: FTDI USB Serial Device converter detected
usb 1-2.2: Detected FT232H
usb 1-2.2: Number of endpoints 2
usb 1-2.2: Endpoint 1 MaxPacketSize 512
usb 1-2.2: Endpoint 2 MaxPacketSize 512
usb 1-2.2: Setting MaxPacketSize 512
usb 1-2.2: FTDI USB Serial Device converter now attached to ttyUSB0
[INFO] mts-io:mts_io_init:829: init: v1.1.1
[INFO] mts-io:mts_id_eeprom_load:754: detected board MTCDT-0.0
[INFO] mts-io:mts_id_eeprom_load:757: sizeof: 512
[INFO] mts-io:mts_id_eeprom_load:758: vendor-id: Multi-Tech Systems
[INFO] mts-io:mts_id_eeprom_load:759: product-id: MTCDT-210L
[INFO] mts-io:mts_id_eeprom_load:760: device-id: 18213182
[INFO] mts-io:mts_id_eeprom_load:761: hw-version: MTCDT-0.0
[INFO] mts-io:mts_id_eeprom_load:768: mac-addr: 00:08:00:4A:01:AE
[INFO] mts-io:mts_id_eeprom_load:770: imei:
[INFO] mts-io:mts_id_eeprom_load:771: capa-gps: no
[INFO] mts-io:mts_id_eeprom_load:772: capa-din: no
[INFO] mts-io:mts_id_eeprom_load:773: capa-dout: no
[INFO] mts-io:mts_id_eeprom_load:774: capa-adc: no
[INFO] mts-io:mts_id_eeprom_load:775: capa-wifi: no
[INFO] mts-io:mts_id_eeprom_load:776: capa-bluetooth: no
[INFO] mts-io:mts_id_eeprom_load:800: uuid: 00000000000000000000000000000000
[INFO] mts-io:load_port:653: no accessory card inserted in port 1
[INFO] mts-io:load_port:688: accessory card 2 vendor-id: Multi-Tech Systems
[INFO] mts-io:load_port:689: accessory card 2 product-id: MTAC-LORA-868
[INFO] mts-io:load_port:690: accessory card 2 device-id: 18211113
[INFO] mts-io:load_port:691: accessory card 2 hw-version: MTAC-LORA-1.0
[INFO] mts-io:load_port:712: accessory card 2 eui: 00:80:00:00:00:00:9A:B0
[INFO] mts-io:lora_setup:42: loading LORA accessory card in port 2
IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
macb f802c000.ethernet eth0: link up (100/Full)
IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
g_serial gadget: Gadget Serial v2.4
g_serial gadget: g_serial ready
ftdi_sio ttyUSB0: FTDI USB Serial Device converter now disconnected from ttyUSB0
ftdi_sio 1-2.2:1.0: device disconnected

The gateway was running fine for 1 day now and just stopped working again. above is the results the commands


#12

Could you add a line to your local_conf.json file?

"logger": true,

That should show information regarding individual packets in the logging.


(Ssozonoff) #13

I have just had the problem again after over 2 days of no issues. I will add the logging as you suggest and see if I can get some more info. My gut feeling at the moment is it is somehow related to a node connecting. I say this because during my testing where I had a single node coming and going off the gateway I saw the problem more frequently. No I have been away for the weekend with no nodes connected and the gateway was "seen" moments ago. The minute I connected my node to try some new code I worked on over the weekend I noticed it was not able to join. Sure enough the "seen" time started increasing past a minute and I had to restart the packet forwarder to get things working again.

So I would see to try and reproduce the problem one should have only one node sending to the gateway and then maybe make it come and go with varying intervals of time..... just a thought


(Arjan) #14

one should have only one node connected to the gateway

Just to be sure: nodes do not "connect to a gateway". They just transmit and hope one or more gateways receive their packet and will forward that to TTN.


(Ssozonoff) #15

Correct, my bad. I was thinking about OTAA. Having said this I cant say at the moment if it might be OTAA related or not but I do use OTAA for the node I am testing.


(Arjan) #16

Well... Same goes for OTAA: a node transmits a Join Request (uplink), and hopefully one or more gateways forward that to TTN. If accepted, after 5 or 6 seconds TTN will command one of the gateways to transmit a Join Accept with the new secrets (downlink). Hopefully one or more nodes receive the gateway's transmission, but only the one that transmitted the original Join Request will be able to decipher it. Again, no "connection".

If ABP works, you could try if the gateway fails after being told to send a downlink. (Or an ACK, or some ADR command.) Maybe it already fails after receiving the downlink command. Maybe while processing it. Maybe it does not return to listening mode after transmitting.

In TTN Console you can look at the gateway's "traffic" page to see what TTN has received and what it has commanded the gateway to transmit.


(Mattkrebs) #17

I have the exact same problem. Since I have installed the new packet forwarder, the gateways stop working within a couple of minutes. I do not think the problem is directly related to OTAA, because no data ist received from activated nodes either. However, it may have something to do with downlinks, because the last message showing up in traffic was indeed a downlink.

Here is the dmesg output:

vizzini 2-3:1.2: ttyXRUSB1: XR21v14x usb uart device
usbcore: registered new interface driver vizzini
vizzini: Exar USB UART Driver for XR21V141x
usb 1-2.2: FTDI USB Serial Device converter now attached to ttyUSB0
[INFO] mts-io:mts_io_init:829: init: v1.1.1
[INFO] mts-io:mts_id_eeprom_load:754: detected board MTCDT-0.0
[INFO] mts-io:mts_id_eeprom_load:757: sizeof: 512
[INFO] mts-io:mts_id_eeprom_load:758: vendor-id: Multi-Tech Systems
[INFO] mts-io:mts_id_eeprom_load:759: product-id: MTCDT-H5-210L
[INFO] mts-io:mts_id_eeprom_load:760: device-id: 18751642
[INFO] mts-io:mts_id_eeprom_load:761: hw-version: MTCDT-0.0
[INFO] mts-io:mts_id_eeprom_load:768: mac-addr: 00:08:00:4A:0A:FF
[INFO] mts-io:mts_id_eeprom_load:770: imei: 356136071886214
[INFO] mts-io:mts_id_eeprom_load:771: capa-gps: no
[INFO] mts-io:mts_id_eeprom_load:772: capa-din: no
[INFO] mts-io:mts_id_eeprom_load:773: capa-dout: no
[INFO] mts-io:mts_id_eeprom_load:774: capa-adc: no
[INFO] mts-io:mts_id_eeprom_load:775: capa-wifi: no
[INFO] mts-io:mts_id_eeprom_load:776: capa-bluetooth: no
[INFO] mts-io:mts_id_eeprom_load:800: uuid: 00000000000000000000000000000000
[INFO] mts-io:load_port:653: no accessory card inserted in port 1
[INFO] mts-io:load_port:688: accessory card 2 vendor-id: Multi-Tech Systems
[INFO] mts-io:load_port:689: accessory card 2 product-id: MTAC-LORA-868
[INFO] mts-io:load_port:690: accessory card 2 device-id: 18948415
[INFO] mts-io:load_port:691: accessory card 2 hw-version: MTAC-LORA-1.0
[INFO] mts-io:load_port:712: accessory card 2 eui: 00:80:00:00:A0:00:01:CC
[INFO] mts-io:lora_setup:42: loading LORA accessory card in port 2
IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
macb f802c000.ethernet eth0: link up (100/Full)
IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
g_serial gadget: Gadget Serial v2.4
g_serial gadget: g_serial ready
ftdi_sio ttyUSB0: FTDI USB Serial Device converter now disconnected from ttyUSB0
ftdi_sio 1-2.2:1.0: device disconnected

According to netstat -a, the connection to the router/bridge is still open:

tcp        0      0 xxxxxxxxxxxxxxxx:51353 52.169.76.203:1883      ESTABLISHED

ps -ef:

UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 14:08 ?        00:00:05 init [5]
root         2     0  0 14:08 ?        00:00:00 [kthreadd]
root         3     2  0 14:08 ?        00:00:00 [ksoftirqd/0]
root         5     2  0 14:08 ?        00:00:00 [kworker/0:0H]
root         6     2  0 14:08 ?        00:00:00 [kworker/u2:0]
root         7     2  0 14:08 ?        00:00:00 [khelper]
root         8     2  0 14:08 ?        00:00:00 [kdevtmpfs]
root         9     2  0 14:08 ?        00:00:00 [writeback]
root        10     2  0 14:08 ?        00:00:00 [bioset]
root        11     2  0 14:08 ?        00:00:00 [crypto]
root        12     2  0 14:08 ?        00:00:00 [kblockd]
root        13     2  0 14:08 ?        00:00:00 [khubd]
root        23     2  0 14:08 ?        00:00:00 [rpciod]
root        25     2  0 14:08 ?        00:00:00 [khungtaskd]
root        26     2  0 14:08 ?        00:00:00 [kswapd0]
root        27     2  0 14:08 ?        00:00:00 [fsnotify_mark]
root        28     2  0 14:08 ?        00:00:00 [nfsiod]
root        29     2  0 14:08 ?        00:00:00 [cifsiod]
root        62     2  0 14:09 ?        00:00:00 [deferwq]
root        74     2  0 14:09 ?        00:00:00 [jffs2_gcd_mtd6]
root        84     1  0 14:09 ?        00:00:00 /lib/udev/udevd -d
root       196     2  0 14:09 ?        00:00:00 [kworker/u2:2]
root       199    84  0 14:09 ?        00:00:00 /lib/udev/udevd -d
root       201    84  0 14:09 ?        00:00:00 /lib/udev/udevd -d
root       269     2  0 14:09 ?        00:00:11 [jffs2_gcd_mtd8]
root       397     1  0 14:09 ?        00:00:00 udhcpc -R -n -p /var/run/udhcpc.eth0.pid -i eth0
999        407     1  0 14:09 ?        00:00:00 /usr/bin/dbus-daemon --system
root       414     1  0 14:09 ?        00:00:00 /usr/sbin/sshd
root       423     1  0 14:09 ?        00:00:00 /sbin/syslogd -n -O /var/log/messages -s 512 -b 4
root       426     1  0 14:09 ?        00:00:00 /sbin/klogd -n
root       478     1  0 14:09 ?        00:00:00 /usr/sbin/crond
root       522     1  0 14:09 ?        00:00:00 /bin/bash /usr/sbin/reset-handler
root       528     1  0 14:09 ttyS0    00:00:00 /sbin/getty 115200 ttyS0
root       529     1  0 14:09 ttyGS0   00:00:00 /sbin/getty 115200 ttyGS0
root       548   414  0 14:12 ?        00:00:03 sshd: root@pts/0
root       551   414  0 14:12 ?        00:00:01 sshd: root@notty
root       552   548  0 14:12 pts/0    00:00:00 -sh
root       554   551  0 14:12 ?        00:00:00 /usr/lib/openssh/sftp-server
root       629     1 15 14:33 ?        00:08:06 /opt/lora/mp_pkt_fwd -c /var/config/lora -l /var/log/lora-pkt-fwd.log
root       648     2  0 14:42 ?        00:00:00 [kworker/0:0]
root       687     2  0 15:10 ?        00:00:00 [kworker/0:1]
root       706   552  0 15:27 pts/0    00:00:00 ps -ef

It seems the output of /var/log/lora-pkt-fwd.log also stopped:

# PULL_DATA sent: 0 (0.00% acknowledged)
# PULL_RESP(onse) datagrams received: 0 (0 bytes)
# RF packets sent to concentrator: 3 (0 bytes)
# TX errors: 0
# TX rejected (collision packet): 0.00% (req:4, rej:0)
# TX rejected (collision beacon): 0.00% (req:4, rej:0)
# TX rejected (too late): 0.00% (req:4, rej:0)
# TX rejected (too early): 0.00% (req:4, rej:0)
### BEACON IS DISABLED!
### [JIT] ###
# INFO: JIT queue contains 1 packets.
# INFO: JIT queue contains 0 beacons.
### GPS IS DISABLED!
### [PERFORMANCE] ###
# Upstream radio packet quality: 80.00%.
# Semtech status report send.
##### END #####
INFO: [status] TTN send status success
INFO: tx_start_delay=1495 (1495.500000) - (1497, bw_delay=1.500000, notch_delay=0.000000)
INFO: [up] TTN lora packet send to server "bridge.eu.thethings.network"
INFO: [down] TTN received downlink with payload
INFO: [TTN] downlink 12 bytes
INFO: tx_start_delay=1495 (1495.500000) - (1497, bw_delay=1.500000, notch_delay=0.000000)
INFO: [up] TTN lora packet send to server "bridge.eu.thethings.network"
INFO: [down] TTN received downlink with payload
INFO: [TTN] downlink 12 bytes
INFO: tx_start_delay=1495 (1495.500000) - (1497, bw_delay=1.500000, notch_delay=0.000000)
INFO: [up] TTN lora packet send to server "bridge.eu.thethings.network"
INFO: [up] TTN lora packet send to server "bridge.eu.thethings.network"
INFO: [down] TTN received downlink with payload
INFO: [TTN] downlink 12 bytes
INFO: tx_start_delay=1495 (1495.500000) - (1497, bw_delay=1.500000, notch_delay=0.000000)
INFO: [up] TTN lora packet send to server "bridge.eu.thethings.network"
INFO: [down] TTN received downlink with payload
INFO: [TTN] downlink 12 bytes
INFO: tx_start_delay=1495 (1495.500000) - (1497, bw_delay=1.500000, notch_delay=0.000000)
INFO: [up] TTN lora packet send to server "bridge.eu.thethings.network"
INFO: [down] TTN received downlink with payload
INFO: [TTN] downlink 12 bytes
INFO: tx_start_delay=1495 (1495.500000) - (1497, bw_delay=1.500000, notch_delay=0.000000)

INFO: Disabling GPS mode for concentrator's counter...
INFO: host/sx1301 time offset=(1494851599s:180515µs) - drift=2126µs
INFO: Enabling GPS mode for concentrator's counter.


##### 2017-05-15 12:41:22 GMT #####
### [UPSTREAM] ###

I hope this output helps...


#18

It looks like there might be an issue with the USB bus access. I'll create a new binary for USB devices and update the package tomorrow.


#19

A new package (r3) is available on github. Requires manual download and install with

opkg install < filename >

Once installed edit local_conf.json and add

    "flush_line": true,

This will push the log to 'disk' after every line of output. Not recommended for busy gateway as it will have performance impact, but better for the curious that want to see the logging while things are happening in stead of minutes later.


(Mattkrebs) #20

Thank you very much.
I have installed it on my test gateway and I will report if there are still issues.