[IMST IC880A + RPi] Gateway "hangs" every few weeks

I have a gateway consisting of an IMST IC880a and a RaspberryPi, connected via SPI and connected to a good (=2.5A) power source.
The gateway usually works well, but every month or so, it stops forwarding messages. The logfile /var/log/daemon.log shows no real problems (but is very verbose, so I might have missed something)

A reboot helps of course. Only doing a sudo service restart ttn-gateway also fixes the problem.

The day before yesterday the gateway seemed to not forward messages from two of my three nodes. Restarting the service fixed this once again.

Are other people experiencing similar problems?

Yes, I do. And @JeroenB did. My gateway runs since the end of december and has stopped about 5 times.
The Pi keeps running OK but the IC880A hangs. Status messages are still being send but the crc-errors on noise stop.
The ā€˜failuresā€™ were not evenly spread in time but happened 3 times in one week and the other two several weeks later.
JeroenB was having a lot more trouble but has resolved it (hopefully) by giving it an other case with proper shielding.
So thatā€™s my plan too. In time.

Iā€™m experiencing exactly the same. SPI communications with the iMST seems to be OK as no errors are logged. The symptom is that I do not receive any packets anymore. Simply restarting the ttn-gateway process ā€œsolvesā€ this issue.

I may also try to get a better case, and have a look at the receptions statistics: if no packets are received within some set time interval restart the packet forwarder. Not nice, but better than missing lots of packets.

Iā€™ll try a real case with some shielding (how much??). The thing is now ā€œmountedā€ in the cardboard box the Raspberry was mailed in.

I can calculate how much of my own packets I miss, but the gateway may be missing other persons packets too; thatā€™s harder to monitor.

Same here. Rpi with iC880a on top in a small plastic box. Would work for hours (at the office, noisy RF environment) or days (at home, less noisy RF environment), then stop receiving radio packets. No errors or anything, just not receiving anything:

RF packets received by concentrator: 0

A restart of the ttn-gateway service solves it temporarily.
I tried another iC880a with the same result.

Iā€™ve ordered an aluminum box, letā€™s see if that helps.

same issue here

Some guys in Zurich had the same problem, and as @arthoo said, the main suspect is the casing because this mostly happened with gateways that were not in a proper enclosure. We have other gateways using the plastic outdoor-ready enclosure that I show on the wiki and those have been running for several months already without issues.

In any case, is something weā€™ll try to start monitoring more in detail in the coming weeks.

A plastic enclosure working well, sounds like the problem is not about shielding for radio interference? Then I wonder how many people above use the gateway indoors, and still run into problems. Like:

ā€¦which is installed inside, I assume?

The cardboard enclosure is standing inside front of a window.
I think the shielding capacities of cardboard are similar to those of plastic, so it might not be a shielding problem.

That leaves the Raspberry itself, the power brick, the ic880a itself and the connections as problem sources.
The fact that restarting the gateway service and not the whole Raspberry suggests a problem with the ic880a or the software. Once I think I saw the gateway forwarding only 1 of my 3 nodes. This could also indicate problems in the ic880a.

Do these gateway components have updatable firmware?

Maybe you should put some shielding around the cardboardbox, just for the fun of it and see what happens.
Can we be sure the plastic box does not have any shielding capabilities? Is there carbon fiber in it?

Another option is to see whether the problem IC880aā€™s are all perpendicular to the antenna or have the antenna directly mounted to it or have a pigtail.
Mine has a pigtail and is laying mostly flat (antenna standing upright of course). => medium problems (~once a month)

Maybe it would be worth making some kind of script that runs

sudo service restart ttn-gateway

once in a while

I think it is possible to configure auto restart directly on the systemd service file with something like:

[Service]
-Restart=on-failure
+Restart=always
+Type=notify
+WatchdogSec=604800

But Iā€™d like to find out more about the root cause of this before going this route.

Cheers

The aluminum box helped quite a lot, but did not solve the problem. Uptime in the (RF-noisy) office went up from several hours to several days. For now cron restarts the ttn-gateway service (which resets the iC880a) every night.

Me too! Has anyone contacted IMST yet?

I did send a mail to the personal mail address of one of the people I had contact with before but did not yet receive a reply. I just sent a message via the contact form in the hope they will reply.

I wonder how it is with the Lorank8 gateways as they are based on the same board AFAIK. Anyone?

The Lorank8 uses the same IMST hardware, and does not have this problem at all. The software is the same as well.

The differences are BaegleBone vs Raspberry, and a (passive) PCB between the BeagleBone and the IMST board.

Thanks Thomas. Could it be ā€œflaky connectionsā€ due to using jumper wires instead of proper soldering or using real headers. However then I wouldnā€™t expect restarting only the software fixes it (although it does reset the iMST board, so maybe that does the trick). Also people use these jumper wires for lots of other prototypes successfully, so it seems unlikely to me (but with RF you never know ;-))

Got a reply from iMST:

Thanks for your Email.

Sometimes, customers had problems when catching an impulse on the reset line.

So iC880A is reset and linux host is still running.

We have started a long term test with several iC880A-USB to reproduce this problem.

We will keep you informed about the results.

For the further communication please use our Ticket System:

https://wireless-solutions.de/support/

This is more easy to handle for us.

Best Regards,

Heinz

Note that I am using the SPI version

I the beginning i had allot of problems, the gateway stops working after a few hours.
Then i build it into an old router case, which solves allmost all problems, but an sporadic hangup is always possible.
I hope iMST can find the cause of this problem !

2 Likes
  1. Would it be possible to get the drawings of that in-between PCB ? - Iā€™d like to take a look - I ā€œsuspectā€ it does the shielding between the 880 and the MCU.
  2. Also using shielded, mono microphone-style cables for connecting the 880 to the Pi (signal cables CLK, MISO, MOSI, NSS and RESET) should help (gut-feeling). This means soldering pins ā€¦ can be a bit tedious. Shields go to the signal GND (not power GND) on the Pi side and are left unconnected at the 880 end.
  3. SSTP Cat6a Cables can also be used. 2 cables are necessary (because each cable has 4 pairs of wires shielded by pair, and we will use only ONE wire in each pair - see this image). One cable will carry CLK, MISO, MOSI and NSS, for example, and the other RESET and 3 left unused. Again, shields and ALL unused core wires go to the GND on the Pi side and left unconnected on the 880 side.

In the earlier days, when we were active on the radio band we built something like this:

<img src="/forum/uploads/default/original/2X/4/47ffb293730ba431eeeacf99d7e18d8edba7da43.jpg" width=ā€œ690ā€ height=ā€œ459ā€
Cheap 2-layer pcb, excellent shielding!

3 Likes