RP3 x IC880a gateway stopped working

Could it be an early life component(*) failure? Have you been able to check/test by substitution? - Swap RPi, Swap i/f board, swap concentrator bd , swap PSU (per Ryan suggestion), etc. Minor/early pin/connector corrosion - try reseating/cleaning with switch cleaner etc., just more for the checklist! :wink:

Also where in the world are you - parts of EMEA having a heatwave in recent days/weeks so is it temp related? May explain why fine originally but now a month later failing…and indeed once failing may continue to fail due to (*) :slight_smile:

Can the issue possibly be SD card (wear/malfuntion) related?

You could try redo from scratch with a different microSD card.

@BoRRoZ, I understand the problem comes from the SPI bus is not running at the expected 8MHz even though the #define SPI_SPEED 8000000 instruction was issued. This is caused by the variable VPU clock rate in the Pi3. As the VPU changes frequency the SPI speed changes. At the RP3 default VPU clock rate you correct by setting the SPI for 5MHz to obtain the required 8 MHz SPI speed. I also understand even with this correction the VPU can still change speed and the connection to the Lora chips stops working and a different correction factor is required. I went about this a different way, rather trying to account for this error I locked the VPU frequency by adding the line “core_freq=250” to /boot/config.txt By doing this the #define SPI_SPEED 8000000 instruction has the correct effect and sets the SPI speed to 8MHz.

it worked in my RP3B+ gateway with a RAK831

(some more detail I found. The VPU can switch from 250MHz to 400MHz in Turbo mode and will drop back under some conditions. The SPI clock rate is linked to the VPU frequency and as it steps up to 400MHz the SPI speed increases. Hence setting SPI to 8MHz means its actually running at 12.8MHz when the VPU is in Turbo mode, so setting it to 5MHz means its actually running at 8MHz (5MHz * 400/250 = 8MHz). The core_freq=250 instructions stops the VPU going into Turbo mode and the SPI speed is then set as expected)

1 Like

Following on from my previous post, the RP3B+ will reduce CPU frequency as the Pi overheats. It may be the daytime ambient temperature that’s driving this, the reason it worked for a month then stopped. You may find it works at night but not during the day. If the enclosure housing the gateway’s electronics is exposed to direct sunlight (eg clamped to the antenna pole) you can expect temperatures inside the enclosure to be 20C higher than ambient air temperature and the chip temperature of the Pi is higher again. We experience ambient air temp (shade temp) of 45C, so think of this ALL the time.

1 Like

Hm, that might be a very good point. My gateway stopped working last Friday and had auto security updates enabled… I haven’t found the time so far to debug it (it’s mounted on the roof and I don’t have power for a monitor up there).

Use Putty and ssh into the Pi from a PC. No need to fall off the roof trying to juggle a screen and keyboard :smiley:

1 Like

I’ve updated however all of mine use a zero or B+ to avoid the SPI issues (And I have an abundance of older models).
I don’t have a traditional backplane to use with my RPi 3 so can’t debug further sorry.

Of course. The issue is that I don’t have direct access (it’s not my network). Instead, the Pi does a reverse SSH tunnel to a server of mine through which I can connect. That tunnel closed down on Friday and did not re-open after a power cycle, so something must be wrong.

It could be that one of the disk partitions is 100 percent full. Most likely the /var/log partition.

1 Like

Ohh, that makes it difficult. Sounds like a site visit is about to occur.

Could be, although the Pi uses Systemd/Journal which does logrotation (with size limits) by default. But maybe some process writes to /var/log directly…

Jep :slight_smile: Unfortunately thunderstorms don’t help (I usually try to avoid opening the enclosure if the humidity is >55-60%RH).

I have installed around 80 Raspi in a range of industrial applications and part of my standard configuration is to run a Cron job to restart the Pi at 2am every day. Never had a problem with a Pi when I do this. They just keep on working, never lock up. Applied this in a Pi1 with the first operating system and right through to a Pi3B+ with systemd.

2 Likes

Could it be that logging to ramdrive is used and logging rotation expects more size than available?

It turned out to be a defective SD card, kernel could not boot…

The file system already broke three weeks prior, but I assumed that simply re-flashing the image will do the job (and did just that). Now that the file system was broken again, I assume it was a defect on the SD card level.

It was one of those cards shipped with the RPi. I hope that the Sandisk Ultra will do better… :thinking:

IRC the ones that ship are Sandisk. Some tips may be to move logging into a RAM directory to produce less writes. Another trick I’ve heard of is that a bigger SD card capacity can last longer as while it doesn’t need say a 16GB Card because it can handle more writes it can last slightly longer or such.

Flash memory cell wear can cause different kinds of symptoms, depending on which files are affected. Therefore it may not be directly clear that problems are wear related.

Assuming that the flash wear levelling management performed by the on-card memory controller is perfect (which I doubt) this would imply that there is a linear relation between flash capacity and wear out life expectancy. So a twice larger capacity would mean a twice longer lifetime.
This seems like a logical assumption, but I think the reality will be (much) more complex. So I wonder how much increasing the capacity will actually result in longer correct operation of the flash card. Will it be more linear or will the gains be much less?

1 Like

Its not clear if SD Cards actually implement Wear Levelling like implemented in a USB. SD cards were created for cameras which write to the same location far less than a Pi. One option is to boot from a USB another is to boot the Pi from a server and not have either an SD Card nor USB. if you stick with the SD Card, you can implement all writing is to RAM disk and the SD Card is write protected so the Pi never writes to the SD card at all.

here is a link to protecting an SD card from sudden power loss https://www.raspberrypi.org/forums/viewtopic.php?t=161416