Overriding clksrc suddenly necessary for Multitech LoRa gateway

Hello,

We saw our Multitech LoRa (MTAC-LORA-H 868 card) MTCDTIP gateway fail to initialize the concentrator after a WAN outage a few weeks ago. The gateway was configured as a LoRa Basics station with TTN Sandbox at the time, hence configuration would have been automatically provisioned. Starting from scratch with a new Basics gateway entry from a new TTS (enterprise) account did not seem to help.

Looking at the logs, a cryptic message regarding firmware verification showed up (PoE brownout? Clock-skew? Unrelated issue with subsequent error-handling procedure?)

Click for preceding logs
INFO: [main] resetting lora **************
INFO: FPGA supported features: [TX filter]  [Spectral Scan] 
ERROR: Version of calibration firmware not expected, actual:4 expected:2
ERROR: [main] failed to start the concentrator, resetting lora and trying again
INFO: FPGA supported features: [TX filter]  [Spectral Scan] 
ERROR: Version of calibration firmware not expected, actual:4 expected:2
ERROR: [main] concentrator cannot be started, exiting
*** Beacon Packet Forwarder for Lora Gateway ***
Version: 4.0.17
*** Lora concentrator HAL library version info ***
Version: 5.0.11;
***
INFO: Little endian host
INFO: found global configuration file global_conf.json, parsing it
INFO: global_conf.json does contain a JSON object named SX1301_conf, parsing SX1301 parameters
INFO: lorawan_public 1, clksrc 1
INFO: no configuration for LBT
INFO: antenna_gain 0 dBi
INFO: Default Temperature Compensation
INFO: Configuring TX LUT with 16 indexes
INFO: radio 0 enabled (type SX1257), center frequency 867500000, RSSI offset -166.000000, tx enabled 1, tx_notch_freq 0
INFO: radio 1 enabled (type SX1257), center frequency 868500000, RSSI offset -166.000000, tx enabled 0, tx_notch_freq 0
INFO: Lora multi-SF channel 0>  radio 1, IF -400000 Hz, 125 kHz bw, SF 7 to 12
INFO: Lora multi-SF channel 1>  radio 1, IF -200000 Hz, 125 kHz bw, SF 7 to 12
INFO: Lora multi-SF channel 2>  radio 1, IF 0 Hz, 125 kHz bw, SF 7 to 12
INFO: Lora multi-SF channel 3>  radio 0, IF -400000 Hz, 125 kHz bw, SF 7 to 12
INFO: Lora multi-SF channel 4>  radio 0, IF -200000 Hz, 125 kHz bw, SF 7 to 12
INFO: Lora multi-SF channel 5>  radio 0, IF 0 Hz, 125 kHz bw, SF 7 to 12
INFO: Lora multi-SF channel 6>  radio 0, IF 200000 Hz, 125 kHz bw, SF 7 to 12
INFO: Lora multi-SF channel 7>  radio 0, IF 400000 Hz, 125 kHz bw, SF 7 to 12
INFO: Lora std channel> radio 1, IF -200000 Hz, 250000 Hz bw, SF 7
INFO: FSK channel> radio 1, IF 300000 Hz, 125000 Hz bw, 50000 bps datarate
INFO: global_conf.json does contain a JSON object named gateway_conf, parsing gateway parameters
INFO: gateway MAC address is configured to [redacted]
INFO: server hostname or IP address is configured to "eu1.cloud.thethings.network"
INFO: upstream port is configured to "1700"
INFO: downstream port is configured to "1700"
INFO: restart interval is configured to 600 minutes
INFO: duty cycle will NOT be enforced
INFO: duplicate packets received with low SNR will NOT be forwarded
INFO: packets received with a valid CRC will be forwarded
INFO: packets received with a CRC error will NOT be forwarded
INFO: packets received with no CRC will NOT be forwarded
INFO: [main] resetting lora **************
INFO: [main] resetting lora **************
INFO: FPGA supported features: [TX filter]  [Spectral Scan] 
ERROR: Version of calibration firmware not expected, actual:4 expected:2
ERROR: [main] failed to start the concentrator, resetting lora and trying again
INFO: FPGA supported features: [TX filter]  [Spectral Scan] 
ERROR: Version of calibration firmware not expected, actual:4 expected:2
ERROR: [main] concentrator cannot be started, exiting
*** Beacon Packet Forwarder for Lora Gateway ***
Version: 4.0.17
*** Lora concentrator HAL library version info ***
Version: 5.0.11;
***
INFO: Little endian host
INFO: found global configuration file global_conf.json, parsing it
INFO: global_conf.json does contain a JSON object named SX1301_conf, parsing SX1301 parameters
INFO: lorawan_public 1, clksrc 1
INFO: no configuration for LBT

I spent a while on this red-herring trying to replace the firmware through firmware upgrades and factory resets, however only upon doing diffs with older configuration backups did the culprit reveal itself (even though it was staring at me with INFO: lorawan_public 1, clksrc 1).

Screenshot_20240502_011322

INFO: [main] resetting lora **************
INFO: FPGA supported features: [TX filter]  [Spectral Scan] 
INFO: [main] concentrator started, packet can now be received

I eventually learned that while most boards follow Semtech reference design for SX1301 where the second SX1257 is used to clock the concentrator and the first radio, Multitech’s SX1301 card follows the SX1302/SX1303 clocking hierarchy where the first radio (radio 0) in the RF chain index is used as the clocking source.

radio_1 provides clock to concentrator for most devices except MultiTech. For MultiTech set to 0

As such, I now have to override the clksrc to 0 using packet forwarder mode with a manually edited config. Perhaps consider adding a gateway parameter for clksrc to the TTN/TTI CLI, so that I can set it up as a Basics station again.

For feature requests you need to log an issue on GitHub. TTI does not read the forum looking for feature requests.

If Multitech have introduced a hardware bug, perhaps they need to work on the fix rather than getting TTI to add a feature for just one variant of a gateway to fix someone else’s hardware design choices.

What does Multitech have to say?

1 Like

It is not a hardware bug, it’s a valid design decision that has been there since at least 2015. Packet forwarders have a configuration item to set the correct value, it is just the TT stack that does not seem to provide a way to set this value to the correct one for these gateways. All MultiTech gateways require the same setting.

I remain perplexed as to how the gateway worked without intervention prior to this. Does multitech firmware automatically override the value? Is there a specific heuristic on TTN’s end that the gateway would trigger during enrolment such that the correct value is sent? Was this an edge case resulting from an extended WAN outage?
This is genuinely the first time I’m seeing this crop up, despite the gateway being in service for a couple of years now on various research projects. I suspect that this might be a regression on my current firmware version.

Fair enough - but an interesting decision for Multitech to make as they are effectively preventing BS use on TTS. I guess that v2 was mostly PF so this wasn’t an issue then.

I guess the TTI team can triage it if an issue is created on GitHub - they do seem to have got to the niche stuff now.

BS allows configuration of the required parameter so the decision is perfectly valid (and the choice precedes all versions of the TTN stack). If the current stack can’t handle it I very much consider that a stack issue.
Both TTI and MultiTech have documentation on how to setup the hardware with BasicStation so if it doesn’t work fixes will be required…