All my gateways disconnected suddenly and none are re-connecting

digidhamu · August 29, 2024, 2:54pm

Issue Summary:

I’m facing a critical issue where all my LoRaWAN gateways have suddenly disconnected, and I haven’t been able to reconnect any of them. Despite trying multiple troubleshooting steps, including setting up a new instance of the LNS (LoRaWAN Network Server) on both DigitalOcean and AWS, none of the gateways are establishing a connection. This issue is persistent across all gateways, and I’m at a loss for what might be causing it.

Environment Details:

Deployment Method: Docker Compose
Operating System: Ubuntu 24.04
LNS (Open Source) Versions Tried: 3.29.x, 3.30.x, 3.31.1

Logs and Troubleshooting Attempts:

Unfortunately, I couldn’t find any relevant logs that might indicate the root cause of the problem. Neither the gateways nor the LNS instances are providing error messages or any logs that could point me in the right direction.

Hardware Involved:

Gateway Model: RAK7268 WisGate Edge Lite 2
Firmware Version: 1.3.9_RAK b186

Steps Taken So Far:

Rebooted Gateways: Rebooted all the gateways multiple times to see if they would reconnect.
LNS Re-deployment: Set up new instances of the LNS on DigitalOcean and AWS, using the same Docker Compose deployment method. I tried with different versions (3.29.x, 3.30.x, 3.31.1) but faced the same issue with each.
Network Checks: Verified that there are no network issues or firewall rules that could be blocking the connection between the gateways and the LNS.
Firmware Verification: Ensured that all gateways are running the correct firmware version (1.3.9_RAK b186) as recommended by RAKwireless.

Request for Help:

Has anyone encountered a similar issue or can provide insight into potential causes? Could it be a compatibility issue with the LNS versions or something related to the Docker Compose setup? Any suggestions on further troubleshooting steps or logs I should be looking into would be greatly appreciated.

descartes · August 29, 2024, 3:58pm

How many is all? 2, 10, 100??

What do you mean by critical error - testing has come to a halt or 10,000 puppies run the risk of starvation?

I’ve not seen any spontaneous bit-rot in TTS OS, something must have changed else where - like an auto-update of the gateways for instance - or a change in Basic Station config if that’s what you are running (are you running that, you don’t say).

As that’s a pretty popular gateway and no one else has reported issues with TTN, there must be something in your mixture that is causing this. So you might want to try putting a gateway on to TTN to see what happens.

digidhamu · August 30, 2024, 12:44am

Thanks, @descartes, for your reply.

The situation is critical because we have numerous sensors deployed across farms to monitor environmental parameters, and these readings directly influence feeding decisions based on the data collected.

I’ve ensured that automatic updates for the gateways are turned off, and there hasn’t been any physical or remote access that could have altered their configuration.

To troubleshoot, I connected the gateways to TTN (Enterprise with a Discovery Subscription), and they are working perfectly, with data flowing as expected.

Any further advice or insights would be greatly appreciated.

descartes · August 30, 2024, 8:46am

If you startup TTS OS from the command line you can see a blizzard of info - I’ve never looked for the gateway activity but that may be worth grepping.

What is the backhaul - the internet connection that the gateways use - could this have changed?

Can you run TTS OS in your office with one of these RAK gateways to make sure there are no firewall or other port issues. If there are, you can then debug that more easily.

Jeff-UK · August 30, 2024, 9:05am

Particularly look at what PF the GW is running, and check what your backhaul comms provider might have started filtering wrt Port numbers (a know historic issues with some service providers) though fact you can connect GW ok to TTN/TTS(Sandbox) suggests this may not be an issue…, check also any ingress Port filtering/set-up to your cloudy services for private LNS instance.

That plus on same network segment if internal network large or segmented! (Tip - I try at home network if I find a field or corporate deployment has a problem!)

digidhamu · August 30, 2024, 2:06pm

One thing I’ve noticed in the TTN OS Docker console’s backend logs is that there’s no inbound communication from any of the gateways to the server. This is quite unusual. The firewall configuration, which was working fine earlier, hasn’t been changed. For troubleshooting purposes, I’ve even allowed all connections to rule out any issues, but unfortunately, that didn’t resolve the problem.

Backhaul: WiFi (Office Network) and WiFi (Home Network) and both has been configured for this troubleshooting

Firewall Config

Jeff-UK · August 30, 2024, 5:19pm

Fine if local testing and LNS on same local network as GW but by backhaul what we are interested in is the internet connection service provision from the premise (Office/Home to “The Internet” and ultimately the cloud service where remote LNS is hosted), so equivalent to say BT, Virgin Media, Vodafone or any of >100 AltNets here in UK, Verizon or Comcast or… in US, France Telecom, Bouyuge Telecom, Free, Orange… in France, Swisscom, Deutch Telecom, Fastnet, SingTel, or whoever is your internet communication service provider in your geography

descartes · August 30, 2024, 9:37pm

This:

was this:

There are other items suggested that haven’t been explored yet:

Asked by Jeff as well.

Just asked by Jeff above.

This is the fundamental test that will eliminate TTS OS & the gateways. Then you can start extending the network in your office between the server & the LNS.

Please consider answering questions, it’s how we can figure out ways to help you. Otherwise it’s all guessing.

digidhamu · August 31, 2024, 2:16am

I have connected my home GW to the internet using ACT Fibrenet as the service provider, with the connection going through a Wi-Fi-Fiber router. The TTN OS LNS is hosted on AWS.

digidhamu · August 31, 2024, 2:28am

How many is all? 2, 10, 100??

Currently about 40 GWs

or a change in Basic Station config if that’s what you are running (are you running that, you don’t say)

No configuration changed

Can you run TTS OS in your office with one of these RAK gateways to make sure there are no firewall or other port issues. If there are, you can then debug that more easily.

I am very sure there is no firewall in between.

I was seeing these logs as you advised via command line. This logs are captured after applying the new token in GW and even I have restarted GW too.

INFO    Setting up core component
INFO    Setting up Identity Server
INFO    Identity Server telemetry is enabled. Check the documentation for more information on what is collected and how to disable it   {"documentation_url": "https://www.thethingsindustries.com/docs/reference/telemetry/identity_server", "namespace": "identityserver"}
INFO    Setting up Gateway Server
INFO    Setting up Network Server
INFO    Setting up Application Server
INFO    Setting up Join Server
INFO    Setting up Console
INFO    Setting up Gateway Configuration Server
INFO    Setting up Device Template Converter
INFO    Setting up QR Code Generator
INFO    Setting up Packet Broker Agent
INFO    Setting up Device Repository
INFO    Setting up Device Claiming Server
INFO    Starting...
WARN    No cluster key configured, generated a random one       {"key": "2f9f9a71a25a98f7043cc305a3e3d1afa5daae74e377d797e0c18e5a38ab9cf6", "namespace": "cluster"}
INFO    Listening for connections       {"address": ":1884", "namespace": "grpc", "protocol": "gRPC"}
INFO    Listening for connections       {"address": ":8884", "namespace": "grpc", "protocol": "gRPC/tls"}
INFO    Listening for connections       {"address": ":1885", "namespace": "web", "protocol": "Web"}
INFO    Listening for connections       {"address": ":8885", "namespace": "web", "protocol": "Web/tls"}
INFO    Listening for connections       {"address": ":1886", "namespace": "interop", "protocol": "Interop"}
INFO    Listening for connections       {"address": ":8886", "namespace": "interop", "protocol": "Interop/tls"}
INFO    Finished unary call     {"duration": 0.0016, "grpc.method": "AuthInfo", "grpc.service": "ttn.lorawan.v3.EntityAccess", "namespace": "grpc", "peer.address": "pipe", "peer.real_ip": "172.225.219.54", "request_id": "01J6K29ECAGN3035G8H9E63021"}
INFO    Request handled {"duration": 0.0079, "http.method": "GET", "http.path": "/api/v3/auth_info", "http.status": 200, "namespace": "web", "peer.address": "172.225.219.54:29587", "peer.real_ip": "172.225.219.54", "request_id": "01J6K29ECAGN3035G8H9E63021"}
INFO    Request handled {"duration": 0.0004, "http.method": "GET", "http.path": "/healthz", "http.status": 200, "namespace": "web", "peer.address": "[::1]:56072", "peer.real_ip": "::1", "request_id": "01J6K2B70E7VNE30NC1FN6KRQC"}
INFO    Request handled {"duration": 0.0001, "http.method": "GET", "http.path": "/healthz", "http.status": 200, "namespace": "web", "peer.address": "[::1]:56662", "peer.real_ip": "::1", "request_id": "01J6K2D1P53GD9XTZK3TX4P1F3"}
INFO    Request handled {"duration": 0.0002, "http.method": "GET", "http.path": "/healthz", "http.status": 200, "namespace": "web", "peer.address": "[::1]:38726", "peer.real_ip": "::1", "request_id": "01J6K2EWBBGBDRB2799R75RY2C"}
INFO    Request handled {"duration": 0.0001, "http.method": "GET", "http.path": "/healthz", "http.status": 200, "namespace": "web", "peer.address": "[::1]:56192", "peer.real_ip": "::1", "request_id": "01J6K2GQ0XZK7ASNPECT70N44E"}
INFO    Request handled {"duration": 0.0001, "http.method": "GET", "http.path": "/", "http.status": 302, "namespace": "web", "peer.address": "185.16.39.118:52216", "peer.real_ip": "185.16.39.118", "request_id": "01J6K2GSB2SQM62JH0ES14C7RQ"}
INFO    Request handled {"duration": 0.0001, "http.method": "GET", "http.path": "/healthz", "http.status": 200, "namespace": "web", "peer.address": "[::1]:47420", "peer.real_ip": "::1", "request_id": "01J6K2JHP2K49B41JNP6CY34DS"}
INFO    Request handled {"duration": 0.0001, "http.method": "GET", "http.path": "/healthz", "http.status": 200, "namespace": "web", "peer.address": "[::1]:44644", "peer.real_ip": "::1", "request_id": "01J6K2MCC0XM50HKECBESSGG2P"}
INFO    Request handled {"duration": 0.0001, "http.method": "GET", "http.path": "/healthz", "http.status": 200, "namespace": "web", "peer.address": "[::1]:37766", "peer.real_ip": "::1", "request_id": "01J6K2P7111FD0FZN57CSCKZA0"}
INFO    Request handled {"duration": 0.0001, "http.method": "GET", "http.path": "/healthz", "http.status": 200, "namespace": "web", "peer.address": "[::1]:40918", "peer.real_ip": "::1", "request_id": "01J6K2R1PDDM1Z5Z9PHKQBFCA5"}
INFO    Request handled {"duration": 0.0001, "http.method": "GET", "http.path": "/healthz", "http.status": 200, "namespace": "web", "peer.address": "[::1]:59794", "peer.real_ip": "::1", "request_id": "01J6K2SWBX36R3TDAX4KYTZMFX"}
INFO    Request handled {"duration": 0.0001, "http.method": "GET", "http.path": "/healthz", "http.status": 200, "namespace": "web", "peer.address": "[::1]:51870", "peer.real_ip": "::1", "request_id": "01J6K2VQ1HDDBMVD4QHRAC4Z55"}
INFO    Request handled {"duration": 0.0001, "http.method": "GET", "http.path": "/healthz", "http.status": 200, "namespace": "web", "peer.address": "[::1]:47654", "peer.real_ip": "::1", "request_id": "01J6K2XHPGXFWNQPPYYW722R0S"}
INFO    Request handled {"duration": 0.0001, "http.method": "GET", "http.path": "/healthz", "http.status": 200, "namespace": "web", "peer.address": "[::1]:42228", "peer.real_ip": "::1", "request_id": "01J6K2ZCCERV45E3ARQGNP23F5"}
INFO    Request handled {"duration": 0.0014, "http.method": "GET", "http.path": "/console/gateways/digidhamu-lorawan-02", "http.status": 200, "namespace": "web", "peer.address": "172.225.219.54:29231", "peer.real_ip": "172.225.219.54", "request_id": "01J6K2ZEK0W1MRATNJ4S74RQRV"}
INFO    Request handled {"auth.token_id": "2LCT2A4OTQ2PXU6URNOXEK5BU6RMTICJYGMRETI", "auth.token_type": "AccessToken", "duration": 719.8783, "http.method": "GET", "http.path": "/api/v3/console/internal/events/", "http.status": 200, "namespace": "web", "peer.address": "172.225.219.54:29964", "peer.real_ip": "172.225.219.54", "request_id": "01J6K29FMCRQ8038M6QJGDWSBX"}
INFO    Finished unary call     {"auth.token_id": "2LCT2A4OTQ2PXU6URNOXEK5BU6RMTICJYGMRETI", "auth.token_type": "AccessToken", "auth.user_id": "admin", "duration": 0.0034, "grpc.method": "AuthInfo", "grpc.service": "ttn.lorawan.v3.EntityAccess", "namespace": "grpc", "peer.address": "pipe", "peer.real_ip": "172.225.219.54", "request_id": "01J6K2ZEPYVWCFXJJJZW0BT4KM"}
INFO    Request handled {"auth.token_id": "2LCT2A4OTQ2PXU6URNOXEK5BU6RMTICJYGMRETI", "auth.token_type": "AccessToken", "duration": 0.0045, "http.method": "GET", "http.path": "/api/v3/auth_info", "http.status": 200, "namespace": "web", "peer.address": "172.225.219.54:29231", "peer.real_ip": "172.225.219.54", "request_id": "01J6K2ZEPYVWCFXJJJZW0BT4KM"}
INFO    Finished unary call     {"auth.token_id": "2LCT2A4OTQ2PXU6URNOXEK5BU6RMTICJYGMRETI", "auth.token_type": "AccessToken", "auth.user_id": "admin", "duration": 0.0062, "grpc.method": "Get", "grpc.request.user_id": "admin", "grpc.service": "ttn.lorawan.v3.UserRegistry", "namespace": "grpc", "peer.address": "pipe", "peer.real_ip": "172.225.219.54", "request_id": "01J6K2ZERCYANR1CY5XS8X8F0R"}
INFO    Request handled {"auth.token_id": "2LCT2A4OTQ2PXU6URNOXEK5BU6RMTICJYGMRETI", "auth.token_type": "AccessToken", "duration": 0.0081, "http.method": "GET", "http.path": "/api/v3/users/admin", "http.status": 200, "namespace": "web", "peer.address": "172.225.219.54:29231", "peer.real_ip": "172.225.219.54", "request_id": "01J6K2ZERCYANR1CY5XS8X8F0R"}
INFO    Request handled {"duration": 0.0025, "http.method": "GET", "http.path": "/assets/logo.svg", "http.status": 200, "namespace": "web", "peer.address": "172.225.219.54:29231", "peer.real_ip": "172.225.219.54", "request_id": "01J6K2ZESZYFZN7C7D428F3JHF"}
INFO    Finished unary call     {"auth.token_id": "2LCT2A4OTQ2PXU6URNOXEK5BU6RMTICJYGMRETI", "auth.token_type": "AccessToken", "auth.user_id": "admin", "duration": 0.006, "grpc.method": "ListRights", "grpc.request.gateway_id": "digidhamu-lorawan-02", "grpc.service": "ttn.lorawan.v3.GatewayAccess", "namespace": "grpc", "peer.address": "pipe", "peer.real_ip": "172.225.219.54", "request_id": "01J6K2ZETQEQGTXG32DR08N2BX"}
INFO    Request handled {"auth.token_id": "2LCT2A4OTQ2PXU6URNOXEK5BU6RMTICJYGMRETI", "auth.token_type": "AccessToken", "duration": 0.0075, "http.method": "GET", "http.path": "/api/v3/gateways/digidhamu-lorawan-02/rights", "http.status": 200, "namespace": "web", "peer.address": "172.225.219.54:29231", "peer.real_ip": "172.225.219.54", "request_id": "01J6K2ZETQEQGTXG32DR08N2BX"}
WARN    No encryption key defined, return stored LBS LNS Secret value   {"grpc.method": "Get", "grpc.service": "ttn.lorawan.v3.GatewayRegistry", "namespace": "identityserver", "peer.real_ip": "172.225.219.54", "request_id": "01J6K2ZEWAGZ7172GW3GQ2X3VR"}
INFO    Finished unary call     {"auth.token_id": "2LCT2A4OTQ2PXU6URNOXEK5BU6RMTICJYGMRETI", "auth.token_type": "AccessToken", "auth.user_id": "admin", "duration": 0.0138, "grpc.method": "Get", "grpc.request.gateway_id": "digidhamu-lorawan-02", "grpc.service": "ttn.lorawan.v3.GatewayRegistry", "namespace": "grpc", "peer.address": "pipe", "peer.real_ip": "172.225.219.54", "request_id": "01J6K2ZEWAGZ7172GW3GQ2X3VR"}
INFO    Request handled {"auth.token_id": "2LCT2A4OTQ2PXU6URNOXEK5BU6RMTICJYGMRETI", "auth.token_type": "AccessToken", "duration": 0.0157, "http.method": "GET", "http.path": "/api/v3/gateways/digidhamu-lorawan-02", "http.status": 200, "namespace": "web", "peer.address": "172.225.219.54:29231", "peer.real_ip": "172.225.219.54", "request_id": "01J6K2ZEWAGZ7172GW3GQ2X3VR"}
INFO    Finished unary call     {"auth.token_id": "2LCT2A4OTQ2PXU6URNOXEK5BU6RMTICJYGMRETI", "auth.token_type": "AccessToken", "auth.user_id": "admin", "duration": 0.0195, "error": "error:pkg/gatewayserver:not_connected (gateway `digidhamu-lorawan-02` not connected)", "error_correlation_id": "21ac0d0a164f4530bf076eca5622f4c9", "error_name": "not_connected", "error_namespace": "pkg/gatewayserver", "gateway_uid": "digidhamu-lorawan-02", "grpc.method": "GetGatewayConnectionStats", "grpc.request.gateway_id": "digidhamu-lorawan-02", "grpc.service": "ttn.lorawan.v3.Gs", "grpc_code": "NotFound", "namespace": "grpc", "peer.address": "pipe", "peer.real_ip": "172.225.219.54", "request_id": "01J6K2ZEZ2JJ8171PSSRHS7X6Y"}
INFO    Client error    {"auth.token_id": "2LCT2A4OTQ2PXU6URNOXEK5BU6RMTICJYGMRETI", "auth.token_type": "AccessToken", "duration": 0.0217, "http.method": "GET", "http.path": "/api/v3/gs/gateways/digidhamu-lorawan-02/connection/stats", "http.status": 404, "namespace": "web", "peer.address": "172.225.219.54:29231", "peer.real_ip": "172.225.219.54", "request_id": "01J6K2ZEZ2JJ8171PSSRHS7X6Y"}
INFO    Finished unary call     {"auth.token_id": "2LCT2A4OTQ2PXU6URNOXEK5BU6RMTICJYGMRETI", "auth.token_type": "AccessToken", "auth.user_id": "admin", "duration": 0.0404, "grpc.method": "ListCollaborators", "grpc.request.gateway_id": "digidhamu-lorawan-02", "grpc.service": "ttn.lorawan.v3.GatewayAccess", "namespace": "grpc", "peer.address": "pipe", "peer.real_ip": "172.225.219.54", "request_id": "01J6K2ZEZ1MGX88VRH525CK0Q4"}
INFO    Request handled {"auth.token_id": "2LCT2A4OTQ2PXU6URNOXEK5BU6RMTICJYGMRETI", "auth.token_type": "AccessToken", "duration": 0.0425, "http.method": "GET", "http.path": "/api/v3/gateways/digidhamu-lorawan-02/collaborators", "http.status": 200, "namespace": "web", "peer.address": "172.225.219.54:29231", "peer.real_ip": "172.225.219.54", "request_id": "01J6K2ZEZ1MGX88VRH525CK0Q4"}
INFO    Finished unary call     {"auth.token_id": "2LCT2A4OTQ2PXU6URNOXEK5BU6RMTICJYGMRETI", "auth.token_type": "AccessToken", "auth.user_id": "admin", "duration": 0.0428, "grpc.method": "ListAPIKeys", "grpc.request.gateway_id": "digidhamu-lorawan-02", "grpc.service": "ttn.lorawan.v3.GatewayAccess", "namespace": "grpc", "peer.address": "pipe", "peer.real_ip": "172.225.219.54", "request_id": "01J6K2ZEZ1Y9541083C2BZJ9FQ"}
INFO    Request handled {"auth.token_id": "2LCT2A4OTQ2PXU6URNOXEK5BU6RMTICJYGMRETI", "auth.token_type": "AccessToken", "duration": 0.0458, "http.method": "GET", "http.path": "/api/v3/gateways/digidhamu-lorawan-02/api-keys", "http.status": 200, "namespace": "web", "peer.address": "172.225.219.54:29231", "peer.real_ip": "172.225.219.54", "request_id": "01J6K2ZEZ1Y9541083C2BZJ9FQ"}
INFO    Finished unary call     {"auth.token_id": "2LCT2A4OTQ2PXU6URNOXEK5BU6RMTICJYGMRETI", "auth.token_type": "AccessToken", "auth.user_id": "admin", "duration": 0.0082, "grpc.method": "List", "grpc.service": "ttn.lorawan.v3.ApplicationRegistry", "namespace": "grpc", "peer.address": "pipe", "peer.real_ip": "172.225.219.54", "request_id": "01J6K2ZMC48W75K7TRYD9SR3V9"}
INFO    Request handled {"auth.token_id": "2LCT2A4OTQ2PXU6URNOXEK5BU6RMTICJYGMRETI", "auth.token_type": "AccessToken", "duration": 0.0094, "http.method": "GET", "http.path": "/api/v3/applications", "http.status": 200, "namespace": "web", "peer.address": "172.225.219.54:29231", "peer.real_ip": "172.225.219.54", "request_id": "01J6K2ZMC48W75K7TRYD9SR3V9"}
INFO    Finished unary call     {"auth.token_id": "2LCT2A4OTQ2PXU6URNOXEK5BU6RMTICJYGMRETI", "auth.token_type": "AccessToken", "auth.user_id": "admin", "duration": 0.0096, "grpc.method": "List", "grpc.service": "ttn.lorawan.v3.GatewayRegistry", "namespace": "grpc", "peer.address": "pipe", "peer.real_ip": "172.225.219.54", "request_id": "01J6K2ZN72SA0J2YZ3157QTA1K"}
INFO    Request handled {"auth.token_id": "2LCT2A4OTQ2PXU6URNOXEK5BU6RMTICJYGMRETI", "auth.token_type": "AccessToken", "duration": 0.0107, "http.method": "GET", "http.path": "/api/v3/gateways", "http.status": 200, "namespace": "web", "peer.address": "172.225.219.54:29231", "peer.real_ip": "172.225.219.54", "request_id": "01J6K2ZN72SA0J2YZ3157QTA1K"}
INFO    Finished unary call     {"auth.token_id": "2LCT2A4OTQ2PXU6URNOXEK5BU6RMTICJYGMRETI", "auth.token_type": "AccessToken", "auth.user_id": "admin", "duration": 0.0038, "grpc.method": "AssertRights", "grpc.service": "ttn.lorawan.v3.GatewayBatchAccess", "namespace": "grpc", "peer.address": "pipe", "request_id": "01J6K2ZNAJGTANPQ8J9HT2R7KS"}
INFO    Finished unary call     {"auth.token_id": "2LCT2A4OTQ2PXU6URNOXEK5BU6RMTICJYGMRETI", "auth.token_type": "AccessToken", "duration": 0.0048, "grpc.method": "BatchGetGatewayConnectionStats", "grpc.service": "ttn.lorawan.v3.Gs", "namespace": "grpc", "peer.address": "pipe", "peer.real_ip": "172.225.219.54", "request_id": "01J6K2ZN8NDNVKHWA9WAQFSYA1"}
INFO    Request handled {"auth.token_id": "2LCT2A4OTQ2PXU6URNOXEK5BU6RMTICJYGMRETI", "auth.token_type": "AccessToken", "duration": 0.0664, "http.method": "POST", "http.path": "/api/v3/gs/gateways/connection/stats", "http.status": 200, "namespace": "web", "peer.address": "172.225.219.54:29231", "peer.real_ip": "172.225.219.54", "request_id": "01J6K2ZN8NDNVKHWA9WAQFSYA1"}
INFO    Request handled {"duration": 0.0001, "http.method": "GET", "http.path": "/healthz", "http.status": 200, "namespace": "web", "peer.address": "[::1]:48646", "peer.real_ip": "::1", "request_id": "01J6K3171RKSDJ9A6KBVXNM548"}
INFO    Request handled {"duration": 0.0001, "http.method": "GET", "http.path": "/healthz", "http.status": 200, "namespace": "web", "peer.address": "[::1]:48094", "peer.real_ip": "::1", "request_id": "01J6K331Q57XV463Q4ZS0DVFEP"}
INFO    Request handled {"duration": 0.0002, "http.method": "GET", "http.path": "/healthz", "http.status": 200, "namespace": "web", "peer.address": "[::1]:42246", "peer.real_ip": "::1", "request_id": "01J6K34WCZF4AZDY244AYX7XKB"}
INFO    Request handled {"duration": 0.0002, "http.method": "GET", "http.path": "/healthz", "http.status": 200, "namespace": "web", "peer.address": "[::1]:47458", "peer.real_ip": "::1", "request_id": "01J6K36Q1ZJKRXHCA6AJYQHB4J"}
INFO    Request handled {"duration": 0.0001, "http.method": "GET", "http.path": "/healthz", "http.status": 200, "namespace": "web", "peer.address": "[::1]:34154", "peer.real_ip": "::1", "request_id": "01J6K38HQ17QNPQJ06MBHKT4J2"}
INFO    Request handled {"duration": 0.0001, "http.method": "GET", "http.path": "/healthz", "http.status": 200, "namespace": "web", "peer.address": "[::1]:37400", "peer.real_ip": "::1", "request_id": "01J6K3ACCEZGFF5W0GJ90YY5K3"}
INFO    Request handled {"duration": 0, "http.method": "GET", "http.path": "/", "http.status": 302, "namespace": "web", "peer.address": "3.92.25.217:35442", "peer.real_ip": "3.92.25.217", "request_id": "01J6K3AKB9RRJEENKBKQC9WYXC"}
INFO    Request handled {"duration": 0.0001, "http.method": "GET", "http.path": "/healthz", "http.status": 200, "namespace": "web", "peer.address": "[::1]:47326", "peer.real_ip": "::1", "request_id": "01J6K3C71G3D114GZBT1MQGY55"}
INFO    Request handled {"duration": 0.0001, "http.method": "GET", "http.path": "/healthz", "http.status": 200, "namespace": "web", "peer.address": "[::1]:59660", "peer.real_ip": "::1", "request_id": "01J6K3E1QA5A65M5K0S3PM8PDN"}
INFO    Request handled {"duration": 0.0002, "http.method": "GET", "http.path": "/healthz", "http.status": 200, "namespace": "web", "peer.address": "[::1]:52380", "peer.real_ip": "::1", "request_id": "01J6K3FWCBT2KV667CZ333DKK1"}
INFO    Request handled {"duration": 0.0001, "http.method": "GET", "http.path": "/healthz", "http.status": 200, "namespace": "web", "peer.address": "[::1]:42898", "peer.real_ip": "::1", "request_id": "01J6K3HQ24KQHH37SE7P47TCW3"}

netstat -nlt output

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
tcp        0      0 127.0.0.54:53           0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:1883            0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:1882            0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:1881            0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:1887            0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:1886            0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:1885            0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:1884            0.0.0.0:*               LISTEN     
tcp        0      0 127.0.0.1:5432          0.0.0.0:*               LISTEN     
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN     
tcp        0      0 127.0.0.1:6379          0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:8883            0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:8882            0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:8881            0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:8887            0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:8886            0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:8885            0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:8884            0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN     
tcp6       0      0 :::443                  :::*                    LISTEN     
tcp6       0      0 :::1883                 :::*                    LISTEN     
tcp6       0      0 :::1882                 :::*                    LISTEN     
tcp6       0      0 :::1881                 :::*                    LISTEN     
tcp6       0      0 :::1887                 :::*                    LISTEN     
tcp6       0      0 :::1886                 :::*                    LISTEN     
tcp6       0      0 :::1885                 :::*                    LISTEN     
tcp6       0      0 :::1884                 :::*                    LISTEN     
tcp6       0      0 :::8883                 :::*                    LISTEN     
tcp6       0      0 :::8882                 :::*                    LISTEN     
tcp6       0      0 :::8881                 :::*                    LISTEN     
tcp6       0      0 :::8887                 :::*                    LISTEN     
tcp6       0      0 :::8886                 :::*                    LISTEN     
tcp6       0      0 :::8885                 :::*                    LISTEN     
tcp6       0      0 :::8884                 :::*                    LISTEN     
tcp6       0      0 :::80                   :::*                    LISTEN     
tcp6       0      0 :::22                   :::*                    LISTEN

Hope I have answered all the questions asked.

Curious that I could not see 8887 port listening for connection in the command line log but I could see the port 8887 appearing in netstat.

Is this an issue in the configuration in YAML?

digidhamu · September 5, 2024, 7:00am

@descartes @Jeff-UK
Any direction or help is much appreciated?

digidhamu · September 7, 2024, 8:31am

I have also tried using TTN OS version 3.32.0 and am encountering the same issue. Additionally, I was able to connect the same gateway with TTI using the Discovery License.

It would be helpful if someone could provide insight into what might be going wrong.

descartes · September 7, 2024, 9:18am

Have you tried setting it all up on the same network in your office - following the notes on using ip addresses and not localhost?

That would be the simplest setup and easy to debug.

I’d also try a different brand of gateway. This is why everyone should own a TTIG or a WaterGate.

It sure would, but it’s 99.9% volunteer end-users on here of which probably 0.9% run TTS OS and this is the official support channel for it - WYSIWYG!

I use TTS OS on the basis that I try not to breath on it - not because it’s unstable, some say that’s me, but because I lack the detailed knowledge of Docker, networking, tracing traffic etc etc to be sure of knowing what I’m doing if anything goes wrong.

I’ve found with an instance in the office and one out on Linode that as long as I keep it up to date that it just works.

I send clients off to use a Discovery instance which they upgrade to a full one when they start using it for real. That €/£/$200 a month is about the same as me charging for admin on a TTS OS instance but I can’t do lots of them and I’d be in big trouble if I got out of my depth. So for that money the client gets the actual developers of TTS to run everything for them.

Now, god forbid as it’s not my thing, I had a client with tens of thousands of devices, I’d still recommend the TTI offering because they provide 24/7 cover - which needs four members of staff scattered around the globe which, because of holidays & sickness, means you need to have eight. There just is no scale where it is more economic to run your own instance unless you already have enterprise level systems then it’s just another part of the server array.

There is the potential of you paying for support with TTI. But as 8 days ago I suggested you run a setup in your office and we don’t have results yet, they too may find that getting things actioned will be hard. If you buy a support contract, they aren’t obliged as such to replicate your setup with the same hardware, so I strongly recommend you try what’s suggested first.