Known issue: Gateways appear as offline

Because of an issue in our NOC component, gateways currently appear as offline on the console and on our maps. We are doing our best to have everything back to normal. Routing services are up and running.

4 Likes

Do the gateways only appear offline or are they actually offline?

1 Like

As I wrote in the first post:

What happened

The NOC

The NOC experienced unexpected timeouts while writing to its database. Instead of backing off, the NOC started even more processes trying to write to the database (an unintended positive feedback loop). This resulted in an explosion of goroutines and memory usage and eventually the NOC just kept crashing/restarting/…

The Proxy

We have an SSL-terminating proxy in front of the NOC. When the NOC crashed and restarted within a second, the proxy did not close existing connections

The Routing Services

Because the connections were not closed by the proxy, the routing services (router, broker, …) that forward metadata to the NOC did not back off to allow the NOC to recover, this also didn’t really help. Instead, they started buffering these messages, dumping a flood of messages onto the NOC when it came back after a crash. For some reason these components also spawned more goroutines, leading to extremely high memory usage and slowdown of message processing.

Mitigation

We temporarily disabled forwaring metadata to the NOC, but as a result the gateways now appear as offline on the console and maps.

Resolution

We are still working on reproducing the issue in a controlled environment, and will post an update when we know more. We aim to re-enable NOC forwarding within a couple of hours, after which the gateway pages should display the correct gateway status again.

7 Likes

We decided to push re-enabling the noc to tomorrow.

Until then, ttnctl gateways status [gatewayID] still shows the actual status of your gateways.

1 Like

Now miraculously my gateway appeared online :wink:

1 Like

Two of our gateways appear online now! Finallyyyy!

Mine too, yippee! :slight_smile:

Alex

Hello, is the problem solved? our gateway status is still “not connected”
Lora traffic is being send but the status in TTN is incorrect.

1 Like

Watch https://status.thethings.network/ for updates.

Thanx Marcelstoer

Are there any predictions about returning online on the map?

Thanks
Alex

The Gateway now appears in the console.

It still doesn’t appear when issuing CLI “ttnctl gateways status …” or shows as inactive with “ttnctl gateways info …”, which I normally would consider more accurate.

Is there any way to fix this?

Did you supply --router-id to ttnctl?

Actions done in the dashboard on gateway and devices are reported by ttnctl in real time? or there is a delay?

Thanks
Alex

ttnctl gateways info eui-ID
and
ttnctl gateways status eui-ID

both working for me when I enter the eui -ID of my gateway.

The problem was with --router-id, as we are using ttn-router-asia-se,

Thanks htdvisser!

This command seemed to work previously without this flag. Perhaps it was assigned a different router prior to the recent changes.