Can't access TTN Console, ttnctl fails, TTN Node-RED library fails, and Kickstarter gateways get stuck after daily reboot

No access to GWs and some of them no traffic at all :frowning:

Check thethingsnetwork.slack.com -> ops channel

console seems to be down complete - getting this now:

502 Bad Gateway

nginx

Those are old messages

The same for me now. 502 bad gateway. I can not enter to the consoel.

502 bad gateway. the servers behind the nginx instance are down.
please fix it fast.

This also affects the Kickstarter The Things Gateway, if that does its daily reboot:

[2020-05-17T10:57:25.238Z] MAIN: Rebooting gateway for firmware update check
...
[2020-05-17T10:57:55.889Z] HTTP: Starting connection
[2020-05-17T10:57:55.900Z] HTTPS: Connection Opened: Starting TLS Negotiation
[2020-05-17T10:57:55.903Z] HTTP: Wait for TLS Connect
[2020-05-17T10:57:56.372Z] HTTP: TLS Connection Opened: Starting Clear Text Communication
...
[2020-05-17T10:58:56.383Z] HTTP: Got 341 bytes
[2020-05-17T10:58:58.385Z] HTTP: Connection Closed
[2020-05-17T10:58:58.392Z] HTTP: Close active socket 1
[2020-05-17T10:58:58.395Z] CONF: Parsing response token: HTTP/1.1 504 Gateway Time-out
[2020-05-17T10:58:58.397Z] ACTI: Object expected
[2020-05-17T10:58:58.399Z]
[2020-05-17T10:58:58.402Z] CNFG: Downloading gateway configuration failed
[2020-05-17T10:58:58.409Z]
[2020-05-17T10:58:58.421Z] CNFG: Load online user config state change to 8
[2020-05-17T10:58:58.423Z]
[2020-05-17T10:58:58.424Z] CNFG: Communication ERROR
...
[2020-05-17T10:59:03.424Z] HTTP: Starting connection
...
[2020-05-17T11:00:05.956Z] CONF: Parsing response token: HTTP/1.1 504 Gateway Time-out
[2020-05-17T11:00:05.961Z] ACTI: Object expected

After showing this error a total of 6 times, it did not try again, but after a final HTTP: Starting connection just got stuck with the 3rd LED slowly blinking (“Still activating”) and only logging stack sizes, heap usage and LoRa packets. Only when I rebooted manually a few minutes ago it showed the above errors again.

So: Kickstarter gateways might need a manual reboot when the server problems have been resolved.

Something is wrong witht he console this morning… I get a bad gateway error when trying to load the console.

Same here in eastern North America … 503 Gateway error. Gateways can’t forward packets.

The same problem in Brazil, since sunday 01h00

Same thing here in Australia, last error message:

502 Bad Gateway

nginx

Same thing here in Perth Australia, last error message:

502 Bad Gateway

nginx

Its been doing this all week so i think its a much larger on going issue. It’ll go down for 4 hrs or so then come back for a few hours. In saying that I haven’t been able to access it now for over a day.

Same here in Spain, Western Europe, since yesterday.

Same in Mexico since 8 hours ago! BAD GATEWAY!

My last received data was 24+ hours at 11:39 on 17/5/2020 (Western Australia). Getting Error 502 Bad Gateway from MESHED console. Any news on a fix?

the same I find here in Switzerland, I can’t access to my console and furthermore my Gateway indicates me that it is in the activating mode (3rd LED flashes slowly).
It seems that Sunday 17th of May at 18:29 (Zurich TIME) I received the last data.
Thank you for setting it up.

The same here in California. Have not had access for +24 hours. This is a showstopper for my project. Lack of response from TTN is even more worrisome!

1 Like

I rebooted one of my gateways Saturday - I hope I didn‘t delete the internet?! #wasntme

“Keep calm and don’t panic!” chaps. :slight_smile:

I confirm the issue having nothing fiddled on my (RAK2245) installation working flawless since months.

Browser failure message when trying to access the web console: “502 Bad Gateway” nginx

Checked via ttnmapper.org: according to “Last heard at 2020-05-18T02:02:37.000Z, Channels heard on: 8” my gateway itself is working as expected.

It seems to be an issue when trying to access the gateway configuration and status pages on the TTN webserver - maybe an user database handling/referencing failure or “console.thethingsnetwork.org” is simply down.

I might have it overlooked but is there a possibility to address incidents to a tracking system outside this forum?

The 502 Bad Gateway started yesterday afternoon for me. I’m on the Australian meshed console. It sounds like Europe was affected a bit later.

I have 2 gateways one has gone offline in TTN Mapper, other is still being seen.

I’m keeping an eye on the #ops channel in Slack to see if there are any updates from the TTN team when Europe wakes up for Monday morning.

1 Like