EU TTn Server Down?

Hi,

I’ve noticed that my 3 Gateways are “Not Connected” since yesterday for 2 of them and since 3 hours for the third one.
2 of them are on my local home network : 1 Rpi3 and 1 Kona Macro form Tektelic
One is at my office : this one is a Kona Macro.

Thoses gateways have routed millions messages for now and it’s the first time i see them down since the bug of October.

What I can say:
A traceroute to router.eu.thethings.network (NS resolved) fails after 104.44.11.72 :

15 ae122-0.icr02.dub07.ntwk.msn.net (104.44.11.72) 208.849 ms ae102-0.icr02.dub07.ntwk.msn.net (104.44.11.56) 208.152 ms ae120-0.icr01.dub07.ntwk.msn.net (104.44.11.76) 208.388 ms
*16 * * **

The logs from packet_forwarder displays a good rythm of uploads but I can see the message :
"INFO: [down] for server router.eu.thethings.network PULL_ACK received in 206 ms"

I don’t know if the port TCP/1700 is opened on the servers but I can’t telnet to them.

The TTn Status for Europe is Green.

Does anyone else has the same problem ?
I haven’t noticed any recent thread talking about this.

Thanks all,

Nicolas

I performed also a trace, after 29 hops i quit the trace, may be a loop in the path. to the server. Also a ping times out.

Johan

Ok JohanD, thank you for your feedback.

It’s very strange that there is so few people concerned, I’m even suspecting my provider to filter this UDP1700 port …
But if I’m not the only one concerned, it’s perhaps a problem on the path as you suppose in your reply.

Nicolas

OK I haven’t tested this before but it seems that the packets are still reaching to the applications, despite the gateways are “Down”.
No big problem so, but something is quite strange anyway …

nicolas

with me the same, multiple gateways have the status not connected, but send data, but can not see anything in traffic.

Hi Nick, we are experiencing the same issue in Barranquilla (Colombia), we have 3 routers in the city and all of them are down since yesterday.

Initially, i believe than was a problem with my router, but, others parnerts in the city have the same issue.

OK Marlon thanks.

Are you also on the EU routers ?

Hi Nick,

Yes, i use that router because is nearest of my city, Brazil have routers but i saw than the response time is higher.

Nick

Can you test your solution with other router, FE: Australia or asia?
tks in advanced.

Hi all,

This link shows than everything is ok.
im a little confused.

https://status.thethings.network/

ok, the problem, if problem is, seems stucked to EU.

I have tried to add an OTAA device on an existaing App in my console, but it fails. i can see the join messages but nothing else.
A new ABP device works well.

Can you try a traceroute from your gateway or from your station to see where it stops ?

Here is mine :

traceroute to router.eu.thethings.network (52.169.76.203), 30 hops max, 60 byte packets

[The first ones are not interesting]

5 78.247.5.109.rev.sfr.net (109.5.247.78) 176.427 ms 176.341 ms 177.148 ms
6 198.244.5.109.rev.sfr.net (109.5.244.198) 181.105 ms 126.29.3.109.rev.sfr.net (109.3.29.126) 178.411 ms 178.463 ms
7 ae62-0.ier02.mrs20.ntwk.msn.net (104.44.14.16) 190.167 ms 189.181 ms 189.625 ms
8 ae26-0.icr01.mrs20.ntwk.msn.net (104.44.238.113) 189.228 ms 190.007 ms ae25-0.icr02.mrs20.ntwk.msn.net (104.44.238.117) 189.146 ms
9 be-120-0.ibr02.mrs20.ntwk.msn.net (104.44.20.63) 206.962 ms 206.921 ms 207.769 ms
10 be-13-0.ibr02.gva20.ntwk.msn.net (104.44.19.222) 207.858 ms be-3-0.ibr01.par30.ntwk.msn.net (104.44.7.47) 203.892 ms be-10-0.ibr01.gva20.ntwk.msn.net (104.44.19.220) 207.847 ms
11 be-4-0.ibr01.zrh20.ntwk.msn.net (104.44.18.50) 207.852 ms be-2-0.ibr02.zrh20.ntwk.msn.net (104.44.17.254) 208.228 ms be-2-0.ibr01.zrh20.ntwk.msn.net (104.44.17.252) 207.362 ms
12 be-6-0.ibr01.fra21.ntwk.msn.net (104.44.18.79) 208.707 ms 207.358 ms be-6-0.ibr02.fra21.ntwk.msn.net (104.44.18.87) 207.970 ms
13 be-8-0.ibr01.ams30.ntwk.msn.net (104.44.19.234) 207.650 ms be-9-0.ibr02.ams30.ntwk.msn.net (104.44.19.236) 208.149 ms 208.158 ms
14 be-7-0.ibr02.dub07.ntwk.msn.net (104.44.17.58) 207.889 ms * be-7-0.ibr01.dub07.ntwk.msn.net (104.44.17.56) 206.863 ms
15 ae102-0.icr02.dub07.ntwk.msn.net (104.44.11.56) 206.913 ms 206.765 ms ae100-0.icr01.dub07.ntwk.msn.net (104.44.11.60) 206.510 ms
16 * * *
17 * * *

Two gateways here are also reported to be down.
Both on completely different internet providers, so it cannot be filtering by the provider.
Also on the TTNmapper I see a lot of gateways offline in the neighborhood.

My gateway: Last heard at 2019-10-23T10:01:53.000Z

This is the Netherlands by the way.
All offline

Pinging/trace routing through the ttn routers doesnt work as the servers are behind load balancers which are configured to not respond to such traffic.

Its the console that is down vs the routers - traffic is passing through to applications ok, unfortunately the database conection to the console front end looks broken - it has been that way off & on for last 2-3 days (I see 2-5 of my 30+ GW’s connected at any time…frustrating. This is not reflected in the status page.

These pages are not monitored by TTI staff so you need to hook up on Slack via the #ops channel. If you look at the forum threads you will see mutiple threads already calling out the issue so no need to set up & feed another thread :wink:

Since yesterday, all our Gw have fallen since the morning, we have reviewed our configurations and everything is fine, someone has any idea if there is a massive failure, it is very strange that all our GW have fallen … I am attentive to any help …

For everyone looking at the gateway console and seeing the gateway seems to be down/not connected, there is another way to check the gateway status.

Go to the ttn cli quick start page and download the appropriate binary for you system.
Use the instructions on the page to get the binary up and running, including ‘Register and Login’.

Once logged in, use ‘ttnctl gateways status < gatewayid >’ to check the status of your gateway. This will provide a ‘Last seen’ value which should be within minutes of the current time for connected gateways. Ignore the Tx/Rx statistics reported, those are not the totals for the lifetime of the gateway.

Example output (redacted):

  INFO Discovering Router...                   
  INFO Connecting with Router...               
  INFO Connected to Router                     
  INFO Received status                          GatewayID=gatewayid

           Last seen: 2019-10-25 01:10:35.388703964 +0200 CEST
           Timestamp: 2529939411
       Reported time: 2019-10-25 01:10:35 +0200 CEST
         Description: Some description
            Platform: IMST + Rpi
       Contact email: email@some.domain
      Frequency Plan: EU_863_870
              Bridge: Mystique MQTT Bridge
            Location: (53.NNNN, 6.NNNN; source unknown)
                 Rtt: 31ns
                  Rx: (in: 37307; ok: 1248)
                  Tx: (in: 1; ok: 1)

This method is more involved, but does not suffer the many outages the console is experiencing lately.

4 Likes

I didn’t knew this tool, it works perfectly for me :

INFO Discovering Router…
INFO Connecting with Router…
INFO Connected to Router
INFO Received status GatewayID=eui-b827ebfffXXXXXXX

       Last seen: **2019-10-25 10:31:21.44390785 +0400 +04**
       Timestamp: 0
   Reported time: 2019-10-25 10:31:20 +0400 +04
     Description: Fleurimont TTN GW 1
        Platform: IMST + Rpi
   Contact email: ttn@airtune.info
  Frequency Plan: EU_863_870
          Bridge: "ttn-bridge-eu-1712" 2.x.x-04b47381cd9bccae430251deb09102a7b218de66 (2018-02-20T16:19:59Z) PacketForwarder Backend
      IP Address: 127.0.0.1
        Location: (21.0XXXXX, 55.2XXXXX; source unknown)
             Rtt: not available
              Rx: (in: 1; ok: 1)
              Tx: (in: 0; ok: 0)

thanks a lot @kersing :slight_smile:

My 2 gateways are not connected as well since 2 days in two different european countries.
One is a dragino single channel Gateway and an Indoor Gateway from ttn.
But there are other gateways are running in Germany. Weird

Hi
My Gateway is also in “not connected” mode for 2 days. But all applications transmit data normally on this Gateway.
CaptureTTN

Hi Jeff,

indeed your analysis seems correct, thanks for that.

Nico

Now my two Gateways seems to work again few minutes ago…appears Green in the console but dont forward Messages…weird.
Edit: and now its gone again.
Edit: and its working…cool