Problem - ttn-router-us-west connection issues

:warning: TTN-router-US-WEST is having issues as of ~5AM EST, where GWs may have been disconnected/not communicating for an hour, then briefly came back in ~7AM EST, then interrupted again and, as of 10AM EST back online FYI. I will continue to monitor thru out the day, if anyone else can validate this please chime in for the rest of the US / elsewhere community - thank you and all be healthy and good! :red_circle:

TTI is looking into the issue.
https://status.thethings.network/
image

1 Like

@MicrochipTTN Thank you for the update! didn’t even know TTN had a ‘Status Page’ - much needed. Here’s our 48 hr timestamp of events as reported by one of our sensors FYI:
image
Thanks! :red_circle:

@MicrochipTTN Any new updates? The problems are still present (spread dots means connection issues >1hr) :arrow_down_small:
image
No updates since 2 days ago:
image
Thanks! :red_circle:

Also seeing issues here …

I have confirmed my gateway is sending data to and from us-west however the gateway
still shows as “disconnected” even though UPD status packets flowing to and from us-west.

I can see lora sensor data being sent up as well yet application page for sensor shows
it has not been seen recently.

10:22:35.846382 IP 192.168.20.104.54193 > 13.66.213.36.1700: UDP, length 187
E…4E@.@.NZ…h.B.$…b…’…yy.{“stat”:{“time”:“2020-03-26 14:22:35 GMT”,“rxnb”:0,“rxok”:0,“rxfw”:0,“ackr”:0.0,“dwnb”:0,“txnb”:0,“pfrm”:“IMST + Rpi",“mail”:"ttn@mydomain.net”,“desc”:“RAK831 TTN gateway”}}
10:22:35.915556 IP 13.66.213.36.1700 > 192.168.20.104.54193: UDP, length 4
E… >.@.3.Q…B.$…h…:…
10:22:37.278764 IP 192.168.20.104.49997 > 13.66.213.36.1700: UDP, length 12
E…(4.@.@.N…h.B.$.M…]g.d…’…yy.
10:22:37.408808 IP 13.66.213.36.1700 > 192.168.20.104.49997: UDP, length 4
E… ?.@.2.Q~.B.$…h…M…z…d…
10:22:42.448738 IP 192.168.20.104.49997 > 13.66.213.36.1700: UDP, length 12
E…(5.@.@.M…h.B.$.M… h.c@…’…yy.
10:22:42.516740 IP 13.66.213.36.1700 > 192.168.20.104.49997: UDP, length 4
E… CX@.2.M…B.$…h…M…=…c@…
10:22:47.548724 IP 192.168.20.104.49997 > 13.66.213.36.1700: UDP, length 12
E…(6p@.@.L…h.B.$.M…’…yy.
10:22:47.617719 IP 13.66.213.36.1700 > 192.168.20.104.49997: UDP, length 4
E… G.@.2.I…B.$…h…M…d…
10:22:52.648754 IP 192.168.20.104.49997 > 13.66.213.36.1700: UDP, length 12
E…(7c@.@.K…h.B.$.M…(x.S8…’…yy.
10:22:52.716839 IP 13.66.213.36.1700 > 192.168.20.104.49997: UDP, length 4
E… J$@.2.G2.B.$…h…M…E…S8…
10:22:57.748785 IP 192.168.20.104.49997 > 13.66.213.36.1700: UDP, length 12
E…(8.@.@.Js…h.B.$.M…9…&…’…yy.
10:22:57.817851 IP 13.66.213.36.1700 > 192.168.20.104.49997: UDP, length 4
E… K.@.2.Ei.B.$…h…M…Vw…&…
10:23:02.351068 IP 192.168.20.104.54193 > 13.66.213.36.1700: UDP, length 235
E…:d@.@.H…h.B.$…’…yy.{“rxpk”:[{“tmst”:3383993300,“time”:“2020-03-26T14:23:02.350784Z”,“chan”:5,“rfch”:1,“freq”:904.900000,“stat”:1,“modu”:“LORA”,“datr”:“SF7BW125”,“codr”:“4/5”,“lsnr”:8.0,“rssi”:-42,“size”:17,“data”:“QP4eAiaA0yIBprSnyQ/kANo=”}]}
10:23:02.858723 IP 192.168.20.104.49997 > 13.66.213.36.1700: UDP, length 12
E…(:.@.@.H…h.B.$.M…-…’…yy.
10:23:02.926655 IP 13.66.213.36.1700 > 192.168.20.104.49997: UDP, length 4
E… Mt@.2.C…B.$…h…M…
10:23:05.843009 IP 192.168.20.104.54193 > 13.66.213.36.1700: UDP, length 187
E…;4@.@.Gk…h.B.$…k…’…yy.{“stat”:{“time”:“2020-03-26 14:23:05 GMT”,“rxnb”:2,“rxok”:1,“rxfw”:1,“ackr”:0.0,“dwnb”:0,“txnb”:0,“pfrm”:“IMST + Rpi",“mail”:"ttn@mydomain.net”,“desc”:“RAK831 TTN gateway”}}
10:23:05.910774 IP 13.66.213.36.1700 > 192.168.20.104.54193: UDP, length 4
E… O.@.3.@u.B.$…h…"…k…
10:23:07.958769 IP 192.168.20.104.49997 > 13.66.213.36.1700: UDP, length 12

1 Like

@tbalon Correct; we’re still seeing ‘network disconnect gaps’ as well within 1,2 and even 3 continuous hours. Doesn’t mean the TTN server was down for 1,2 or 3 hours, but that it was not processing packets correctly / dropping packets within those hours:
image
:red_circle:

@tbalon UPDATE : TTN is still investigating :arrow_down:
image
:red_circle:

Yes, I can see data is occasionally showing up on the TTN application device
data page.

However, when using mosquitto_sub to connect to us-west.thethings.network,
no data is broadcast.

Tom

I can’t register any new devices

Hi everyone,

My gateway was connected for about an hour yesterday but got disconnected and has been disconnected for more than 24 hours now.

My gateway is shown as disconnected, but I can see on the thethingsnetwork site my application node is sending activation requests. Is there any thing we can do to help this issue? My application and gateway have been up a couple of years.

Experimenting the same problems right here in us-west.thethings.network, Gateway seems to be disconected since yesterday, but still receiving packages, devices uplinks are intermitent, this morning packages were lost from 12am to 8am.
Seems that I am not the only one with the problem.

@rlgjr562 Sadly there’s not a lot we can do - this is on TTN’s side.

Something simmilar happened last year around late October. I had a conversation with TTN, who explained to me that TTN “…is for testing and development purposes.”, which I completely understand. But that is precisely the point and why so many of us (including :red_circle: ) only use it for testing and development and NOT for scientific or commercial purposes. Nevertheless, we argue, if you’re a coffee shop with a ‘Free WiFi’ sticker on the door as an invitation to come in and buy your coffee as well and, once inside, I learn the WiFi is not working then we think we’ve earned the right to ask questions about it. Especially when, in our view, TTN is also meant to be the ‘springboard’ to a paid subscription (aka the cup of coffee) to TTI which WILL guarantee SLA, QoS, tech support, etc.

Is understandable that a ‘free’ community-based Network Server (TTN) will not be managed the same way as a PAID commercial-private one (TTI), but with just ONE server in the US (as far as we’re concerned), there’s not a lot the rest of us can do to answer the ‘global community’ call to ‘test and develop’ with our own resources, gateways, backhaul, etc when the Network Server itself goes offline or its capacity is crammed with too many sensor’s request and begins dropping packets. We’ll have to wait for TTN/TTI to figure it out and, hopefully, add more servers in the US Or we can pay TTI (or other NS providers for that matter, aka other ‘coffee shops’) - all of which, of course, are options to us all. :red_circle:

1 Like

Everything seems to be back to normal since March 30 :
image
:red_circle: