Not seen gateway status on ttn

I see that things are broken - I’ll go and put some washing on, do the hoovering etc and come back later.

https://status.thethings.network/

Thanks Hylke, I checked over breakfast earlier and saw ‘Last Seen’, Traffic tab and GW Overview Connected/Not Connected colums all back :slight_smile:

Are there other items we should be aware get hidden in these circumstances (to save us trawling or having to respond to user cry’s for help)? We can get them documented and flagged on the forum so users know not to panic!

So to clarify what you are saying is if NOC goes down the various pages will automagically remove the page elements related to NOC - as called out - last seen, traffic tab and for GW overview page the con/not con column? When NOC back up they automagically re-appear?

As Jac says this is inconsistent UI/experience so would be good if the repective elements remained in place but instead called out something like ‘sorry there is a noc issue’ so user knows it’s not their GW or their browser or whatever. E.g. the Traffic tab could still be shown but with a line saying ‘noc is down, no current data available for display’ or some such, GW overview lines could substitute ‘noc down’ for connected/not connected.

As mentioned on other threads and #ops last nigh the noc also stopped responding to browers with connection timed out error suggesting noc was down/not responding… can that be scripted and checked with a simple watchdog and then if down for more that say 10 mins (to allow for some off time for maintenance/updates) then automagically trigger a server/noc restart - that would save you the hastle of having to go beat it with a stick as and when needed - appreciate this may not be the fun part of your job or a priority therefore automating makes sense? Doing this automatically would also stop the flood of forum or #ops posts when one of the pages starts to noc has thrown a wobbly again :wink: (note though data coming through on consle I checked noc url (noc.thethingsnetwork.org:8085/api/v2/gateways) for overview of gateways and for some individual known good gw’s earlier and still getting connection time out problem)

As Andrew says we absolutely have to have a way of monitoring GW status, I would say ~1/4-1/3rd of my personal deployed GWs do not carry traffic for me regularly (hourly/daily) - some may see my traffic within a given month but that is no use for status monitoring via data received (they are depoyed for community benefit). These days I try to follow best practice of locating a canary (often a chosen functional data gathering node) close to a gw so full path to backend can be monitored and gw status verified, but only adopted that approach after some time on TTN and after a few harsh lessons - and ofcourse those early canaryless deployments are all on V2! :wink:

1 Like

Maybe awkward formulation from my side - I agree with observations. TTN have made a lot of problems with this step. I’m scared what will be the next such step

hi
in gateway side everything is working fine but i can not see my packets in application-data part.
empty
is this a problem of mine or a general problem?

I see application traffic for my devices so suspect it’s your end… is node joined and transmitting ok, do you see activity in your gw log and traffic page etc.

it is working now,
i was not able to see packets on application part for about one hour but now it is ok,
thank you.

Looking deeper it depends on when you were checking - there was significant outage of V2 earlier which was recovered but then seemed to go into decline again
image

1 Like

yes i had difficulty exactly in the mentioned period

Things are up and down and up and down and up and down at present.

Rinse, repeat.

Etc

Nice day for a walk …

might be worth you bookmarking: https://status.thethings.network/ often doesnt show minor wrinkles but such outage usually pops up or is visible if you keep an eye on the graphs. also subscribe to the ttn slack ops channel…

1 Like

been there done that, phographed the bluebells, drank the coffee…nose now back on grindstone! :rofl:

Indeed.

I see at the status page that the incident in NOC is reported as “unreliable information”. Actually the entry point is unreachable since it times out.

I’ve read the thread and we also monitor our community gateways using NOC. @htdvisser: I’m upset when I read that it is not a “critical” component. It is for a lot of people. We might be enrolled in non-monetizing projects we still have the commitment to keep the gateways online and this makes us effectively blind to issues on the gateways.

Overall, The Things Network / The Things Industries is losing credibility with this rough transition to v3. I completely agree with @Jeff-UK post above. v2 breaking often, the lack of a well defined migration procedure and timing, information missing on the packet broker,… these won’t help us move faster to v3. For some people quite the opposite.

Back into now: I’m missing information on the status page about when you estimate NOC will be available again.

2 Likes

@htdvisser Will the NOC API be back or has this been permanently retired? - eg http://noc.thethingsnetwork.org:8085/api/v2/gateways/eui-00800000a000284e

No, we are not working on getting http://noc.thethingsnetwork.org:8085/api/v2 back online.

This endpoint is not part of our official API, and was never actually intended to be exposed/used publicly (otherwise we would have enabled https and documented it). Therefore we’re also not going to spend our time on getting it back online.

@htdvisser

It is for me and a lot of other users a tool to measure if a gateway is online, now with V2 still operational and a lottt of users still havin gateways and devices running on v2 the NOC is a tool that can be missed.

Please give it a little effort to bring it back, and if v2 shuts down, send it off to the bin :slight_smile:

When.

As you do Node-Red, why not script something that looks at uplinks and checks for your gateway id’s.

IMO this is SEVERELY PROBLEMATIC to not fix V2 since many of us are using TTIG-915 gateways which AFAIK are STILL NOT SUPPORTED ON V3

So WTF are we supposed to do when we have issues? I have nearly 20 TTIG915 gateways deployed?!

Use the gateway console?

If you can politely explain what extra benefit you get from the NOC tool, it may be possible to find an alternative solution as an interim.

One can easily monitor gateway status by querying the gateway-data entrypoint (https://www.thethingsnetwork.org/gateway-data/country/es, for instance) and parse the response. The only thing I’m missing here is the info about received and transmitted packets I had using the NOC. Not a big deal thou.

Or to just get your own data:

https://www.thethingsnetwork.org/gateway-data/gateway/eui-de5ca70000000000

for instance.

As I was alluding to, @dnear1 and anyone else who’s getting agitated about the perfectly normal bit-rot in v2 that they use for free, solutions are out there, please ask nicely.

1 Like