Gateway status not connected for a month

Hello folks,

I write with certain discomfort because it does not seem like anyone cares about this.
It seems some (or all?) of the original kickstarter gateways (TTG) are showing up as offline for a month already! I don’t really know what’s the deal and why nobody seems to care. Is this a general issue? Is it only this gateway somehow? Application data still flows through it.

image

Cheers

1 Like

This is a known issue. TTN does not want to invest time to fix it.

You can ask operations staff to try resetting some components to see if that fixes the issue. I expect this gateway is connected to the Swiss back-end infrastructure?

Yes, this is connected to the swiss cluster, and we’ve tried resetting all components possible, but no luck.

I understand the no-fix-legacy approach but without an ETA for v3 upgrade and with over one year of delay, this is pretty frustrating.

4 Likes

Welcome to the club. How do you think the moderators feel after having to explain this over and over and over… I don’t know how much time a fix would take, but on the forum we spend several days worth of hours on this issue. Not counting the numerous hours the TTN community members lost because they’re trying to fix a gateway that isn’t broken.
So all in all it might not have taken a lot of time of the team in Amsterdam but it has already wasted a lot of equally valuable time for others.

9 Likes

Darn… I spent half a day troubleshooting and re-installing the gateway from scratch… I should have started with checking the forum. :disappointed_relieved:

The best strategy is to have a simple device in range of a gateway with an alert setup if an uplink isn’t received every 60 minutes or so.

Then you would have never got to re-installing as you’d know the gateway was connected, just not being reported as such on the console.

Problem comes in areas of high GW ‘densification’ or good coverage as even if your GW is down packets will still get through to the monitoring app in TTN. So make sure Tx power dialed right down - say +5-7dbm safety headroom to no reception on your GW if poss, set to SF7 to minimise time on air/interference and if you know there are other GW’s close by try to set monitor node on side away from the others, and set range >>3m-4m, pref behind some absorber (e.g. a window or a wall etc.) up to say 50m(*) away from GW, then most likely if signal caught it will be through your GW…otherwise you have to put in place more complicated monitor system to parse app metadata to look for the monitored GW only.

(*) In areas where I ‘know’ there are few or no other GWs in range I tend to push out to >50 - ~250m if I can find a suitable site as then you also get some indication of if GW suddenly goes deaf (e.g. from nearby lightinng strike, failed RF front end, etc.) as RSSI will drop very low (when using a low Tx pwr) and may even loose signal…then its time to grab the toolkit & spares box! :slight_smile:

Always good if monitor/heartbeat node also provided a useful piece of info/data vs just taking up spectrum so I often use e.g. a T&H monitor for added value…

The best strategy is that this basic feature works.

Anything else is unnecessary friction for everyone (If you’re new to the community, you will not know this; if you’re experienced, it might anyway trick you if you’re not actively aware of the issue, if you are a member that cares about gateway uptime and took the time to setup monitoring, it will screw you, etc etc etc).

2 Likes

Totally Agree, and very frustrating… worst is when whole console down vs just one GW not showing correctly…and you have a training course scheduled for same time! :frowning: :scream:

Still a good idea to set up a monitor though :wink: (and an integration to say look for heartbeat and if not seen for a few cycles ( I would heartbeat more often than Nick’s suggested one hour on a ‘critical’ GW where you want to avoid longer hours of outage) then send GW owner a text/email notification to go double check (e.g. in Console - if up and running, - via ttnctl, simply checking Apps ofother nodes normally received - as monitor node might be out! - or whatever…)

Agreed. However if you want to know your gateway performs anything apart from the traffic to the back-end having a canary node in place helps. I’ve seen gateways happily uplinking status packets while the radio wasn’t operational. TTN console shows it life, no data flowing. Very puzzling.

1 Like

All the other things you add, sure, but I check the uplink data for gateway ids so I know that it was heard by the gateway(s) it is providing backup for.

Most of the time a canary isn’t required, you just have to nominate one or more of the closer devices with a smaller payload as “it”.

For a weather station in the High Peak, we detect wind speed > 80mph followed by 0mph - tells us the anemometer has come off (again).

No it’s not. You are relying on a green dot on a website that you have to log in to observe. Sure, ideally we’d like reassurance when we (occasionally) log in to the TTN console, but having your own mechanism that checks end to end and checks at each step along the way so you can be alerted there is an issue and where the data stops flowing is so stupidly cheap that why wouldn’t you do it.

I’d only trigger a gateway status check if the end check doesn’t arrive on schedule, no point loading TTN with constant gateway queries.

Given some of the £3 GSM modules from Ali and buying the right bag of SIMs at the right time, you can have additional parallel backup for less than £10 a year.

1 Like

Not all gateway operators own sensors. You’re totally missing the point of TTN, which is created as a low-barrier-of-entry community network, if you think that putting up a gateway implies you also need to setup your own end-to-end monitoring system and use additional nodes.

There is a lot of people in the community that contribute gateways simply because of the premise of “buy/plug/do good to the world”. This broken green dot on the website, as you call it, is what enables that to happen. Without it, this doesn’t work. People buy the device, plug the device, the device seemingly does not work and then unplug, throw it in the drawer and move on to the next thing fancy technology available.

5 Likes

Erm, actually that wont always work - where it does thats what I do (e.g. the T&H ‘Canary’ :slight_smile: ) As Gonzalo calls [people in the community that contribute gateways simply because of the premise of “buy/plug/do good to the world”] out there are many who just deploy…and I count myself in that group - many of my GW’s - and many I have deployed for others - are deployed in areas where I do not have or may not always have a node in place or in targeted use - hence need for a monitor that I can look at. Right now I would say ~20 - 25% of my deployed and active GW’s are where I do not have a node! I have deployed as a ‘pro bono’ contribution to the network to help extend and expand and to encourage others to try/adopt the technology … as an evangelist for LoRa/LoRaWAN if you will. I’ve made a good living out of tech over the years and just do this as a way of giving something back. I hear of people interested in trying/testing and if they are in a place I think may be a good site location or where use cases can be deployed I just configure a Gw and go install for them :slight_smile: Have done this in Cheshire, The Tyne Valley, Mersey Valley and locally here in Thames Valley. The GW I ended up having to remove/relocate from Nottingham before the summer break was another example… didnt get much traffic but I know a few people who occassionally used it even though I didnt have a node stationed within >50 miles of it :slight_smile:

You are saying there are people that are aultristic enough about a relatively niche technology that has a community network that they go and buy a gateway, set it up and because there’s no green dot today, throw it away? It’s hard enough to find a location to setup a gateway as well as get it going, I doubt they are that much of the FaceBook generation that they’d give in just because of one indicator.

From my experience these altruistic people are real edge cases (sorry @Jeff-UK) who would know to look further than the green dot.

As an ambulance first responder volunteer (just getting my halo more polished than Jeff’s), offering free CPR / First Aid training falls on fallow ground and I can assure you, the majority of Community Access Defibrillators are installed AFTER they would have been useful, so I know how uninvolved most people can be.

As for Jeff, and in the very near future for me with two gateways on hand to start to fill out Bolton’s coverage, why deploy a gateway out the kindness of your heart and not stick a £10 canary a few metres away - even very literally in a bird box nailed to a municipal tree. That way you know your contribution is working and you get auto-halo-polishing as a bonus.

I so totally agree the green dot should work. I think it’s a critical error by TTI to neglect some of the ragged edges. I’ve redirected numerous “gateway down, end of civilisation as we know it” to what was a sticky AND asked for it to be restuck but TTI have decided not to. I think it would make most people think hard about TTI as a commercial offering if they can’t keep something like that operational. I think they should look to offer a community-commercial tier so those that don’t need their own infrastructure can at least get some internal support from TTI. At present there isn’t really any progression from TTN to TTI and I suspect a measurable number of opportunities are lost because they equate LoRaWAN with TTN and all the nut jobs posting on the forum at half-past-midnight.

On Friday night, the team finally solved the problem! It might not be a permanent solution, that I don’t know, but at least it resolves the pain point for now!

Thanks!

It surely is not. From Slack:

@adrianmares 2020-10-09 20:20

I expect the NOC to stay responsive…for a while. We will probably have to do this whole restart dance again the future, but that’s the state of affairs for v2 in general.