Gateways data unavalible

Did something happen last night? at midnight NL time.

I monitor the number of gateways reported here: https://www.thethingsnetwork.org/gateway-data/country/gb

For the UK has contained a pretty constant 1077 gateways.

Last night it dropped to 790.

Also lots of the metadata is now missing

also GWs that are online and public do not appear in the list.

Any chance those gateways are using Digital Catapult?

Some are but not all

1 Like

So here is an example. This GW, one next to my desk, is on line and taking traffic and shows up as on line on the TTN dashboard. eui-58a0cbfffe800ee0

However it does not appear in this feed.
https://www.thethingsnetwork.org/gateway-data/country/gb

Also i am sure that the ttn map used to show the gateway name.

Picture1

That’s probably as https://www.thethingsnetwork.org/gateway-data/gateway/eui-58a0cbfffe800ee0 shows no coordinates*. (Of course, still not okay, unless those have been set to be private.)

Aside, the Kickstarter The Things Gateway has its coordinates missing in the uplink metadata since May 11th, but gateway-data/gateway/id still includes those.

Are all new troublesome gateways TTIGs?

*) or: no "country_code"
) and also "country_code"

Yes but there are other attributes for other GWs that are missing too and over 300 GWs have dropped off since last night

I’ve now reported this in the appropriate Slack channel. But of course the working week in Europe is over now. :crossed_fingers:

No, seems to be a problem for all gateways (including my Dragino LPS8), at least in the Duesseldorf area. Only altitude and brand name is shown when clicking on the icon.

Ah indeed, details are missing for mine (in The Netherlands) too, though those happen to still include their coordinates.

Also, they show "last_seen": "2020-10-09T14:27:37Z" (and times close to that) and everything seems offline for me too, but that is a known problem: TTN V2 Console not working properly all the time So, for me they still forward data.

The output for /gateway-data is cached for 6 hours, so if this is due to the NOC failing, then maybe that will degrade even more soon:

My gateway in Wiltshire, UK has also “disappeared” from the TTN Mapper:

GatewayEvaporatedFromMap
I initially rebooted the gateway thinking it was just me, but I assume it is the same root cause others are observing.

I can confirm my gateway is now missing from https://www.thethingsnetwork.org/gateway-data/country/gb and I also observe “last_seen”:“2020-10-09T14:27:45Z” from https://www.thethingsnetwork.org/gateway-data/gateway/ but it is actively in use since then.

Ah, that also refers to attributes on, say, https://www.thethingsnetwork.org/gateway-data/gateway/eui-58a0cbfffe800ee0 ? So, it does not refer to actual uplinks/downlinks not being seen in, say, the MQTT API? (I don’t recall seeing those numbers in that API, but I still see up-to-date numbers in the NOC right now.)

Sometimes the NOC throws:

{"error":"Failed to execute add commands in pipeline: Failed to execute pipeline: redis: connection pool timeout","message":"Failed to execute add commands in pipeline: Failed to execute pipeline: redis: connection pool timeout","code":2}

Meanwhile the NOC has been restarted, and for me "last_seen" is now accurate again in both the NOC and /gateway-data/gateway/id (EU868, The Netherlands). Also, Last Seen, Transmitted Messages and Received Messages in TTN Console look okay right now.

Some fields such as description, owner and gateway brand/type are still missing from, e.g, https://www.thethingsnetwork.org/gateway-data/gateway/eui-0000024b080606d8.

I’m personally not getting any response now from TTN Mapper
TTNMapper

Earlier on the browser console was reporting lots of “location is on null island” for many gateways:

IsOnNullIsland

And meanwhile, @adrianmares also fixed that, for me.

@peterkarney, I don’t know if https://www.thethingsnetwork.org/gateway-data/gateway/eui-58a0cbfffe800ee0 should show more than below?

{
  "eui-58a0cbfffe800ee0": {
    "id": "eui-58a0cbfffe800ee0",
    "attributes": {
      "brand": "The Things Industries",
      "frequency_plan": "EU_863_870",
      "model": "The Things Indoor Gateway",
      "version": "2.0.0"
    },
    "last_seen": "2020-10-09T18:44:04Z"
  }
}

Surely, there’s still no country or location. Is this using Digital Catapult?

And that one seems fixed too:

{
  "eui-58a0cbfffe800ee0": {
    "id": "eui-58a0cbfffe800ee0",
    "description": "Plug 1 - 58 A0 CB FF FE 80 0E E0",
    "owner": "peterkarney",
    "owners": [
      "peterkarney"
    ],
    "location": {
      "latitude": <redacted>,
      "longitude": <redacted>,
      "altitude": 10
    },
    "country_code": "gb",
    "attributes": {
      "antenna_model": "Internal",
      "brand": "The Things Industries",
      "frequency_plan": "EU_863_870",
      "model": "The Things Indoor Gateway",
      "placement": "indoor",
      "version": "2.0.0"
    },
    "last_seen": "2020-10-09T19:11:40Z"
  }
}

So, assuming this is correct:

I think all is fixed?

Thanks, @adrianmares!

Yes I were referring the the API attribute, up/downlink.

I did mention I still can see my trackers data into the server, so the nodes were still up link to the gateway and the gateway to the server, so did not lose any data.

The problem seem to have been related to certain gateways brands that the attributes were missing off. I have a Multitech and Mikrotik gateway, the Mikrotik arrributes in the API were missing.

Yes all the attributes are back now.

I’ve had 8 gateways (out of 17) show offline as of 5:45pm UTC.
All use the meshed australia server, but tree different models of gateways across 4 different network connections, so seems to be a TTN issue

Web site back for me…

TTNMapperUp

…and missing gateway back on the map:

GatewayReappearedOnMap

So all is well in West Swindon, Wiltshire

:unamused:

chrome_2020-10-10_11-56-42