Console shows not connected, traffic goes through, how can I use the TTN mapper?

This is strange: data
gww

now the question: how can I check the GW coverage when it is not shown on TTN map (errorneusly reported as not connected) ???

No it is not, the magic word is timezone.

Basically you are out of luck.

@jpmeijers I just checked for my gateways and it seems ttnmapper no longer shows gateways which the console reports off-line where ttnctl provides the correct (working) status. Any hint on what to do except trying to get TTN fix the problem? (@johan @htdvisser ttnmapper not showing active gateways and coverage is annoying to say the least. Any progress on fixing this? All my MQTT connected gateways show offline in the console for 9 days now)

1 Like

Hi @s54mtb, I follow the normal industrial approach of “don’t trust the infrastructure to monitor the infrastructure”. I use the “coal-mine canary” approach.

I co-locate a low-cost LoRaWAN GPS tracker with each of my gateways. The trackers are fixed to uplink once every 30mins using SF7 with power of 1mW. The uplinks go to a dedicated application and MQTT feed to a cloud-based virtual machine. I get the following delivered completely outside the LoRaWAN infrastructure:

  • The location of the gateway so I can map independently with no admin.
  • The operational status of the gateway; up and running, or not.
  • The stability of the gateway RF receiver via metadata RSSI and SNR.
  • The operational status of the end-to-end delivery - LoRa–stuff–MQTT; up and running or not.

I have software that detects if the “coal-mine canary” stops singing and notifies me. Standard industrial exception reporting.

3 Likes

My canary stopped singing the day before yesterday after singing for more than 3 years while th coal mine is still operating. Now i am blind again

1 Like

You are completely correct! Canaries need care and feeding.

For monitoring that works like a charm and I would recommend such an approach for ‘critical’ infrastructure.
However it doesn’t help to keep the gateway on ttnmapper as it still considers the gateway to be down. And for the community network keeping the map up to date and have running gateways displayed on the map is important imho. Sadly about 1600 of the gateways in Europe (according to the numbers presented at the virtual conference) won’t be shown as all MQTT connected gateways are reported to be down it seems (at least all mine are while the UDP gateways are fine). Ttnctl reports them to be up however that doesn’t help with ttnmapper.

1 Like

Hi @kersing, you are correct to point out the tension between the commercial/industrial use of LoRaWAN and community use. Most of the commercial/industrial LoRaWAN that I see (in the UK) is on networks like Actility, Loriot and Orbiwise and is not normally visible to TTN users.

Most commercial/industrial users are naturally very selfish; they want to see their stuff, they want SLAs and they have zero interest in helping open-source or community efforts. I have chosen to put all the commercial/industrial work that I control on TTI cloud-hosted LoRaWAN. The “coal-mine canary” monitoring that I described is ideal for them. For community work, I fully agree that TTN and tools like TTN Mapper are the most appropriate.

1 Like

It seems like it could be possible for TTN Mapper to use ttnctl to get the gateway statuses. I’ve tried logging in with my user and I can see the status of a gateway I do not own and I’m not a contributor to. Easiest way would be to iterate through all known gateways in TTN Mapper and one by one request the status via ttnctl. This would however take a very long time and cause a lot of unneccesary load on the TTN servers, as a couple of api calls need to be done per gateway. And there are around 60 000 known gateways.

I would prefer not to have to wast time on writing a piece of code like this, as I want to focus on getting TTN Mapper comaptible with V3.

If anyone of you feel like it, please contribute.

The code that currently gets the gateway statuses from the TTN website is here:

And an example of how to get a list of known gateways from the TTN Mapper database can be seen here:

One problem with using ttnctl I see is that we won’t be discovering new gateways. For that to happen we need to change this micro service to also add gateways to the TTN Mapper database if they don’t exist yet. This would also cause much more load on the TTN Mapper database - something that is already a problem.

If someone could write a python script that takes in a single gateway ID and returns the last seen time, that would help tremendously. Then I can add the database and TTN Mapper-specific logic.

I think the most difficult part is either wrapping ttnctl, using valid login details which should be stored somewhere, or talking to the API endpoints directly.

Ignoring any ttnctl login difficulties :wink: … you could use this first suggestion:

from subprocess import check_output
from datetime import datetime, timedelta

def getGatewayLastSeenDateTimeFromEui(argGatewayEui):
    """
    Returns the last seen datetime (UTC Time) of a gateway acquired by ttnctl as datetime object, or 'None' if gateway not available. 

    ttnctl must be callable from the shell. User must be logged in. 
    Expects one status output line (if gateway available) in the format: Last seen: 2020-05-24 16:54:00.04129753 +0200 CEST

    Parameters:
    argument1 (string): Takes Gateway EUI as string

    Returns:
    datetime: Last seen datetime in UTC time or None

    """
    statusoutput = ""
    datetimeObj = None
    try:
        statusoutput = check_output('ttnctl gateways status '+ argGatewayEui, shell=True)
    except Exception:
        pass
    #print(statusoutput) # for debugging or curiosity
    indexLastSeen = statusoutput.find("Last seen: ")
    if (-1 != indexLastSeen):
        dateStr = statusoutput[indexLastSeen+11:statusoutput.find("\n",indexLastSeen)]
        datetimeObj = datetime(int(dateStr[0:4]), int(dateStr[5:7]), int(dateStr[8:10]), int(dateStr[11:13]), int(dateStr[14:16]), int(dateStr[17:19]))
        indexOfTimeZoneDiffMinus = dateStr.find("-", 20)
        indexOfTimeZoneDiffPlus = dateStr.find("+", 20)
        if (-1 != indexOfTimeZoneDiffMinus) or (-1 != indexOfTimeZoneDiffPlus):
            indexDiff = indexOfTimeZoneDiffMinus + indexOfTimeZoneDiffPlus + 2
            offsetToZuluTime = timedelta(hours=int(dateStr[indexDiff:indexDiff+2]), minutes=int(dateStr[indexDiff+2:indexDiff+4]))
            if (-1 != indexOfTimeZoneDiffMinus):
                datetimeObj = datetimeObj + offsetToZuluTime
            else:
                datetimeObj = datetimeObj - offsetToZuluTime
    return datetimeObj
2 Likes

Ahh excellent. Thanks a lot @Basler. I’ll try and get this running as soon as possible (tomorrow).

1 Like

It’s running, but it’s very slow. Need to iterate through 47000 gateway IDs, and it takes about one second each. We’ll have to be patient.

I don’t think this is a sustainable solution.

1 Like

Yes it is slow, seems to be response time from ttnctl…
Lot’s of gateway icons disappeared from the map now, even though their blue coverage lines still appear.

A little faux-AI could be introduced - each time you do a run, increment a score for any gateway still online - once you know who’s mostly online, you can then choose to check those that were recently online but have apparently gone off, to see if they are back, check new gateways until they build up their confidence score, check a % of the intermittent gateways and do a % of the known-good gateways as a well.

My gateway disappeared 13 days ago, at the same day as Jac’s.
Sad canary is sad.