TTIG does not support downlinks when not frequently receiving uplinks?

[Vaelid]

So to summarise it is an issue and there are what I see as 3 solution.

1. Keep the web socket Connections permanent open. - consumes resources.

• Action by TTI on TTN Servers.

2. Keep sending traffic to node say every 20 seconds with short payload.This seem wasteful in air time and on TTN server resources but is the only option open to gateway / node owners.This is easy to set up with a spare node.
3. TTI Change server software to take account of the disconnect of web socket in regard to handling down links and intersection to the TTIG.- Action by TTI.

I hope TTI looks seriously at this issue and resolve it as it causes issues with ADR and Acknowledge responses and packet loss. To have all of the TTIG owner sending dummy traffic to keep gateway open is very wasteful on TTN.

Related to above when will there be update software for TTIG?

[bluejedi]

I don’t know if there’s any GitHub project where this might be/have been reported on?

During Semtech’s Basicstation (workshop) presentation held at The Things Conference last January, it was mentioned that Basicstation itself is open source, but the implementation of the ‘Platform Layer’ and ‘Radio Layer’ building blocks for ESP8266 (used on the TTIG) are closed source.
This makes availability of an open TTIG project on Github less likely.

[rharte]

@didzis

Is there a way to widen the RX window with LMIC library?

Well I guess the answer is yes and no… Search in the radio part of the code for the register LORARegSymbTimeoutLsb. It defines the timeout in RX1 and RX2 in terms of number of symbols. For instance in RX2, receiving SF9, if you set this value to 80 symbols, it will be about 300 ms for the LoRa module to timeout if no preamble is received. If you set a higher value, the LoRa module will search longer for a preamble, and would also capture a transmission that may have started somewhat late.
But remember you can’t shift RX2, it needs to start at 2 seconds after TX_ready, so the only thing you can do is extend the timeout. And the maximum would be 255 symbols.

[cslorabox]

And the maximum would be 255 symbols.

Actually the maximum is 1023 but the upper two bits are in a different register.

[rharte]

Aha, I see, they are mapped in RegModemConfig2

Never noticed it and never needed it But thank you for your addition!

[arjanvanb (Arjan)]

This makes availability of an open TTIG project on Github less likely.

Indeed, but for, e.g., the closed source NOC, integrations and TTN Console in V2, issues could be reported on https://github.com/TheThingsNetwork/ttn/issues/

@KrishnaIyerEaswaran2 (@laurens, @rish1), is there any place where possible bugs for the TTIG can be reported?

[Nobyte]

Is there anything new on the subject? I came across this problem with my TTI Gataway. If my node sends an uplink every 10 minutes it is obviously too long and many values ​​are lost. Now I have additionally installed a dummy node that sends every 2 minutes and now all values ​​are transmitted without loss even at 10 or 15 minute intervals. But that doesn’t seem to be a really good solution to me, if you should actually keep the traffic low.

[kersing (Jac Kersing)]

During the conference I got confirmation that work on ongoing on new firmware, however progress is very slow. TTN does not own the sources and can’t do the work themselves and they can’t release the sources to allow the community to improve things which is a shame as I know there are several avid ESP developers in the community that should be able to help out.

[Nobyte]

Too bad. Thanks anyway for the answer and the information.

[arjanvanb (Arjan)]

During the conference I got confirmation that work on ongoing on new firmware, however progress is very slow.

Do you happen to know if this implies that it might not be the server that is causing the WebSocket connection to be dropped?

Recv failed: SSL - The peer notified us that the connection is going to be closed

…

So it seems to be the server what closes connection.

Or, @KrishnaIyerEaswaran2, could you tell us who’s dropping the connection?

TTIG : The Things Indoor Gateway

If there’s no traffic on your gateway for about 60s, the WebSocket(WS) connection to the network is dropped intentionally and the gateway will re-connect.

[kersing (Jac Kersing)]

Do you happen to know if this implies that it might not be the server that is causing the WebSocket connection to be dropped?

Sorry, no. I would have to speculate so let’s wait for an authoritative answer.

[KrishnaIyerEaswaran2 (Krishna Iyer)]

Hey @kersing/ @arjanvanb,
It is indeed the server that drops the connection. The TTIG currently runs 2.0.0 of the Basic Station packet forwarder which does not implement WS Pong. With the upgrade to 2.0.4 which we are planning, we’ll be able use this to not terminate active gateway connections.

Quick Note: We increased the keepalive time from 60s to 600s. So your gateways will now only reconnect every 10m if there’s no traffic through it.

[arjanvanb (Arjan)]

Quick Note: We increased the keepalive time from 60s to 600s.

I guess you’re keeping an eye on https://github.com/TheThingsNetwork/lorawan-stack/issues/1730 but just to be sure: this introduced new problems with internet providers that reset the connection every 24 hours (and maybe also assigns a new IP address lease). From Slack:

@JackGruber Today at 7:47 AM

Every morning the same problem The TTIG Gateway does not transmit data anymore

@strenker Today at 10:34 AM

Thats exactly the same what i have seen as well. My ISP resetted the internet connection (which always includes new IP addresses) in the night from 2020-01-29 to -30 and my TTIG became trouble. Before that date, ISP resets were never a trouble.

Such an ISP-reset of the internet line takes normally appr. 90 seconds to recover.

[KrishnaIyerEaswaran2 (Krishna Iyer)]

Hey yeah indeed.
We (TTI) should improve how we communicate these changes.

[KrishnaIyerEaswaran2 (Krishna Iyer)]

We’ve deployed a potential fix here: TTIG Problems, - no location data, wrong date/time, wrong channel and stability issues
We’ve currently deployed the update only to the US West cluster. EU soon to follow.

If you still have issues, please post your observations in this thread (not in the one I posted).

Thanks,
Krishna

[arjanvanb (Arjan)]

@KrishnaIyerEaswaran2

we’re going to deploy another update where the server supports WebSocket Pings. This will keep TTIG connections alive even when there’s no upstream data. The ping interval is set to 30s.

Krishna, apparently this does not work for all TTIGs, or some regression has occurred since February? A few instances that seem related:

@dellekom

One works fine, the other one loses its connection very often, for many hours. Admittedly where it sits are not many packets around (pretty sure it is the most southern GW in Germany) but I would expect it to hold the connection to the Things servers.

@kersing

My TTIG still disconnects when there is no LoRaWAN traffic so the change doesn’t work for all devices…

@Martin_Kautenburger

I have the disconnect problems since a month or more with my 3 gateways.

…for which on Slack some more details are given:

@Martin_Kautenburger 2020-05-07 11:31 PM

Hi, i own 3 TTIG, the devices disconnects a few times a day. A few month ago the ttig works very good. But now it is not usable. I see on ttnmapper there are many ttig with the same problem.

[arjanvanb (Arjan)]

@KrishnaIyerEaswaran2

If you still have issues, please post your observations in this thread

One more:

@Ejanse64

My 2 TTIG gateways are also losing the TTN connection multiple times a day, while WiFi is stable. It looks that it only occurs when receiving no node traffic.

[KrishnaIyerEaswaran2 (Krishna Iyer)]

So thanks for posting these here @arjanvanb.

I haven’t made any updates to the bridge in a while so there aren’t any regressions.

The received status messages is a strong indication of how often gateways reconnect and based on the graph below, it’s non-periodic and non-synchronised (as I expect).

[Excerpt from our metrics for the last 24 hours]

The disconnection could be due to;

1. Client-side disconnects (need to check with the gateway guys)
2. Flaky internet connectivity (but I don’t think this is the case for all)

I’ll do some tests locally and revert.

In the meantime, a disconnection will not affect your ability to forward traffic when the gateways reconnect.
In our previous update, we re-wrote the connection logic with special attention to unclean client-side disconnects.

[arjanvanb (Arjan)]

@KrishnaIyerEaswaran2

In our previous update, we re-wrote the connection logic with special attention to unclean client-side disconnects.

That doesn’t seem to match the following?

@KrishnaIyerEaswaran2

we’re going to deploy another update where the server supports WebSocket Pings. This will keep TTIG connections alive even when there’s no upstream data. The ping interval is set to 30s.

I still see people referring to not getting downlink due to inactivity:

@Jeff-UK

If TTIG has been offline/disconnected its websockets connection with the server due to time out then uplinks inc Join Req’s may get ‘missed’ until websockets connection reinitialised and established.

It’s unclear to me if that should have been fixed.

[KrishnaIyerEaswaran2 (Krishna Iyer)]

I tried testing this locally but I’m unable to reproduce it. Even with the shitty WiFi in my apartment, the TTIG does not disconnect periodically or when it does, comes back online and can route packets immediately.

@arjanvanb: What doesn’t match? The reconnection logic and server side WS pings work together and are not opposite features.

My debugging is not being helped by the fact that a lot of people are reporting (here and on Slack) that the gateways are disconnected with it’s only the NOC/Console not displaying the connection.

Can someone who’s facing multiple disconnections (i.e., the gateway LED rapidly blinking often before stabilising) please send me your EUI so I can trace it on the server?