Downlink error in v3 UI - message not arriving at gateway nor device

TTN V2 to V3 Upgrade V3 Integrations
1

Hi all,

we have successfully migrated our acceptance environment to the new v3 stack and so far nearly everything is working fine. Unfortunately one important part is not, which is sending messages to the devices (downlink).

We can schedule the downlink messages and we can see them being received and put onto the queue in the v3 live data view but when it comes to actually delivering them to the device, they do not arrive. Neither are the downlink messages seen on the Gateway, which we watched locally as well as on TTN.

On TTN we get the following error message in the Gateway console:

image

But the data json itself looks fine to me so I am asking here, is this something that has to do with the data we are sending or is this something on TTN side of things?

We tested both, scheduling the downlink message through the TTN UI as well as through the API
image
image

I have attached the json messages from the TTN UI for reference.

Maybe you can point me in the right direction to get this working… :thinking:

Thanks for any help.

ttnv3-receive-downlink.json (1.0 KB) ttnv3-forwards-downlink.json (1.0 KB)
ttnv3-downlink-error.json (1.6 KB)

2

This started a few day ago…
My two gateway that are on Ver3 are seeing an odd error. both are LPS8 with current firmware…
traffic seems to continue working…

There was an error and the event cannot be displayed. The raw event can by viewed by clicking this row.

below is the event display…

{
  "name": "gs.down.send",
  "time": "2021-04-22T23:30:56.661966274Z",
  "identifiers": [
    {
      "gateway_ids": {
        "gateway_id": "lps8-9old",
        "eui": "A840411EF1C04150"
      }
    }
  ],
  "data": {
    "@type": "type.googleapis.com/ttn.lorawan.v3.DownlinkMessage",
    "raw_payload": "YIwaDCagQhh6PYln",
    "request": {
      "downlink_paths": [
        {
          "uplink_token": "ChcKFQoJbHBzOC05b2xkEgioQEEe8cBBUBCT4pjJBBoLCLCOiIQGEJvu3WUguOS+6dulKQ=="
        }
      ],
      "rx1_delay": 5,
      "rx1_data_rate_index": 13,
      "rx1_frequency": "923900000",
      "rx2_data_rate_index": 8,
      "rx2_frequency": "923300000",
      "priority": "HIGHEST",
      "frequency_plan_id": "US_902_928_FSB_2"
    },
    "correlation_ids": [
      "gs:conn:01F3RMZEY908DECZBP7J6JJ3WR",
      "gs:up:host:01F3RMZF2QS579B24RX21DFAQ0",
      "gs:uplink:01F3XYR52NAQVD1RP85FZ9T93N",
      "ns:downlink:01F3XYR5GM15EY5RGHE0GDRZY1",
      "ns:uplink:01F3XYR52PC167YG1RJTP5DYNR",
      "rpc:/ttn.lorawan.v3.GsNs/HandleUplink:01F3XYR52P8ZAEW384285JC7SW",
      "gs:conn:01F3RMZEY908DECZBP7J6JJ3WR",
      "rpc:/ttn.lorawan.v3.NsGs/ScheduleDownlink:01F3XYR5GNEFECW3ZYP5GF0YT9"
    ]
  },
  "correlation_ids": [
    "gs:conn:01F3RMZEY908DECZBP7J6JJ3WR",
    "rpc:/ttn.lorawan.v3.NsGs/ScheduleDownlink:01F3XYR5GNEFECW3ZYP5GF0YT9"
  ],
  "origin": "ip-10-101-13-99.us-west-1.compute.internal",
  "context": {
    "tenant-id": "CgN0dG4="
  },
  "visibility": {
    "rights": [
      "RIGHT_GATEWAY_TRAFFIC_READ"
    ]
  },
  "unique_id": "01F3XYR5GNTZ60HC80K66W4DYC"
}
3

Moved to relevant topic, @trlafleur, please search the forum and please consider the content of the thread you are adding to for relevance. Cheers.

4

Looking at the content of the error message kindly provided by @trlafleur, it would appear that it is all about the UI not being able to display the message, not that the message itself is indicating a problem.

Can one of you please file a bug report on GitHub so this can be resolved.

5

Hi all,

is there already a solution for @sqippa-online problem? I Migrate from V2 to V3 with the UDP Packet Forwarder and have the same Issue.

Thanks for help

6

As I said, this appears to be a UI bug.

Are uplinks are flowing through your gateway?

7

Gateway Traffic:

Gateway Traffic

Node Traffic:

Node Traffic

I thought the error was with the downlink but the uplink works and I also get data in my MQTT integration.
However, I do not send any data as a downlink but only an acknowledgement.

Thanks for your Help!

8

I’m noticing the same thing and see that an issue has already been raised for this, you can track progress on the fix at: Warnings in the TTS Console for `gs.down.send` event · Issue #4078 · TheThingsNetwork/lorawan-stack · GitHub

9

FWIW, I’ve now seen this whilst testing a new device - data was passed on with no problems.

1 Like
10

Good morning all,

@descartes: My problem is not just a UI issue, the data is not arriving at the client, nor at the gateway on the client location.

@philmcm Thanks for linking to the UI github issue.

@MarkusKa Do you see the messages arrive at your device, or are you having the same issue that they don’t arrive at the device nor the local gateway?

11

Morning,

Do you mean that the downlink messages do not arrive at your device?
I only get a confirmation as a downlink that the uplink was successful and that works. I do not send any data as downlink.

The Uplink Work and so does my MQTT Integration.

12

@MarkusKa Yes correct. I am sending commands to the device to do something (e.g. restart) and I do get the schedule downlink confirmation message in the TTN console but on the device itself, nothing arrives. Also on the local gateway, the downlink message does not arrive.

13

Hi @descartes,

can I ask if you are seeing uplink data or downlink data on the device? Because it seems everyone on this thread is talking about uplink, but my issue is related to downlink messages.

So from TTN to the device

14

Hi @sqippa-online and everyone, I am having similar issue with downlink after device migrated from V2 to V3. When gateway is connected via GSM network, some times downlink reached the device, most of the time it doesn’t.

When gateway is connected via LAN, sometimes when set to a confirmed downlink, even after the device received it, the downlink msg keeps being resent. I checked with CLI that there is no Downlink queue, at first I thought maybe it is being queued on the Gateway? I restarted the Gateway and the same downlink still repeating. I checked the device logs and it did received a downlink even when CLI shows no downlink in queue and the Gateway has been restarted.

To further eliminate the possibility of queue/cache via the Internet provider since the Gateway is connected to HomeBroadband, I connect the Gateway to a Mini-Router with Vodafone GSM cellular network and the repeating downlink stopped for a few Uplink cycle and then … it re-appeared. As I mentioned earlier with GSM connection, downlink will only reach the device from time to time.

When I switch back to HomeBroadband LAN connection to Gateway, the repeating downlink returns. When check with CLI, there is nothing in downlink queue.

The weirdest behaviour is … after a different downlink was sent successfully to the device, the previous repeating downlink continue after that. I even used CLI to perform a CLEAR QUEUE but the downlink still continue being sent to the device.

The device has not been touched for all the above tests so the final test is to restart the device and see whether is it something to do with the device. Restarted the device and the downlink is still repeating.

Do apologies for the above long behaviour observations, I am trying to understand TTN V3 downlink since TTN V2 downlink works nicely. It does looks like it is some kind of synchronisation issue and maybe a bug?

Any suggestion/help would be much appreciated. Thank you very much.

15

Downlinks aren’t good. Confirmed downlinks are very very bad.

If your device doesn’t confirm the downlink the network server will retry forever, using up computer resources thereby contributing to global warming which means you will be taken off Greta’s Christmas Card list.

The console will not show a downlink awaiting confirmation.

What is your device?
Is it on OTAA or ABP?

No such thing - but if the GSM network is running slow, the downlink may not reach the gateway in time. Is the gateway on v2 or v3.

I have a test outstanding on clearing blocked confirmations which I may be able to look at later today.

16

Hi Nick (@descartes) ,

Thank you for getting back to me quickly, really appreciate it.

noooo … don’t want to be taken off the Christmas list :sweat_smile:

Could you point me to the right direction on implementing a Downlink response please? Are you referring to the u1_t confirmed parameter value set to 1 when sending the next uplink request? We updated the device code to send the next uplink with u1_t confirmed value = 1, but still getting the previous downlink.

So far we have been using LMIC library with V2 and a confirmed downlink will normally be replaced by a non-confirmed downlink and once that is sent it stop repeating.

The device is a heltec esp32 lora unit. We are using ABP, downlink is just for initial installation where signal strength and confirmation on uplink msgs are received and confirmed. Then transmissions will be set to 30 mins interval via downlink and most probably never use downlink again. However it is vital during initial installation and setup.

Gateway is Laird RG186 on V3. I tried setting the RX1 delay to different values (1 to 5) but it doesn’t seems to have any effect and downlink still unable to reach device when using GSM network. Any suggestion on how to diagnose and a good way to overcome this problem? Thanks.

Thank you and looking forward to getting downlink working reliably :blush:

17

Out of curiosity I wondered … “What if I send a different confirmed downlink … would this new confirmed downlink repeat or the previous confirmed downlink will return after …” ?

With LAN (Broadband connection to Gateway) I scheduled a different downlink via TTN V3 console, after the device received this new downlink once, the previous downlink came back again … something very strange is happening … mmm

Maybe I should try with a new device next?

18

That is saying “send uplink as confirmed”. The acknowledgement of a downlink is done automatically from within LMIC. And as such there are notes regarding ensuring you don’t sleep before LMIC has finished. EV_TXDONE is only an event telling us that it has transmitted an uplink, it doesn’t tell us if the MAC still has other work to do which could take some time if there are series of MAC commands to process - as it may have a confirmed downlink to acknowledge that it transmits that receives a further downlink that needs processing and some sort of response.

LMIC isn’t hugely helpful about indicating that it has work to do - you have to call os_queryTimeCriticalJobs() to find out if there are any jobs pending which is a bit obtuse. See EV_TXDONE and transmission complete · mcci-catena/arduino-lmic · Discussion #640 · GitHub

It’s all cool, downlinks have their time & place, just not as a routine thing IMO.

It won’t do / that’s just advanced guessing.

You will have to change DELAY_DNW1 in lorabase.h for the downlink to work with ABP on 5s which is preferable but if you leave it as is in the code base you’ll need to set that to 1s in the device settings. I’m not sure if I should be proud or depressed that my brain retains this level of detail.

MCCI LMIC 3.2.0 onwards is highly compliant - as in it processes MAC commands which is an expectation of v3. You should use LoRaWAN version 1.0.3. I’m using 3.3.0 for real but have test devices running nicely on 4.0.0. I don’t deploy client devices using LMIC, but if further testing is satisfactory and given the world-wide shortage of modules I may well do.

Anyway, here’s the exciting command I’ve just tested.

I sent my device a confirmed reboot downlink which means it never confirms command and just keeps getting the downlink again.

Using:

ttn-lw-cli dev set your-application-id the-offending-device-id --unset mac-state.pending-application-downlink

it clears the downlinks and all is well.

19

Not strange at all, you asked for a downlink to be confirmed, it hasn’t yet, so TTS NS is resending.

Hopefully the CLI command above will reset the state of affairs.

If you need to know that a device has received a downlink it is preferable to put some logic in to the firmware & your backend systems.

To emphasise how non-PC confirmed downlinks are, TTI have only just put a message on the NS to tell us if a device has confirmed a downlink. Up until now we had no indication at all that one had been confirmed.

20

Hi Nick,

Thank you. I executed the command you provide and the repeating downlink stopped … phew … :pray:

I will check my current LMIC version and upgrade it to 3.2.0 and test again.

Thank you very much.