Experiencing Packet Loss on v3 but not v2

@mat89 & @Jeff-UK - moved this as case specific and not related to migrating OTAA devices.

I’ve lost track of where I said it, but I have a project folder for hacking the latest mp_packet_forwarder to provide much clearer logs that I can then correlate with a device with an SD Card on it, a copy of FireHose.js* logging the gateway console (as we don’t appear to have that), a copy of FireHose.js logging the realtime device console, data storage, a Webhook or two and possibly an MQTT for good measure. If I can figure out how to run an SDR I can use something to log the actual RF as well, may be able to get a local Radio peep who likes tracking balloons to help.

This should then allow end to end tracing of everything. But this is NOT going to happen quickly - probably be over Q4 with data analysis in the quiet of the Mid-Winter Solstice of Excess.

* FireHose.js is a Node.js variant of my web console so it can write to file. Still need to trap for disconnects.

1 Like

Unlikely, but yes, there is a chance, then the v2 node should also suffer from packetloss…
And, the same packetloss happens if I first tunnel the messages from the v3 gateway through a tunnel to Germany (Frankfurt). I see all packets sent in Frankfurt ‘sent towards TTN’ but the loss is still the same - so I don’t think it is a local ISP issue.

Sorry, I need to be more clear here.
v3 node = v3 application and v3 device (new not migrated node)
v2 node = v2 application and v2 device (old not migrated node)

If I see something interesting in the console, I will do that.
Yes all three gateways are contributing to message handling according to the metadata:

  {
      "gateway_ids": {
        "gateway_id": "packetbroker"
      },
      "packet_broker": {
        "message_id": "01FEDYPDQ707KKHCBCS245JZJ9",
        "forwarder_net_id": "000013",
        "forwarder_tenant_id": "ttnv2",
        "forwarder_cluster_id": "ttn-v2-ch",
        "forwarder_gateway_eui": "C0EE40FFFF2940EC",
        "forwarder_gateway_id": "eui-c0ee40ffff2940ec",
        "home_network_net_id": "000013",
        "home_network_tenant_id": "ttn",
        "home_network_cluster_id": "ttn-eu1"
      },
      "rssi": -75,
      "channel_rssi": -75,
      "snr": 7.5,
      "location": {
        "latitude": 47.431269,
        "longitude": 8.465581
      },
      "uplink_token": "eyJnIjoiWlhsS2FHSkhZMmxQYVVwQ1RWUkpORkl3VGs1VE1XTnBURU5LYkdKdFRXbFBhVXBDVFZSSk5GSXdUazVKYVhkcFlWaFphVTlwU25kaE1VNU1WbGhhZEZGdGVITmtSMnhEVFVodk0wbHBkMmxrUjBadVNXcHZhVTB3VmxwWU0xSnZZakJ3VEZsWFRuaFVNVlpJWTBVMWVsZEdPREZWVTBvNUxqRXlSbFJ5TUZwYVpHbGlRbFJvU25vMGMwbHVUM2N1WkRoSlRqRnRTQzFFTUhGYVVuTkVXQzVsVGxWUmEweHRkVEJCVXkxMGEwMUJZM1Z6VTI5WlkwUndRVGN0YlZaemJqZHphbVJRVFdkaFVGOTVlWG8zVTNSbVgxcEdYMGhFYkZwd1NqZHphemRYVEVjeFRXUkhWREZ1Um05TlFVbHlPQzFOVkcwMWNtcEdOVkZrYzJRellUVlNOMGM1VTNaV1JrSktOUzB3YUVVd1F6TkpTMlZzYUdSUFlrMUdRMmh6YkVKdmRGSjZXRjltTlhNdFZGbHdha3B5VDB0aWNIWldOMGROTTFGdGFsUmpSekF6TTJjeldHTlVPVlF0U2pWWkxsWk5PR1JzYzB4Sk5WTlJXSE5uYUhWeWVGOXFkMUU9IiwiYSI6eyJmbmlkIjoiMDAwMDEzIiwiZnRpZCI6InR0bnYyIiwiZmNpZCI6InR0bi12Mi1jaCJ9fQ=="
    },
    {
      "gateway_ids": {
        "gateway_id": "packetbroker"
      },
      "packet_broker": {
        "message_id": "01FEDYPDQC6QWG016AX4NKNHM0",
        "forwarder_net_id": "000013",
        "forwarder_tenant_id": "ttnv2",
        "forwarder_cluster_id": "ttn-v2-ch",
        "forwarder_gateway_eui": "313330371D005500",
        "forwarder_gateway_id": "eui-313330371d005500",
        "home_network_net_id": "000013",
        "home_network_tenant_id": "ttn",
        "home_network_cluster_id": "ttn-eu1"
      },
      "time": "2021-08-31T11:14:48.337980Z",
      "rssi": -68,
      "channel_rssi": -68,
      "snr": 7,
      "uplink_token": "eyJnIjoiWlhsS2FHSkhZMmxQYVVwQ1RWUkpORkl3VGs1VE1XTnBURU5LYkdKdFRXbFBhVXBDVFZSSk5GSXdUazVKYVhkcFlWaFphVTlwU2xwaVJrWmFWRlJqZUZkWE1UUlRTRVkyV2xkM01rbHBkMmxrUjBadVNXcHZhV0ZUTVhSa1JXaERWRVJTV2s0emF6QlBWV1JPVkVkR01XRXpTbE5WVTBvNUxsOU1iMHhCZHpabmEwYzViVFpNYkV4a1pXUkVTM2N1TVZKblJESkxZMUJ4T0VsR2JVOVNaQzVNUVRKQlZXWlZaRlZCZWpWTVNsUnJSMHM0VnpNemVUWXlNelZXUVhGbFV6VklSRjlDVmxWcU5scGhTWEZIVkVsQlpsZG1ZMll3VldaRmVtWnZjRVZoUmtRek1uTmFTR1JwZEd4QlExZG1NREZZYWpaSWRFMVZUbkZVY0hKRk16Rk9MVXBTZGtaRloxZE5NRTVCTWtvMWRVRnRhVlUwZVRGb1IwaEpNM2xsYW05UmNDMTNWbmwwYW14YU1VaHNZMFJ0WkRCVFVuSnhSREpYT0VZeU9FeHhMV1E1WDAxU1QwdFFhVVJqVDB4ekxsbEZPR05vTFVRNGRqQnBkR2hRVWtKc0xTMXdOMUU9IiwiYSI6eyJmbmlkIjoiMDAwMDEzIiwiZnRpZCI6InR0bnYyIiwiZmNpZCI6InR0bi12Mi1jaCJ9fQ=="
    },
    {
      "gateway_ids": {
        "gateway_id": "v3-testgw",
        "eui": "C0EE40FFFF2940E9"
      },
      "timestamp": 3272836931,
      "rssi": -89,
      "channel_rssi": -89,
      "snr": 7.8,
      "uplink_token": "ChcKFQoJdjMtdGVzdGd3EgjA7kD//ylA6RDDjs6YDBoMCKieuIkGEPngmMYCILi7oaKgnRY=",
      "channel_index": 2
    }
  ],

@Jeff-UK so I discovered today loss in another situation…

Device and App registered to EU1v3, one gateway in reach, also registered to EU1v3.
Gateway is a Mikrotik UDP gateway again (yes I know, UDP…) with LTE (yes I know, LTE, baaaad!).
I see the uplink from the node in the TTN gateway ‘Live data’ tab, but not in the TTN device ‘Live tab’.
Also, the packet is missing on all my three Webhook endpoints (which are in different datacenters).

So for me it looks cleary that TTN is loosing messages within their own system which should not happen, at least not on a regular basis… and no packet broker involved… and yes, the packet arrived at TTN side :wink:

Metadata of the correctly processed uplink:

{
  "name": "gs.up.receive",
  "time": "2021-08-31T20:03:49.745908419Z",
  "identifiers": [
    {
      "gateway_ids": {
        "gateway_id": "hir-ttn01v3"
      }
    },
    {
      "gateway_ids": {
        "gateway_id": "hir-ttn01v3",
        "eui": "4836372047001D00"
      }
    }
  ],
  "data": {
    "@type": "type.googleapis.com/ttn.lorawan.v3.UplinkMessage",
    "raw_payload": "QEIKCyYA3CcBE01ET2RMCTrWtOHygexSMP0=",
    "payload": {
      "m_hdr": {
        "m_type": "UNCONFIRMED_UP"
      },
      "mic": "7FIw/Q==",
      "mac_payload": {
        "f_hdr": {
          "dev_addr": "260B0A42",
          "f_ctrl": {},
          "f_cnt": 10204
        },
        "f_port": 1,
        "frm_payload": "E01ET2RMCTrWtOHygQ=="
      }
    },
    "settings": {
      "data_rate": {
        "lora": {
          "bandwidth": 125000,
          "spreading_factor": 7
        }
      },
      "coding_rate": "4/5",
      "frequency": "868300000",
      "timestamp": 2394773225,
      "time": "2021-08-31T20:03:50.991572Z"
    },
    "rx_metadata": [
      {
        "gateway_ids": {
          "gateway_id": "hir-ttn01v3",
          "eui": "4836372047001D00"
        },
        "time": "2021-08-31T20:03:50.991572Z",
        "timestamp": 2394773225,
        "rssi": -73,
        "channel_rssi": -73,
        "snr": 9.25,
        "uplink_token": "ChkKFwoLaGlyLXR0bjAxdjMSCEg2NyBHAB0AEOmt9fUIGgwIpZa6iQYQ8dDP4wIgqPztnNnrHQ==",
        "channel_index": 1
      }
    ],
    "received_at": "2021-08-31T20:03:49.745793649Z",
    "correlation_ids": [
      "gs:conn:01FEB3H9WXPK7GD0AZ7K6AZVRK",
      "gs:uplink:01FEEWZ2VHYCPYDT6SE3J9PJYT"
    ]
  },
  "correlation_ids": [
    "gs:conn:01FEB3H9WXPK7GD0AZ7K6AZVRK",
    "gs:uplink:01FEEWZ2VHYCPYDT6SE3J9PJYT"
  ],
  "origin": "ip-10-100-5-46.eu-west-1.compute.internal",
  "context": {
    "tenant-id": "CgN0dG4="
  },
  "visibility": {
    "rights": [
      "RIGHT_GATEWAY_TRAFFIC_READ",
      "RIGHT_GATEWAY_TRAFFIC_READ"
    ]
  },
  "unique_id": "01FEEWZ2VHFM930T1MTXMT9QFX"
}

Metadata of the missing uplink:

{
  "name": "gs.up.receive",
  "time": "2021-08-31T20:08:49.148113249Z",
  "identifiers": [
    {
      "gateway_ids": {
        "gateway_id": "hir-ttn01v3"
      }
    },
    {
      "gateway_ids": {
        "gateway_id": "hir-ttn01v3",
        "eui": "4836372047001D00"
      }
    }
  ],
  "data": {
    "@type": "type.googleapis.com/ttn.lorawan.v3.UplinkMessage",
    "raw_payload": "QEIKCyYA3ScBkpJWAk1cyoAv+Gti376IhpE=",
    "payload": {
      "m_hdr": {
        "m_type": "UNCONFIRMED_UP"
      },
      "mic": "voiGkQ==",
      "mac_payload": {
        "f_hdr": {
          "dev_addr": "260B0A42",
          "f_ctrl": {},
          "f_cnt": 10205
        },
        "f_port": 1,
        "frm_payload": "kpJWAk1cyoAv+Gti3w=="
      }
    },
    "settings": {
      "data_rate": {
        "lora": {
          "bandwidth": 125000,
          "spreading_factor": 7
        }
      },
      "coding_rate": "4/5",
      "frequency": "868500000",
      "timestamp": 2694196729,
      "time": "2021-08-31T20:08:50.415727Z"
    },
    "rx_metadata": [
      {
        "gateway_ids": {
          "gateway_id": "hir-ttn01v3",
          "eui": "4836372047001D00"
        },
        "time": "2021-08-31T20:08:50.415727Z",
        "timestamp": 2694196729,
        "rssi": -78,
        "channel_rssi": -78,
        "snr": 8.25,
        "uplink_token": "ChkKFwoLaGlyLXR0bjAxdjMSCEg2NyBHAB0AEPnb2IQKGgsI0Zi6iQYQrZnIRiCoqY7VtPQd",
        "channel_index": 2
      }
    ],
    "received_at": "2021-08-31T20:08:49.147983533Z",
    "correlation_ids": [
      "gs:conn:01FEB3H9WXPK7GD0AZ7K6AZVRK",
      "gs:uplink:01FEEX877WCQEYNSYM67DA29CP"
    ]
  },
  "correlation_ids": [
    "gs:conn:01FEB3H9WXPK7GD0AZ7K6AZVRK",
    "gs:uplink:01FEEX877WCQEYNSYM67DA29CP"
  ],
  "origin": "ip-10-100-5-46.eu-west-1.compute.internal",
  "context": {
    "tenant-id": "CgN0dG4="
  },
  "visibility": {
    "rights": [
      "RIGHT_GATEWAY_TRAFFIC_READ",
      "RIGHT_GATEWAY_TRAFFIC_READ"
    ]
  },
  "unique_id": "01FEEX877W5FJJ11Z7ED57Z1CS"
}

And the screenshots:

Gateway live data:
2021-08-31 22_18_34-Gateway data - hir-ttn01v3 - The Things Network

Device live data:
2021-08-31 22_19_15-Live data - fi-weather-dev01 - The Things Network

2 Likes

Cool, something to dig in to.

Do you have data storage turned on? If not, please do, as that’s all internal to the servers which can be cross-referenced with the Webhooks and any future examples you get screen shots of.

No, never used it.
I’ve enabled it now… but have to check the docs how to get some useful infos out of it…

This works well: TheThingsStack-Integration-Starters/DataStorage-to-Tab-Python3 at main · descartes/TheThingsStack-Integration-Starters · GitHub

You need to fill in the top two lines. It’s a starter script so not so bright so it will download whatever you ask, it won’t deduplicate or look at previous downloads. Data storage only holds about 36 hours worth of data on v3 so needs to be kicked off at least once a day.

In the repro is also the same sort of functionality for Webhooks - no configuration required. I can set one up on a server if you’d like. There is also the same thing but with MQTT.

1 Like

Today morning, for the node which is in reach by the two v2 gateways and in reach by one v3 gateway I’ve captured a missing uplink.
Again it is shown in the ‘live data’ tab of the v3 gateway, but not in the devices view ‘live data’.

Metadata of the correctly processed uplink:

{
  "name": "gs.up.receive",
  "time": "2021-09-01T03:58:08.100765704Z",
  "identifiers": [
    {
      "gateway_ids": {
        "gateway_id": "v3-testgw"
      }
    },
    {
      "gateway_ids": {
        "gateway_id": "v3-testgw",
        "eui": "C0EE40FFFF2940E9"
      }
    }
  ],
  "data": {
    "@type": "type.googleapis.com/ttn.lorawan.v3.UplinkMessage",
    "raw_payload": "QFNPCyYA/gIBkK+R/kYLjG9glRc8+bznOpU=",
    "payload": {
      "m_hdr": {
        "m_type": "UNCONFIRMED_UP"
      },
      "mic": "vOc6lQ==",
      "mac_payload": {
        "f_hdr": {
          "dev_addr": "260B4F53",
          "f_ctrl": {},
          "f_cnt": 766
        },
        "f_port": 1,
        "frm_payload": "kK+R/kYLjG9glRc8+Q=="
      }
    },
    "settings": {
      "data_rate": {
        "lora": {
          "bandwidth": 125000,
          "spreading_factor": 7
        }
      },
      "coding_rate": "4/5",
      "frequency": "867300000",
      "timestamp": 4175510243
    },
    "rx_metadata": [
      {
        "gateway_ids": {
          "gateway_id": "v3-testgw",
          "eui": "C0EE40FFFF2940E9"
        },
        "timestamp": 4175510243,
        "rssi": -91,
        "channel_rssi": -91,
        "snr": 10,
        "uplink_token": "ChcKFQoJdjMtdGVzdGd3EgjA7kD//ylA6RDj9YTHDxoLCND0u4kGEITX/i8guI24/sLwAw==",
        "channel_index": 4
      }
    ],
    "received_at": "2021-09-01T03:58:08.100641668Z",
    "correlation_ids": [
      "gs:conn:01FEF7V0BJQHC31NP9WF5RWG18",
      "gs:uplink:01FEFR3J74YCBH3SZVE285GRK3"
    ]
  },
  "correlation_ids": [
    "gs:conn:01FEF7V0BJQHC31NP9WF5RWG18",
    "gs:uplink:01FEFR3J74YCBH3SZVE285GRK3"
  ],
  "origin": "ip-10-100-5-46.eu-west-1.compute.internal",
  "context": {
    "tenant-id": "CgN0dG4="
  },
  "visibility": {
    "rights": [
      "RIGHT_GATEWAY_TRAFFIC_READ",
      "RIGHT_GATEWAY_TRAFFIC_READ"
    ]
  },
  "unique_id": "01FEFR3J74J8WCMG6JBPJEWC6H"
}

Metadata of the missing uplink:

{
  "name": "gs.up.receive",
  "time": "2021-09-01T04:03:07.693527097Z",
  "identifiers": [
    {
      "gateway_ids": {
        "gateway_id": "v3-testgw"
      }
    },
    {
      "gateway_ids": {
        "gateway_id": "v3-testgw",
        "eui": "C0EE40FFFF2940E9"
      }
    }
  ],
  "data": {
    "@type": "type.googleapis.com/ttn.lorawan.v3.UplinkMessage",
    "raw_payload": "QFNPCyYA/wIBD3aTUlLREeufRkdWg9KWK1w=",
    "payload": {
      "m_hdr": {
        "m_type": "UNCONFIRMED_UP"
      },
      "mic": "0pYrXA==",
      "mac_payload": {
        "f_hdr": {
          "dev_addr": "260B4F53",
          "f_ctrl": {},
          "f_cnt": 767
        },
        "f_port": 1,
        "frm_payload": "D3aTUlLREeufRkdWgw=="
      }
    },
    "settings": {
      "data_rate": {
        "lora": {
          "bandwidth": 125000,
          "spreading_factor": 7
        }
      },
      "coding_rate": "4/5",
      "frequency": "868500000",
      "timestamp": 180152179
    },
    "rx_metadata": [
      {
        "gateway_ids": {
          "gateway_id": "v3-testgw",
          "eui": "C0EE40FFFF2940E9"
        },
        "timestamp": 180152179,
        "rssi": -83,
        "channel_rssi": -83,
        "snr": 8,
        "uplink_token": "ChcKFQoJdjMtdGVzdGd3EgjA7kD//ylA6RDzzvNVGgwI+/a7iQYQ2ri5ygIguLKgj5/5Aw==",
        "channel_index": 2
      }
    ],
    "received_at": "2021-09-01T04:03:07.693001306Z",
    "correlation_ids": [
      "gs:conn:01FEF7V0BJQHC31NP9WF5RWG18",
      "gs:uplink:01FEFRCPSDWT0HEYVPDJ8PX5KP"
    ]
  },
  "correlation_ids": [
    "gs:conn:01FEF7V0BJQHC31NP9WF5RWG18",
    "gs:uplink:01FEFRCPSDWT0HEYVPDJ8PX5KP"
  ],
  "origin": "ip-10-100-5-46.eu-west-1.compute.internal",
  "context": {
    "tenant-id": "CgN0dG4="
  },
  "visibility": {
    "rights": [
      "RIGHT_GATEWAY_TRAFFIC_READ",
      "RIGHT_GATEWAY_TRAFFIC_READ"
    ]
  },
  "unique_id": "01FEFRCPSD8FW5WW5HRHHH84TN"
}

And the screenshots:

Gateway live data:
2021-09-01 10_12_38-

Device live data:
2021-09-01 10_13_25-

Nick @descartes has flagged your issue/experience to the TTI core team to investigate, so if you see more examples please post as the more ‘evidence’ & instances we have the better to try and trap if issue is real/repeatable and why…

One more from today morning, for the node which is in reach by the two v2 gateways and in reach by one v3 gateway I’ve captured a missing uplink.
Again it is shown in the ‘live data’ tab of the v3 gateway, but not in the devices view ‘live data’.

Metadata of the correctly processed uplink:

{
  "name": "gs.up.receive",
  "time": "2021-09-01T08:07:46.726035599Z",
  "identifiers": [
    {
      "gateway_ids": {
        "gateway_id": "v3-testgw"
      }
    },
    {
      "gateway_ids": {
        "gateway_id": "v3-testgw",
        "eui": "C0EE40FFFF2940E9"
      }
    }
  ],
  "data": {
    "@type": "type.googleapis.com/ttn.lorawan.v3.UplinkMessage",
    "raw_payload": "QFNPCyYAMQMBFej8rkkOHT9DxKyoUCfnqyg=",
    "payload": {
      "m_hdr": {
        "m_type": "UNCONFIRMED_UP"
      },
      "mic": "J+erKA==",
      "mac_payload": {
        "f_hdr": {
          "dev_addr": "260B4F53",
          "f_ctrl": {},
          "f_cnt": 817
        },
        "f_port": 1,
        "frm_payload": "Fej8rkkOHT9DxKyoUA=="
      }
    },
    "settings": {
      "data_rate": {
        "lora": {
          "bandwidth": 125000,
          "spreading_factor": 7
        }
      },
      "coding_rate": "4/5",
      "frequency": "868100000",
      "timestamp": 1973979307
    },
    "rx_metadata": [
      {
        "gateway_ids": {
          "gateway_id": "v3-testgw",
          "eui": "C0EE40FFFF2940E9"
        },
        "timestamp": 1973979307,
        "rssi": -88,
        "channel_rssi": -88,
        "snr": 9.2,
        "uplink_token": "ChcKFQoJdjMtdGVzdGd3EgjA7kD//ylA6RCrkaKtBxoMCNLpvIkGEJncycQCIPi319K5pAc="
      }
    ],
    "received_at": "2021-09-01T08:07:46.680685081Z",
    "correlation_ids": [
      "gs:conn:01FEF7V0BJQHC31NP9WF5RWG18",
      "gs:uplink:01FEG6CNS6X84CKTJHS8P59XA1"
    ]
  },
  "correlation_ids": [
    "gs:conn:01FEF7V0BJQHC31NP9WF5RWG18",
    "gs:uplink:01FEG6CNS6X84CKTJHS8P59XA1"
  ],
  "origin": "ip-10-100-5-46.eu-west-1.compute.internal",
  "context": {
    "tenant-id": "CgN0dG4="
  },
  "visibility": {
    "rights": [
      "RIGHT_GATEWAY_TRAFFIC_READ",
      "RIGHT_GATEWAY_TRAFFIC_READ"
    ]
  },
  "unique_id": "01FEG6CNS6A7WYG6EZ13WRWRN9"
}

Metadata of the missing uplink:

{
  "name": "gs.up.receive",
  "time": "2021-09-01T08:12:45.971283817Z",
  "identifiers": [
    {
      "gateway_ids": {
        "gateway_id": "v3-testgw"
      }
    },
    {
      "gateway_ids": {
        "gateway_id": "v3-testgw",
        "eui": "C0EE40FFFF2940E9"
      }
    }
  ],
  "data": {
    "@type": "type.googleapis.com/ttn.lorawan.v3.UplinkMessage",
    "raw_payload": "QFNPCyYAMgMBbgnj4bWAmssoJDR2gTmkcmc=",
    "payload": {
      "m_hdr": {
        "m_type": "UNCONFIRMED_UP"
      },
      "mic": "OaRyZw==",
      "mac_payload": {
        "f_hdr": {
          "dev_addr": "260B4F53",
          "f_ctrl": {},
          "f_cnt": 818
        },
        "f_port": 1,
        "frm_payload": "bgnj4bWAmssoJDR2gQ=="
      }
    },
    "settings": {
      "data_rate": {
        "lora": {
          "bandwidth": 125000,
          "spreading_factor": 7
        }
      },
      "coding_rate": "4/5",
      "frequency": "867300000",
      "timestamp": 2273462987
    },
    "rx_metadata": [
      {
        "gateway_ids": {
          "gateway_id": "v3-testgw",
          "eui": "C0EE40FFFF2940E9"
        },
        "timestamp": 2273462987,
        "rssi": -94,
        "channel_rssi": -94,
        "snr": 9,
        "uplink_token": "ChcKFQoJdjMtdGVzdGd3EgjA7kD//ylA6RDLlYm8CBoMCP3rvIkGELSth88DIPjR0KeVrQc=",
        "channel_index": 4
      }
    ],
    "received_at": "2021-09-01T08:12:45.971101876Z",
    "correlation_ids": [
      "gs:conn:01FEF7V0BJQHC31NP9WF5RWG18",
      "gs:uplink:01FEG6NT0K2SRE99FAAA9FZ69G"
    ]
  },
  "correlation_ids": [
    "gs:conn:01FEF7V0BJQHC31NP9WF5RWG18",
    "gs:uplink:01FEG6NT0K2SRE99FAAA9FZ69G"
  ],
  "origin": "ip-10-100-5-46.eu-west-1.compute.internal",
  "context": {
    "tenant-id": "CgN0dG4="
  },
  "visibility": {
    "rights": [
      "RIGHT_GATEWAY_TRAFFIC_READ",
      "RIGHT_GATEWAY_TRAFFIC_READ"
    ]
  },
  "unique_id": "01FEG6NT0KTX5CZ6928XH0GPQH"
}

And the screenshots:
Gateway live data:
2021-09-01 13_32_50-10.0.0.20 - Remotedesktopverbindung

Device live data:
2021-09-01 13_33_28-10.0.0.20 - Remotedesktopverbindung

Does it appear in Data Storage?

And one more missing from today morning for the node which is only connected through a v3 gateway.

Metadata of the correctly processed uplink:

{
  "name": "gs.up.receive",
  "time": "2021-09-01T09:17:24.611990238Z",
  "identifiers": [
    {
      "gateway_ids": {
        "gateway_id": "hir-ttn01v3"
      }
    },
    {
      "gateway_ids": {
        "gateway_id": "hir-ttn01v3",
        "eui": "4836372047001D00"
      }
    }
  ],
  "data": {
    "@type": "type.googleapis.com/ttn.lorawan.v3.UplinkMessage",
    "raw_payload": "QEIKCyYAeygBublRh2GZrgjfAsQDPVf8Wrc=",
    "payload": {
      "m_hdr": {
        "m_type": "UNCONFIRMED_UP"
      },
      "mic": "V/xatw==",
      "mac_payload": {
        "f_hdr": {
          "dev_addr": "260B0A42",
          "f_ctrl": {},
          "f_cnt": 10363
        },
        "f_port": 1,
        "frm_payload": "ublRh2GZrgjfAsQDPQ=="
      }
    },
    "settings": {
      "data_rate": {
        "lora": {
          "bandwidth": 125000,
          "spreading_factor": 7
        }
      },
      "coding_rate": "4/5",
      "frequency": "868100000",
      "timestamp": 2764948561,
      "time": "2021-09-01T09:17:25.929362Z"
    },
    "rx_metadata": [
      {
        "gateway_ids": {
          "gateway_id": "hir-ttn01v3",
          "eui": "4836372047001D00"
        },
        "time": "2021-09-01T09:17:25.929362Z",
        "timestamp": 2764948561,
        "rssi": -77,
        "channel_rssi": -77,
        "snr": 9,
        "uplink_token": "ChkKFwoLaGlyLXR0bjAxdjMSCEg2NyBHAB0AENGIt6YKGgwIpIq9iQYQ/KyEoAIg6LibnrzVKA=="
      }
    ],
    "received_at": "2021-09-01T09:17:24.604051068Z",
    "correlation_ids": [
      "gs:conn:01FEB3H9WXPK7GD0AZ7K6AZVRK",
      "gs:uplink:01FEGAC5R3EH3Q1W2Y698CSNBY"
    ]
  },
  "correlation_ids": [
    "gs:conn:01FEB3H9WXPK7GD0AZ7K6AZVRK",
    "gs:uplink:01FEGAC5R3EH3Q1W2Y698CSNBY"
  ],
  "origin": "ip-10-100-5-46.eu-west-1.compute.internal",
  "context": {
    "tenant-id": "CgN0dG4="
  },
  "visibility": {
    "rights": [
      "RIGHT_GATEWAY_TRAFFIC_READ",
      "RIGHT_GATEWAY_TRAFFIC_READ"
    ]
  },
  "unique_id": "01FEGAC5R3R327JN6V6AW4SQXM"
}

Metadata of the missing uplink:

{
  "name": "gs.up.receive",
  "time": "2021-09-01T09:22:24.106628075Z",
  "identifiers": [
{
  "gateway_ids": {
    "gateway_id": "hir-ttn01v3"
  }
},
{
  "gateway_ids": {
    "gateway_id": "hir-ttn01v3",
    "eui": "4836372047001D00"
  }
}
  ],
  "data": {
"@type": "type.googleapis.com/ttn.lorawan.v3.UplinkMessage",
"raw_payload": "QEIKCyYAfCgBDVwadgF8GAsEz3XIAW/04W8=",
"payload": {
  "m_hdr": {
    "m_type": "UNCONFIRMED_UP"
  },
  "mic": "b/Thbw==",
  "mac_payload": {
    "f_hdr": {
      "dev_addr": "260B0A42",
      "f_ctrl": {},
      "f_cnt": 10364
    },
    "f_port": 1,
    "frm_payload": "DVwadgF8GAsEz3XIAQ=="
  }
},
"settings": {
  "data_rate": {
    "lora": {
      "bandwidth": 125000,
      "spreading_factor": 7
    }
  },
  "coding_rate": "4/5",
  "frequency": "868500000",
  "timestamp": 3064348225,
  "time": "2021-09-01T09:22:25.329676Z"
},
"rx_metadata": [
  {
    "gateway_ids": {
      "gateway_id": "hir-ttn01v3",
      "eui": "4836372047001D00"
    },
    "time": "2021-09-01T09:22:25.329676Z",
    "timestamp": 3064348225,
    "rssi": -73,
    "channel_rssi": -73,
    "snr": 8,
    "uplink_token": "ChkKFwoLaGlyLXR0bjAxdjMSCEg2NyBHAB0AEMH8mLULGgwIz4y9iQYQwtHw1AMg6NuMy5feKA==",
    "channel_index": 2
  }
],
"received_at": "2021-09-01T09:22:23.983312578Z",
"correlation_ids": [
  "gs:conn:01FEB3H9WXPK7GD0AZ7K6AZVRK",
  "gs:uplink:01FEGANA7A7C1CHHN81XTW19AQ"
]
  },
  "correlation_ids": [
"gs:conn:01FEB3H9WXPK7GD0AZ7K6AZVRK",
"gs:uplink:01FEGANA7A7C1CHHN81XTW19AQ"
  ],
  "origin": "ip-10-100-5-46.eu-west-1.compute.internal",
  "context": {
"tenant-id": "CgN0dG4="
  },
  "visibility": {
"rights": [
  "RIGHT_GATEWAY_TRAFFIC_READ",
  "RIGHT_GATEWAY_TRAFFIC_READ"
]
  },
  "unique_id": "01FEGANA7AN90YFHGG36CC64EW"
}

And the screenshots:
Gateway live data:
gw
Device live data:
node

I will check that later and update here once more :slight_smile:

Hello, the script downloads some rows yes, but then exits with an error:

Traceback (most recent call last):
File “TTS.DataStorage.Tab.ch-weather.py”, line 76, in
f_port = uplink_message[“f_port”];
KeyError: ‘f_port’

So I was at least able to download the interesting rows, and there I can clearly see, that the missing uplinks are also missing in the storage integration.

Yes it will, it’s an homage to my favourite grumpy programmer:

If you want to fix it, I use this in production code:

f_port = theJSON.get(“f_port”, 0)

You’ll need to do the same for any other fields that may return as zero, null, blank or are uninitialised.

In my not very humble opinion it’s a mistake in the design of Google Protocol Buffers translation utilities to JSON that has been perpetuated by programmers half my age because they all think Google has been around forever so much be right whereas I was coding for hire before the web even existed.

@johan and @htdvisser, per my PM, we have two examples and some correlation from Data Storage, so something appears to be amiss here.

I page them on Slack as well.

@mat89, can you confirm which connection / join method you are using.

I know you originally posted this in the “How to Migrate OTAA Devices from V2 to V3” topic but support have noted that if it there is some channel / frequency configuration issue - both dropped packets are on 868.5 - then the network server may not be able to process the uplink correctly.

I’m using OTAA, I attached the ‘Network Layer’ settings I’m using on EU1v3 for the devices as screenshot.
2021-09-02 19_11_21-General settings - fi-weather-dev01 - The Things Network - Brave

My nodes are based on Arduinos with an RFM95 module, the library I use is ‘MCCI LoRaWAN LMIC library’, the version of the library I’m using is v4.0.0, so it is the latest version.
The libary is of course configured for 868MHz in ‘lmic_project_config.h’ :

// project-specific definitions
#define CFG_eu868 1
//#define CFG_us915 1
//#define CFG_au915 1
//#define CFG_as923 1
// #define LMIC_COUNTRY_CODE LMIC_COUNTRY_CODE_JP	/* for as923-JP */
//#define CFG_kr920 1
//#define CFG_in866 1
#define CFG_sx1276_radio 1
//#define LMIC_USE_INTERRUPTS

I checked in my backend, at least I see also processed uplinks with 868.5mhz, so not all 868.5mhz uplinks are lost.

Good info, thanks.

And this too!

@descartes here an example of a lost uplink which is not at 868.5MHz.
EU1 v3 Application/Device with one UDP gateway in reach which is also registered in EU1 v3.

Metadata of the correctly processed uplink:

{
  "name": "gs.up.receive",
  "time": "2021-09-03T18:33:46.886265454Z",
  "identifiers": [
    {
      "gateway_ids": {
        "gateway_id": "hir-ttn01v3"
      }
    },
    {
      "gateway_ids": {
        "gateway_id": "hir-ttn01v3",
        "eui": "4836372047001D00"
      }
    }
  ],
  "data": {
    "@type": "type.googleapis.com/ttn.lorawan.v3.UplinkMessage",
    "raw_payload": "QPOzCyYATgABO/LC8hhbcH2f",
    "payload": {
      "m_hdr": {
        "m_type": "UNCONFIRMED_UP"
      },
      "mic": "W3B9nw==",
      "mac_payload": {
        "f_hdr": {
          "dev_addr": "260BB3F3",
          "f_ctrl": {},
          "f_cnt": 78
        },
        "f_port": 1,
        "frm_payload": "O/LC8hg="
      }
    },
    "settings": {
      "data_rate": {
        "lora": {
          "bandwidth": 125000,
          "spreading_factor": 7
        }
      },
      "coding_rate": "4/5",
      "frequency": "868100000",
      "timestamp": 2553430643,
      "time": "2021-09-03T18:33:47.139654Z"
    },
    "rx_metadata": [
      {
        "gateway_ids": {
          "gateway_id": "hir-ttn01v3",
          "eui": "4836372047001D00"
        },
        "time": "2021-09-03T18:33:47.139654Z",
        "timestamp": 2553430643,
        "rssi": -73,
        "channel_rssi": -73,
        "snr": 9,
        "uplink_token": "ChkKFwoLaGlyLXR0bjAxdjMSCEg2NyBHAB0AEPOEycEJGgwIitXJiQYQsvrGpgMguKLOoqiIFg=="
      }
    ],
    "received_at": "2021-09-03T18:33:46.886160690Z",
    "correlation_ids": [
      "gs:conn:01FEKJJD5D0GTQX6M9SSC6H9X1",
      "gs:uplink:01FEPF0BM6FQRHCZESF9PBJ4BW"
    ]
  },
  "correlation_ids": [
    "gs:conn:01FEKJJD5D0GTQX6M9SSC6H9X1",
    "gs:uplink:01FEPF0BM6FQRHCZESF9PBJ4BW"
  ],
  "origin": "ip-10-100-5-46.eu-west-1.compute.internal",
  "context": {
    "tenant-id": "CgN0dG4="
  },
  "visibility": {
    "rights": [
      "RIGHT_GATEWAY_TRAFFIC_READ",
      "RIGHT_GATEWAY_TRAFFIC_READ"
    ]
  },
  "unique_id": "01FEPF0BM6AG2MTX9RFAGXAVV8"
}

Metadata of missing uplink:

{
  "name": "gs.up.receive",
  "time": "2021-09-03T18:28:49.987885825Z",
  "identifiers": [
    {
      "gateway_ids": {
        "gateway_id": "hir-ttn01v3"
      }
    },
    {
      "gateway_ids": {
        "gateway_id": "hir-ttn01v3",
        "eui": "4836372047001D00"
      }
    }
  ],
  "data": {
    "@type": "type.googleapis.com/ttn.lorawan.v3.UplinkMessage",
    "raw_payload": "QPOzCyYATQABUk9iGW7UtTck",
    "payload": {
      "m_hdr": {
        "m_type": "UNCONFIRMED_UP"
      },
      "mic": "1LU3JA==",
      "mac_payload": {
        "f_hdr": {
          "dev_addr": "260BB3F3",
          "f_ctrl": {},
          "f_cnt": 77
        },
        "f_port": 1,
        "frm_payload": "Uk9iGW4="
      }
    },
    "settings": {
      "data_rate": {
        "lora": {
          "bandwidth": 125000,
          "spreading_factor": 7
        }
      },
      "coding_rate": "4/5",
      "frequency": "867700000",
      "timestamp": 2256425572,
      "time": "2021-09-03T18:28:50.133441Z"
    },
    "rx_metadata": [
      {
        "gateway_ids": {
          "gateway_id": "hir-ttn01v3",
          "eui": "4836372047001D00"
        },
        "time": "2021-09-03T18:28:50.133441Z",
        "timestamp": 2256425572,
        "rssi": -72,
        "channel_rssi": -72,
        "snr": 9.25,
        "uplink_token": "ChkKFwoLaGlyLXR0bjAxdjMSCEg2NyBHAB0AEOSk+bMIGgwI4dLJiQYQ/6PEqgMgoK3H69X/FQ==",
        "channel_index": 6
      }
    ],
    "received_at": "2021-09-03T18:28:49.894505471Z",
    "correlation_ids": [
      "gs:conn:01FEKJJD5D0GTQX6M9SSC6H9X1",
      "gs:uplink:01FEPEQ9P3P1XKTDTNZFGGNANE"
    ]
  },
  "correlation_ids": [
    "gs:conn:01FEKJJD5D0GTQX6M9SSC6H9X1",
    "gs:uplink:01FEPEQ9P3P1XKTDTNZFGGNANE"
  ],
  "origin": "ip-10-100-5-46.eu-west-1.compute.internal",
  "context": {
    "tenant-id": "CgN0dG4="
  },
  "visibility": {
    "rights": [
      "RIGHT_GATEWAY_TRAFFIC_READ",
      "RIGHT_GATEWAY_TRAFFIC_READ"
    ]
  },
  "unique_id": "01FEPEQ9P348AGN1H2WXKE46FZ"
}

Hi,

I will follow this problem with more than normal interest : it looks identical to what I posted a few weeks ago.

I counted a loss of 6 % - also messages that were send by my GW, arrived at TTN, but did not appear in the webhook. (but finally adapted my application as 10 % was considered as acceptable)