# Missing messages between gateway and application **URL:** \ **Category:** TTN Network and Services\ **Created:** [May 22, 2020, 12:23pm UTC](https://www.thethingsnetwork.org/forum/t/missing-messages-between-gateway-and-application/37261 "2020-05-22T12:23:11Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![DefProc](https://www.thethingsnetwork.org/forum/user_avatar/www.thethingsnetwork.org/defproc/32/35889_2.png) [@DefProc](https://www.thethingsnetwork.org/forum/u/DefProc)\ **Post date:** [May 22, 2020, 12:23pm UTC](https://www.thethingsnetwork.org/forum/t/missing-messages-between-gateway-and-application/37261/1 "2020-05-22T12:23:11Z")
I’ve got a curious problem with messages that I can see arriving on the gateway, and all but a couple of them don’t arrive at the application. I’ve tried a bunch of things to be able to understand what might be going on, including spending hours with the sensor manufacturer’s application engineer to try and find out what might be happening, with no idea. But at least the problem is reproducable! This is with a new to the market internal air quality sensor, so I don’t want to mention the manufacturer straight away, as this could just be teething trouble with a new device. I’ve tried with both OTAA and ABP with the same results, and tried with ADR on and off. What’s the best way to try and find the cause of this packet loss? Setup: The sensor is turned on an working as it expects to sending (in this case) every 2 minutes (although I started at every 20 minutes), using OTAA the join works as expected, or using ADB the sensor starts sending as expected. While the gateway local to me (on the roof of the building the sensor is in) shows receiving every expected message, the message `cnt` matches the sensor’s expectations, and I can decode all the LoRaWAN messages correctly using [`lora-packet`](https://github.com/anthonykirby/lora-packet) with the appkey and devkey from the application, only 2 or 3 of the packets actually arrive at the application and the rest go missing somewhere between the gateway and the application. Expected Result: Every packet for the application that the gateway receives should be decoded and arrive at the application. Current Result: Only 2 or 3 of the messages arrive at the application, the rest go missing. These screenshots are from rebooting the sensor just now and watching the sensor traffic only (uses ADB): ![image](https://www.thethingsnetwork.org/forum/uploads/default/original/3X/a/3/a30b77d1c1d1e15d0509f694768759adc1ce6a29.png) Sometimes, packet 1 makes it through, and occasionally I’ve seen number 4, but never anything beyond that. Payloads: ```auto 2: 0175640367FF0004686106659300C8004700056A2B00077DF203087D000009735F2703670A0104684E06659300C8004700 3: 056A2B00077DF203087D000009735F27 ``` And the local gateway traffic, filtered by device ID: ![image](https://www.thethingsnetwork.org/forum/uploads/default/original/3X/c/f/cffe2be83e5f5c780803239b5e1a312b7f2b2215.png) Detail: ```auto Device Address: 260117E6 Network Session Key: 0583B50A5A0EEE11A1504A5C20822694 Application Session Key: FCB12639D8E7341EDFAAD71711DD5E2E ``` Physical payloads as seen by the gateway: ```auto 1: 40E61701260101000D55369CDEDAAA342470773F2637282DB54DD92F3829526465A60D01EE6ECB3DB58543 2: 40E6170126000200555350BCB55AD3E62C49247A1633491D52EB9828596734FFCF5B7BA601DE35527196BEC33DD3EB3677155019BEFCB88ED8CA246395B8 3: 40E6170126000300554FB8CD427EB690DDADF5AE07D942D066FDD34396 4: 40E61701260104000D553A548BE9C7E26CAF9D76635F57A0B33DB81530F6E4ACB392E35C8F83DAD7101B23A402 5: 40E61701260105000D559A94A00DD7FF486EC7DCF6715467A26EAC05F4FA5548A7A54EB34D8FCDC5FA3B53DB4F 6: 40E61701260106000D55769EA7804EFECD84656A8C920E8DC0220D8F05283A12F9AADE422CA2B974CCC7D5F086 7: 40E61701260107000D5587360B69BFBB7C854D80F4F3FCF463D5CD9B2121E8D519ABF76FBF30BC0EBBC91F638D 8: 40E61701260108000D553DCB5B7D3EC7A8D7BE4E4D25F4826BC531B56772DD592922BD7F4AB14F45464931FA7F 9: 40E61701260109000D550A7B826F8E5D4AF1682DBC327D8F065ABFD12D53599B21BC1483C59E17FD3956A2D422 10: 40E6170126010A000D559055A1F59326FE9D9E743B11850183EA824362F4BC94DFB28318AD72B93898694630E0 ``` And the decoded versions, using the keys above: [1](https://lorawan-packet-decoder-0ta6puiniaut.runkit.sh/?data=40E61701260101000D55369CDEDAAA342470773F2637282DB54DD92F3829526465A60D01EE6ECB3DB58543&nwkskey=0583B50A5A0EEE11A1504A5C20822694&appskey=FCB12639D8E7341EDFAAD71711DD5E2E), [2](https://lorawan-packet-decoder-0ta6puiniaut.runkit.sh/?data=40E6170126000200555350BCB55AD3E62C49247A1633491D52EB9828596734FFCF5B7BA601DE35527196BEC33DD3EB3677155019BEFCB88ED8CA246395B8&nwkskey=0583B50A5A0EEE11A1504A5C20822694&appskey=FCB12639D8E7341EDFAAD71711DD5E2E), [3](https://lorawan-packet-decoder-0ta6puiniaut.runkit.sh/?data=40E6170126000300554FB8CD427EB690DDADF5AE07D942D066FDD34396&nwkskey=0583B50A5A0EEE11A1504A5C20822694&appskey=FCB12639D8E7341EDFAAD71711DD5E2E), [4](https://lorawan-packet-decoder-0ta6puiniaut.runkit.sh/?data=40E61701260104000D553A548BE9C7E26CAF9D76635F57A0B33DB81530F6E4ACB392E35C8F83DAD7101B23A402&nwkskey=0583B50A5A0EEE11A1504A5C20822694&appskey=FCB12639D8E7341EDFAAD71711DD5E2E), [5](https://lorawan-packet-decoder-0ta6puiniaut.runkit.sh/?data=40E61701260105000D559A94A00DD7FF486EC7DCF6715467A26EAC05F4FA5548A7A54EB34D8FCDC5FA3B53DB4F&nwkskey=0583B50A5A0EEE11A1504A5C20822694&appskey=FCB12639D8E7341EDFAAD71711DD5E2E), [6](https://lorawan-packet-decoder-0ta6puiniaut.runkit.sh/?data=40E61701260106000D55769EA7804EFECD84656A8C920E8DC0220D8F05283A12F9AADE422CA2B974CCC7D5F086&nwkskey=0583B50A5A0EEE11A1504A5C20822694&appskey=FCB12639D8E7341EDFAAD71711DD5E2E), [7](https://lorawan-packet-decoder-0ta6puiniaut.runkit.sh/?data=40E61701260107000D5587360B69BFBB7C854D80F4F3FCF463D5CD9B2121E8D519ABF76FBF30BC0EBBC91F638D&nwkskey=0583B50A5A0EEE11A1504A5C20822694&appskey=FCB12639D8E7341EDFAAD71711DD5E2E), [8](https://lorawan-packet-decoder-0ta6puiniaut.runkit.sh/?data=40E61701260108000D553DCB5B7D3EC7A8D7BE4E4D25F4826BC531B56772DD592922BD7F4AB14F45464931FA7F&nwkskey=0583B50A5A0EEE11A1504A5C20822694&appskey=FCB12639D8E7341EDFAAD71711DD5E2E), [9](https://lorawan-packet-decoder-0ta6puiniaut.runkit.sh/?data=40E61701260109000D550A7B826F8E5D4AF1682DBC327D8F065ABFD12D53599B21BC1483C59E17FD3956A2D422&nwkskey=0583B50A5A0EEE11A1504A5C20822694&appskey=FCB12639D8E7341EDFAAD71711DD5E2E), [10](https://lorawan-packet-decoder-0ta6puiniaut.runkit.sh/?data=40E6170126010A000D559055A1F59326FE9D9E743B11850183EA824362F4BC94DFB28318AD72B93898694630E0&nwkskey=0583B50A5A0EEE11A1504A5C20822694&appskey=FCB12639D8E7341EDFAAD71711DD5E2E) The payload data is a little esoteric, but it’s using a variant of the Cayenne, except message 1 that encodes device/firmware info. But what’s confusing, is that all of the packet data can be decoded properly. What am I missing here? What’s the best way to go about troubleshooting to find these missing messages? ---
**Author:** ![cultsdotelecomatgmai](https://www.thethingsnetwork.org/forum/user_avatar/www.thethingsnetwork.org/cultsdotelecomatgmai/32/35697_2.png) [@cultsdotelecomatgmai](https://www.thethingsnetwork.org/forum/u/cultsdotelecomatgmai)\ **Post date:** [May 22, 2020, 2:04pm UTC](https://www.thethingsnetwork.org/forum/t/missing-messages-between-gateway-and-application/37261/2 "2020-05-22T14:04:08Z")
Hi @DefProc, please can I suggest that you take a look at the trace data in the web console / gateway / traffic page. This first image selects a specific uplink by clicking on the blue up marker that I have shown with a red line. ![ttn-forum-01](https://www.thethingsnetwork.org/forum/uploads/default/original/3X/3/2/320076efcbb3f8214e1af16b48732272b91d2114.jpeg) Then scroll down using the scroll bar that I have marked with a red line in this image. ![ttn-forum-02](https://www.thethingsnetwork.org/forum/uploads/default/original/3X/5/5/5542c451e25b685cfe248ceaead25f096de81d9f.jpeg) This should take you down to the Trace section and you should be able to see if the system encountered problems routing the uplink. ---
**Author:** ![arjanvanb](https://www.thethingsnetwork.org/forum/user_avatar/www.thethingsnetwork.org/arjanvanb/32/46626_2.png) [@arjanvanb](https://www.thethingsnetwork.org/forum/u/arjanvanb)\ **Post date:** [May 22, 2020, 3:09pm UTC](https://www.thethingsnetwork.org/forum/t/missing-messages-between-gateway-and-application/37261/3 "2020-05-22T15:09:00Z")
Are you only looking at TTN Console? Or do you have some actual application outside of TTN that also only gets the first few uplinks? If you don’t have such application, then I’d very much recommend using [an MQTT client](https://www.thethingsnetwork.org/docs/applications/mqtt/quick-start.html) during debugging as well. Also, in TTN Console, the device’s “Status” might reveal if TTN actually routed the uplinks without TTN Console showing them in the Data pages. Do you have any integration enabled? (I don’t recall if failing integrations might cause the Data page to not show the uplinks.) Anything different in the event data when clicking the different uplinks in the gateway Traffic? Aside: I failed to see _any_ uplinks in the Data page at all when I started to write this answer. But the device’s Status was updated just fine, and MQTT worked just fine at that time too. Also querying the Data Integration worked as expected, though I did not [see “historical data”](https://www.thethingsnetwork.org/forum/t/why-application-data-is-shown-as-historical/8793) in TTN Console’s Data page either. Meanwhile, all is fine again for me, but the Data and Traffic pages in TTN Console fail on me a lot. ---
**Author:** ![DefProc](https://www.thethingsnetwork.org/forum/user_avatar/www.thethingsnetwork.org/defproc/32/35889_2.png) [@DefProc](https://www.thethingsnetwork.org/forum/u/DefProc)\ **Post date:** [May 22, 2020, 4:40pm UTC](https://www.thethingsnetwork.org/forum/t/missing-messages-between-gateway-and-application/37261/4 "2020-05-22T16:40:21Z")
I have the Data Storage integration enabled, for example, a query for the last 12hrs gives the same 2 messages as above. ```auto [ { "device_id": "6128a1757429", "raw": "AXVkA2f/AARoYQZlkwDIAEcABWorAAd98gMIfQAACXNfJwNnCgEEaE4GZZMAyABHAA==", "time": "2020-05-22T11:44:12.975415847Z" }, { "device_id": "6128a1757429", "raw": "BWorAAd98gMIfQAACXNfJw==", "time": "2020-05-22T11:45:27.367185239Z" } ] ``` I’ve watched traffic over MQTT using `mosquitto_sub`, and it only recieves the same messages as the console. The device status seems to match the last recived message (currently showing: `22/05/2020 12:45:27`, the same recieve time as packet 3), and the `frames up` counter is on 3. The gateway info doesn’t seem to show anything unusual (this is the most recent failed packet #153): ![image](https://www.thethingsnetwork.org/forum/uploads/default/original/3X/d/a/da122145608b60f210d0156aa9d444717ae2786a.png) ---
**Author:** ![DefProc](https://www.thethingsnetwork.org/forum/user_avatar/www.thethingsnetwork.org/defproc/32/35889_2.png) [@DefProc](https://www.thethingsnetwork.org/forum/u/DefProc)\ **Post date:** [May 22, 2020, 4:53pm UTC](https://www.thethingsnetwork.org/forum/t/missing-messages-between-gateway-and-application/37261/6 "2020-05-22T16:53:30Z")
It looks to be routing correctly, this is packet 153, which failed to appear at the application: ![image](https://www.thethingsnetwork.org/forum/uploads/default/original/3X/3/b/3bc6bfc5cd10855c786de56f1cc1afa4b026efac.png) ---
**Author:** ![arjanvanb](https://www.thethingsnetwork.org/forum/user_avatar/www.thethingsnetwork.org/arjanvanb/32/46626_2.png) [@arjanvanb](https://www.thethingsnetwork.org/forum/u/arjanvanb)\ **Post date:** [May 22, 2020, 5:13pm UTC](https://www.thethingsnetwork.org/forum/t/missing-messages-between-gateway-and-application/37261/7 "2020-05-22T17:13:52Z")
Has it always been the same application and device in TTN Console? So: did you ever start from scratch in TTN Console? (Don’t delete the gateway!) Any chance you can program a different DevEUI and AppEUI in the device? Then you could create an additional application, with a new device, and see if that behaves the same. (You could then keep the existing thing as is, in TTN Console.) I know, it shouldn’t matter, but maybe there’s some duplicate device, somehow…? Are you using the EU handler (`ttn-handler-eu`) for the application, and EU router (`ttn-router-eu`) for the gateway, and has that always been like that? I’ve seen reports about _changing_ the router not always being effective. (In those cases yielding weird downlink frequencies.) ---
**Author:** ![arjanvanb](https://www.thethingsnetwork.org/forum/user_avatar/www.thethingsnetwork.org/arjanvanb/32/46626_2.png) [@arjanvanb](https://www.thethingsnetwork.org/forum/u/arjanvanb)\ **Post date:** [May 22, 2020, 5:27pm UTC](https://www.thethingsnetwork.org/forum/t/missing-messages-between-gateway-and-application/37261/8 "2020-05-22T17:27:30Z")
> [@arjanvanb](#): > > Any chance you can program a different DevEUI and AppEUI in the device? Or program the current DevEUI, AppEUI and AppKey into a totally different device? (If you don’t have any other devices, then I could do that too, if you’d like.) ---
**Author:** ![cslorabox](https://www.thethingsnetwork.org/forum/user_avatar/www.thethingsnetwork.org/cslorabox/32/35697_2.png) [@cslorabox](https://www.thethingsnetwork.org/forum/u/cslorabox)\ **Post date:** [May 22, 2020, 6:30pm UTC](https://www.thethingsnetwork.org/forum/t/missing-messages-between-gateway-and-application/37261/9 "2020-05-22T18:30:30Z")
You are missing the packets where FCtrl = 1 indicates that FOpts contains one byte, and where that byte is 0x0D. 0x0D is “DeviceTimeReq” which is a LoRaWAN v1.1 MAC command not supported by TTN - there are several other threads on that here. Because MAC commands are variable in length, it is in general _impossible_ for an implementation to safely skip over unknown ones, as it does not know how many bytes to skip. In theory an implementation could simply skip processing anything remaining in FOpts once encountering something undefined and move on to the rest of the packet (since the total length of FOpts is known), however it would appear that the packet is being rejected entirely, and that may well be what the specification formally expects in the case of undefined input. In practical terms, you will need to work with the device manufacturer to get its firmware to cease sending DeviceTimeReq if you wish to use it on a pre-1.1 network such as TTN. Hopefully this would be part of a more general “make it operate in legacy LoRaWAN mode” as otherwise there may well be other areas where you will trip over incompatibilities between LoRaWAN 1.1 and 1.0x too. ---
**Author:** ![arjanvanb](https://www.thethingsnetwork.org/forum/user_avatar/www.thethingsnetwork.org/arjanvanb/32/46626_2.png) [@arjanvanb](https://www.thethingsnetwork.org/forum/u/arjanvanb)\ **Post date:** [May 22, 2020, 9:05pm UTC](https://www.thethingsnetwork.org/forum/t/missing-messages-between-gateway-and-application/37261/10 "2020-05-22T21:05:12Z")
Great find! (And I feel bad for not peeking into the actual messages, while @DefProc provided such good details!) The 1.0.2 specifications [state to stop processing](https://lora-alliance.org/sites/default/files/2018-05/lorawan1_0_2-20161012_1398_1.pdf#page=23): > Note: The length of a MAC command is not explicitly given and must be implicitly known by the MAC implementation. Therefore unknown MAC commands cannot be skipped and the first unknown MAC command terminates the processing of the MAC command sequence. …and for nodes: > It is therefore advisable to order MAC commands according to the version of the LoRaWAN specification which has introduced a MAC command for the first time. This way all MAC commands up to the version of the LoRaWAN specification implemented can be processed even in the presence of MAC commands specified only in a version of the LoRaWAN specification newer than that implemented. Indeed, some have [claimed not seeing the data](https://www.thethingsnetwork.org/forum/t/mac-command-devicetimereq-supported/19785/10) in TTN Console when `DeviceTimeReq` was used. But others say data _was_ still displayed, though reading that claim again today raises new questions: > [@Cannot see data in device dashboard, while OTAA works and uplinks show in gateway's traffic](https://www.thethingsnetwork.org/forum/t/cannot-see-data-in-device-dashboard-while-otaa-works-and-uplinks-show-in-gateways-traffic/20119/20): > > Yes I saw raw data in the device dashboard without doing any modification…then I remove the DeviceTimeReq on the Node side, and I saw good Cayenne LPP payload . Still, just to be sure: there is no Decoder involved in the Payload Formats in TTN Console, right? (The successful uplinks show no sign of that.) In short: of course @cslorabox is very likely right that this is the culprit, even though earlier reports differ. 🙂 I’ve heard that V3 makes debugging easier… (In an earlier version of this answer I referred to [my November 2018 issue report #748](https://github.com/TheThingsNetwork/ttn/issues/748), in which I thought TTN was not stopping processing. Looking at [the code](https://github.com/TheThingsNetwork/ttn/blob/v2.11.0/core/networkserver/uplink_mac.go#L39-L87) again, I now see that the actually parsing was already done [elsewhere](https://github.com/brocaar/lorawan/blob/master/mac_commands.go), not in the code I referred to. So while I might have reported the symptoms in that issue, I think I wrongly interpreted their cause.) ---
**Author:** ![DefProc](https://www.thethingsnetwork.org/forum/user_avatar/www.thethingsnetwork.org/defproc/32/35889_2.png) [@DefProc](https://www.thethingsnetwork.org/forum/u/DefProc)\ **Post date:** [May 27, 2020, 3:16pm UTC](https://www.thethingsnetwork.org/forum/t/missing-messages-between-gateway-and-application/37261/11 "2020-05-27T15:16:30Z")
This is fantastic information thank you both (@arjanvanb and @cslorabox). From that analysis, it looks like the device is requesting network time information with all but those two messages. Even as a stop gap measure, it looks like just not requesting the time _every_ message would get me most of the data in this case. Yes, there was no decoder for these messages, I tried it with one, and got the same results.