Mikrotik Gateway - Offline - not reconnecting

Hi,

Yesterday morning my Mikrotik gateway went off line, I connected with my phone to the WiFi AP and have access to the internet.

I check all the settings and seem fine to me, I rebooted it, but the gateway did not reconnect.

I subsequently have rested to factory default and reconfigured - still not connecting the the TTN server, but I can access the internet via the AP and I can log in from my router side (LAN) into the gateway.

Any suggestions?

image

image

image

As can be seen there are some LoRa traffic

image

image

Are you basing your ā€œnot connectedā€ on the Last Seen message?

If so, did you see this at the top of the forum home page?

If not, do applications or single devices get information through which is the real test.

Thereā€™s multiple join accepts e.g. TX in response to RX, in that traffic log so I think your assessment is 100% correct, although the existence of the proprietary line there reminds me this list shows CRC-error packets by default e.g. noise that looks like LoRa. </pet peeve>

1 Like

Hi,

From WinBox I can see my packet on gateway.

image

But on my application on TTN the data comes from another Gateway close by. So my gateway are definitely not connected the the TTN network.

image

So the traffic screen shot shows a pile of downlinks which could only come from a network server that you are connected to.

Maybe your device (what is it, where is it?) just didnā€™t happen to pickup your gateway - maybe something physical has change - obstruction, interference, antenna not quite connected?

With the different options on the Mikrotik, does it show the gateway to server exchanges / traffic?

The screenshot could be less clipped, but those actually look to be receive records, ie, transmissions sent by another gateway which have been received by this one.

I will turn the other gateway off, as it does belonging to me and see.

Why not connect both gateways (only, one at a time) through a router on which you can run tcpdump or tshark, look at the UDP traffic and compare. Particularly look to see if the cycle of push/pull and their corresponding acks is working.

Oh, incidentally, two UDP based gateways behind the same NAT router is apparently a known issue for some NAT implementationsā€¦ though that tends to affect the downstream side not the upstream, unless the packet forwarded gives up when it doesnā€™t get any acks.

So, the gateway definitely do receive Lora packets, it defiantly receive traffic from the TTN server, but some how it does not connect the two together.

I connected the gateway to where my other gateway were connected same behavior, plug the Multiteck gateway back in it works.

Then took it to a mates network, plugged it in, can connect to the Wifi AP and then to the net, cant connect from the lan to the gateway, can ping it, see it on DHCP.

Retested it four times, then could connect with WinBox via the LAN.

So some where in the firmware there are some issue.

And people wonder why being able to build the image from source is importantā€¦

Even though Mikrotikā€™s stuff is proprietary, it probably could be made to run an MT7621 OpenWrt which may not differ much from RAKā€™s MT7628 one.

But thatā€™s a big project.

Hi @cslorabox, different worlds!

In my world [of commercial operations at scale] if a working system fails then a technician would remove the failed unit and replace it with an identically configured spare unit,

If this restores service then itā€™s a simple unit failure and the failed unit is returned to quarantine for repair, maintenance-contract replacement or scrap.

If this does not restore service then itā€™s a wider system-environment failure and that is escalated to an engineer.

All the work would be managed by the operations supervisor through the Maintenance Management System (MMS).

ā€¦ which is a very nice description of sitting around waiting for someone else to fix the problem.

Meanwhileā€¦ downtime.

Do you have an actual suggestion for getting this userā€™s gateway back online? You might note Iā€™ve made concrete suggestions for understanding the issue faced by the existing software, and not just jumped to the idea of replacing it - in fact it was Johann, not I, who blamed the incumbent software, my point is as much that the closedness of components makes understanding the problem challenging, not that Iā€™m ready to conclude the factory firmware is actually responsible for the failureā€¦

Unfortunately I am not in a industrial or Telco where I can replace the unit and then late diagnose the problem, so I need to figure it out and fix it.

1 Like

Thanks for your help so far, I will ask on the Mikrotik forum aswell.

Iā€™m not saying it doesnā€™t, but could you be specific about exactly what piece of evidence or test leads you to conclude it is getting responses back from TTN servers?

Unless Iā€™m mistaken, the leftmost ā€œRXā€ column on the ā€œJoin Acceptā€ messages in the earlier log are things it received over the air, not items queued into it by TTN which it transmitted over the air.

If there is UDP traffic going both ways, Iā€™d be really curious to pull it apart. Maybe some condition changed causing one end to format things in a way the other canā€™t handle, or maybe it is just too spotty with dropped packets.

Does Miktrok speak the ordinary Semtech UDP protocol, or a custom variation?

The ordinary protocol is documented on the semtech packet forwarder repo and itā€™s pretty easy to separate the binary header from the textual JSON remainder. Even strings would pull out the textual part, but Iā€™d want to take a careful look to make sure the binary headers were conformant and properly matched tokens in both directions. etc.

Also worth wondering if something is going on with your gateway EUIā€¦ if it is derived from a network interface MAC could it have changed? Could someone have, er, ā€œstolenā€ it by accident for theirs? Youā€™ll see the EUI in the raw UDP, too.

These Join-Accept packets were the leading me to believe there are traffic from the TTN to the gateway.

image

These Join-Accept packets were the leading me to believe there are traffic from the TTN to the gateway.

If the left column thatā€™s cropped off now says ā€œTXā€ Iā€™d entirely agree.

But in your previous post from an earlier run, it clearly said ā€œRXā€

(Granted, I canā€™t rule out the possibility of a web GUI bug mislabeling everything as ā€œRXā€)

But then itā€™s possible to note that those downlinks donā€™t seem to be time-correlated to any received uplinks recieved by this gateway which could have triggered them, so Iā€™m still learning towards intercepting the transmissions of another gateway.

Kinda odd that traffic isnā€™t sorted by timestamp, either. But even looking through that Iā€™m not seeing uplink/downlink pairings, except perhaps with something clipped off.

Yes, after looking now again I think you are correct, this is traffic from the node to the gateway.

image

Actually itā€™s downlink traffic from a gateway to a node.

Itā€™s just such transmitted by another gateway than yours, and merely overheard by your gateway.

The Dev Addr confirms that my node are transmitting and been received by the gateway

image

I donā€™t see the packet in my application.

From where my node is, it cant see any other gateway, but my gateway do pick up other nodes.

I moved this afternoon my Multiteck gateway to the location of where my Mikrotik is and it does see my node and forwards the packet to the application.

As can bee seen the RSSI is +/- -70, so not a level issue.