# Australia cluster - Infrastructure upgrade

**URL:** <https://www.thethingsnetwork.org/forum/t/australia-cluster-infrastructure-upgrade/41748>\
**Category:** TTN Network and Services\
**Created:** [November 30, 2020, 2:50am UTC](https://www.thethingsnetwork.org/forum/t/australia-cluster-infrastructure-upgrade/41748 "2020-11-30T02:50:34Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![Maj](https://www.thethingsnetwork.org/forum/user_avatar/www.thethingsnetwork.org/maj/32/47732_2.png) [@Maj](https://www.thethingsnetwork.org/forum/u/Maj)\
**Post date:** [November 30, 2020, 2:50am UTC](https://www.thethingsnetwork.org/forum/t/australia-cluster-infrastructure-upgrade/41748/1 "2020-11-30T02:50:35Z")

</div>

The Australia cluster will undergo an infrastructure upgrade tomorrow, Tue 1-Dec at 4:00pm AEST. If all goes well, outage time will be less than 10 minutes.  
I note that after the previous infrastructure upgrade a number of gateways could not reconnect to the server due to stale connections. We’ll be on the lookout for this issue and can address it at the server end if it happens again.  
Please reply to this topic on the day if there are any issues, or for more immediate communication join us on the #Australia Slack channel.

---

<div class="post-metadata">

**Author:** ![Maj](https://www.thethingsnetwork.org/forum/user_avatar/www.thethingsnetwork.org/maj/32/47732_2.png) [@Maj](https://www.thethingsnetwork.org/forum/u/Maj)\
**Post date:** [December 1, 2020, 7:46am UTC](https://www.thethingsnetwork.org/forum/t/australia-cluster-infrastructure-upgrade/41748/2 "2020-12-01T07:46:12Z")

</div>

Unfortunately we are experiencing issues with the TTN services (even before we started the intended upgrade). Currently waiting on the TTN team to advise.

---

<div class="post-metadata">

**Author:** ![wikicheedia](https://www.thethingsnetwork.org/forum/user_avatar/www.thethingsnetwork.org/wikicheedia/32/25844_2.png) [@wikicheedia](https://www.thethingsnetwork.org/forum/u/wikicheedia)\
**Post date:** [December 2, 2020, 7:15am UTC](https://www.thethingsnetwork.org/forum/t/australia-cluster-infrastructure-upgrade/41748/3 "2020-12-02T07:15:03Z")

</div>

I’ve been pulling my hair out for the last 24 hours thinking I had managed to stuff up every sketch I was working on somehow. 🤦‍♂️  
Thanks for the hardwork. Hopefully we’re back online soon. 😀

---

<div class="post-metadata">

**Author:** ![Malcman](https://www.thethingsnetwork.org/forum/user_avatar/www.thethingsnetwork.org/malcman/32/36035_2.png) [@Malcman](https://www.thethingsnetwork.org/forum/u/Malcman)\
**Post date:** [December 3, 2020, 3:49pm UTC](https://www.thethingsnetwork.org/forum/t/australia-cluster-infrastructure-upgrade/41748/4 "2020-12-03T15:49:28Z")

</div>

Ah, thanks for the info, well that more than likely explains my MAJOR forwarding issues in the last 2 days. I too have had a sleepless night looking for errors in code and set-up. How is the best way to get updates on the “issues status”?

---

<div class="post-metadata">

**Author:** ![Maj](https://www.thethingsnetwork.org/forum/user_avatar/www.thethingsnetwork.org/maj/32/47732_2.png) [@Maj](https://www.thethingsnetwork.org/forum/u/Maj)\
**Post date:** [December 3, 2020, 9:51pm UTC](https://www.thethingsnetwork.org/forum/t/australia-cluster-infrastructure-upgrade/41748/5 "2020-12-03T21:51:48Z")

</div>

Apologies, I never closed the loop on this issue. Problems started at 4pm AEST on 1-Dec, before we even got a chance to upgrade. The issue was a DDoS attack on a Netherlands ISP, which caused one of the global TTN services to become unavailable.

The issue was resolved about 3 hours later, we performed the upgrade and brought the services back online by about 8pm AEST on 1-Dec.

---

<div class="post-metadata">

**Author:** ![Maj](https://www.thethingsnetwork.org/forum/user_avatar/www.thethingsnetwork.org/maj/32/47732_2.png) [@Maj](https://www.thethingsnetwork.org/forum/u/Maj)\
**Post date:** [December 3, 2020, 9:51pm UTC](https://www.thethingsnetwork.org/forum/t/australia-cluster-infrastructure-upgrade/41748/6 "2020-12-03T21:51:57Z")

</div>

@malcman What issues are you seeing?

---

<div class="post-metadata">

**Author:** ![EvolvePlus](https://www.thethingsnetwork.org/forum/user_avatar/www.thethingsnetwork.org/evolveplus/32/39030_2.png) [@EvolvePlus](https://www.thethingsnetwork.org/forum/u/EvolvePlus)\
**Post date:** [December 3, 2020, 10:05pm UTC](https://www.thethingsnetwork.org/forum/t/australia-cluster-infrastructure-upgrade/41748/7 "2020-12-03T22:05:06Z")

</div>

We are testing office sensors and people counter sensors (about 3 weeks) and they have been working really well until early this week (1st December)  
 ![Annotation 2020-12-04 090224](https://www.thethingsnetwork.org/forum/uploads/default/original/3X/8/6/8682a1e816f2c9ac369a4608a8d6b81b90e8f5b0.jpeg)

Since the 1st December no payload packets are passing through from any of our test sensors - Sensors/Gateway all appear to be working as expected however the device status only shows the last connection as 3 days ago.

---

<div class="post-metadata">

**Author:** ![Malcman](https://www.thethingsnetwork.org/forum/user_avatar/www.thethingsnetwork.org/malcman/32/36035_2.png) [@Malcman](https://www.thethingsnetwork.org/forum/u/Malcman)\
**Post date:** [December 4, 2020, 12:09am UTC](https://www.thethingsnetwork.org/forum/t/australia-cluster-infrastructure-upgrade/41748/8 "2020-12-04T00:09:41Z")

</div>

My nodes are able to join OK and uplink packets are appearing in the gateway console but packets are not appearing in the device console for any of my 2 nodes, nor is data being forwarded to my platform. This started happening about Dec 1. The the frame counters seem to be correct on the gateway console but show ‘0’ on the device console for up and down.  
After some research I decided to change the router from eu to asia-se on both my gatway and nodes but still the same result. I have reset my nodes several times and I have have no problems getting a join accept showing on the gateway console but nothing on the device console. As my platform has not seen any new data since Dec 1 I can only assume the router is not forwarding the packets.

---

<div class="post-metadata">

**Author:** ![EvolvePlus](https://www.thethingsnetwork.org/forum/user_avatar/www.thethingsnetwork.org/evolveplus/32/39030_2.png) [@EvolvePlus](https://www.thethingsnetwork.org/forum/u/EvolvePlus)\
**Post date:** [December 4, 2020, 1:53am UTC](https://www.thethingsnetwork.org/forum/t/australia-cluster-infrastructure-upgrade/41748/9 "2020-12-04T01:53:31Z")

</div>

I can confirm that I am experiencing the exact same issue as outlined by Malcman. Thanks David for your explanation of the issue. Still an issue for me at the moment no new data has been received since Dec 1 - hopefully it will be resolved over the weekend.

---

<div class="post-metadata">

**Author:** ![andyau](https://www.thethingsnetwork.org/forum/user_avatar/www.thethingsnetwork.org/andyau/32/25844_2.png) [@andyau](https://www.thethingsnetwork.org/forum/u/andyau)\
**Post date:** [December 4, 2020, 2:00am UTC](https://www.thethingsnetwork.org/forum/t/australia-cluster-infrastructure-upgrade/41748/10 "2020-12-04T02:00:42Z")

</div>

I can also say I have the same issue across at least 3 gateways. Our gateways and apps are on asia-se so nothing to do with Australian cluster upgrade.

---

<div class="post-metadata">

**Author:** ![Maj](https://www.thethingsnetwork.org/forum/user_avatar/www.thethingsnetwork.org/maj/32/47732_2.png) [@Maj](https://www.thethingsnetwork.org/forum/u/Maj)\
**Post date:** [December 4, 2020, 2:06am UTC](https://www.thethingsnetwork.org/forum/t/australia-cluster-infrastructure-upgrade/41748/11 "2020-12-04T02:06:24Z")

</div>

We’ve been working on this throughout the morning. Unfortunately we haven’t found the issue yet. None of our test devices or gateways are affected. It’s affecting some devices on some gateways, but unfortunately it’s inconsistent.

Note that unless your gateway is configured for meshed-router and your application for meshed-handler, we are probably talking about different things.

@Malcman, @EvolvePlus, can you provide a DevEUI and GatewayEUI for an affected device

---

<div class="post-metadata">

**Author:** ![Maj](https://www.thethingsnetwork.org/forum/user_avatar/www.thethingsnetwork.org/maj/32/47732_2.png) [@Maj](https://www.thethingsnetwork.org/forum/u/Maj)\
**Post date:** [December 4, 2020, 2:20am UTC](https://www.thethingsnetwork.org/forum/t/australia-cluster-infrastructure-upgrade/41748/12 "2020-12-04T02:20:57Z")

</div>

After hearing from someone else with similar issues I’m starting to think this is a cross-region routing issue.

See

> [@Meshed-handler](https://www.thethingsnetwork.org/forum/t/meshed-handler/14652/55):
>
> In you’re in Australia you should use: meshed-router for gateways meshed-handler for applications console.thethings.meshed.com.au for the Console when configuring integrations. You can use the regular console at all other times. The problem with using asia-se is that other people in Australia may have applications registered on meshed-handler trying to use your gateway and the cross-region routing is intermittent and won’t be resolved until V3 is implemented. warning See also [below](https://www.thethingsnetwork.org/forum/t/meshed-handler/14652/63) for a [J…](https://www.thethingsnetwork.org/forum/t/meshed-handler/14652/69)

Does anyone have a meshed-handler/meshed-router combination that is failing?

---

<div class="post-metadata">

**Author:** ![EvolvePlus](https://www.thethingsnetwork.org/forum/user_avatar/www.thethingsnetwork.org/evolveplus/32/39030_2.png) [@EvolvePlus](https://www.thethingsnetwork.org/forum/u/EvolvePlus)\
**Post date:** [December 4, 2020, 2:24am UTC](https://www.thethingsnetwork.org/forum/t/australia-cluster-infrastructure-upgrade/41748/13 "2020-12-04T02:24:54Z")

</div>

Thanks for your assistance Andrew.  
The affected device DevEUI is: A81758FFFE03FD23 and the Gateway EUI is eui-c0ee40ffff297f46. The gateway is configured for meshed-router  
 ![Annotation 2020-12-04 131941-Gateway](https://www.thethingsnetwork.org/forum/uploads/default/original/3X/7/2/7203a94d16cc653f85869041079b9308cd3d81ec.jpeg)

---

<div class="post-metadata">

**Author:** ![Maj](https://www.thethingsnetwork.org/forum/user_avatar/www.thethingsnetwork.org/maj/32/47732_2.png) [@Maj](https://www.thethingsnetwork.org/forum/u/Maj)\
**Post date:** [December 4, 2020, 2:29am UTC](https://www.thethingsnetwork.org/forum/t/australia-cluster-infrastructure-upgrade/41748/14 "2020-12-04T02:29:48Z")

</div>

@EvolvePlus …and the app on meshed-handler?

---

<div class="post-metadata">

**Author:** ![EvolvePlus](https://www.thethingsnetwork.org/forum/user_avatar/www.thethingsnetwork.org/evolveplus/32/39030_2.png) [@EvolvePlus](https://www.thethingsnetwork.org/forum/u/EvolvePlus)\
**Post date:** [December 4, 2020, 2:43am UTC](https://www.thethingsnetwork.org/forum/t/australia-cluster-infrastructure-upgrade/41748/15 "2020-12-04T02:43:07Z")

</div>

Oh sorry Andrew - I forgot to mention: Handler = ttn-handler-asia-se  
 ![Annotation 2020-12-04 131941-AppHandler](https://www.thethingsnetwork.org/forum/uploads/default/original/3X/8/2/8265249eee577e4051c5d3aa41d6ac4bae050707.jpeg)

---

<div class="post-metadata">

**Author:** ![Malcman](https://www.thethingsnetwork.org/forum/user_avatar/www.thethingsnetwork.org/malcman/32/36035_2.png) [@Malcman](https://www.thethingsnetwork.org/forum/u/Malcman)\
**Post date:** [December 4, 2020, 2:47am UTC](https://www.thethingsnetwork.org/forum/t/australia-cluster-infrastructure-upgrade/41748/16 "2020-12-04T02:47:31Z")

</div>

DevEUI: A8610A34343D8512  
GatewayEUI: eui-dca632fffe08e182

---

<div class="post-metadata">

**Author:** ![Malcman](https://www.thethingsnetwork.org/forum/user_avatar/www.thethingsnetwork.org/malcman/32/36035_2.png) [@Malcman](https://www.thethingsnetwork.org/forum/u/Malcman)\
**Post date:** [December 4, 2020, 3:53am UTC](https://www.thethingsnetwork.org/forum/t/australia-cluster-infrastructure-upgrade/41748/17 "2020-12-04T03:53:54Z")

</div>

I now have my nodes running again! they are uploading and forwarding to the platform.  
I changed from “asia-se” to “thethings.meshed.com.au” on both gateway and nodes. Thank you so very much Maj.  
I am hoping the au meshed routers are more reliable than the others for us in Australia.  
What was the issue with the other routers except for the cross-region routing you mentioned?

---

<div class="post-metadata">

**Author:** ![Maj](https://www.thethingsnetwork.org/forum/user_avatar/www.thethingsnetwork.org/maj/32/47732_2.png) [@Maj](https://www.thethingsnetwork.org/forum/u/Maj)\
**Post date:** [December 4, 2020, 4:09am UTC](https://www.thethingsnetwork.org/forum/t/australia-cluster-infrastructure-upgrade/41748/18 "2020-12-04T04:09:48Z")

</div>

The problem is with the code that routes from one region’s gateways to another region’s apps. It’s not a problem with any particular region. Now that you’re on one region only, cross-region routing is not required, so everything works 🙂

---

<div class="post-metadata">

**Author:** ![EvolvePlus](https://www.thethingsnetwork.org/forum/user_avatar/www.thethingsnetwork.org/evolveplus/32/39030_2.png) [@EvolvePlus](https://www.thethingsnetwork.org/forum/u/EvolvePlus)\
**Post date:** [December 7, 2020, 3:53am UTC](https://www.thethingsnetwork.org/forum/t/australia-cluster-infrastructure-upgrade/41748/19 "2020-12-07T03:53:44Z")

</div>

YES! Thank you so very much Maj. Switching over to “thethings.meshed.com.au” for the devices has solved my problem. Cheers!

---

<div class="post-metadata">

**Author:** ![andyau](https://www.thethingsnetwork.org/forum/user_avatar/www.thethingsnetwork.org/andyau/32/25844_2.png) [@andyau](https://www.thethingsnetwork.org/forum/u/andyau)\
**Post date:** [December 9, 2020, 12:49am UTC](https://www.thethingsnetwork.org/forum/t/australia-cluster-infrastructure-upgrade/41748/20 "2020-12-09T00:49:57Z")

</div>

In regards to the asia-se issue. The issue solved itself. Nothing was changed on our side it just started working again.
