2000 gateways offline

The SLA is clear, you are owed exactly the amount you pay for this service. If you want certainty, pay for an instance or self-host. Consider that providing updates that almost none of us will understand the technicals of is a distraction from getting things fixed.

In the meanwhile, TTI have an outstanding record of keeping this free service up and as harsh as it may be on the users of the 1,000 gateways that are offline, that’s only 4% of the total, important, but not like the whole network is down.

To underline this commitment, the updates on the 2nd were to enable the continued connectivity of the original TTIGs - in an age where Micro$oft will abandon millions of perfectly working PCs because [insert your own plausible reason here, because I can’t think of one].

I’d also note that there are third parties involved in the automagic provisioning / boot up of a TTIG, so it may take more than just rolling back a bit of code on the TTN servers if indeed there is still a problem and we just have to be patient whilst updates propagate.

1 Like

I would like to comment on the way critical questions and reports are currently being handled here.

To be honest, the response that users are ‘owed exactly the amount they pay’ and have no right to expect anything is something I find highly arrogant and lacking in appreciation. Also, the repetitive way of dismissing concerns—often implying that users should just ‘read the manual’ „RTFM“ —is counterproductive for a community that thrives on participation and voluntary engagement.

It is important to remember that we are not just ‘users’ of a free service. I personally operate three gateways at three different locations, paying for the gateways and the internet connections and investing my own time—not only to use the network myself but to actively expand it for everyone. For me, this investment holds significant value.

The ‘community-driven’ principle only works if there is a respectful exchange on equal terms. The argument that volunteer gateway operators have no right to open communication or serious error analysis contradicts the very foundation of building a shared public network together. We are not just a ‘4% statistic’—we are individuals investing our own resources and dedication into this project.

6 Likes

And we are volunteer moderators that have worked hard on keeping TTN as accessible as it still is when it became apparent that there were many commercial users profiting off the back of TTN. Most of this has been settled for the last couple of years, so it would be so much better not to rock this boat.

I’m reflecting back - indeed pushing back - on the assumption that we have some sort of right to any service because we absolutely do not. That we purchased TTIGs and then deployed them and pay for power & backhaul to use a free service was our choice. This hardware & its resource requirements pales in to insignificance to the costs of the AWS instances TTS requires and people expecting more than is being given freely has been a bone of contention in the past. And we’d rather not revisit that.

I (we) do have full appreciation of this issue. This affects @Jeff-uk far more than me as he has far more TTIGs deployed, well in to double digits and it has impacted our various community projects too.

openssl s_client -showcerts -connect eu1.cloud.thethings.network:443
Connecting to 52.212.223.226
CONNECTED(00000003)
depth=3 C=US, O=Internet Security Research Group, CN=ISRG Root X1
verify return:1
depth=2 C=US, O=ISRG, CN=Root YR
verify return:1
depth=1 C=US, O=Let's Encrypt, CN=YR1
verify return:1
depth=0 CN=eu1.cloud.thethings.network
verify return:1
---
Certificate chain
 0 s:CN=eu1.cloud.thethings.network
   i:C=US, O=Let's Encrypt, CN=YR1
   a:PKEY: RSA, 2048 (bit); sigalg: sha256WithRSAEncryption
   v:NotBefore: Jul  2 09:42:48 2026 GMT; NotAfter: Sep 30 09:42:47 2026 GMT
...

this is the current certificate chain they offer on port 443 (https) - I assume as long this will not change, TTIG-868 with 2.0.4 firmware are lost.

Imho they cannot use newest LE certificates for IoT devices having older firmware but I would wonder whether it’s not possible to get from another by device trusted CA a certificate which fit. Sure it will cause some expenses, but is much more eco-friendly than creating electronic waste.

One reason why long running IoT eco systems needs more care in designing secure transport…with a private CA truststore in the devices this would not have been happened…

That might be, however it is exactly what you signed up for. Please keep in mind TTI can not spend money on support engineers to support over tens of thousands of registered TTN users that do not generate any income for them as well as pay the substantial fee to supply the free clusters. That is not a viable business model.

That being said, TTI is aware there are issues and is working on them. It just takes time.

(With >20 gateways I’m footing the bills for just to provide community coverage I fully understand this doesn’t feel right, there are times I it doesn’t feel right to me either)

1 Like

I assume the root cause is still the same a mentioned The Things Network Status - LBS Gateway Disconnections in TTS Sandbox Clusters on June, 8 - has anyone seen since then an request from the team like:

Dear community, we found an incompatibility issue in newer TLS server certificates issued by Let’s Encrypt. To avoid that thousands of gateways turn into electronic waste we request immediate funding for expenses to get TLS server certificates from a different CA which is still compatible to your gateways and also to circumvent this later we need funding to create a firmware update which contains an up-to-date version of the TLS client library and also insertion of an additional private truststore for at least fallback TLS connection trust. In addition the margin of the sold community gateways and marketing budget from TTI ran out and therefore we request funding/donations for keeping the community service up-and-running.

???

At least with such information sent out 3 weeks in the past all community service users would have been warned that their TTN usage is on high risk.

Same issue here. I have two The Things Indoor Gateways and both are offline since 03.07.2026. Restart does not help.

I presume you are being rhetorical here because the best place to put this information would be on this forum and we all know that didn’t happen.

Your analysis has only one major flaw in it, and that is the cost of a certificate is peanuts compared to the cost of the servers rented on AWS for TTN to run, so your assumption is unlikely.

The status from the update is:

" During this maintenance window, we will deploy changes to how The Things Indoor Gateways (non-pro versions only) connect to The Things Stack to ensure their continued operation.
The Things Indoor Gateways will have to reconnect for the changes to apply. Since TTIGs have a backoff mechanism, it may take a few minutes for them to reconnect.
This change was successfully verified in TTS Sandbox au1 cluster and will be applied to the nam1 and eu1 cluster.
Jul 1, 16:30 CEST"

Which says “we know there is an issue, we have a resolution, we tested it at scale on the Australians and we are deploying it for everyone else”.

Unfortunately a small percentage of gateways didn’t reconnect.

Around 300 gateways came back online between 08:15 and 08:35 this morning.

Working on SysOps on a Sunday morning is indicative of a company working to resolve an issue as soon as it can.

TTI made a commitment to keep the community version running for free and we haven’t heard otherwise.

Almost every single free internet service that has needed funding or donations or some such at a late point in its existence has never managed to bridge the gap. I think we see TTNMapper as the perfect example - despite the real possibility of it having to close and some substantial down time, I’m not aware of there being a sizeable uptick in donations or Patreon subscriptions.

There is work & effort involved in collecting money, as I mention above, anything less than 2€ would actually cost more than it makes to process & do admin on.

And one of the best business lessons I was taught 30+ years ago is that the problem with charging for a service is that people expect something for their money. So it has to be priced at a level that allows the service to be delivered, it can’t just pay for hardware or rental, all the costs need to be covered with some left over for a rainy day, plus ongoing upgrades etc etc.

Which is why the TTI entry level instance is 190€/month for up to 1,000 devices, rather than some graduated scale of charges starting at just 1 device. Because if you have a user with 9 devices on some entry level tier, say 2€/device/month, and they have a problem, telling them they haven’t paid enough to be able to get personal support usually involves a long enough exchange to fully wipe out the 18€ they did pay, but several times over.

And that 190€/month doesn’t get you help desk support - it is very much RTFM time plus knowing the LW spec plus RTFM on the hardware you deploy. If you want to ask someone questions, that’s the 300€ option, but even then only a limited number of times with a response time of 8 hours. Because just one support question is likely to require the whole of that 110€ to service. It is described as “Limited scope and maintenance, suited for experienced LoRaWAN developers” - fundamentally you need to be mostly self-sufficient. The 600€/month for the support element gets you what most people imagine to be a help desk.

And in all this, all the “interesting” cheap & cheerful device & gateway hardware that people can buy that tend to work OK are out of scope for support, see Service Level Agreement | The Things Industries

LoRaWAN was developed to service thousands of devices which is why it’s all big & corporate and has specs with numbers on each line and some serious charges for support because if you need it, it’s likely to be complicated and need people with serious chops to be able to sort stuff out.

TTN is mostly users with one or two gateways and a handful of devices. We are the polar opposite of the normal customer in so many aspects. Sometimes it is best not to look a gift horse in the mouth.

2 Likes

Hi,

I see there is a lot of gateways connected today. But mines are not. I m behind a firewall, is there a new port connexion to accept ?
best regards,

Apparently, there’s an automated process there that brings around 300 gateways online every day between 9 and 10 am. Mine is still offline, so I’m hoping it’ll be back online tomorrow or on Wednesday.

1 Like

Nothing has been published and the pattern I spotted last night is expanded upon by @mertens observation above, so it looks like patience is the order of the day.

1 Like

That has been unofficially confirmed. Every day additional gateways should recover.

My TTIG is up too, I appreciate the effort that was required to get this resolved.

@descartes - thank you very much for your explanation! Mine is also up since 11 hours now. Credits to the team which worked behind the scene to get this fixed.

Hello, No new gateway online today and mines are always off. any news?

Ok, I set my gateway on a direct link to the internet not through a firewall, and know I see it connected. So the update must use a new configuration port to communicate with the TTN server. Have a documentation about that to configure our firewall in the right way?

Rather like this:

It’s not about State Secrets or the recipe for Pepsi, when something is known, it will be posted.

However you can look at your router to see what ports are being used.

Hi All,
I have unclaimed and deleted my TTIG gateway from TTN

I then reset my TTIG and reconfigured it and tried setting it up again on TTN but its still flashing orange green.

Will this device eventually re-connect again or is it over for my TTIG?

Thank you

Yoda says “Do or Do Not, there is no try”.

Were you successful when you re-claimed it?

Everything anyone knows is covered in this thread. So it may well be that it will just have to wait.

Im in the same boat, where i deleted the gateway when it wasnt working and tried to add it again. It all seems file on the website, but just to double check, the WiFi password is the Owner Token right? It has been too long since i set it up.