TTN MQTT integration is not working

Very strange issue we are facing since 2nd September,2020. MQTT integration not working only for one of my application, it has been working without any issue so far. MQTT settings, access key all are same but no data over mqtt integration from this ttn application. Is there any change or outage for ttn mqtt services in europe region. Please share.
Is it because of this ? image

Hi @interravikram, no problem here with my EU region MQTT subscriptions.

I suggest that you try subscribing interactively using the mosquitto_sub utility with the -d [ebug] flag set and observe if the MQTT connect and subscribe works and if there is a flow of uplinks as you expect.

Have you fully stopped and re-started your MQTT subscriber to ensure that it is connecting to the right MQTT server since the incident on Sept. 2nd?

1 Like

Yes i have restarted and its strange because for all other applications mqtt integration is working . Issue is only on one application.

Issue not resolved, hence shifted to http Integration.

This sounds very similar to a serious issue we’re having. Actually existing TTN MQTT connections are still receiving data, but any attempt at a new connection hangs without receiving any new data.

Any hints or help will be much appreciated as currently unable to create any new subscription to the data means we are ultimately going to fail in a serious way.

We have data coming from ~30 sensors and can see the data in the archive from the MQTT subscribers that are still running, but:

  1. if we access the console and select “Applications -> appname -> Data” then no data appears there

  2. if we access the console and select “Applications -> appname -> Devices -> devicename -> Data” then no data appears there.

  3. If we subscribe with a new client, we get no messages, e.g. with mosquitto_sub -d … :

mosquitto_sub -v -h ‘eu.thethings.network’ -p 1883 -u ‘cambridge-sensor-network’ -P ‘ttn-account-v2.rest-of-the-key’ -t ‘+/devices/+/up’ -d

The ‘-d’ output is a continuous sequence as below with no data and the sequence repeating once-per-second with the ‘Mid’ number incrementing (even though the existing connected clients are receiving sensor data messages ok):

Client mosqsub|32618-tfc-app6 sending CONNECT
Client mosqsub|32618-tfc-app6 received CONNACK
Client mosqsub|32618-tfc-app6 sending SUBSCRIBE (Mid: 1, Topic: +/devices/+/up, QoS: 0)
Client mosqsub|32618-tfc-app6 sending CONNECT
Client mosqsub|32618-tfc-app6 received CONNACK
Client mosqsub|32618-tfc-app6 sending SUBSCRIBE (Mid: 2, Topic: +/devices/+/up, QoS: 0)
… this continues indefinitely

Hi @CambridgeSensorNetwo, I have just tried a new console connection (W10, chrome) and a new MQTT subscribe (mosquitto_sub -d…) to an existing app with existing devices on EU.

Both are working normally, showing uplink data as expected.

I guess that the fault that you are seeing is only affecting some users or applications or connections.

thanks - all info is helpful. I’ll test if another TTN ‘application’ for my user also sends MQTT data ok - I’m guessing it will.

We’ve been running sensors on TTN for multiple years without any real issues, so this is new for us.

I’m considering two fallback plans (I have an additional 400 sensors delivered this week, so I have to do something)

  1. Implement a HTTP POST receiver like the OP .
  2. Find out if I can download all my sensor registrations into a file and upload them into another application and move on from there.

Of course I’d prefer a way to kick the MQTT feed back into life.

Anyone got any advice/clues at all? Either to fix the application MQTT integration or somehow reset it (I know all this is an open-source freebie but is there any email address I could try?). This permanent failure of an application is really a problem for us, and I’m about to deploy 400 sensors in a research project but really should hold off until this is worked out.

Thanks.

Just as side thought and given the scale of what you are doing and no doubt your reliance on the infrastructure, would it make sense to be working with the The Things Industries - we have dozens of nodes but 400 plus really needs the platform backup support in place I would have thought.

1 Like

As @jezd says, or run your own backend …

… OR add BOTH data storage AND HTTP Integration to backup the MQTT.

It seems issue is resolved now, MQTT integration working back on all of my applications . Please check if its same on your end too. Meanwhile i have been shifted to http integration and keeping that too as a backup if MQTT stops working in future .

Just run all three if you have a community benefit system critical application.

If it’s commercial and system critical, run your own servers!

I wonder:

Yup, just seen & replied to that!

Thanks for the helpful posts here. Our ‘TTN application’ still fails to send data to new MQTT subscriptions (although oddly the existing connections from before ~Sept 1st continue to run).
I’ve contacted TheThingsIndustries to see what we might do with them. This isn’t as simple as building our own network server and adding a user-friendly front-end (we did that in previous generations of our network) because we’re covering a large region with our gateways and want to continue to support public access - in practice that helps research usage across the university also as TTN is a well-known LoraWAN point-of-access and we’re happy to support it with our gateways.

If we fall back to the HTTP connection (which we used before MQTT) I’ll really need to implement our own “MQTT-HTTP bridge to TTN” as we have messages flowing around on our platform using Mosquitto/MQTT (e.g. messages coming from Wifi and Zigbee sensors) - if anyone’s aware of an existing open-source implementation please let me know. I know the basic idea is simple (a POST arriving from TTN results in a PUB to Mosquitto, and something similar in reverse) but writing all this stuff from scratch adds up a lot of work (e.g. we have an ok decoders system as an alternative to the TTN javascript per-application decoder).

Have you tried to add an additional, new access key to the TTN Application experiencing the MQTT issues and then use the new access key for new connections, just to see if that works?

Are you currently using MQTT bridging?

Using your own MQTT server for your applications makes them less dependent on TTN’s MQTT server. Each message will then have to be transferred from TTN only once - from TTN’s MQTT server to your own (local) MQTT server. Your applications can then communicate with your (local) MQTT server.
But for the bridging to work also requires that new MQTT connections for your TTN Application (from your MQTT server to TTN’s MQTT server) must work successfully.


To prevent naming confusion:

  • Application is any kind of software application.
  • TTN Application is the ‘application’ that you define on the TTN console, which serves as ‘container/parent’ for its configured devices.

Thanks bluejedi - good suggestion but unfortunately using the new key immediately fails with “Connection Refused: not authorised” and the client (mosquitto) disconnects.

Attempted new MQTT ‘mosquitto_sub’ connections to receive messages from the TTN Application with the original (default) access key continue to hang, with no messages being returned.

I am using Mosquitto MQTT ‘in’ bridging as the preferred method of connecting my Linux servers to the TTN MQTT broker to receive messages, these are (unsurprisingly) failing to connect to TTN in the same way as the Linux terminal ‘mosquitto_sub’.

Existing connections to the TTN MQTT broker are still receiving the data but understandably these are gradually dropping off for local reasons and failing to reconnect.

So the TTN Application MQTT access seems fairly seriously borked, and I’ve deferred my install of the additional sensors until I have an effective migration plan if this happens again. I can run a HTTP client but as the MQTT access seems permanently dead then I have a new dependency on HTTP so I still need a plan B. It would be easy enough to run our own server/platform but we want to continue to support public use of our gateways via the TTN console as that works really well not only for our region but also within the university for the majority of projects which are typically small-scale (like single sensors…).

I’m talking to The Things Industries and they have a proposed solution which I need to look into more carefully.

In any case I’m assuming I should maintain the list of our sensor registration data and implement a program to register all those sensors via a TTN api, so I can re-register them easily to another TTN application if this failure occurs a second time. I’m not sure how the sensors (mainly Elsys, RadioBridge) will behave on a TTN Application swap though, but I can test that.

Hi @CambridgeSensorNetwo

It sure looks like it! The question is what’s borked?
There is no sign on the forum that this is affecting other users.

I have a test TTN application that has a single GPS tracker device on it that sends its location every 15mins. I use MQTT to subscribe to this feed both from Linux/mosquitto_sub and also from a mosquitto broker/server acting as a bridge so that I can change the topic prefix. Both of these are working correctly and I have rebooted both in the last 24hrs with no issues.

If you want to pm me, I will set up a new API key for you to access the TTN MQTT feed for this test application and you can see if it works.
If it works then the bork looks to be in TTN associated with your user/application.
If it doesn’t work then the bork may be in your infrastructure.