A Python program to listen to your devices with MQTT

A python program to listen to your devices with mqttn with credentials

I will upload a Spark version when ready.

# node.py
# Simple Python client to show node activity from ttn MQTT brooker with credentials
# Author: R.Schimmel
# www.schimmel-bisolutions.nl
# first install paho.mqtt.client: pip install paho-mqtt
# 
import paho.mqtt.client as mqtt

#Call back functions 

# gives connection message
def on_connect(client,userdata,rc):
    print("Connected with result code:"+str(rc))
    # subscribe for all devices of user
    client.subscribe('+/devices/+/up')

# gives message from device
def on_message(client,userdata,msg):
    print"Topic",msg.topic + "\nMessage:" + str(msg.payload)

def on_log(client,userdata,level,buf):
    print("message:" + str(buf))
    print("userdata:" + str(userdata))
    
mqttc= mqtt.Client()
mqttc.on_connect=on_connect
mqttc.on_message=on_message

mqttc.username_pw_set("App-eu registered op ttn dashboard","key registered op ttn dashboard")

# BEWARE, outdated; see the follow up posts to use eu.thethings.network instead
mqttc.connect("staging.thethingsnetwork.org",1883,10)

# and listen to server
run = True
while run:
    mqttc.loop()
2 Likes

Ruud,

I’m a LoRa newbie so bear with me please.

I have a LoRa node (Sodaq Explorer) that send out data every now and than. I can see this data comming in at the MQTT server with this command:

mosquitto_sub -h eu.thethings.network -t '+/devices/+/up' -u '<Application ID>' -P '<Access Keys>' -v

Gives the messages:

<Application ID>/devices/<Device ID>/up {"app_id":"<Application ID>","dev_id":"<Device ID>","hardware_serial":"0004A30B001BABF7","port":1,"counter":877,"payload_raw":"SGVsbG8gWzEwOV0A","metadata":{"time":"2017-08-19T13:52:16.030688605Z","frequency":868.1,"modulation":"LORA","data_rate":"SF7BW125","coding_rate":"4/5","gateways":[{"gtw_id":"eui-b827ebffffe8ddf2","timestamp":2376170187,"time":"","channel":0,"rssi":-48,"snr":9,"latitude":52.34849,"longitude":5.21332,"altitude":-5}],"latitude":52.34853,"longitude":5.2133574,"altitude":-2,"location_source":"registry"}}

So, adopted your python program in the hope to be able to read and decode the messages from my node … but it does not!

I changed the mqttc.username_pw_set() to:

mqttc.username_pw_set("<Application ID>","<Acces Keys>")

and:

mqttc.connect("eu.thethings.network",1883,60)

Running the program states:

Connected with result code:5

Resetting the connect statement to "staging.thethingsnetwork.org" gets rid of the error code 5 but nothings shown on the output…

I obvious are missing something.
Help would be very much apriciated!

From the library’s documentation:

The value of rc indicates success or not:

0: Connection successful 1: Connection refused - incorrect protocol version 2: Connection refused - invalid client identifier 3: Connection refused - server unavailable 4: Connection refused - bad username or password 5: Connection refused - not authorised 6-255: Currently unused.

Are you sure you used the application id (the one you entered yourself, and like you also used in mosquitto_sub), not the AppEUI (the one that TTN generates)?

(It’s not a secret; anyone can see that id. The Python example works fine for me, though with a more recent version of the library one needs 4 parameters in def on_connect(client, userdata, flags, rc): )

Thanks for your reply!

If I enter everything as I do with the bare mosquitto_sub command, I get the code 5 error.

If I use the mqtt broker as in the example, I do nót get an error but nothing shows op in the output (although the mosquitto_sub in an other window does show messages passing by).

According to the examples I found the user name should be the AppEUI…
I have made an application with the name “loraexplora” and the device has the same name.
The password of the application is generated by TTN and shows at the AppKeys.

I’m really confused by the functions of all these keys and ID’s :neutral_face:

But, as far as I do understand Mosquitto, what I type in the cli version should be the same as what I type in the python script … but that does not seem the case here…

Hope you can enlight me some more.

No; see https://www.thethingsnetwork.org/docs/applications/mqtt/quick-start.html#credentials (You might be looking at old examples, which refer to the obsolete staging.thethingsnetwork.org.)

The access key you’re using should at least have access to the messages. You could generate an additional key just for your MQTT client.

Indeed, and so it does for me. I’d expect that too. (I have not tested with mosquitto_sub today.) The user name is the Application ID you’ve chosen yourself and the key starts with ttn-account-v2:

mqttc.username_pw_set("loraexplora", "ttn-account-v2.Cz...<redacted>...")
mqttc.connect("eu.thethings.network", 1883, 10)

Maybe update the Python MQTT library: pip install paho-mqtt --upgrade (and add the 4th parameter to on_connect like shown earlier.)

Ok, thanks again for all the help!

It seems TTN authentification does not like an Application and Device with the same name :frowning:

So I added an other application with the name “my_second_lora_app” and connected the original “loraexplora” device at it.

Than, with this program I can connect to the MQTT server and decode the send messages:

#!/<path-to>/python
# Get date from MQTT server

import paho.mqtt.client as mqtt
import json
import base64

APPEUI = "70B3D57EF00069A3"
APPID  = "my_second_lora_app"
PSW    = 'ttn-account-v2.<lots-of-chars>'

#Call back functions 

# gives connection message
def on_connect(mqttc, mosq, obj,rc):
    print("Connected with result code:"+str(rc))
    # subscribe for all devices of user
    mqttc.subscribe('+/devices/#')

# gives message from device
def on_message(mqttc,obj,msg):
    x = json.loads(msg.payload)
    device = x["dev_id"]
    payload_raw = x["payload_raw"]
    payload_plain = base64.b64decode(payload_raw)
    datetime = x["metadata"]["time"]
    #rssi = x["metadata"]["gateways"]["rssi"]  # <== this raises an error (why?)
    rssi = -1
    print(device + ": " + payload_raw + " ==> " + payload_plain + ", RSSI ["+ str(rssi) + "] @" + datetime )

def on_publish(mosq, obj, mid):
    print("mid: " + str(mid))

def on_subscribe(mosq, obj, mid, granted_qos):
    print("Subscribed: " + str(mid) + " " + str(granted_qos))

def on_log(mqttc,obj,level,buf):
    print("message:" + str(buf))
    print("userdata:" + str(obj))
    
mqttc= mqtt.Client()
# Assign event callbacks
mqttc.on_connect=on_connect
mqttc.on_message=on_message

mqttc.username_pw_set(APPID, PSW)
mqttc.connect("eu.thethings.network",1883,60)

# and listen to server
run = True
while run:
    mqttc.loop()
1 Like

because gateways is an array,
this: rssi = x[“metadata”][“gateways”][0][“rssi”]

works fine.

I have improved the code a bit to work in Python3 and provide a simple CSV output showing packages received and signal strengths for each gateway.

Output example: > 2018-04-15T13:52:31.521984942Z, drone-mapper-new, 319, eui-3535303229005f00, -37, {‘alt’: 252, ‘lat’: 46.565055642429314, ‘lon’: 15.658404568338653, ‘hdop’: 2.2}

# https://www.thethingsnetwork.org/forum/t/a-python-program-to-listen-to-your-devices-with-mqtt/9036/6
# Get data from MQTT server
# Run this with python 3, install paho.mqtt prior to use

import paho.mqtt.client as mqtt
import json
import base64

APPEUI = "70B3D57EF00069A3"
APPID  = "my_second_lora_app"
PSW    = 'ttn-account-v2.<lots-of-chars>'

#Call back functions

# gives connection message
def on_connect(mqttc, mosq, obj,rc):
    print("Connected with result code:"+str(rc))
    # subscribe for all devices of user
    mqttc.subscribe('+/devices/+/up')

# gives message from device
def on_message(mqttc,obj,msg):
    try:
        #print(msg.payload)
        x = json.loads(msg.payload.decode('utf-8'))
        device = x["dev_id"]
        counter = x["counter"]
        payload_raw = x["payload_raw"]
        payload_fields = x["payload_fields"]
        datetime = x["metadata"]["time"]
        gateways = x["metadata"]["gateways"]
        # print for every gateway that has received the message and extract RSSI
        for gw in gateways:
            gateway_id = gw["gtw_id"]
            rssi = gw["rssi"]
            print(datetime + ", " + device + ", " + str(counter) + ", "+ gateway_id + ", "+ str(rssi) + ", " + str(payload_fields))
    except Exception as e:
        print(e)
        pass

def on_publish(mosq, obj, mid):
    print("mid: " + str(mid))

def on_subscribe(mosq, obj, mid, granted_qos):
    print("Subscribed: " + str(mid) + " " + str(granted_qos))

def on_log(mqttc,obj,level,buf):
    print("message:" + str(buf))
    print("userdata:" + str(obj))

mqttc= mqtt.Client()
# Assign event callbacks
mqttc.on_connect=on_connect
mqttc.on_message=on_message

mqttc.username_pw_set(APPID, PSW)
mqttc.connect("eu.thethings.network",1883,60)

# and listen to server
run = True
while run:
    mqttc.loop()
1 Like

Beware that many examples declare some log function, but don’t actually make the Paho MQTT library use it. As Paho tends to swallow many exceptions in callbacks, enabling logging surely helps finding even basic typing errors.

For the above examples, add the last line below:

mqttc = mqtt.Client()
# Assign event callbacks
mqttc.on_connect = on_connect
mqttc.on_message = on_message
# Log all MQTT protocol events, and the exceptions in callbacks
# that have been caught by Paho
mqttc.on_log = on_log

When using Python’s logging, you can convert the log level of the Paho message to Python’s logging level, using:

import logging
...

def on_log(self, client, userdata, level, buf):
    """
    Log all MQTT protocol events, and the exceptions in callbacks
    that have been caught by Paho.
    """
    logging_level = mqtt.LOGGING_LEVEL[level]
    logging.log(logging_level, buf)

Note that on_connect does not guarantee one is indeed connected:

def on_connect(self, client, userdata, flags, rc):
    logging.info("MQTT connected: result code=%i", rc)
    if rc == 0:
        res = client.subscribe("+/devices/+/up")
        if res[0] != mqtt.MQTT_ERR_SUCCESS:
            raise RuntimeError("the client is not connected")

    if rc == 1:
        raise RuntimeError("connection failed: incorrect protocol version")
    if rc == 2:
        raise RuntimeError("connection failed: invalid client identifier")
    if rc == 3:
        raise RuntimeError("connection failed: server unavailable")
    if rc == 4:
        raise RuntimeError("connection failed: bad app_id or access_key")
    if rc == 5:
        raise RuntimeError("connection failed: not authorised")
    # 6 - 255: currently unused

Also, you might want to use TLS (and port 8883 rather than the non-TLS 1883). Just replacing the line for mqttc.connect with the following will do:

# By default, on Python 2.7.9+ or 3.4+, the default certification
# authority of the system is used.
mqttc.tls_set()
mqttc.connect("eu.thethings.network", 8883, 60)

Alternatively, explicitly refer to the Certificate Authority chain as used by the Let’s Encrypt server certificate, which is available in https://console.thethingsnetwork.org/mqtt-ca.pem:

mqttc.tls_set(ca_certs="mqtt-ca.pem")

For all of the above, when looking at the (deprecated) TTN SDK, add something like:

import logging
import paho.mqtt.client as mqtt
...
class MQTTClient:
    ...
    def connect(self):
        self.__client.on_log = self._on_log
        ...

    def _on_log(self, client, userdata, level, buf):
        """
        Log all MQTT protocol events, and the exceptions in callbacks
        that have been caught by Paho.
        """
        logging_level = mqtt.LOGGING_LEVEL[level]
        logging.log(logging_level, buf)

And for TLS add the following somewhere before self.__client.connect:

self.__client.tls_set()
1 Like