Download data using python

How to get started
1

Hello,

i am new in the topic. i would like to download data from ttn (v3) via python. I have created an ApiKey and have tried the example code from descartes https://github.com/descartes/TheThingsStack-Integration-Starters/blob/main/MQTT-to-Tab-Python3/TTS.MQTT.Tab.py.

In principle, the program starts up well and generates the following output:

Imports:
Functions:
Body of program:
Init mqtt client
Assign callbacks
Connect

Log: Sending CONNECT (u1, p1, wr0, wq0, wf0, c1, k60) client_id=b''
Subscribe

Log: Sending SUBSCRIBE (d0, m1) [(b'#', 0)]
And run forever
.
Log: Received CONNACK (0, 0)

Connect: rc = 0
.
Log: Received SUBACK

Subscribe: 1 (0,)
......
Log: Sending PINGREQ
..
Log: Received PINGRESP

....
ERROR:root:Caught exception in on_message: 'data_rate_index'

Log: Received PUBLISH (d0, q0, r0, m0), 'v3xxxx@ttn/devices/xxxxx/up', ...  (1481 bytes)

Message: v3/xxx/devices/xxxx/up 0
{
    "end_device_ids": {
     xxxxxx
    },
    
    "received_at": "2023-12-24T06:29:44.829554629Z",
    "uplink_message": {
        "session_key_id": ".....",
        "f_port": 2,
        "f_cnt": 6064,
        "frm_payload": "......",
        "decoded_payload": {
            "Ext": 1,
            "Hum_SHT": 100,
            "Systimestamp": 1703399375,
            "TempC_DS": 327.67,
            "TempC_SHT": 7.46
   ... here is a lot more output but ...
    }
    }
}

Log: Caught exception in on_message: 'data_rate_index'

First:
what does both error messages / exceptiona are saying to me and how to solve it?

Second:
How can i get an array (time + values) of the decoded payload (eg TempC_SHT)?

Third:
i am not sure what to do with the following line

print(os.path.basename(__file__) + " " + VER)

do i have to exchange “file” to a self chosen filename?

Last:
the program is not producing an outputfile.

Can you help?

Thanks in advance
Frank

2

In the JSON there somewhere where there ‘… here is a lot more output but …’. You will see ‘uplink_message’ and that contains ‘settings’ and that have the ‘data_rate’.

Have you compared the TTN Sandbox console application JSON to your application JSON?

"uplink_message": {
      ...
      "settings": {
        "data_rate": {
          "lora": {
            "bandwidth": 125000,
            "spreading_factor": 7,
            "coding_rate": "4/5"
          }
        },
        "frequency": "868100000",
        "timestamp": 345085483
      },
 ...
    }
3

They are the same and relate to the data_rate_index but as you’ve not included that in your JSON it’s a bit hard to tell from here (mobile device) what the problem might be.

The code doesn’t expand out the decoded_payload as that is unique to each users application. Other parts show how to access the JSON so you could use that as a reference to do that. However it will only ever have one set of results in it because you are only sending one line at a time which is normal, so if you want an array of results, you’ll have to let incoming messages accumulate which means waiting for a number of messages to arrive. What do you want to do with the array?

Nothing, or delete the whole line or …

All it does is print the filename of the script and the version number - Googling os.path.basename & __file__ brings up lots of info on this.

It won’t do - the data_rate_index issue will prevent the saveToFile function from completing.

It’s a pretty bare bones script designed to just dump the LoRaWAN data to a tab delimited text file which can be used to import in to Excel or similar - and the console output lets you know what’s going on when developing. Think of it as a starter for ten, it’s functional & has some use but it rarely fits each individual need. So if you can tell us what you want to do with your data, we can expand on it.

4

Hello,

thanks to both of you. If i go to ttn i get the folllowing information:

      "settings": {
        "data_rate": {
          "lora": {
            "bandwidth": 125000,
            "spreading_factor": 7,
            "coding_rate": "4/5"
          }
        },
        "frequency": "868100000",
        "timestamp": 2273245347,
        "time": "2023-12-24T12:09:45.405481100Z"

in the json output i get

        "settings": {
            "data_rate": {
                "lora": {
                    "bandwidth": 125000,
                    "spreading_factor": 7,
                    "coding_rate": "4/5"
                }
            },
            "frequency": "868100000",
            "timestamp": 2273245347,
            "time": "2023-12-24T12:09:45.405481100Z"
        },

It looks equal.

I thougth to need the array to work with the data. But possibly the csv-file will solve to problem.

Thanks
Frank

the complete json output (partly changed)

Message: v3/xxx@ttn/devices/xxxx/up 0
{
    "end_device_ids": {
        "device_id": "xxx",
        "application_ids": {
            "application_id": "xxx"
        },
        "dev_eui": "xxx",
        "join_eui": "xxx",
        "dev_addr": "260B51BA"
    },
    "correlation_ids": [
        "gs:uplink:01HJDVPC8E38QSDANVW6N4QG8A"
    ],
    "received_at": "2023-12-24T12:09:45.692774290Z",
    "uplink_message": {
        "session_key_id": "xxx",
        "f_port": 2,
        "f_cnt": 6082,
        "frm_payload": "A38D6H//AWWIH38=",
        "decoded_payload": {
            "Ext": 1,
            "Hum_SHT": 100,
            "Systimestamp": 1703419775,
            "TempC_DS": 327.67,
            "TempC_SHT": 8.95
        },
        "rx_metadata": [
            {
                "gateway_ids": {
                    "gateway_id": "xxx",
                    "eui": "xxx"
                },
                "time": "2023-12-24T12:09:45.405481100Z",
                "timestamp": 2273245347,
                "rssi": -83,
                "channel_rssi": -83,
                "snr": 9.75,
                "location": {
                    "latitude": xx.424,
                    "longitude": xx.045,
                    "altitude": 45,
                    "source": "SOURCE_REGISTRY"
                },
                "uplink_token": "xxxx",
                "received_at": "2023-12-24T12:09:45.461280569Z"
            }
        ],
        "settings": {
            "data_rate": {
                "lora": {
                    "bandwidth": 125000,
                    "spreading_factor": 7,
                    "coding_rate": "4/5"
                }
            },
            "frequency": "868100000",
            "timestamp": 2273245347,
            "time": "2023-12-24T12:09:45.405481100Z"
        },
        "received_at": "2023-12-24T12:09:45.487116875Z",
        "consumed_airtime": "0.061696s",
        "locations": {
            "user": {
                "latitude": xx.4242,
                "longitude": xx.0455,
                "altitude": 45,
                "source": "SOURCE_REGISTRY"
            }
        },
        "network_ids": {
            "net_id": "000013",
            "ns_id": "EC656E0000000181",
            "tenant_id": "ttn",
            "cluster_id": "eu1",
            "cluster_address": "eu1.cloud.thethings.network"
        }
    }
}
5

How is your Python programming?

Can you alter the python to extract the JSON value for data_rate?

Or your could comment that line out (and remove it from the output).


You’ve not actually told us what you want to do with it - and the TAB file will not give you the decoded payload in its present form so even once you’ve got the program working, it’s not going to help yet.

No point redacting gateway locations, they are publicly available by API. But particularly when you leave your user location in the file!

6

do you require the data_rate for your application

i know some do collect it to measure the performance

did you specify the path correctly

7

Hello,

at Descartes :

  • yes parsing json is new for me. I used time last hours to read into it.
  • i usually use fortran to process the data. I can now process the data with the output file.

at Piet Pillay:

  • i only need the decoded payload. removing all unneeded lines solved the problem.

I have now writen a version which is parsing the json until a string that includes the decoded payload und writes this to the files.

# Write uplink to tab file
def saveToFile(someJSON):
    end_device_ids = someJSON["end_device_ids"]
    device_id = end_device_ids["device_id"]
    application_id = end_device_ids["application_ids"]["application_id"]
    received_at = someJSON["received_at"]
    DecPayLoad = someJSON["uplink_message"]["decoded_payload"]; 

            
    # Device log
    pathNFile = application_id + "__" + device_id + ".txt"
    print(pathNFile)
    if (not os.path.isfile(pathNFile)):
        with open(pathNFile, 'a', newline='') as tabFile:
            fw = csv.writer(tabFile, dialect='excel-tab')
            fw.writerow(["received_at", "DecPayLoad"])
    
    with open(pathNFile, 'a', newline='') as tabFile:
        fw = csv.writer(tabFile, dialect='excel-tab')
        fw.writerow([received_at, DecPayLoad])

The output looks now like this.

received_at	DecPayLoad
2023-12-25T14:09:49.494325419Z	{'Ext': 1, 'Hum_SHT': 79.9, 'Systimestamp': 1703513375, 'TempC_DS': 327.67, 'TempC_SHT': 10.23}
2023-12-25T14:29:49.541348787Z	{'Ext': 1, 'Hum_SHT': 78.8, 'Systimestamp': 1703514575, 'TempC_DS': 327.67, 'TempC_SHT': 10.14}

Frank