I have an application with a LoRaWAN device and I want to use the storage integration to store the data. The storage integration is enabled by opening Integrations → Storage Integration → Enable and I can simulate live data which gets stored.
I can see the data by using the API, but the data from the device aren´t stored in the database.
How can I forward my device data into my database to store them?
Which API and which database are you referring to?
Data Storage you fetch the data, other integrations forward.
MQTT or WebHooks give you the maximum amount of information. MQTT you need an application running to listen for messages & process them. WebHooks you can write a web server script to receive the data & process them.
Data Storage you request the data for an application or just a device & process them.
Hi @descartes,
I´m referring to this database
I thought that the storage integration will store the data in my account and then I can access them as I wish.
And you can access it as you wish - not only is it all detailed in the documentation but there are even some sample web addresses being displayed on the console for you.
What do you mean when you say you can see your data by using the API?
How do you know the data is not being stored in the database?
If you can see your data, surely that means it is stored in the database!
Hi,
I use the following Python snipped to access the data:
import json
import requests
Application = "kampi-smartmetering"
APIKey = ""
Fields = "up.uplink_message.decoded_payload,up.uplink_message.frm_payload"
NumberOfRecords = 30
URL = "https://eu1.cloud.thethings.network/api/v3/as/applications/" + Application + "/packages/storage/uplink_message?order=-received_at&limit=" + str(NumberOfRecords) + "&field_mask=" + Fields
Header = { "Accept": "text/event-stream", "Authorization": "Bearer " + APIKey }
print("\n\nFetching from data storage...\n")
r = requests.get(URL, headers = Header)
JSON = "{\"data\": [" + r.text.replace("\n\n", ",")[:-1] + "]}";
print("URL: {}\n".format(r.url))
print("Status: {}\n".format(r.status_code))
print("Response: {}\n".format(r.text))
print("JSON: ")
print(json.dumps(json.loads(JSON), indent = 4))
print()
The script only returns the data which I have simulated (4 data points) over the Messaging tab
but not the data from the device over the last hours (18 messages).
Somewhat frustrating that you just didn’t start with this as that’s either my code or you are a miracle mind-reader!
My code is working. Your code is working with my application & key.
I don’t know how long is storing uplinks in Data Storage in v3. Can you trigger an uplink and then try?
Have you tried removing the limit just in case?
Hi,
removing limit doesn´t change something. My device is sending new data every hour. The simulated data are older than the last data from my device.
But I have configured a payload formatter in my device. May this formatter cause this issue?
If it is generating an issue I’d still expect the raw part of the payload to be stored.
I’d try removing the sort and the field filters, again, worth a try.
Then I’d try turning off the payload formatter, again just to see.
Yes it can. What happens if you reset to the default formatted?
I will check it.
Edit: I have checked it and it seems that the formatter was the issue. The data is stored correctly after I removed it.
Hi,
I am having the same issue and removing the formatter didn’t work.
I believe the v3 stores and shows the incoming data but I couldn’t manage to access the data that I took last week.
Do you have any suggestions?
Thank you!
What have you tried? What are you expecting in the way of accessing data?
I took some measurements last week, and I was able to see the data on the console and through the sample web addresses provided at the storage integration tab when I was taking the measurements. However, I didn’t save them, and there is nothing on the console/web address now.
I tried the above code, but it didn’t return anything. When I use my other applications reporting frequently, it shows only the real-time data that you can see through the web address, but there is no historical data, e.g., two days or a week old.
I need what I measured; raw or processed doesn’t matter.
The Data Storage only holds data for up to about 36 hours.
You may want to look at these for ways of exporting Data Storage with a cron job or using MQTT or Webhooks:
Okay, thank you. I am checking it now.
Just to confirm, it won’t be possible to get those data back, right?
Correct, if it’s not there, it’s gone.
It’s not a primary data source for the stack, it’s there mostly if your other integration(s) go down you can grab the lost data from there.
My backup Webhook is on a £7/year server, so some PHP web space shouldn’t cost much and doesn’t even have to be in country.
This shows setup:
Thank you for your help. I will surely do that.
is there a example for javascript ?
What do you want to do?
More details other than just the Python being translated in to JavaScript please.
hi i get this Data but how I can fetch it with the api key in javascript ? I don´t know how i use the api key to make a authorized
the result :
{“result”:{“end_device_ids”:{“device_id”:“eui-70b3d57ed004bf93”,“application_ids”:{“application_id”:“jjjj”},“dev_eui”:“70B3D57ED004BF93”,“dev_addr”:“260BA217”},“received_at”:“2022-01-29T21:18:17.682195763Z”,“uplink_message”:{“f_port”:1,“frm_payload”:"/////wXDAAAADQACAAU=",“decoded_payload”:{“Etage”:2,“Gebaeude”:13,“Lichtintensität”:1475,“Luftfeuchtigkeit”:65535,“Raum”:5,“Reed-Kontakt(1=Geschlossen, 0=Geoeffnet)”:0,“Temperatur”:65535},“rx_metadata”:[{“gateway_ids”:{“gateway_id”:“lorawangatewayy”,“eui”:“58A0CBFFFE8047F5”},“time”:“2022-01-29T21:18:17.416009902Z”,“timestamp”:2478936795,“rssi”:-54,“channel_rssi”:-54,“snr”:10.25,“location”:{“latitude”:54.34513235377525,“longitude”:10.092231893285023,“source”:“SOURCE_REGISTRY”}}],“settings”:{“data_rate”:{“lora”:{“bandwidth”:125000,“spreading_factor”:7}},“coding_rate”:“4/5”,“frequency”:“868100000”,“timestamp”:2478936795,“time”:“2022-01-29T21:18:17.416009902Z”},“received_at”:“2022-01-29T21:18:17.476084832Z”,“consumed_airtime”:“0.066816s”,“network_ids”:{“net_id”:“000013”,“tenant_id”:“ttn”,“cluster_id”:“ttn-eu1”}}}}