descartes
(Nick McCloud)
May 3, 2021, 2:37pm
2
Which API and which database are you referring to?
Data Storage you fetch the data, other integrations forward.
MQTT or WebHooks give you the maximum amount of information. MQTT you need an application running to listen for messages & process them. WebHooks you can write a web server script to receive the data & process them.
Data Storage you request the data for an application or just a device & process them.
Kampi
(Kampi)
May 3, 2021, 3:03pm
3
Hi @descartes ,
I´m referring to this database
I thought that the storage integration will store the data in my account and then I can access them as I wish.
descartes
(Nick McCloud)
May 3, 2021, 3:08pm
4
And you can access it as you wish - not only is it all detailed in the documentation but there are even some sample web addresses being displayed on the console for you.
What do you mean when you say you can see your data by using the API?
How do you know the data is not being stored in the database?
If you can see your data, surely that means it is stored in the database!
Kampi
(Kampi)
May 3, 2021, 3:12pm
5
Hi,
I use the following Python snipped to access the data:
import json
import requests
Application = "kampi-smartmetering"
APIKey = ""
Fields = "up.uplink_message.decoded_payload,up.uplink_message.frm_payload"
NumberOfRecords = 30
URL = "https://eu1.cloud.thethings.network/api/v3/as/applications/" + Application + "/packages/storage/uplink_message?order=-received_at&limit=" + str(NumberOfRecords) + "&field_mask=" + Fields
Header = { "Accept": "text/event-stream", "Authorization": "Bearer " + APIKey }
print("\n\nFetching from data storage...\n")
r = requests.get(URL, headers = Header)
JSON = "{\"data\": [" + r.text.replace("\n\n", ",")[:-1] + "]}";
print("URL: {}\n".format(r.url))
print("Status: {}\n".format(r.status_code))
print("Response: {}\n".format(r.text))
print("JSON: ")
print(json.dumps(json.loads(JSON), indent = 4))
print()
The script only returns the data which I have simulated (4 data points) over the Messaging
tab
but not the data from the device over the last hours (18 messages).
descartes
(Nick McCloud)
May 3, 2021, 3:53pm
6
Somewhat frustrating that you just didn’t start with this as that’s either my code or you are a miracle mind-reader!
My code is working. Your code is working with my application & key.
I don’t know how long is storing uplinks in Data Storage in v3. Can you trigger an uplink and then try?
Have you tried removing the limit
just in case?
Kampi
(Kampi)
May 3, 2021, 4:26pm
7
Hi,
removing limit
doesn´t change something. My device is sending new data every hour. The simulated data are older than the last data from my device.
But I have configured a payload formatter in my device. May this formatter cause this issue?
descartes
(Nick McCloud)
May 3, 2021, 4:41pm
8
If it is generating an issue I’d still expect the raw part of the payload to be stored.
I’d try removing the sort and the field filters, again, worth a try.
Then I’d try turning off the payload formatter, again just to see.
kersing
(Jac Kersing)
May 3, 2021, 5:13pm
9
Yes it can. What happens if you reset to the default formatted?
Kampi
(Kampi)
May 3, 2021, 5:24pm
10
I will check it.
Edit: I have checked it and it seems that the formatter was the issue. The data is stored correctly after I removed it.
1 Like
Hi,
I am having the same issue and removing the formatter didn’t work.
I believe the v3 stores and shows the incoming data but I couldn’t manage to access the data that I took last week.
Do you have any suggestions?
Thank you!
descartes
(Nick McCloud)
August 3, 2021, 11:40am
12
What have you tried? What are you expecting in the way of accessing data?
I took some measurements last week, and I was able to see the data on the console and through the sample web addresses provided at the storage integration tab when I was taking the measurements. However, I didn’t save them, and there is nothing on the console/web address now.
I tried the above code, but it didn’t return anything. When I use my other applications reporting frequently, it shows only the real-time data that you can see through the web address, but there is no historical data, e.g., two days or a week old.
I need what I measured; raw or processed doesn’t matter.
descartes
(Nick McCloud)
August 3, 2021, 12:28pm
14
The Data Storage only holds data for up to about 36 hours.
You may want to look at these for ways of exporting Data Storage with a cron job or using MQTT or Webhooks:
Okay, thank you. I am checking it now.
Just to confirm, it won’t be possible to get those data back, right?
descartes
(Nick McCloud)
August 3, 2021, 2:13pm
16
Correct, if it’s not there, it’s gone.
It’s not a primary data source for the stack, it’s there mostly if your other integration(s) go down you can grab the lost data from there.
My backup Webhook is on a £7/year server, so some PHP web space shouldn’t cost much and doesn’t even have to be in country.
This shows setup:
1 Like
Thank you for your help. I will surely do that.
safe-99
(Safe 99)
January 30, 2022, 11:57am
18
is there a example for javascript ?
descartes
(Nick McCloud)
January 30, 2022, 12:19pm
19
What do you want to do?
More details other than just the Python being translated in to JavaScript please.
safe-99
(Safe 99)
January 30, 2022, 12:28pm
20
hi i get this Data but how I can fetch it with the api key in javascript ? I don´t know how i use the api key to make a authorized
https://eu1.cloud.thethings.network/api/v3/as/applications/jjjj/devices/eui-70b3d57ed004bf93/packages/storage/uplink_message
the result :
{“result”:{“end_device_ids”:{“device_id”:“eui-70b3d57ed004bf93”,“application_ids”:{“application_id”:“jjjj”},“dev_eui”:“70B3D57ED004BF93”,“dev_addr”:“260BA217”},“received_at”:“2022-01-29T21:18:17.682195763Z”,“uplink_message”:{“f_port”:1,“frm_payload”:"/////wXDAAAADQACAAU=",“decoded_payload”:{“Etage”:2,“Gebaeude”:13,“Lichtintensität”:1475,“Luftfeuchtigkeit”:65535,“Raum”:5,“Reed-Kontakt(1=Geschlossen, 0=Geoeffnet)”:0,“Temperatur”:65535},“rx_metadata”:[{“gateway_ids”:{“gateway_id”:“lorawangatewayy”,“eui”:“58A0CBFFFE8047F5”},“time”:“2022-01-29T21:18:17.416009902Z”,“timestamp”:2478936795,“rssi”:-54,“channel_rssi”:-54,“snr”:10.25,“location”:{“latitude”:54.34513235377525,“longitude”:10.092231893285023,“source”:“SOURCE_REGISTRY”}}],“settings”:{“data_rate”:{“lora”:{“bandwidth”:125000,“spreading_factor”:7}},“coding_rate”:“4/5”,“frequency”:“868100000”,“timestamp”:2478936795,“time”:“2022-01-29T21:18:17.416009902Z”},“received_at”:“2022-01-29T21:18:17.476084832Z”,“consumed_airtime”:“0.066816s”,“network_ids”:{“net_id”:“000013”,“tenant_id”:“ttn”,“cluster_id”:“ttn-eu1”}}}}
descartes
(Nick McCloud)
January 30, 2022, 12:55pm
21
Sorry, I meant are you:
Using Node.js?
Building it in to a web page?
Something else?
We have no big picture so no context so no idea how to direct you.