I’ve read some docs, but I am having more trouble than I’d expect understanding how to access the data from a free account. I am sure that I do not understand some of the finer points, but I am working on it.
I have got my device connected and I have set up storage integration, but5 I am havign issues with the API variable names ({device_id} and {type}) ???
I’ve looked and I cannot find exactly how to do this as the tutorials that would normally dumb it down are mostly in V2/Swagger implementation.
My end goal is to set up an azure function to hit endpoint (x) every 60 seconds and stick the data in a table or similar.
At this point every time I try and hit that endpoint, I get “invalid type: value must be in list”, I obvioulsy cannot understand the docs and there’s a heap of methods in there I do not really understand how we got there, if anyone could help or give a simple explanation as to how to access this data once the integration has been enabled, that would be great. Thank you.
@htdvisser, can we have rate limiting on the Data Storage API - one “all the data” query every 24 hours and one “records since with number of records constrained” per hour would be a nice balance.
Sure, I definitely do not want to ‘melt’ anything.
Would you be able to point to a resource that might help? I looked at webhooks, but I wasn’t sure exactly what was going on in that section?
I half expected I could hit an endpoint, get the data in JSON and proces through an Azure serverless function and move on with my day. I watched a Video form TTN mentioning 30 second intervals, so I never considered that this might load the network. I am only working on a small application to track my hiking/riding so my wife knows I am safe, wouldn’t have thought the load was that big a deal, but I appreciate you pointing it out, thanks.
Any tips to proceed? I’m learning as I go here. Thanks again, Nick
The only context I had was the use of Azure which implies potential scaling of OMG proportions. But even for a smaller scale setup, consider if 1,000 TTS CE users (that’s less than 1% of the registered base) all had a single device retrieving data at 60 second intervals but to be kind, they implemented the data since timestamp so they only got recent records, I’m not sure which would be worse, the query load or just returning all the records.
Then there is the other implication of 60 second download - it is almost impossible to send geographic co-ordinates in a payload less than every 5 minutes assuming you are in an area with many gateways that ensure a low DR. So database updates every 60 seconds would be pointless unless the device was breaching the Fair Use Policy.
I’d say you are trying to solve the wrong problem.
Now if you are in a remote area for hiking or cycling, tracking by LoRaWAN isn’t a thing - if you live on a hill top and you put an antenna on a pole you may do OK but you will likely hit all sorts of DR adjustment issues and end up with no tracking or lots of airtime (again, impacting the FUP). If when you say riding you mean on a horse, from my experience it’s best to be followed by an ambulance
If you have an iPhone, then there are some good apps for that which would do the job nicely via the conveniently located network of mobile masts unless you live in the most visited national park in the UK, where mobile coverage isn’t considered useful by the networks.
You could try LoRa point to point and test that out - particularly on 433MHz but that’s off topic for here.
Loverly as they all are, there is a disconnect between marketing, the developers & reality on a good day. To be fair, marketing are pitching TTI instances upon which you are only constrained by legal limits, not the community Fair Use Policy for airtime and the shared resource of free servers that are provided.
Could you use MQTT / Python to create a desktop app so you can then go out and see what the coverage is actually like?
Have you looked on the gateway maps to see what coverage is available where you hope to go?
I see what you are saying and I appreciate the feedback. To clarify. I only ride for a few (maybe 4-5) hours once/twice a month, not all day, every day. I am not trying to start a commercial business. 60 Secs is the refresh rate in Azure is what I might use as a signal from the device that something was wrong if I come off or got badly injured or stuck under the bike etc, I would try to use those signals and insert into a seperate function to txt message my wife or whoever. I have no intention of breaking the fair-use policy. This is all about experimentation and prototyping, right?
I have good LoRaWAN coverage in my area and although I am unsure about entire coverage, having a position to start looking in an emergency is what I am hoping for even at 5min intervals. Coverage looks good for most of my regular trails. I have a webhook into TTNmapper and am so far impressed with the results of the received data in my area.
So getting back to where I was originally curious, can anyone point to any advice on accessing the API from storage integration? The docs are confusing me and my original question remains. I just need a hand to access my api from data storage integration, cheers.
And if you can point me to a way to do it in MQTT or other, than I am on board. I’m just struggling to access the data so I can use it, that’s the only real goal here. How can I access the data storage integration API?
To be clear, I’m speaking as a community user with some experience of going outside to do stuff, this is nothing to do with moderating which is nothing to do with technical issues.
It’s a tricky area - prototyping is one thing, but philosophically it’s like saying that the three pints of Stella were a trial run officer, it’s still naughty. And we still have the multiplication effect where less than 1% of the user base implement something impactful on the servers at any part - uplinks, payload formatting or data storage - and we end up with slow downs.
And that then leads back to using a device for safety tracking. If the servers have a whole sequence of mini-perfect storms just at the point you go straight over the handlebars and your emergency message is delayed or doesn’t get through because of all this extraneous activity, your wife won’t be impressed.
Then there is the no small matter of how poor coverage is on the ground so you’d best do some mapping at 10cm AGL to see what wheat, a clump of trees, wet grass or the spokes of a wheel does to signal strength.
I’m not being overly protective of the community resource here, I’ve tried the tracking thing already. Until recently I lived in the heart of the Peak District with scant mobile coverage, which is a bugger when you are volunteer ambulance service. I walked, cycled and flew (sailplanes) and my wife had a horse (my hobby injuries zero, hers, several). I launch & track High Altitude Balloons (think Met balloons) with schools using LoRa point to point - easy to track in the sky, but you need to know where they were at about 250m up before they land as once they are on the ground you need to be within about 1km of them to get an form of signal.
The tracker I ended up with was point to point LoRa for the majority of fixes, using a SIM808 module that provided GPS and when a fall was detected or the button pressed, it sent lots of SMS’s as they are, like people’s hearing, the last thing to go. The LoRa P2P had a simple “speak louder” command sent to it if signal strength waned at the cost of battery life. My home gateway’s antenna was on a 20m mast on the top of a hill. At the bottom of the village there was a dead patch along the main street due to the buildings. So I told the wife never to fall off behind a Derbyshire dry stone wall as they turn out to be perfect signal blockers. Thankfully Billy only threw her on a medium-hard grass at the roadside so she was able to walk him home and we never tested the device for real.
I was looking to setup some solar powered P2P relays on high hills and see if running a mesh between them and any on person trackers would be considered reliable enough - there are plenty of DoE expeditions that application would help as well as plenty of day walkers who would have benefited from some better geolocation when they phoned up my Mountain Rescue colleagues. But Covid.
As for more context, if you respond in remote areas for the ambulance service, fly and your main business makes electronics for security systems, you end up with some firm views on the nature of safety & redundancy.
All that said, try any one of these:
MQTT will give an almost instant response and has quality of service baked in (ie some level of guarantee of delivery), hint hint! It is almost ready to go out the box - adding an on screen log with a Google Maps link would be a nice evenings activity and the following two evenings you can make the map live.
As this is an interesting use case that has come up before (last major discussion was for personal attack alerts / kidnap tracker), maybe a grown up like @Jeff-UK could comment, he’ll certainly set me straight! It’s not just the FUP / impact on community servers, it’s the real time personal tracking viability in the great outdoors.
Thanks. I appreciate your concern, but safety isn’t really the end goal here, like I said, I am just playing around as a community user, trying to see how and where this could work. I have a Garmin InReach if I am riding alone, no one needs to be concerned about my safety, although I appreciate you pointing out the inadequacies, I’m just playing with the technology. And the coverage in the areas I ride suggests TTN will cover a large part of it, so I am just trying to push the network and see where I can get a hit while I explore this technology as it’s quite new to me.
I will look further into fair-use and appreciate the heads up, I am obviously out of my depth there, and I thought a few hundred packets @ .05 seconds would be ok, but I obvioulsy need to learn more about that and I will ensure I am more familiar with the rules moving forward.
The code samples you provided are awesome, thanks for sharing, you’ve obvioulsy got a lot of knowledge in this area. I am trying to work my way through MQTT, but I had never heard of it until recently, so I am trying to work out the differences between it being a transmission protocol and how it’s integrated with existing systems like Azure.
Ideally, I’m looking for an end-point that can be hit or will be pushed from the data source to Azure or whomever.
I’d also never heard of Node-Red, but I’m not real keen on drag and drop anything, so I’ll try to work out what’s what in that regards, thanks again for your help. You obvioulsy have a good idea oof what’s going on in this space. Thanks.
Being a thicked skin quasi-northerner, it’s not any one individual I’m concerned about, it’s the collective and I’d rather not read about someone thinking they can build themselves a radio tracker and coming a cropper off the back of any one thread positing some ideas.
As for the community aspects, the v2 servers are suffering because some people have overloaded them with connections, there is a post about it from a senior TTI engineer Friday, consequently some functionality that is rather useful to migration has been disabled. So anyone hoping to “hit” as you so frequently say, the world wide free-to-use servers that some communities rely on for flood warnings and other useful stuff are going to get some encouragement to re-design their application.
So please don’t try to “push the network” until you understand the details of what you are pushing and how it may have consequences that make others uncomfortable.
50ms is enough time to transmit 3 bytes, which if you send some cunningly crafted delta’s could just about fit a geolocation in - but even then you’d need to be close to a gateway to get the DR to 5 and you’d be limited to 583 uplinks in 24 hours on TTN.
This may feel like nitpicking on the semantics of what you are typing, but the juxtaposition of your choice of words and the data storage query rate point towards the wrong end of the use spectrum for this community resource.
To understand uplink limits, this is a very useful calculator, you should bear in mind that with longer distances you will have to find a way to automagically set the DR lower with the consequential impact on transmission frequency:
And for downlinks, it’s 10 a day per device but if you are doing much more than 1 a week there may be something wrong with your use of LoRaWAN - gateways are stone deaf whilst transmitting so any uplinks are lost so the best designs are the ones that do not rely on downlinks to be delivered in a timely fashion.
Node-RED is a thing in some circles of data presentation - some systems seem to clump together - so Node-RED to InfluxDB to Granfana (dashboard) is a solution set that’s in common use.
The very nicest integration to use, as advised by the aforementioned senior engineer, is web hooks which are so stupidly simple that they don’t even need configuring in the code I’ve supplied in the repro above but can produce output that can be copied & pasted in to Excel once it’s up & running. I’m not really in to the bigger iron stuff (AWS, Azure, GCP), I tend to get younger minds with more patience to make me one, but all of them have ‘serverless’ endpoints that can stuff the data in to any one of a bewildering range of databases which you can then BigData all over the place and draw charts and sometimes, maybe, some conclusions.
It is exciting tech, I have multiple back burner projects of all types & sizes littering my office/workshop, but it’s also got lots of moving parts that take time for the understanding to bake in. Just getting a device sending a number (not Hello World, we don’t transmit text here) and fishing it out the other end is a big milestone. Adding in a GPS & making the payload compact is a big step up. Creating a decent integration that then displays the GPS co-ordinates is another. Adding in Google Maps or some such gets a working version. With that sort of progression you’ll start to see the wrinkles, foibles & gotchas and have a much more in-depth understanding of it all.
Go and get a TTGO T-Beam 868MHz LoRa with GPS from Amazon, a gateway (highly recommended for when you want to debug your own devices, it lets you see what’s actually going on), a TTIG is OK unless you want to fiddle with it, then come back & search for low cost gateway recommendations and use LMIC-node for your firmware.
Hi @slim_pickins , the documentation is a little confusing. I wrote a primitive python script – GitHub - terrillmoore/ttn_storage_api – that shows all the details, and can be added to your own scripts. As everyone says, you may be happier with a more complete data service, but this works, and I have users who are happy with it.
