Integrations - Data Storage / API generates a CORS error when accessed from script running in my browser

[marcovanschagen (Marcovanschagen Junior IOT)]

Hi,
As part of the Junior Internet of Things Challenge for school teams in Alkmaar, we are developing a simple GPS tracker unit based on the RFM95, an Arduino Pro mini, a small GPS board and a small LiPo. Each team will try to build a tracker and we are planning to launch these using a few balloons, very similar as was done at the Sodaq event in Hilversum.
Feel free to take a look at wiring diagram, code and other bits and pieces in our Kaasfabriek Github!

Now, as part of the new TTN Data Storage option I have started to pull the data from the API into Excel using WinHttpRequest and a bit of good old VBA. I must say, this is a very cool option to allow you to quickly create your personalized graphs and data tables. The API approach simplifies things a lot for our junior teams.

Now I wanted to add some real-time mapping of waypoints using Google maps and some javascript. We are also using the ttnmapper.org - however the Data Storage API could allow us a few tweaks, such as connecting the dots of the different teams in their own color.

However, in javascript I have tried to use XMLHTTPRequest to read data from the TTN Data Storage API, this generates an error. To my understanding, the browser recognizes this as a CORS, Cross Origin Resource Sharing, from one domain to another. Such call is blocked by the browser as it does not see an approval from the API, an approval (as I understand) the API can communicate by adding specific response headers. My apologies if this wild story is a result of a coding error on my part…!

My very prelim code is visible at http://dataschrift.nl/ballon_map.html
The error displayed is:
XMLHttpRequest cannot load https://.data.thethingsnetwork.org/api/v2/query?last=2h. Response to preflight request doesn’t pass access control check: No ‘Access-Control-Allow-Origin’ header is present on the requested resource. Origin ‘http://dataschrift.nl’ is therefore not allowed access. The response had HTTP status code 404.

A few questions:

• Is TTN planning to allow access to this API from within browsers, and can such CORS headers be added?

• In the client side scripting, our access key is plainly vsible. There is no privacy issue as we planned the node data as ‘safe for public use’. However, I am wondering if there is a nice way to obscure the key a bit.

• Is there a policy on API usage and traffic? What if many people would start hitting the page, will the API start to blacklist our account?

Again, apologies if I did a coding error - actually that would give us the quickest solution… Otherwise, I am very interested to see a few responses!

Thanks,
Marco

[marcovanschagen (Marcovanschagen Junior IOT)]

@johan

[arjanvanb (Arjan)]

You first two bullet points could only be implemented together. Without some way to explicitly make (some) data public without specifying credentials, CORS is simply needed. Like if one can authenticate using a cookie (either now or in the future) then I could be authenticated on data.thethingsnetwork.org while visiting any other site. If that other site makes a request from my browser to data.thethingsnetwork.org then CORS stops that site from getting the visitor’s data.

Note that CORS does not apply to anything in your backend (like in PHP code), as your backend cannot access any cookie my browser has for data.thethingsnetwork.org. And you can also keep your secrets for yourself in the backend. So my wild guess would be that this integration is not meant for use in web pages. (But I’d love to be surprised, and then I’d have a 4th question: can TTN please add WebSocket support? ;-))

As an aside: the LiPo might never be found again Also note that your city’s regulations prohibit attachments that exceed 30 grams.

[marcovanschagen (Marcovanschagen Junior IOT)]

Arjan, thanks!

Thanks for looking into the details of our plan! The city of Alkmaar requires a permit if payload is above 30 grams, balloon over 75 cm, balloons in high quantities and such. For our handfull of balloons we barely reach the weighth threeshold, so we have kindly requested the permit. Also, I am guiding the teams to discuss all aspects of the plan, technical and environmental, find alternatives and compare with other balloon events. And yes, the LiPo still bothers me, we are using the tinyest available and are asking any finders to send back our device.
All combined this is giving my Junior teams and myself a wonderfull deep-dive into the world of IoT. In the TTN discussion however I hope to focus on The Things.

My first two bullets were mentioned separately as I believe it is up to the data publisher to decide. The first bullet is about tech feasibility, and I hope this triggers a good tech discussion, especially to convince TTN to add this feature…!

As for the second bullet, the key only gives read-only access to sensor data, also can easily be revoked if need arises. At this stage it could be too much to ask TTN for a config field to define which domains (the “origin” in the request) can request data.

And indeed, if TTN can handle a bit of traffic, your 4th question will likely change the way Things are done

Pinging @johan for feedback…

[johan (Johan Stokking)]

@marcovanschagen

• Is TTN planning to allow access to this API from within browsers, and can such CORS headers be added?

• In the client side scripting, our access key is plainly vsible. There is no privacy issue as we planned the node data as ‘safe for public use’. However, I am wondering if there is a nice way to obscure the key a bit.

We don’t encourage application developers to call the Storage API directly from the client application for two reasons. First, it doesn’t scale on our end. It’s a free service, and the potential size of a week of data of an application is really high so this is a practical limit. We need the TTN public community network infrastructure to scale. In fact, the reason why the storage integration came this late, is exactly for this reason. Second, you don’t want your application access key in your client applications, especially not in browsers where they’re plain and visible in code or in any browser builtin network analyser. With the key’s messages right (the bare minimum for the storage integration), you can schedule downlink. It’s a very bad practice, regardless of how public your data is.

@marcovanschagen

• Is there a policy on API usage and traffic? What if many people would start hitting the page, will the API start to blacklist our account?

Currently there isn’t any. If we introduce this, we will announce that in time and probably contact application developers that are violating future defined new rules. Today it’s “best effort” as with anything in the public community network.

For the CORS, I would suggest a proxy that runs in your domain that fetches data from the Storage API and makes it available to your clients. You can do this server less in AWS and Azure, for example. You could do the same with MQTT to web sockets, although not server less. Both approaches allow you to use your own security mechanism.

[marcovanschagen (Marcovanschagen Junior IOT)]

Thanks, it makes a lot of sense to go for best efford. We will adjust accordingly.