# Storage Integration data is delayd **URL:** \ **Category:** V3 Integrations\ **Created:** [July 4, 2024, 8:23am UTC](https://www.thethingsnetwork.org/forum/t/storage-integration-data-is-delayd/68469 "2024-07-04T08:23:32Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![nico11](https://www.thethingsnetwork.org/forum/letter_avatar_proxy/v4/letter/n/b487fb/32.png) [@nico11](https://www.thethingsnetwork.org/forum/u/nico11)\ **Post date:** [July 4, 2024, 8:23am UTC](https://www.thethingsnetwork.org/forum/t/storage-integration-data-is-delayd/68469/1 "2024-07-04T08:23:33Z")
Hi everybody, I wounder if anybody else is experiencing delays in the storage integration? I fetch data from storage integration via a bash-script in regular intervals and store it in a txt-file. Since approximately one week or so, data seems to be coming in in batches instead of being countinously added. Is there a specific reason for this behavior? Am I doing somthing wrong here? I noted chunks of data comming in at arround midnight (Berlin Timezone) and the earlier evening. Here the curl comand I use to retrieve the data: ```auto json_output=$(curl -G "https://eu1.cloud.thethings.network/api/v3/as/applications/xxxxx/packages/storage/uplink_message" -H "Authorization: Bearer xxxxxxxxxxxxxxxxxxx" -H "Accept: text/event-stream" -d "limit=${ups_count}" \ -d "field_mask=up.uplink_message.decoded_payload") ``` I use the variable _ups\_count_ to define the number of uplinks to fetch. Currently set to 100. Any Ideas on that issue? Thx ---
**Author:** ![Jeff-UK](https://www.thethingsnetwork.org/forum/user_avatar/www.thethingsnetwork.org/jeff-uk/32/35697_2.png) [@Jeff-UK](https://www.thethingsnetwork.org/forum/u/Jeff-UK)\ **Post date:** [July 4, 2024, 8:37am UTC](https://www.thethingsnetwork.org/forum/t/storage-integration-data-is-delayd/68469/2 "2024-07-04T08:37:03Z")
> [@nico11](#): > > I fetch data from storage integration Firstly an observation wrt SI’s purpose and roll… as I understand it SI is intended as a short term back-up source of data in the event that your own real-time capture processes fail or fall over; e.g. if you discover a missed message (perhaps based on fcount sequence or skipped timestamp) and need to pull a copy from this short term back up. What process do you have in place (Webhook, MQTT feed, or what ever) to capture and store the principal feed in your own storage instance? As such SI is intended for occasional vs regular access. If all users and all devices were taking regular SI feeds I believe we would quickly saturate and exhaust this valuable community resource and loads would be unsustainable (much as with the internal payload decoder function this is a convenience and not a guaranteed service and is load dependent - if I’m wrong in this description no doubt a fellow mod, contributor or member of TTi core will step in and correct! 😉 ) That said > [@nico11](#): > > via a bash-script in regular intervals and store it in a txt-file. How regular? 1/hr, 1/day…1/per when needed? > [@nico11](#): > > data seems to be coming in in batches instead of being countinously added I would not be suprised to hear of a change in behaviour if your access is being rate limited or as a general change to avoid: > [@nico11](#): > > being countinously added Just sayin’… ---
**Author:** ![descartes](https://www.thethingsnetwork.org/forum/user_avatar/www.thethingsnetwork.org/descartes/32/47220_2.png) [@descartes](https://www.thethingsnetwork.org/forum/u/descartes)\ **Post date:** [July 4, 2024, 8:39am UTC](https://www.thethingsnetwork.org/forum/t/storage-integration-data-is-delayd/68469/3 "2024-07-04T08:39:04Z")
> [@nico11](#): > > Am I doing somthing wrong here? Yes, you are using it as a primary data store - it’s only there as a backup if your primary integration - like HTTP - has any down time at your end and you need to fill in some gaps. DS is always delayed and committed to the database as & when, says so in the docs. And if everyone is doing what you do, it will increase delays on the **free to use community system**. There is a limit to the number of servers that TTI can fund for free. You also may be rate limited. How many devices are you doing this with? Why are you fetching 100 - why not fetch since the last uplink time - if there’s only 5 that’s a lot less load. But mostly, just stop and use HTTP. ---
**Author:** ![descartes](https://www.thethingsnetwork.org/forum/user_avatar/www.thethingsnetwork.org/descartes/32/47220_2.png) [@descartes](https://www.thethingsnetwork.org/forum/u/descartes)\ **Post date:** [July 4, 2024, 8:40am UTC](https://www.thethingsnetwork.org/forum/t/storage-integration-data-is-delayd/68469/4 "2024-07-04T08:40:26Z")
> [@Jeff-UK](#): > > if I’m wrong in this description no doubt a fellow mod, contributor or member of TTi core will step in and correct! Despite not being a softie, you are completely on the button here. I’m surprised we even get DS in TTN and having heard of some spectacular loads due to DS abuse, wouldn’t oppose its removal. ---
**Author:** ![Jeff-UK](https://www.thethingsnetwork.org/forum/user_avatar/www.thethingsnetwork.org/jeff-uk/32/35697_2.png) [@Jeff-UK](https://www.thethingsnetwork.org/forum/u/Jeff-UK)\ **Post date:** [July 4, 2024, 8:47am UTC](https://www.thethingsnetwork.org/forum/t/storage-integration-data-is-delayd/68469/5 "2024-07-04T08:47:23Z")
> [@descartes](#): > > wouldn’t oppose its removal. Please dont put ideas in their heads! 😉 Its a valuable resource for occational use and server load/storage burden was reduced somewhat when nominal storage time reduced from days to hours a while back…but even so we are dependent of users like @nico11 behaving reasonably, reading the supporting docs and not abusing. Anyone treating as a (near) live stream and (primary) feed should indeed be ‘policed’ and corrected as needed… can’t be sure but think that in \>6 years on TTN my devices and demoes have used it maybe 6 or 7 times… did a fantastic job when needed 🙂 ---
**Author:** ![nico11](https://www.thethingsnetwork.org/forum/letter_avatar_proxy/v4/letter/n/b487fb/32.png) [@nico11](https://www.thethingsnetwork.org/forum/u/nico11)\ **Post date:** [July 9, 2024, 8:48am UTC](https://www.thethingsnetwork.org/forum/t/storage-integration-data-is-delayd/68469/6 "2024-07-09T08:48:00Z")
Sorry for the long absence, I got the point. I’ll use MQTT instead querying data from the Si. Sorry for implementing this in the wrong way. I never thought this would do any harm. I think I’ll switch then to a backup use of SI only. Thanks for the hints. I hope you don’t mind, that not everybody is able to consider all these effects instantaneously. So it was not obvious (at least to me) that the SI is not intended as a regular interface for retrieving data. Thanks everybody! ---
**Author:** ![system](https://www.thethingsnetwork.org/forum/uploads/default/original/3X/6/2/6250ea44fd2b0d6259489e1a92748df681e1f970.png) [@system](https://www.thethingsnetwork.org/forum/u/system)\ **Post date:** [July 10, 2024, 8:48am UTC](https://www.thethingsnetwork.org/forum/t/storage-integration-data-is-delayd/68469/7 "2024-07-10T08:48:59Z")
This topic was automatically closed 24 hours after the last reply. New replies are no longer allowed.