Compression strategies for LoRaWAN packets

LoRaWAN LoRaWAN Discussions
11

Doh! Should have spotted the name. Perhaps I should look at some disposable launches, I’ve got gas begging to be used stuck in the shed all this past year.

But you / we / UKHAS have loads of data that can be used - or even made up based on your own prior launches. Both Peter & Steve have run some LoRaWAN flights this year, Steve definitely fed his in to HABHub so there should be some data there. Put some lines in to a spreadsheet, get a beverage and stare at it.

Not if the first frame in a payload is an absolute fix, which doesn’t actually need to be the full digits as a reference point based on a moving window coupled with last direction can be used.

After some analysis, it may be that the quantisation errors are relatively insignificant over a run of fixes depending on how many decimals accuracy you want the location to run to. Again, model it in a spreadsheet, you never know what may turn up.

As you are solar powered, if you can charge a supercap, then the first frame can have the absolute time and subsequent frames can be assumed to have happened on schedule. The timestamp could be two bytes with a base line of first transmission at turn on, giving you 45 days before roll over - if the balloon goes quiet for that long it should be feasible to figure out where it in its cycle it is when it’s next heard.

12

Just caught up with the ICSS blog - you’ve been keeping busy!

Maybe with the geofencing & results thus far you could store a fix a day whilst over the no-coverage areas and then find a scheme to weave them in to the uplinks when over better areas. Or add some FRAM to keep more of the data.

Additionally, if the panels are generating enough power and are over the right areas, you could open the RX windows to try to get some feedback and purge some of the prior fixes.

1 Like
13

That’s something you most certainly do not want to do! You want to be able to use the longer range, slower SF’s for such a purpose.

the current position as well as 12 previously saved positions

You should use maybe 4, spaced exponentially further into the past

The idea of sending deltas of each position not very attractive to me because the quantisation error adds up. It would only make sense to use SF7 for something like uploading historic data, after getting a reply that indicated you were in range of a gateway.

Send them each as deltas from the current position so that you only hit that once

14

That’s something you most certainly do not want to do! You want to be able to use the longer range, slower SF’s for such a purpose.

From experience on flights, the high data rate is not a problem at all; it’s the curvature of the earth getting in the way being more of a problem. Our tranmissions are regularly picked up at the edge of the line of sight limit. We have had transmissions picked up 700km away when the balloon was at 10.7km up. Gateway was 4km up on the Matterhorn.

We are better off using a high data rate(SF7 or SF8) and transmitting more regularly, while sticking to the TTN airtime limits.

Refer to image below for 700km+ range:

1262×802 1.55 MB

15

Steve Randall launched ICSPACE23 on 17 Dec but it is yet to show up again. It’s internal EEPROM can save 650 positions/timestamps which is plenty I think. The challenge is to send everything down over areas of TTN coverage. Global TTN coverage is limited to Europe, Japan and America. It may be up to 10 days before it reaches these areas.

Currently, in each transmission, it sends the current position and 12 semi-randomly selected past positions(with no repeats) to give us an idea of the past flight path. Subsequent transmissions will fill in the gaps.

Using a downlink(tx from gnd to balloon) to purge the non volatile memory of already received position fixes is a good idea. On the last flight, while over Belarus, only one of my downlinks was acked by the balloon. Getting packets up to the balloon seems more difficult.

Medad

16

Perhaps get it to send a one-a-day fix for the time it was over no-coverage area so you have a better idea of where it was and then it can progressively fill in some more of the detail rather than semi-random.

Whilst sticking to the lower SF to push the data through expeditiously, you may want to send at a higher SF every so often with a smaller payload - and if it picks up a downlink with reasonable RSSI/SNR, perhaps get busy sending a few more packets.

17

Why do you need to include the timestamp for each position in the payload - it takes up a lot of space, even as minutes since the year start? Is there a reason why you can’t use the time from the metadata that comes for free with your received package?
If you really need the timestamp, do you have to be accurate to the minute, or would every 10 mins do and reduce it to 2 bytes per position?

18

I believe that’s because a packet can include multiple distinct measurements from previous portions of the flight

Strategic resolution is indeed important.

It’s possible with a fixed stages back in time scheme (especially the exponential intervals I was arguing for) that the individual times don’t need to be sent, since they’d be known intervals before the transmission time.

And as along as the system is open loop, a fixed scheme is really as good as any other. Even a “random” repetition could be pseudo-random, in particular encryption of the measurement index offset can be a way to turn an ordinal sequence [1, 2, 3…] into an effectively random one which visits all of the same points. A dithering algorithm is probably better than a random one. But given there are going to be very long periods with both good and bad coverage, I continue to think that a sort of exponential history is probably going to be best.

19

Yes indeed we get one timestamp for “free” which is from the metadata. Notice that it is only one timestamp, and can conveniently be used to timestamp the current position of the balloon. However, in each packet, I send 12 previously recorded positions of the balloon so that I get an idea of where the balloon was when it was out of range of TTN gateways. Each position fix will need a timestamp so that I can plot the flight path. Notably, there is no TTN coverage over the oceans and anywhere outside Europe, America and Japan.

The reason for minute precision is we calculate the sun elevations at which the balloon transmits at. The tracker is powered by 6 solar cells and the power output depends on solar elevation. When the sun is directly shining head on at the solar cells(elevation of 90 degrees), it will generate maximum power. At sunrise and sunset, the solar elevation will be lowest. We want to know what is the lowest solar elevation it can operate at. To do that, if we know the position and timestamp accurately, we can calculate the elevation of the sun. From our last flight, ICSPACE23, we found that the tracker’s GPS is operational at just 1.36 degrees sun elevation, which works out to just 2% of the power of the sun if it was shining head on.

A picture of our tracker is below.

4128×3096 601 KB

20

That’s superb Medad!
So rather than calculate minutes since 1/1/20 could you calculate minutes before transmission time for the 12 previous positions? This would be a much smaller number so you could fit into 2 bytes (or less) rather than 3?

Regards, Mark

21

Yes, and given there are only 1440 minutes in a day, some bit packing could bring this down to 1.5 bytes, using the other 4 bits for something else.

1 Like
22

You really don’t need to transmit the time differences at all; just make them fixed offsets.

If there are multiple possible history schemes, use multiple ports for different packet types.

1 Like
23

Your are right @mark-stanley. 3 bytes for timestamp can be shortened to 2 bytes. With 2 bytes, I can send a position/timestamp up to 45 days before the current time. That should be enough for the balloon application.

1 Like
24

Fixed offsets will not directly work because there may have been no data stored from certain times. e.g. when its night time. The tracker is powered solely by the sun.

Therefore, if I use offsets of, for example, 24 hours, I may have data from 24 hours ago, 48 hours ago but not 72 hours ago. Note that the balloon is rapidly traveling across timezones.

However, with a little improvement, this can work. Instead of using fixed offsets, I can get the most recent position/time fixes to these offsets. For example, I can send the closest position/time fix to 24 hours ago, 48 hours ago, 72 hours ago etc. I just have to send the time delta between the recorded timestamp and 24 hours ago, 48 hours ago etc. It may just take 1.5 bytes to encode the time delta in minutes.

25

First of all, you should change that with some on-board power storage. But it also exposes that you can apparently tolerate huge gaps in your data, which really reduces how much you need to move - if you can tolerate overnight gaps, then it would seem you really only need a few points a day.

Therefore, if I use offsets of, for example, 24 hours, I may have data from 24 hours ago, 48 hours ago but not 72 hours ago. Note that the balloon is rapidly traveling across timezones.

You could certainly work out a sensible scheme; given a location and calendar, you know approximately when the sun is up. And you’ve not moving that quickly that it’s going to have been more than a few hours different on previous days.

26

Should? It could be added, but as a decent supercap at 1F is 1.2g, that’s adding 20% to the tracker weight so it’s not a trivial change.

1 Like
27

First of all, that’s an interesting project :slight_smile:

I just have to send the time delta between the recorded timestamp and 24 hours ago, 48 hours ago etc

Alternately you can just send the deltas between each successive recorded timestamps:

For the first recording, you don’t need to send anything because you can use the date/time from the metadata.
For each successive recording send just the delta between it and the previous one.
Using 10 bits would give you a flexible window of 1024 minutes between recordings (17h)

You will need to take a bit of care when rounding the numbers, so the errors do not accumulate, but it’s possible to keep the error below 1 min for each recording:
For i =1 to n calculate diff(i) = t(i) - t(0) - sum(diff(0…i-1))
This should cancel out the errors from the previous diffs!

You can extend this range further in two ways:

  1. add 1 more bit to get 34h
  2. you can shift out some unused values
    For example, if it doesn’t make sense to send too recent data (<4h) you can just subtract 4h from the diff end encode the result … this way your range is effectively 4h-21h with just 10 bits

Another option is to use a “variable precision” encoding/mapping.
This would work best with a scheme like you described, where you encode the diff to the beginning of 1 day ago, 2 days ago, 3 days ago etc.
You map each possible value of a byte to 256 “fixed” minutes of a day, you can adjust the mapping table to map more values where you need more precision (sunrise, sunset), and have some larger jumps in the parts of the day that don’t matter much (noon?, night).
Of course, you can use more than 8 bits if needed

You can apply the diff strategy to the other fields (lat/lng).
Only the first frame needs to record with 2 bytes precision, subsequent frames record just the diff with the previous one.
Depending on how fast these balloons go you might be able to use just 1 byte for the diffs.
A quick calculation at 10km altitude with 20m/s wind speed, you get 1728 km/day which is about 16 degrees of lat/lng
I think you could safely pack this in 1 byte (if -180/+180 fits ok in 2 bytes).

You might be able to pack together the time-lat-lng-alg diff in 4 bytes (11, 7, 7, 7 bits).

28

Thanks @danmana for the ideas.

What would happen if there is a 2-3 day gap in the data? If I restrict all the deltas to a certain size, e.g. 2 bytes, if the actual delta is greater than that, how to deal with that?

The naive solution is to make the size of the delta bigger(e.g. 2 bytes to 3 bytes) but then its size becomes too big to be useful.

A multi-day gaps in data is a possibility. If the balloon flies over the Arctic region, the sun does not come up very high during the winter time which means the balloon may not be powered for days and save positions to its non-volatile storage.

Its not possible to guarantee at least one position fix a day.

Medad

29

A 2 byte delta at the minute level is 45 days - 12 bits is 2.8 days, 14 bits is 11.3 days …

Some sort of header with bit level flags to indicate which payload format / sequence is being used. At worst, you use bit flags to to indicate absolute or relative values.

Most of this just needs a really big envelope to sketch out a plan on. Plus beer. And snacks. And a kebab after.

30

That’s why I’ve been arguing for an exponential increase in the age of each measurement in a packet… Even if you want to use deltas and not mere position in the packet, an exponential scheme could still make sense - the staler it is, the less precision of the time really matters.