Best practices when sending GPS location data [HowTo]

Application Development
1

I’m trying to combine some information found in other posts, as a reference for a future FAQ or Wiki page.

It seems:

  1. Moving nodes should switch off Adaptive Data Rate (or do they?), and always send with the slowest rate (SF12) (or do they? as fixed SF11 or SF12 is not allowed!), and hence are bound to a maximum application payload length of 51 bytes?

  2. On SF12 a maximum frame of 51 bytes will take almost 2.5 seconds to transmit?

  3. On SF12, when all 51 bytes are needed, so when sending for 2.5 seconds, then TTN’s fair access policy of 30 seconds a day would only allow for (on average) sending one location every two hours. But even sending just 6 bytes for two 24 bit coordinates would need 1.3 seconds air time and one could, on average, send fewer than one GPS coordinate per hour on The Things Network. But also an empty “alive” packet would probably still need the 13 bytes LoRaWAN header, and still take 1.16 seconds air time on SF12.

  4. If a GPS unit gives you decimal degrees, then its actual precision might only be 4 decimals, even if it gives you some 32 bit value. How to measure the accuracy of latitude and longitude? claims:

    • The sign tells us whether we are north or south, east or west on the globe.
    • […]
    • The tens digit gives a position to about 1,000 kilometers. It gives us useful information about what continent or ocean we are on.
    • The units digit (one decimal degree) gives a position up to 111 kilometers (60 nautical miles, about 69 miles). It can tell us roughly what large state or country we are in.
    • The first decimal place is worth up to 11.1 km: it can distinguish the position of one large city from a neighboring large city.
    • The second decimal place is worth up to 1.1 km: it can separate one village from the next.
    • The third decimal place is worth up to 110 m: it can identify a large agricultural field or institutional campus.
    • The fourth decimal place is worth up to 11 m: it can identify a parcel of land. It is comparable to the typical accuracy of an uncorrected GPS unit with no interference.
    • The fifth decimal place is worth up to 1.1 m: it distinguish trees from each other. Accuracy to this level with commercial GPS units can only be achieved with differential correction.
    • The sixth decimal place is worth up to 0.11 m: […] This can be achieved by taking painstaking measures with GPS, such as differentially corrected GPS.
    • […]

  5. When only sending a few 24 bit numbers, using Google’s [Protocol Buffers encoding]((Encoding | Protocol Buffers Documentation) does not seem to decrease the data size (but it does require more computing power).

So, I guess best practices are about limiting the packet size, and to not require more than a few coordinates a day:

  • Like with all use cases: never send JSON, or not even ASCII, but encode your data into pure bytes.

    • In decimal degrees, a longitude with 4 decimals, -180.0000…+180.0000 might need 9 bytes when sending as plain characters (or 8 when leaving out the decimal dot), and probably another byte for some separator. But it also nicely fits in 3 bytes (like -8,388,608 to 8,388,607 as a 24 bit signed integer if you first multiply by 10,000). When one needs more decimals, using 4 bytes for a standard 32 bit float, or multiplying by 100,000 and sending as a standard 32 bit signed long, will give more than 7 decimals.

    • With a custom binary encoding one is not limited to an exact multiple of 8 bits.

  • Do not send altitude for nodes that are not airborne. (Like for nodes following roads, your application should be able to determine the sea-level altitude given its coordinates, unless you’re afraid someone might take it inside a tall building. But then you might not get a GPS fix to start with.)

  • Include position error information if needed. (Good thinking, darren-oc.)

  • Do not send a precision that is higher than your use case needs, or the GPS offers.

    • For a regular GPS module, 24 bits might be good enough to match its actual accuracy, so just 6 bytes are needed to send both latitude and longitude. (As an alternative calculation: with the equatorial circumference of the earth being 40,075,017 meters, the accuracy of a 24 bit integer would be 40,075,017 / 2^24, about 2.39 meters.)

    • For some use cases, like a pet finder, one can assume the target is within 50 km of some known place, like the known location of a mobile phone. An application can then figure out the missing details if the node only sends the four decimals, without any sign. Those 4 decimals would fit in the 2 bytes of an unsigned integer (0…65,535) (see discussion below) so only 4 bytes are needed to determine both latitude and longitude. Alternatively, a node could send the full coordinates only, say, every 10th time.

    • If the location of the gateway is known (true for The Things Network) and given the limited range of gateways and nodes then the above even applies to most, if not all, use cases.

  • If on the lowest data rate sending 6 bytes for two 24 bit coordinates takes 1.3 seconds air time and hence is limited to only 23 coordinates per day: be smart about when to send the coordinates.

    • If it’s about finding lost things (rather than continuously tracking something), assuming a LoRaWAN Class A device (where a device can receive some data right after sending some), and assuming the node will not get out of range: make the node send a small “alive” packet quite often, and only if the application responds in some specific way, power the GPS (and some light and a buzzer?) and start sending coordinates more frequently (it seems even “alive” packets are very limited). This saves both bandwidth and battery.

Any other thoughts, or errors in my assumptions?

11 Likes
3

I totally agree with this.

I’m sure we can safely assume the node is in a 40km radius of the gateway that received the packet. If we choose a maximum+minimum latitude at which the system should work, we can calculate the maximum number of degrees 80km will fall in. Only transmitting the least significant part of the coordinates as the most significant part can be deducted from the location of the gateway.

2 Likes
4

Good thinking, @jpmeijers, I didn’t even consider the known locations of gateways (added now), but was only thinking about the known location of the (mobile) application showing the position.

I was wrong about the sign. When using decimal degrees it is needed, for use around the equator and the prime meridian (the “zero” values for latitude and longitude). The 50 km I mentioned was simply the 111 km “weight” of 10 times 11.1 km for the first decimal, divided by two to ensure one can figure out the sign. But that won’t work. And maybe one also needs to divide by two as the range is a radius.

Also, I just read that most GPS devices will output strings defined by the National Marine Electronics Association protocol (NMEA), which gives one latitude and longitude in degrees, minutes and decimal minutes.

So: maybe one can still come up with some 2 byte value that can be enhanced by the application. But it’s not as simple as just sending the 4 decimals of some decimal degrees value, and the range will be smaller than the 50 km I first assumed.

(To be continued.)

5

Today I was thinking of a binary coordinate “compression” system I was looking at last year. The method I used was to see the earth as a rectangle divided into a grid. Longitude being the index of the location on the x-axis, and latitude being the index on the y-axis. With this in mind I recreated this grid to consist of 2^(bits) squares. This is a naive system as it provides a high resolution at the poles, but a low resolution at the equator, just like normal coordinates. Optimising the system to have a uniform “low” resolution only saved me one bit. The mathematics became a lot more complicated, so I decided to stick with the naive approach.

I initially looked at using 48 bits, giving a resolution of 0.00002° (±2.2m at equator). Adjusting the formulas to use 24 bits gave a resolution of 0.06° (±6.6km at equator). 24 bits is therefore not enough to store longitude and latitude, but 48 bits seems decent.

I’m linking a spreadsheet containing the formulas and a few test calculations for my system. I wanted to type out the equations here, but it will be unreadable without formatting.

Play around with the number of bits and the formulas, and please let me know if there is a better way to do it.

1 Like
6

Without much calculations, it seems two 24 bit values would also fit longitude and latitude, when limiting to 4 decimals in decimal degrees. That would only give one an accuracy of 11 meters, but in my above quote someone claims (emphasis mine):

  • The fourth decimal place is worth up to 11 m: it can identify a parcel of land. It is comparable to the typical accuracy of an uncorrected GPS unit with no interference.
  • The fifth decimal place is worth up to 1.1 m: it distinguish trees from each other. Accuracy to this level with commercial GPS units can only be achieved with differential correction.

Are you using a more precise GPS, or do GPS units actually give more accurate results than assumed above? (Or is the above quote wrong? Wikipedia gives the same numbers, but without any claims.)

(Note that I am no expert; just trying to help documenting this.)

7

Another simple idea:
First message (after reboot of device or after reconnect to gateway after a break) contains complete Lat Lon location following one of the scheme above.
Following messages contains offsets only. For a 40KM range that is about 4bytes (2 for lat +2 for long).
Practically, no vehicle (and device) can travel more than 15KM in 10 min at 90kmph. This is much less in urban area due to traffic.

Venkat.

4 Likes
8

As requested on slack I will repeat my statement about SF12 here as well.

Moving nodes should switch off Adaptive Data Rate (or do they?), and always send with the slowest rate (SF12), and hence are bound to a maximum application payload length of 51 bytes2?

As you are developing a product for a moving object, it might not be a good option to use SF12. It is true SF12 can cover more distance but this has to come from somewhere. The thing with SF12 is, every symbol is coded with 4096 bits (also called chips with spread spectrum modulation). This results in a coding gain which can be received under the noise level, thus enabling greater distances to be covered. However this causes the package and airtime to be extremely long. Even with a few bytes it will be very sensitive for interference and as a moving object, it is likely to come across interfering objects.

SF11 would already be much better because the airtime is almost half of SF12 thus less chance for package loss due interference. However ideally SF7 would be used because this has the shortest air time. In addition SF7 uses less energy because of the shorter air time and of course you are able to send packages more frequently.

I actually still have to measure the difference in distance between SF7 and SF12. I am planning to do this within the next two weeks or so. I will share my findings here. If SF12 truly generates a significantly larger distance I would suggest that a moving node would first try to use SF7 and if that fails, it would then try a higher SF.

4 Likes
9

I have done some range testing by driving a node around in a car. Most transmissions using SF12 were not received while in motion. Transmissions while stationary (parked) were received reliably. I haven’t found the time to drive the same circuit using SF7 yet.
Transmissions with SF12 while walking were no problem.

3 Likes
10

I’m sorry, but I have to interject and say that is a genius idea, I was about to post another idea but that probably beats it hands down. I’ll put it here anyway.

In the UK they have an alternative geo system called the National Grid, it was used quite a lot in Government. It features 2 sets of numbers known as Eastings and Northings,
This grid sets the bottom left hand corner of the British Isles (a rock off the Scilly Isles) as 0,0. Go 10 metres to the East and 50 metres north and you are at 10,50. And so on till you get to the top of Scotland.

You could create a similar grid for a 40km area and map lat/lng to East/North. This then assumes your node is smart enough to do the donkey work itself, and you take into account the battery use this will cause.

Which brings me onto another question, seeing as I’m in “application development” anyway.

If we stick with the “pet lost locally” idea, what if you do not want to hear anything UNTIL the pet is lost, if the uplink message is just an hourly checkin (ping) will we be able to send back to the node a signal which can then be interpreted as “OK, wake the GPS up, now I want to know your exact location”. I mean, have I understood how LoRa in TTN will operate? (I have no kit, awaiting my kickstarter kit).

Thanks

11

Yes; after each uplink you can send data back to the node. And when only sending a ping once per hour and only responding when the pet is lost, that seems reasonable.

But a pet can run quite far in an hour, so you’d have to do some calculations about how often you can send such ping, or implement some geofencing to only send pings when not at home or when not close to the pet owner’s Bluetooth-enabled mobile phone. Also beware that both the uplink and the downlink might not be received (especially for a moving node), so the application might need to repeat the downlink if it still doesn’t get location data after it sent a downlink earlier. So one ping per hour might really be too few for this use case.

As an aside: for technical reasons, using downlinks will really need to be limited. The TTN Fair Access Policy might only allow 10 confirmed uplinks and downlinks per day (and other operators should impose such limits as well; LoRaWAN is just not suited for downlinks very well). But I guess pets don’t get lost that often, so most pings would not need a response.

See also Is anyone already working on the PetFinder use case?

12

…but beware that atmospheric circumstances can yield unexpected long ranges!

See the part about “Ducting”, most often happening at night and possibly yielding ranges of 200 km+, in @nmap’s very, very nice TTN webcast, Crash Course in Radio Theory.

13

Just for future reference, here is the c code for packing the GPS data into 48 bits.

uint8_t coords[6]; // 6*8 bits = 48
int32_t lat = float_latitude * 10000;
int32_t lon = float_longitude * 10000;

// Pad 2 int32_t to 6 8uint_t, skipping the last byte (x >> 24)
coords[0] = lat;
coords[1] = lat >> 8;
coords[2] = lat >> 16;

coords[3] = lon;
coords[4] = lon >> 8;
coords[5] = lon >> 16;

LMIC_setTxData2(1, coords, sizeof(coords), 0);

And here is some example PHP-code to decode it back into floats:

// Unpack into 6 bytes      
$data = unpack('C*', $payload);

// Combine each 3 bytes into a 24bit int      
$lat = ($data[1] + ($data[2] << 8) + ($data[3] << 16)) / 10000;      
$lon = ($data[4] + ($data[5] << 8) + ($data[6] << 16)) / 10000;      
var_dump( $lat, $lon );
4 Likes
14

…or to decode using TTN’s payload functions, to get an additional human readable fields in the MQTT payload:

See below for an improved version to support negative decimal coordinates.

// Version 2 (production) needs a function name; see post below for a fix
function (bytes) {
  // Does NOT handle negative values; see post below for a fix
  var lat = (bytes[0] + (bytes[1] << 8) + (bytes[2] << 16)) / 10000;
  var lng = (bytes[3] + (bytes[4] << 8) + (bytes[5] << 16)) / 10000;
  return {
    location: {
      lat: lat,
      lng: lng
    },
    love: "TTN payload functions"
  };
}

…to return something like the following in the MQTT message:

{
  "app_id": "my-app-id",
  "dev_id": "my-node",
  "hardware_serial": "42CF4601045CE277",
  "port": 1,
  "counter": 19,
  "payload_raw": "ANX6nBIX",
  "payload_fields": {
    "location": {
      "lat": -33.8688,
      "lng": 151.2092
    },
    "love": "TTN payload functions"
  },
  "metadata": {
    "time": "2017-01-16T22:59:20.373495512Z",
    "frequency": 868.1,
    "modulation": "LORA",
    "data_rate": "SF8BW125",
    "coding_rate": "4/5",
    "gateways": [
      {
        "gtw_id": "eui-5ccf7ff42f1970ec",
        "timestamp": 494534657,
        "time": "2017-01-16T22:59:20.296723Z",
        "channel": 0,
        "rssi": -46,
        "snr": 11,
        "latitude": 52.373141,
        "longitude": 4.892435,
        "altitude": 3
      }
    ]
  }
}
2 Likes
15

Even better! :slight_smile:

16

I think it is a very interesting approach, but it won’t work with negative integers… so forget about any coordinate to the west of Greenwich or to the South of the Ecuador… any suggestion to fix it?

1 Like
17

same problem

18

You could add 200 and then subtract them on the server/TTN side… that way you would always treat with positive numbers!

2 Likes
19

Great, we seem to have come to a good solution to transfer coördinates. Is there any to automatically get this data on a coverage map? Say on pade.nl/lora. Is there an application that we can assign our trackers to?

20

Check TTNmapper.org

21

When sending 3 bytes, we’re basically not sending the 4th byte of each integer, which should be 0xFF for negative numbers. Like for New York (40.712784, -74.005941, which would be sent as integers 407127 and -740059) we would send:

When decoding these 6 bytes back into two 32 bits signed integers, we need to compute the missing 4th and 8th bytes ourselves, to make JavaScript properly convert the negative values for us. Those bytes should be 0xFF if the most significant bytes have their “high bit” set, which is called “sign extending”:

// LSB, Least Significant Bit/Byte first
// Sign-extend the 3rd and 6th bytes into a 4th and 8th byte:
lat = (b[0] | b[1]<<8 | b[2]<<16 | (b[2] & 0x80 ? 0xFF<<24 : 0)) / 10000;
lng = (b[3] | b[4]<<8 | b[5]<<16 | (b[5] & 0x80 ? 0xFF<<24 : 0)) / 10000;

// MSB, Most Significant Bit/Byte first
// Sign-extend the 1st and 4th bytes into leading bytes:
lat = ((b[0] & 0x80 ? 0xFF<<24 : 0) | b[0]<<16 | b[1]<<8 | b[2]) / 10000;
lng = ((b[3] & 0x80 ? 0xFF<<24 : 0) | b[3]<<16 | b[4]<<8 | b[5]) / 10000;

Alternatively, shift the most significant byte 8 bits too far to the left, and then shift it back, which will do the sign extension on the fly, as the bitwise operator >> is the sign-propagating right shift:

lat = (b[0]<<24>>8 | b[1]<<8 | b[2]) / 10000;
lng = (b[3]<<24>>8 | b[4]<<8 | b[5]) / 10000;

Meanwhile, for the new production environment, payload functions should also include the function name, Decoder. So, to support 6 byte coordinates with possible negative values:

function Decoder(b, port) {

  // Amsterdam: 52.3731, 4.8924 = MSB 07FDD3 00BF1C, LSB D3FD07 1CBF00
  // La Paz: -16.4896, -68.1192 = MSB FD7BE0 F59B18, LSB E07BFD 189BF5
  // New York: 40.7127, -74.0059 = MSB 063657 F4B525, LSB 573606 25B5F4
  // Sidney: -33.8688, 151.2092 = MSB FAD500 17129C, LSB 00D5FA 9C1217

  // LSB, Least Significant Bit/Byte first! Your node likely sends MSB instead.

  // Sign-extend the 3rd and 6th bytes into a 4th and 8th byte:
  var lat = (b[0] | b[1]<<8 | b[2]<<16 | (b[2] & 0x80 ? 0xFF<<24 : 0)) / 10000;
  var lng = (b[3] | b[4]<<8 | b[5]<<16 | (b[5] & 0x80 ? 0xFF<<24 : 0)) / 10000;

  return {
    location: {
      lat: lat,
      lng: lng
    },
    love: "TTN payload functions"
  };
}

Alternatively, as coordinates in decimal degrees are -90…+90 for latitude, and -180…+180 for longitude: one could add 90 to the latitude and 180 to the longitude before sending, then send the positive values, and reverse that in the payload function.

And life can be made easy using libraries such as https://github.com/thesolarnomad/lora-serialization (though that one does not support 3 byte coordinates).

6 Likes