Payload Format / Decode for TM-901 / N1C2 sensor from KCS TraceME

End Devices (Nodes)
1

Hi all,

I am discovering TTN and IoT with sensors. At the moment I have my own sensor working and it sends my data (payload) well to the TTN console.

This sensor measures the position based on the magnetometer and returns the X, Y and Z coordinates.

Now I want to decipher my payload to “real” values. I do manage to see values of the Accelorometer and read the values of the compass.

But now I have to convert these values of the compass (X, Y, Z) to “something”. On the internet I have found that it is possible with “Math.atan2”. Only I try this I get this message:

Error ("json: unsupported value: NaN")

What I’ve done so far:

function Decoder(bytes, port) {       
            var accel_x = (bytes[16] << 8) * 2/32768 ;
            var accel_y = (bytes[17] << 8) * 2/32768 ;
            var accel_z = (bytes[18] << 8) * 2/32768 ;
            
            var compass_x = (bytes[19]) | bytes[20] ;
            var compass_y = (bytes[21]) | bytes[22] ;
            var compass_z = (bytes[23]) | bytes[24] ;
            
            var heading = Math.atan2(compass_y, compass_x) * 180 / Math.Pi;

            
            return {
              
              accel_x:accel_x
              accel_y:accel_y
              accel_z:accel_z

              compass_x:compass_x
              compass_y:compass_y
              compass_z:compass_z
              heading:heading

              
             
            }
        }

But when I return this “heading”, I get that error. The vendor of the sensor gave me this PHP example for decoding the compass.

$compassX = convertToReal(hexdec(substr($payload,38,4)));
$compassY = convertToReal(hexdec(substr($payload,42,4)));
$compassZ = convertToReal(hexdec(substr($payload,46,4)));

But after having these values in my code, I’ve to do more with this to decode these numbers to a “real” value.

Found this and this article and much more about converting the three magnetometer axis.

Does anyone have experience deciphering a magnetometer? Or someone who can push me in the right direction.

2

You’re not doing any shifting here. See Working with Bytes.

Also, without seeing payloads and expected values: are you sure compass_x and all could not be negative values? If they could be negative then you’ll need sign extension in JavaScript, to ensure it’s working with 32 bits values. See Decrypting messages for dummies - #4 by arjanvanb and assuming MSB, maybe use something like:

var compass_x = bytes[19]<<24>>16 | bytes[20];

Also, the above assumes that values are sent as integer numbers (like 1.2 is sent as 12), which (if true) might need to be divided by some factor to get decimal numbers.

As for the result for heading we can only guess, without seeing any actual values. Note that you can write debug messages to TTN Console while testing; see Payload Formats [HowTo] - #3 by arjanvanb

3

I read the post of the byte shifting, but don’t understand the byte shifting that much. Is the example of you valid to use? And can I use it also for the Y and Z compass values?

An example payload:
24 00 00 04 A7 00 00 00 00 00 00 00 00 00 00 87 CC DE 13 FF FF FD 09 00 07 01 6E 34 02

I assume that the values can be negative:

  • X: represents the magnetic field strength in roughly the direction of the north magnetic pole. A positive x-value means that part of the magnetic field is pointing north. A negative x-value means that part of the magnetic field is pointing south.
  • Y: represents the magnetic field strength 90 degrees from the x-direction in the “magnetic east” direction. A positive y-value means that part of the magnetic field is pointing towards magnetic east. A negative y-value means that part of the magnetic field is pointing towards magnetic west.
  • Z: represents the magnetic field strength in the local nadir direction (vertically down).

Hope you can help me in the right direction.

4

We cannot tell. What’s the source of your quoted explanation above? Where did you find the indexes such as bytes[19]? (For future readers, please also provide the device brand and type.)

If the quote is taken from documentation for your sensor, and if you’ve got the byte indexes correct, and if the values are sent MSB first, and if the values are sent as signed integer numbers, and if they don’t need to be divided by some factor to get decimal numbers, then: yes. But we cannot tell from the information you’re providing.

Looking at the example payload (for which we still don’t know the expected values either!), it seems that byte[19] is (hexadecimal) 0xFF, and so is byte[20]. (Beware that JavaScript starts counting from zero, not from one, just like many programming languages do as well.) That’s probably not a valid reading for compass_x?

It’s crucial you understand the shifting. Let’s decipher the next two bytes, those for compass_y: FD and 09, when displayed in the human readable hexadecimal notation. In binary notation, those are the bit sequences 11111101 and 00001001.

  • A left shift of 8 bits in 11111101 << 8 yields 11111101 00000000 (or, as it’s actually always 32 bits in JavaScript: 00000000 00000000 11111101 00000000).
  • Next, a bitwise OR in 11111101 00000000 | 00001001 yields 11111101 00001001.

Please follow the links above and read the documentation.

As we’re shifting 8 bits (which is a full byte) at a time, the above is much easier to read in hexadecimal:

  • For the byte FD, the shifting FD << 8 yields 00 00 FD 00.
  • Next, 00 00 FD 00 | 09 yields 00 00 FD 09.

When doing bitwise operations, JavaScript will interpret its operands as 32-bits signed integers, so will interpret the above 00 00 FD 09 as a large positive number, 64.777. Instead, if it can indeed be negative, you’ll need the sign extension that I linked to above. That would result in:

Note that the above applies to bitwise operators in JavaScript, as used in a TTN Console payload format. Other programming languages are unlikely to always assume 32-bits signed integers for bitwise operators.

Now, applying the above to byte[19] and byte[20] one would get 0xFFFFFFFF for compass_x, which is exactly -1. That might be correct, but I cannot tell. Finally, for compass_z you would get 0x00000007 (here, no sign propagating applies).

Finally, even with the above shifting and sign extension, Math.atan2(0xFFFFFD09, 0xFFFFFFFF) * 180 / Math.Pi will still give you NaN, Not-a-Number. So, JavaScript cannot do that math given these values. Maybe those are edge cases, maybe the formula is wrong, maybe you’re looking at the wrong bytes.

If those are expected edge cases, then you could use the following to suppress the NaN which TTN does not like:

return {
  ...,
  heading: isNaN(heading) ? null : heading,
  ...
}

That’s the ternary operator already mentioned in Decrypting messages for dummies - #4 by arjanvanb as well.

In your case, the formula is wrong. Debugging will show that Math.atan2(-759, -1) * 180 yields -282.98 just fine. However, Math.Pi is undefined, and -282.98 / undefined is NaN. You’ll need the uppercase Math.PI instead. (And you still might want to check all of the above.)

Bonus, to limit the number of decimals, use:

// Unary plus operator to cast string result of toFixed to number
heading: isNaN(heading) ? null : +heading.toFixed(1)
2 Likes
5

Thanks for the explanation @arjanvanb

The device is a TM-901 / N1C2 sensor from KCS TraceME, linke here

The indexes such as bytes[19] took that from the PHP code, this was the PHP code where they split the payload and send by the vendor to me:

$compassX = convertToReal(hexdec(substr($payload,38,4)));
$compassY = convertToReal(hexdec(substr($payload,42,4)));
$compassZ = convertToReal(hexdec(substr($payload,46,4)));

The expected value is for also not clear what’s expected. For the byte shifting I’ll have to read this multiple times to understand :slight_smile: .

I took a dive in the PHP code what was sent by the vendor, there were two files:
1.NC1x_data.php
2.lora_decode_example.php

In the NC1x_data.php were lines for the compass:

private function CorrectCompassXYZ(&$a) { $g = $a['compassX'];
if ($g >= 0x8000) $g -= 0x10000;
$a['compassX'] = $g * 0.92;
$g = $a['compassY'];
if ($g >= 0x8000) $g -= 0x10000;
$a['compassY'] = $g * 0.92;
$g = $a['compassZ'];
if ($g >= 0x8000) $g -= 0x10000;
$a['compassZ'] = $g * 0.92;
}

So it looks like I’ve multiple the values by 0.92.

And two parts that are not clear to me, looks like is only a check:

$a = unpack( "Cidentifier/ntimestamp/nts2/nlongitude/nlon2/nlatitude/nlat2/Cspeed/Cheading/CIO/caccelX/caccelY/caccelZ/vcompassX/vcompassY/vcompassZ/ntemp", $Data);
$this->CorrectTimestamp($a, $TimeStamp);
$this->CorrectLatLon($a);
$a['speed'] *= 1.852;
$this->CorrectAccelXYZ($a);
$this->CorrectCompassXYZ($a);
$this->CorrectTempCelsius($a);
$this->maProperties = $a;
return $this->mValid = true;

And the second part of the code:

$a = unpack( "Cidentifier/ntimestamp/nts2/nlongitude/nlon2/nlatitude/nlat2/Cspeed/Cheading/CIO/caccelX/caccelY/caccelZ/vcompassX/vcompassY/vcompassZ/ntemp/vbattery", $Data);
$this->CorrectTimestamp($a, $TimeStamp);
$this->CorrectLatLon($a);
$a['speed'] *= 1.852;
$this->CorrectAccelXYZ($a);
$this->CorrectCompassXYZ($a);
$this->CorrectTempCelsius($a);
$a['battery'] *= 0.006406;
$this->maProperties = $a;
return $this->mValid = true;

And when I look in the second file (lora_decode_example.php) there are this code blocks:

if ($cN1cx_data->CanGetProperty("compassX")) 
    {
        $Payload_Compassx = $cN1cx_data->GetProperty("compassX", null);
        $Payload_Compassy = $cN1cx_data->GetProperty("compassY", null);
        $Payload_Compassz = $cN1cx_data->GetProperty("compassZ", null);
    }

For better understanding, I’ve added these PHP files as TXT to this post, PHP was not possible. So you have to change the exentsion back to .php

N1Cx_data.txt (11.6 KB) lora_decode_example.txt (14.7 KB) .

6

Ah, I’ve seen you on Slack with a different name, where you posted the following two weeks ago. I’d use that as a starting point, rather than trying to reverse engineer some PHP code.

N1C2 record layout

7

Indeed, that was me! But the compass isn’t in that table, so I guess I’ve to do something with the PHP code, right?

8

Ah, right. And report a bug to the manufacturer or whoever created that table…

Yes.

As for temperature, as that only uses 12 bits, I’d assume something like:

// 12 bits, so to allow for negative values first shift 28 bits
// to the left to get the full 32 bits (discarding bits 7..5),
// followed by a 20 bits sign-propagating right shift
var temperature = (bytes[25]<<28>>20 | bytes[26]) * 0.0625;

That would need you to test with negative temperatures.

Aside: did you see the HEADING in the table?

9

The past few days I have been in contact with the supplier. Unfortunately, the “Heading” cannot be used for this use case. The “Heading” only goes together with the GPS when something is moving. So I have to focus on the accelerators and the compass.

The answer was:

You can use the acceleration sensor to stand upright, the x, y, z indicate whether the post is upright.
The direction relative to magnetic north can be read with the compass function.

What do you think is the best step to take now?

Using the temperature logic for the accelerators? And then translate the PHP code to Java? So the calculations etc.

// 12 bits, so to allow for negative values first shift 28 bits
// to the left to get the full 32 bits (discarding bits 7..5),
// followed by a 20 bits sign-propagating right shift
var temperature = (bytes[25]<<28>>20 | bytes[26]) * 0.0625;
10

Hi @arjanvanb,

I have an update in the right direction, I think. I first wrote a working java code. This gives the same result as the PHP code of the supplier.

Only in the TTN console can not this code completely.

package convertphp2java;

public class vendor {

    public static int convertToReal(int value) {

        if (value > 32767) {

            value = value - 65536;

        }

        return value;

    }

    public static void main(String[] args) {

        String payload = "2400002B9E0000000000000000000083EB003BFF67FE89FF8F01563B02";

        

        String x = payload.substring(38, 42);

        String y = payload.substring(42, 46);

        String z = payload.substring(46, 50);

        int compassX = convertToReal(Integer.parseInt(x, 16));

        int compassY = convertToReal(Integer.parseInt(y, 16));

        int compassZ = convertToReal(Integer.parseInt(z, 16));

        

        // bereken vector en hoek. x is vertical.

        // cartesian to polar: see https://www.mathsisfun.com/algebra/vectors.html

        // almost at bottom of page.

        // $r = sqrt(z*z+y*y)

        double angle;

        if (compassZ == 0) {

            angle = 90;

        } else {

            angle = Math.atan((double)compassY / compassZ) * 180 / Math.PI;

            if (compassY < 0) {

                angle = angle + 180;

            } else if (compassZ < 0) {

                angle = angle + 360;

            }

        }

        int noCompassData = 0;

        if (compassX == -1 && compassY == 255 && compassZ == 0) {

            noCompassData = 1;

            System.out.println("No Compass Data in payload:");

        } else {

            System.out.println("angle: " + angle);

        }

    }
}

Only when I want to use this code do I notice that I cannot use pieces yet.

  • The substring, but I solved that with this:

      var compassX = bytes[19] << 8 | bytes[20];
  var compassY = bytes[21] << 8 | bytes[22];
  var compassZ = bytes[23] << 8 | bytes[24];

  • The piece with converttoreal I still have to apply. From the entire java code it does not work this way. I have written it like this, but I think this can be done more efficiently, is that correct?

if (compassX > 32767) {
compassX = compassX - 65536;
}
//return compassX;

if (compassY > 32767) {
	compassY = compassY - 65536;
}
//return compassY;

if (compassZ > 32767) {
	compassZ = compassZ - 65536;
}
  • How do I declare a double? Now only with a var X it seems.

If I now compare my PHP code with my Java code, there is a difference of 6 degrees.

My Java code till now (with a difference of 6 degrees)

function Decoder(bytes, port) {
	// Decode an uplink message from a buffer


	// (array) of bytes to an object of fields.
	var compassX = bytes[19] << 8 | bytes[20];
	var compassY = bytes[21] << 8 | bytes[22];
	var compassZ = bytes[23] << 8 | bytes[24];

	if (compassX > 32767) {
		compassX = compassX - 65536;
	}
	//return compassX;

	if (compassY > 32767) {
		compassY = compassY - 65536;
	}
	//return compassY;

	if (compassZ > 32767) {
		compassZ = compassZ - 65536;
	}
	//return compassZ;

	var compass_x = (compassX / 10) * 0.92;
	var compass_y = (compassY / 10) * 0.92;
	var compass_z = (compassZ / 10) * 0.92;

	var angle = 0.0;

	if (compassZ == 0) {
		angle = 90;
	} else {
		angle = Math.atan(compassY / compassZ) * 180 / Math.PI;
		if (compassY < 0) {
			angle = angle + 180;
		} else if (compassZ < 0) {
			angle = angle + 360;
		}
	}

	var noCompassData = 0;

	if (compassX == -1 && compassY == 255 && compassZ == 0) {
		noCompassData = 1;

		//LogMessage('No Compass Data in payload: '. "");
	} else {
		noCompassData
		//LogMessage('angle: '.$angle);
	}


	return {
		angle: angle
		compassX: compassX
		compassY: compassY
		compassZ: compassZ
	};
}

Hope you can help me a bit more in the right direction :smiley:

11

No response yet? I’d really urge the manufacturer or vendor to fix their documentation, rather than trying to reverse engineer their PHP code.

The 6 degrees probably apply to a specific payload; it might be much worse for other sensor readings. Especially if readings can be negative.

Also, where are you using the following results?

See, e.g., Here is what you need to know about JavaScript’s Number type. And remember:

1 Like
12

I contacted them and they said they only have PHP classes. And that these compass values could be read in this way. Hence I received this specific PHP code from them.

Ah, that was a typo in my code. But get the same results with 6 degrees difference.

Thanks for both links.

What would be convenient steps for me to get closer to my end result?

This is now my code:

function Decoder(bytes, port) {
	// Decode an uplink message from a buffer

	var accel_x = (bytes[16] << 8) * 2 / 32768;
	var accel_y = (bytes[17] << 8) * 2 / 32768;
	var accel_z = (bytes[18] << 8) * 2 / 32768;

	// (array) of bytes to an object of fields.
	var compassX = bytes[19] << 8 | bytes[20];
	var compassY = bytes[21] << 8 | bytes[22];
	var compassZ = bytes[23] << 8 | bytes[24];

	if (compassX > 32767) {
		compassX = compassX - 65536;
	}
	//return compassX;

	if (compassY > 32767) {
		compassY = compassY - 65536;
	}
	//return compassY;

	if (compassZ > 32767) {
		compassZ = compassZ - 65536;
	}
	//return compassZ;

	var compass_x = (compassX / 10) * 0.92;
	var compass_y = (compassY / 10) * 0.92;
	var compass_z = (compassZ / 10) * 0.92;

	var angle = 0.0;

	if (compass_z == 0) {
		angle = 90;
	} else {
		angle = Math.atan(compass_y / compass_z) * 180 / Math.PI;
		if (compass_y < 0) {
			angle = angle + 180;
		} else if (compass_z < 0) {
			angle = angle + 360;
		}
	}

	var noCompassData = 0;

	if (compass_x == -1 && compass_y == 255 && compass_z == 0) {
		noCompassData = 1;

	} else {
		noCompassData
	}


	return {
		angle: angle
	compassX: compass_x
	compassY: compass_y
	compassZ: compass_z
};}