Registering a device using OTAA

tomsimmons · October 17, 2023, 6:19pm

Certainly, the code I’ve used can be found here…

GereZoltan/LoRaWAN: Possibly complete LoRaWAN implementation in micropython with SX1262 (github.com)

As you can see, it’s well used, no activity in 2 years and I’m the only person that has commented. As I say I have found nothing else regarding using this module to connect to LoRaWAN written in MicroPython. WaveShare do provide a C version, but the rest of the project is written in MicroPython.

The OTAA function can be found in the LoRaWANHandler.

Good to know that the spec is/should be the same, I was going to ask whether everyone has the same OTAA request because as I say there appears to be an extra blank byte on the front of this string. When I pick up again later I’m going to take the string and stick it in the decoder mentioned above.

kersing · October 17, 2023, 8:04pm

Not should, is. Both ChirpStack and The Things Stack implement a ratified standard.

And as a result there shouldn’t be any changes to the code required to switch from one to the other. It would be interesting to see if the code still works with ChirpStack, perhaps a change in MicroPython causes issues…

tomsimmons · October 19, 2023, 11:49am

So using the decoder mentioned above, it does seem to need a preceding blank byte, which looks to be called MHDR?

Also, the DevEUI and AppEUI do need to be reversed when building the join string to get a valid decode.

Which leaves me now needing to figure out when OTAA isn’t working

kersing · October 19, 2023, 12:42pm

Would it be helpful if you take a look at the LoRaWAN specification?

On page 15 you will find the definition of MHDR, The Join request is specified on page 34.
(Page 70 has a glossary should you encounter fields not mentioned like RFU)

Given the definition of MHDR I very much doubt is should be set to 0 for a join request.

tomsimmons · October 19, 2023, 3:17pm

Looks like it should be 1C to join?

Wasn’t aware of this, I’d been looking at this…

End Device Activation | The Things Network

kersing · October 19, 2023, 4:26pm

As you already found that page does not list the full LoRaWAN packet. It’s better to always check the formal specification as well (if available, which for LoRaWAN is the case).

Don’t know how you came to that value, but MHDR is
Bit 7…5 4…2 1…0
MType RFU Major

For a join request message type is 000. RFU is always all zeros. And Major is (strangely) 00 for LoRaWAN V1.

So I was wrong in my previous message and 00 seems to be the correct value after all (another reason to always check the specification)

tomsimmons · October 19, 2023, 6:05pm

I got the M Type as 000

My take on

was that RFU was 0111 and Major was 00

Giving
0000111000

Which I must then have typed into calculator incorrectly, one 0 short on the right, giving 1C, should have been 38.

But you say it should be 00

tomsimmons · October 19, 2023, 6:28pm

From the decoder, it looks like 00 is correct

And the rest of the values look correct, so I’m left wondering whether:

there is something wrong in the code
something wrong in the transmitter
something wrong with the gateway

As I don’t have anything else to transmit with on 868, I’m struggling at the moment.

Can anyone recommend a super cheap 868 off the shelf node? A temperature sensor or similar that I could use to rule out the gateway? Cheapest I’ve seen is best part of £30 on Amazon, maybe that is as cheap as I’ll get?

kersing · October 19, 2023, 7:17pm

Can you check the gateway console at TTN to check the details of the received join request? The console should show the EUIs and the frequency.
(EUIs are public information transmitted unencrypted so don’t bother blacking them out. The AppKey is secret and won’t be part of the request)

Please post the details.

For an off the shelf product, search LDS02. Over here one of the cheapest reliably working sensors.

tomsimmons · October 19, 2023, 7:55pm

The gateway console on TTN ticks away every minute, very occasionally a spurious entry will appear in there, no idea from where because it happens even when my node isn’t powered up.

I never see a any form of message detected when I try the node.

Even the LDS02 is £30 when delivered over here!

Seriously thinking of ditching the Pico and this module and using an Arduino or something, any thoughts or guidance, need to be able to connect some sensors to it for a weather station. Did ponder a T-Beam, but they aren’t so cheap, but some have integrated charging which could be useful.

kersing · October 19, 2023, 8:40pm

It helps if you show a screen shot so we can check what happens.

Then either the node is not transmitting or not using the right frequencies.

Don’t get an Arduino UNO or alike module with the 8 bit chips. They lack the memory required for a full LoRaWAN stack and sensors.

If you check there might be some old stock lopy (from pycom) available if you want to use python with LoRaWAN.
If you are up for embedded C programming there is a multitude of options. Make sure to choose on based on a controller with sufficient memory.
The easiest solution might be to hook a Grove E5 module to your Pico and use the uart with ascii commands. The other boards from Seeed with the same chipset can be programmed with the
Arduino IDE and STMduino and it’s LoRaWAN stack.

tomsimmons · October 19, 2023, 8:54pm

The gateway feed…

I suspect the node either simply doesn’t transmit or what it does transmit simply isn’t valid so the gateway ignores it?

I was using MicroPython because I wrote all the code for the weather sensor components on the Pico in that.

Grove E5 and the Pico is an option, or as you say something in the more powerful Arduino range, I’ve not looked at the options, but I’d guess there must be something with the LoRa onboard.

Ichthus_College_Info · October 19, 2023, 9:27pm

Not related to the actual question at hand but I really cannot recommend Pycom modules. That ship has sunken too far and is at Titanic level now. And bankrupt. I have 60 LoPy4 in deployment and that is no good experience. -100/10
I mean… there’s communication. I receive uplinks. And that’s about it. Not mentioning a lot of other points about their whole uPy branch.

kersing · October 19, 2023, 9:48pm

Gateways only ignore LoRaWAN packets that fail the CRC check. Everythings else is forwarded to the network server.

Those are at this time still the best python powered solution I’ve come across. However there are far better solutions when you can skip python.

tomsimmons · October 26, 2023, 7:59pm

Received my Grove E5, will take a look for some examples of using Micropython with Pico to connect to TTN using OTAA and transfer data.

tomsimmons · November 10, 2023, 11:26pm

Got the E5 working in short order, then once again work and the likes got in the way.

Back to it tonight and have again sent the test outlined in this page…

Raspberry Pi Pico with LoRaWAN and The Things Network - Makerverse LoRa-E5 Module - Transmit Weather Data into IoT - Tutorial Australia (core-electronics.com.au)

Which should contain 2 payloads (if you search for “Payload 1” you find it). Looking in TTN Gateway Console I can see 3 uplinks, each followed by a downlink - which makes sense. What I can’t see in any of the uplinks is a payload that in anyway matches the 2 payloads I’m meant to be seeing. Sadly the screen shots aren’t clear, but it looks like they are actually looking under the application, but I can’t see them there either.

The following is the Gateway live data…

I’m assuming I should find the payload in the detail panel when clicking on the 3 “receive uplink message” lines, but I don’t - unless it has become encoded into the raw in what looks like Base64

1st

{
  "name": "gs.up.receive",
  "time": "2023-11-10T22:28:25.971716942Z",
  "identifiers": [
    {
      "gateway_ids": {
        "gateway_id": "mintlaw-gateway-01",
        "eui": "B827EBFFFE563AC0"
      }
    }
  ],
  "data": {
    "@type": "type.googleapis.com/ttn.lorawan.v3.GatewayUplinkMessage",
    "message": {
      "raw_payload": "AAYAAAAAAACAXr0wMiDx9yyQu8dpgJU=",
      "payload": {
        "m_hdr": {},
        "mic": "x2mAlQ==",
        "join_request_payload": {
          "join_eui": "8000000000000006",
          "dev_eui": "2CF7F1203230BD5E",
          "dev_nonce": "BB90"
        }
      },
      "settings": {
        "data_rate": {
          "lora": {
            "bandwidth": 125000,
            "spreading_factor": 12,
            "coding_rate": "4/5"
          }
        },
        "frequency": "868500000",
        "timestamp": 3497045146,
        "time": "2023-11-10T22:28:25.960534095Z"
      },
      "rx_metadata": [
        {
          "gateway_ids": {
            "gateway_id": "mintlaw-gateway-01",
            "eui": "B827EBFFFE563AC0"
          },
          "time": "2023-11-10T22:28:25.960534095Z",
          "timestamp": 3497045146,
          "rssi": -78,
          "channel_rssi": -78,
          "snr": 7.75,
          "location": {
            "latitude": 57.52503521045234,
            "longitude": -2.004565000534058,
            "source": "SOURCE_REGISTRY"
          },
          "uplink_token": "CiAKHgoSbWludGxhdy1nYXRld2F5LTAxEgi4J+v//lY6wBCa2cKDDRoMCInduqoGEMW8os8DIJDziMHj/CI=",
          "received_at": "2023-11-10T22:28:25.953815107Z"
        }
      ],
      "received_at": "2023-11-10T22:28:25.971546181Z",
      "correlation_ids": [
        "gs:uplink:01HEXNKJQKE4KHB28SBYF08GH9"
      ]
    },
    "band_id": "EU_863_870"
  },
  "correlation_ids": [
    "gs:uplink:01HEXNKJQKE4KHB28SBYF08GH9"
  ],
  "origin": "ip-10-100-7-161.eu-west-1.compute.internal",
  "context": {
    "tenant-id": "CgN0dG4="
  },
  "visibility": {
    "rights": [
      "RIGHT_GATEWAY_TRAFFIC_READ"
    ]
  },
  "unique_id": "01HEXNKJQKAM2X6CAXW7JDP3DW"
}

2nd

{
  "name": "gs.up.receive",
  "time": "2023-11-10T22:28:36.468274375Z",
  "identifiers": [
    {
      "gateway_ids": {
        "gateway_id": "mintlaw-gateway-01",
        "eui": "B827EBFFFE563AC0"
      }
    }
  ],
  "data": {
    "@type": "type.googleapis.com/ttn.lorawan.v3.GatewayUplinkMessage",
    "message": {
      "raw_payload": "QNEnCyaAAAAI4gNpEF4GaFcC6O1sFz7ALg==",
      "payload": {
        "m_hdr": {
          "m_type": "UNCONFIRMED_UP"
        },
        "mic": "Fz7ALg==",
        "mac_payload": {
          "f_hdr": {
            "dev_addr": "260B27D1",
            "f_ctrl": {
              "adr": true
            }
          },
          "f_port": 8,
          "frm_payload": "4gNpEF4GaFcC6O1s"
        }
      },
      "settings": {
        "data_rate": {
          "lora": {
            "bandwidth": 125000,
            "spreading_factor": 12,
            "coding_rate": "4/5"
          }
        },
        "frequency": "868500000",
        "timestamp": 3507541181,
        "time": "2023-11-10T22:28:36.457109928Z"
      },
      "rx_metadata": [
        {
          "gateway_ids": {
            "gateway_id": "mintlaw-gateway-01",
            "eui": "B827EBFFFE563AC0"
          },
          "time": "2023-11-10T22:28:36.457109928Z",
          "timestamp": 3507541181,
          "rssi": -77,
          "channel_rssi": -77,
          "snr": 9.25,
          "location": {
            "latitude": 57.52503521045234,
            "longitude": -2.004565000534058,
            "source": "SOURCE_REGISTRY"
          },
          "uplink_token": "CiAKHgoSbWludGxhdy1nYXRld2F5LTAxEgi4J+v//lY6wBC9qcOIDRoMCJTduqoGEJ2um98BIMiE/M2K/SI=",
          "received_at": "2023-11-10T22:28:36.422155853Z"
        }
      ],
      "received_at": "2023-11-10T22:28:36.468113181Z",
      "correlation_ids": [
        "gs:uplink:01HEXNKWZMNV3QNVS0R2G8KY5Z"
      ]
    },
    "band_id": "EU_863_870"
  },
  "correlation_ids": [
    "gs:uplink:01HEXNKWZMNV3QNVS0R2G8KY5Z"
  ],
  "origin": "ip-10-100-7-161.eu-west-1.compute.internal",
  "context": {
    "tenant-id": "CgN0dG4="
  },
  "visibility": {
    "rights": [
      "RIGHT_GATEWAY_TRAFFIC_READ"
    ]
  },
  "unique_id": "01HEXNKWZM9HCHYMGS1H5D4S3B"
}

3rd

{
  "name": "gs.up.receive",
  "time": "2023-11-10T22:28:44.594079793Z",
  "identifiers": [
    {
      "gateway_ids": {
        "gateway_id": "mintlaw-gateway-01",
        "eui": "B827EBFFFE563AC0"
      }
    }
  ],
  "data": {
    "@type": "type.googleapis.com/ttn.lorawan.v3.GatewayUplinkMessage",
    "message": {
      "raw_payload": "QNEnCyaFAQADBwb/CgiM1Sr8v1svvT0l0DRatRp4KTNCVQ==",
      "payload": {
        "m_hdr": {
          "m_type": "UNCONFIRMED_UP"
        },
        "mic": "KTNCVQ==",
        "mac_payload": {
          "f_hdr": {
            "dev_addr": "260B27D1",
            "f_ctrl": {
              "adr": true
            },
            "f_cnt": 1,
            "f_opts": "AwcG/wo="
          },
          "f_port": 8,
          "frm_payload": "jNUq/L9bL709JdA0WrUaeA=="
        }
      },
      "settings": {
        "data_rate": {
          "lora": {
            "bandwidth": 125000,
            "spreading_factor": 8,
            "coding_rate": "4/5"
          }
        },
        "frequency": "867300000",
        "timestamp": 3515690416,
        "time": "2023-11-10T22:28:44.582701921Z"
      },
      "rx_metadata": [
        {
          "gateway_ids": {
            "gateway_id": "mintlaw-gateway-01",
            "eui": "B827EBFFFE563AC0"
          },
          "time": "2023-11-10T22:28:44.582701921Z",
          "timestamp": 3515690416,
          "rssi": -77,
          "channel_rssi": -77,
          "snr": 16.5,
          "location": {
            "latitude": 57.52503521045234,
            "longitude": -2.004565000534058,
            "source": "SOURCE_REGISTRY"
          },
          "uplink_token": "CiAKHgoSbWludGxhdy1nYXRld2F5LTAxEgi4J+v//lY6wBCw27SMDRoMCJzduqoGEN6mmpsCIIDv6fuo/SI=",
          "received_at": "2023-11-10T22:28:44.561212519Z"
        }
      ],
      "received_at": "2023-11-10T22:28:44.593924958Z",
      "correlation_ids": [
        "gs:uplink:01HEXNM4XJ0DNEEJYRB3RVP0AT"
      ]
    },
    "band_id": "EU_863_870"
  },
  "correlation_ids": [
    "gs:uplink:01HEXNM4XJ0DNEEJYRB3RVP0AT"
  ],
  "origin": "ip-10-100-7-161.eu-west-1.compute.internal",
  "context": {
    "tenant-id": "CgN0dG4="
  },
  "visibility": {
    "rights": [
      "RIGHT_GATEWAY_TRAFFIC_READ"
    ]
  },
  "unique_id": "01HEXNM4XJCMW6JJB6PJPHQH6R"
}

descartes · November 11, 2023, 12:07am

I see you’ve popped your uplink cherry.

So only four fundamental issues:

According to that tutorial your data travelled all the way to Amsterdam and back again. Now that tutorial is from an Australian company. So I suspect that it was bathed in Fosters and then had a puff of the weed on the way through AMS and all the signals got scrambled.

The other three issues are the blocks of text that you took a screen shot of - please please please post text as text using the </> tool. Console screen shots are fair game.

The fifth issue is that even whilst under the influence, the Fosters/Weed tutorial shows the payloads being visible in the application/device console. Not the gateway console which shows traffic through the gateway which is, at that point, encrypted and, big difference, not encoded. Encoding comes later. The gateway will show all uplinks for TTN that it hears, some/many/all may not be for you. This is how LoRaWAN works.

Sixth issue is that the difference between a join and an uplink is a crucial fundamental. Scroll to the top of the page, click Learn in the menu and read the LoRaWAN material - it’s the minimum knowledge required. So now you are actually up & running with some hardware, any question you ask that is answered in that section will likely result in you being directed to it.

Last and ultimately critical issue is that the script does two naughty things, one is illegal. We don’t send text, except for our first uplink, then we never do it again, it’s hugely inefficient. But most importantly, we don’t breach the Fair Use Policy by sending uplinks so fast. You can read the details in the Learn section. Sending too fast / too often can also involve a car with an orange stripe, a court appearance and fines as there are legal limits on how often you can send. The Learn section will fill you in.

As for the mixed bag that the Tutorial of Oz brings to the party, lots of screen shots that can’t be expanded to see properly but with abbreviated descriptions lead to this exact situation. Maybe it’s them that tried a tug on a spliff as having chosen a data centre in Australia it seems a very odd thing to send the packet via Amsterdam. In fact no packets go via Amsterdam. All the EU1 packets go via Dublin and the EU2 packets via London (because Brexit, some uplinks only have a new style UK passport (that was printed in France)).

Pretty much every tutorial on the interwebs for LoRaWAN has something left-field or downright wrong about it - or is written by someone how doesn’t know what details other people don’t know. So always best to read them with a pinch of salt, read more than one to compare & contrast, cross-reference what they say with what the Learn + Docs sections say. And ask informed questions here. But not if it’s in the Learn section …

tomsimmons · November 11, 2023, 10:01am

Images duly replaced with text

I whole heartedly agree that there dirth of good examples/tutorials on how to do most of this stuff, hence I guess so many people, like me, end up following not so good examples, making mistakes and infringing guidelines/laws.

From your reply I understand that the gateway console isn’t the place to look as the data is still encrypted at this point, which is why it looks “at a glance” to Base 64 I guess? I did mention in my post that it looks like they are looking at the application, however, I can’t see anything here either. From Overview > See all activity I can see the join, then 2 uplink messages

However, clicking either of the messages I get

{
  "name": "as.up.data.forward",
  "time": "2023-11-10T22:28:44.813062499Z",
  "identifiers": [
    {
      "device_ids": {
        "device_id": "eui-2cf7f1203230bd5e",
        "application_ids": {
          "application_id": "test-pen-01"
        }
      }
    }
  ],
  "context": {
    "tenant-id": "CgN0dG4="
  },
  "visibility": {
    "rights": [
      "RIGHT_APPLICATION_TRAFFIC_READ"
    ]
  },
  "unique_id": "01HEXNM54D71VW6YMD18WWEX7P"
}

which obviously doesn’t have a payload, going via the device yields the same message.

kersing · November 11, 2023, 2:29pm

That is the encoding. However the data is AES encrypted as well.

You need to have the application console open when the message arrives. Opening it afterwards will only display a minimal amount of meta data.

descartes · November 11, 2023, 4:12pm

To reinforce, as this is a common gotcha, the LNS/console is not a data store, so you have to have the live data console for the device or application open at the time the uplink occurs. Otherwise you just get a summary of recent activity.

Your next step is to capture the uplinks so you can do something with them once you have byte array encoded data being sent no more than once every 5 to 10 minutes or so

As no doubt you’ll have read all the Learn section, here’s the Working with Bytes link:

https://www.thethingsnetwork.org/docs/devices/bytes/

It’s OK to ignore the pink box, all the info on that page is of a general nature but super useful for efficient transfers.

As set of starter files to capture your uplinks you can rely on because the TTN tutor for integrations wrote it and you can ask him questions (after trying things) here:

The WebHook puts the least load on the servers and as it’s usually on external webspace just keeps on trucking, unlike anything at home that gets kiboshed 37.89 seconds after someone cranks up Call of Duty on the home router.

Or you can pass it on to any number of the web dashboard integrations. I wouldn’t bother asking for a recommendations, prices vary wildly from free to lots, best get a beer and spend another evening just looking at the tutorials & docs & see what speaks to you - for all of them they are the official docs for that integration (wot with it being on that actual companies website) so should be fairly reliable & in sync with TTN.

There is also the Storage Integration which saves the last 24 hours of uplinks with mildly less meta info - a backup plan but not something to use as a primary source as it’s not realtime.