Disable Frame Counter Checks in V3

TTN Network and Services The Things Stack v3
#59

Sure! apologies if it seemed like a request! It was not my intention at all! I found this post detailing the same problem I have encountered and would like to help so it can be reviewed if possible. Happy to help sharing any details that could lead to detect what’s happing :slight_smile:

Thanks a lot! I´ll stay tuned for any advance.

PS: Yeah! weekends should definitely be a time to ‘disconnect’ :sweat_smile:, in my case it is when I can invest some of my spare time to look into this :smile: Thanks for your fast feedback. Really appreciated.

#60

So I used one of my test ABP setups in v2 and took that for a spin as a base line using my on-desk Adafruit Feather M0 with RFM95 using LMIC 4.0.0 which says it is LW 1.0.3 compliant (no reason to doubt it).

Then created the same ABP device but using a slightly different AppSKey in v3 using the CLI (to save on typing in all the frequencies) but without setting Resets Frame Counters to Enabled.

Uplinks were ignored if I reset the device until it reached the last max uplink count.

When I ticked the Enabled checkbox, uplinks were received regardless.

I created a new ABP setup using the CLI but set the Resets Frame Counter and re-flashed the Feather - it received all uplinks regardless of Frame Counter.

ESP deep sleep generally means that RAM is lost, therefore state is lost. But that shouldn’t preclude the reset counters option.

Which LMIC are you using?

My TTGO cannot connect with ABP to V3
#61

Just confirming from my side is still an issue.

My setup is as follows:

  • Server Stack: TTS V3.13.2 (our own stack, but TTN shows the same effect)
  • IDE/Code Environment: Arduino IDE
  • Library: MCCI LoRaWAN LMIC
  • LoRaWAN MAC: V1.0.3
  • Module: Murata CMWX1ZZABZ-091
  • Region: AU915

Me and my teammates are developing a system that is just a proof of concept.
It has been ongoing for 6 months now, and was starting to come together really well.

It is just a proof of concept and we can’t prioritise super mature firmware.

Yes, I understand for proper implementation we should be storing the Frame Counter in NVM (e.g. EEPROM), recalling it when the device wakes up, and adding that into the frame.
Or better yet - ditching ABP altogether and using OTAA, storing the join state variables, and recalling those…

But that’s not an option for us, due to our limited firmware resources, fixation on the LMIC library, and timeline.

Our system worked perfectly, seamlessly, and flawlessly on TTN V2 with the “Frame Counter Checks” disabled.

The new “Resets Frame Counters” button (whatever that means) does not do anything it seems.

Also, why is there no button for resetting the frame counts?
I can’t even find one on V3.
You would think there would be something here?

whynot

So, when prototyping/developing/testing, after our device sends 1 packet (and fcnt=1), we basically need to delete the device and recreate it again, if we want to see more data and test, since our device only transmits one time per hour.

Any help would be appreciated - I am not sure how we proceed without delaying our project 1-2 months to bring on extra firmware resources and implementing this, for such a basic proof of concept device we just want to get in the field…:frowning:

#62

The mind boggles as to why you are using MCCI LMIC on a Murata module.

Or indeed why you aren’t using OTAA which works very well with LMIC and is almost the flip of a switch.

Or why you can’t use the CLI to manage the counter reset for a PoC.

Or why you got to a point of deleting devices AFTER sending ONE uplink without stopping for a sanity check.

Apart from that, it’s still working very well for me, see my post above …

1 Like
#64

Sure, no problem, happy to answer your questions/mind boggles!

  1. I am using MCCI LMIC because that is what most support, examples, and use cases are based on for beginners getting started with LoRaWAN, and because it has full support for the STM32 B-L072Z-LRWAN1 Discovery Kit (which is what the Murata Module is used on…)

  2. I have tested OTAA several times with MCCI LMIC and cannot get it to work in AU915. This is because the Join Accept message is not being received, so my product keeps sending Join Requests over and over. I’ve even done some spectrum analyser grabs, and confirm that the gateway sends the Join Accept message just fine down to the RF level. But still no luck.

Untitled

  1. We are looking into the CLI to manage this, sure, but still seems like a poor and impractical solution.

  2. No, I have done many sanity checks. I am just saying, as I am testing our sensors and systems and new prototypes, in order to check a few packets without waiting for ages - since I can’t use the reset frame counters button - my only option to reset the counters is to literally delete and recreate the device again (yes, I know the CLI may work, but again this will take some time for us to investigate and explore).

I am sure in the past many people have used the “reset frame counters” button dozens of times on the old TTN v2, when doing experiments with ABP and just getting started out with this.
And furthermore, many teams, developers, and hobbyist experimenting with low power modes/shutdowns have unticked the “Frame Counter Checks” box to test out their designs before implementing a non-volatile memory system and diving into the details of the LoRaWAN MAC.

We are a small team still learning the fundamentals of LoRaWAN and this is causing us a huge roadblock that others wouldn’t have faced 2-3 years ago :confused: I do not think this “needs” to be fixed, I understand why this is a feature. I am just voicing my troubles and our issues, that is all.

#65

I’d consider my self pretty clued up on MCCI LMIC and this would be the first time someone has used it on the LRWAN1 which has a nice certified module with the ‘official’ LoRa-node code base. Both code bases have their foibles so YMMV and given the relatively clear separation between sensor/device/app code and the LoRaWAN code in both, porting back should be feasible.

This would appear to be the crux of the matter. MCCI are moving towards certification so it not working for a region seems unlikely. Have you set it for sub-band 1? Any commentary on the MCCI LMIC GitHub regarding AU915 that may give you any clues?

I’m more firmware & use the expertise on here for RF, but I’m not sure how you’d get two blips on a scope like that as one device is transmitting and then another one is transmitting and it’s not clear from the divisions what the time gap is. Normally people post pictures of an SDI which shows us the actual chirp. If anyone else is going to comment, perhaps some more info would be useful.

I heartily agree - but it is a quick win as an alternative to delete & add. But as I say, I can add a device and not have the counter get in the way. Here’s the CLI that I use for LMIC:

ttn-lw-cli dev create **Your-App-ID** **Desired-Device-ID**
  --dev-eui **An-EUI**
  --frequency-plan-id EU_863_870_TTN
  --lorawan-version 1.0.3
  --lorawan-phy-version 1.0.3-a
  --abp  --session.dev-addr **260B-Not-Sure-Find-One**
  --session.keys.app-s-key.key **Random-Key**
  --session.keys.f_nwk_s_int_key.key **Another-Random-Key**
  --mac-settings.factory-preset-frequencies 867100000,867300000,867500000,867700000,867900000,868100000,868300000,868500000
  --mac-settings.rx1-delay RX_DELAY_5 
  --mac-settings.supports-32-bit-f-cnt 
  --mac-settings.resets-f-cnt

Obviously you’ll need to alter this for the AU915 band. You can grab a random EUI and some keys from here: Random EUI or Key generator. I’m still trying to elicit a response from TTI about the generation of valid DevAddr on the CLI.

I mean’t here - a totally different set of eye balls.

Compared to delete & re-add??? Not really that much investigating or exploring to do:

https://www.thethingsindustries.com/docs/getting-started/cli/

ttn-lw-cli end-devices reset [application-id] [device-id] --session.last-f-cnt-up

And by searching the forum: Uplink messages not arriving anymore - #12 by descartes

But for the very quickest win, there is this:

which is created by a forum member so you can use LMIC and ask questions, the second bit being the most important as you appear to have dug yourself a deep deep hole before asking for help.

#66

How far apart are your node & GW?

I assume your RF grab is a seperate receiver smewhere in close proximity to both node & GW?

Can you give more details on your set up please,

Trying to guestimate timing for the bursts and seperations - how have you configured RX1 & RX2 window timing?

1 Like
#67

For the purposes of fixing the Join Accept reception, yes, very good point, doh!

But we could do with bottoming the ABP issue too as ABP is less painful for testing and doesn’t end up with a post regarding JoinNonce’s and other magic.

#68

Also which GW are you using? EUI/MAC ID looks like Laird and that is part of description but model called out as a 1702 (Match-X) - both node and gw set for correct local sub bands under AU915?. How connected to Inet? (Backhaul method?). Have you tested against any of Maj’s local Meshed gw’s?

#69

Tests carried out (I use MCCI LMIC library):

  • If I activate the deep sleep function of the ESP8266, the message is only received the first time.

  • Without deep sleep function, the messages are received perfectly after the time set by the TX_INTERVAL parameter (Resets Frame Counters Enabled :white_check_mark:)

The process is as follows:

  1. The first message is handled by the gw as Unconfirmed Data Up and another message as Unconfirmed Data Down.
  2. The next message the gateway handles it again with a couple of messages and restarts the FCnt.
  3. It continues to send the messages perfectly.

From the tests, I conclude that the problem is in the Deep Sleep function…and as you states, something related with RAM and some value that’s neeeded to ‘save’.

So finally, I change the code of my nodes to OTAA to get them working with Deep Sleep, which now I can confirm are working as expected.

BUT… I would like to know, the root cuase why my previous code using ABP (and deep sleep) was working on V2, but on V3 it doesn’t as commented. I’m too curious and need to understand it :sweat_smile: :smile:

Thanks a lot for your tests.

#70

For OTAA make sure to save the LMIC state in deep sleep mode as you have only 64k possible join nonces. If you join each time you wake you will run out of them (and when using random ones you will encounter used nonces fairly soon), you will have a join and send data cycle every time you uplink burning battery like crazy (two transmissions where one should be used), and the gateway(s) near the node will be transmitting far to often.

#71

As Jac says above, you need to save things because deep sleep on ESP isn’t deep sleep which carries the implication of dreaming aka retention of the past.

“Deep sleep” is “Turn off with a sub-system still running to turn the majority back on when a certain event arrives” but that doesn’t look so great on the marketing materials. Nor would the side note that “turn the majority back on” actually means reboot / reset / start from the beginning.

Which means, as per Jac above, you have your work cut out for you saving all the right things and then reinstating them. Something I’m not aware of there being a canonical implementation unless @bluejedi knows of one. But the bits are there to do so if you care to have a try.

#72

Both ESP8266 and ESP32 have RTC memory. Its contents are retained during deep sleep. Without adding additional hardware RTC memory is the preferred place to temporarily store LMIC state for deep sleep.
Another option is to store state in flash memory using the EEPROM emulation library but flash has a relatively limited number of write cycles so not suitable for storing frequently changing values.

Unfortunately there currently is no support in the MCCI LMIC library itself for a uniform method for storing and restoring LMIC state to/from non-volatile memory (NVM).
It is on the wish list for some future version but currently not yet implemented.

For ESP32 there is a fork of MCCI LMIC by @blem named ttn-esp32 which even supports storing/restoring LMIC state to/from NVM (I assume RTC memory will also be supported). Unfortunately it is written for the ESP-IDF framework and does not support the Arduino framework like MCCI LMIC does.

ttn-esp32 appears to be not recently maintained so will not include the latest changes of MCCI LMIC.

[Update]

ttn-esp32 was recently updated:

New in version 4.0 (prerelease)

  • Verified compatibility with ESP-IDF v4.3
  • Upgraded underlying library mcci-catena/arduino-lmic to v4.0.1 (improved channel shuffling)
  • C API
  • Support for sub-bands
2 Likes
#73

Hi everybody.

I think I’ve found the problem, or maybe the misunderstanding, I must say.

In V2, the option “Frame Counter Checks” allowed the node not to use a frame sequence. Therefore, the node could send always the same frame number without problem.

The code almost everybody uses to learn to program the ESP32 uses deep_sleep calls to schedule the transmissions. The deep_sleep destroys RAM, so the ESP32 uses always the same frame number, but thanks to the option “Frame Counter Checks” everything runs good.

In V3, the option “Resets Frame Counters” allows the device to start the frame sequence from a number lower than the last frame counter it used. But, and that’s the key, THE DEVICE CAN’T USE ALWAYS THE SAME FRAME NUMBER.

In other words, you can start your frame sequence where you want, but there must be a sequence.

So, in order to use ABP in V3 with the option “Resets Frame Counter” you must store frame counter in RTC RAM after a TX to keep it during deep_sleep, and recover it just before perform a new TX.

Other possible modification is to use light_sleep instead of deep_sleep, because light sleep doesn’t destroy RAM and the ESP continues execution at the same point it went to sleep.

1 Like
New ESP ABP device do not get to online
#74

Not sure what to you mean with “The code almost everybody uses to learn to program the ESP32” here.

First, people learning to program the ESP32 don’t start with deep sleep nor start with scheduling any transmissions but start with the basics. One of the (more advanced) basic topics is understanding the implications of the ESP32’s sleep modes.

Second, the same holds for learning to program LoRaWAN. You don’t directly start using deep sleep (which has all kinds of implications on ESP32) but start with the basics.
The preferred choice “almost everyone uses” is the MCCI LoraWAN LMIC library.
The preferred examples to start with are ttn-abp.ino and ttn-otaa.ino and since recently there is a better example called LMIC-node (external, not included with the LMIC library).

Only when you have mastered these ESP32 and LoRaWAN basics should you continue with more advanced topics like using deep sleep with LoRaWAN (and/or use an ESP32 specific LoRaWAN library which may not be recently maintained).

So referring to:

That is just not the case.

#75

I didn’t use the right words, sure.

What I wanted to say is that the ESP code for Lora boards that many people (beginners) use is derived from examples found on libraries, and that code uses deep_sleep to schedule transmissions.

With the migration to v3, we found that the code used in v2 doesn’t work, and the reason is that the v3 option “Resets Frame Counter” doesn’t has the same effect that the v2 option “Frame Counter Checks”.

While in v2 frame number could be repeated, in v3 they must follow a sequence. And that’s the reason why the example code fails.

Thanks.

1 Like
#76

@ jmarino
100% agree in V2 the frame counter could stay at 0 or whatever now it needs to increment.
This is an issue for devices that do not have eeprom or nvram like the SAMD21 that utilize deep sleep or restarts.
The move to V3 has stopped all these devices from operating (for me anyway)
In a perfect world getting V3 to operate as V2 did would be a fantastic solution. (for the frame counter issue of course)

Regards

#77

most of ESP32 RAM , but not the 8KB so called ‘RTC-RAM’ (cf @bluejedi )

Jack Gruber uses that ‘nvram’ here (tested OK with MCCI Catena LMIC V4 library)
https://jackgruber.github.io/2020-04-13-ESP32-DeepSleep-and-LoraWAN-OTAA-join/

No problem to store Frame Counter …and more … during ESP32’ deep sleep with a recent LIMC library

#78

Yes, RTC RAM is the solution, by now.

But I think v3 is the perfect reason to study other non-volatile alternatives, just as F-RAM, to start using OTAA the right way.

#79

If developing with the Arduino framework and using FRAM with I2C interface you may be interested in the following library:

The library is not yet fully documented and currently lacks examples but it is functional, easy to use and supports a wide range of different I2C FRAM chips and sizes.