NVM context management

[Adamzat]

Hi all,

In OTAA mode, only DevNonce and the JoinNonce are needed to be stored in EEPROM after the Join success. The LoraWan Stack (4.5.1), reads and writes “All data” very frequently (about each Uplink). All data takes about 1452 bytes.
Our products use STM32L1 and the maximum number of cycling (erase / write) of the EEPROM is about 300Kcycle. This number can be easily reached (300K frames, for not powered devices).

Is there a possibility to enhance this feature (saving only needed data + reducing the frequency)?

And regarding the ABP mode, is there a possibility to reduce the frequency of storing all data (for example each 100 or 1000 uplinks) ? What could happen if we lose the context in ABP (reset, etc.) ?

Thank you for your reply.

[mluis1]

This subject has already been discussed on several Issues/Discussions.

• Restoring NVM contexts require kilobytes of stack and is unnecessarily complicated  #882
• NvmCtxMgmtStore L073 #947
• Persistent context storage management #959
• NvmMangement in the LoraWan Stack #1120
• minimization of persisted data possible? -> battery lifetime optimization #1138

In conclusion, we have already analyzed and tried to reduce the amount of data to be stored on the NVM memory.
The current amount is what we believe to be the minimum.

Nowadays, the MAC layer notifies the application layer to only store what changed since last notification.
In the provided example we decided to store each time the MAC layer notifies the application layer. However, nothing prevents you to not do so (not recommended).

The full amount is only written one time (First boot or when the NVM data is reset). Then after every uplink around 80 bytes (EU868) are stored. Depending on MAC commands sent by the Network Server the amount can be bigger.
The biggest amount of data being stored is related to the secure-element soft emulation (keys, nonce, etc...). The secure-element data does not change very often. In case an hardware secure-element is used the amount of stored data will be greatly reduced.

One of the reasons we decided to update the NVM after each uplink is to be able to power off the end-device and just continue the normal operation once powered up (without joining the network again).

In the provided example we use the MCU internal FLASH memory because it is the only non volatile memory that is available on this project supported platforms examples.

There are a few possibilities to increase the NVM life time

• Implement a wear leveling algorithm to improve the internal FLASH memory life time.
• Use an external I2C/SPI non volatile memory (i.e.: FRAM, EEPROM, etc...). Implementing a wear leveling algorithm for this kind of memories can also be useful.

In the future it would be nice if you could post this kind of questions on the project Discussions tab. It is a better place to engage discussions and then we can agree if it is an issue or not.
If you don't mind I move this issue to the Discussions tab.

[bwesen]

Sorry to hijack the thread but I haven't been able to get a clear answer to this anywhere else or in any doc. If I use OTAA, can I simply skip all the NVM persisting? Our device does a JOIN every time it powers up or resets anyway. In my Makefile I set CONTEXT_MANAGEMENT_ENABLED=0 and define SOFT_SE (to emulate a secure element in RAM) for the latest 4.5.2 release of the stack. This does seem to work.. but after a re-JOIN, it seems to take a few minutes before traffic is 100% trustworthy again..

So was wondering if the re-JOIN (after a complete reset and no NVM) can typically cause some trouble at the Lora gateway side that maybe confuses it for a while? I'm using a Multitech Conduit.

[Pirakutu]

It can indeed, as you would start joining with the start count again every time you reset (this is just an asumption), which would lead to your network server avoiding that join request because of a possible replay attack. For it to work under 1.0.2/1.0.3 (as 1.0.4 should check on the integrity of the devNonce, if it is increasing and so on) you would need to randomize the devNonce again (as done in previous protocol versions). Like that you would have it running correctly if im not mistaken.

[bwesen]

Very interesting, yes it makes sense. Indeed if the gateway keeps track of more than the last one used for the device, the problem could even accumulate over time so more and more JOIN requests might have to be done by the device until it reaches a number the gateway hasn't used yet (but I kind of doubt the GW actually tries to keep track of all nonces ever used, this would expose the GW to a horrible storage situation in case of misbehaving devices).

Would be cool if there was simply a #define or something to just make everything work like 1.0.3 even in the latest v4.5.2 release. I'm trying to track down a parallell problem introduced by 4.5.2, my device doesn't reply to Class C downlinks from the GW even though I'm setting it as Class C, and was worried this also has something to do with 1.0.4 (or it's "just" a regression...).

BTW how does the GW figure out if the device is supposed to be treated like a 1.0.4 device? Is this info sent along in the JOIN req?

[Pirakutu]

The gateway doesnt save this info, the network server does. And yes, the more join reqs you send the longer it will take, at least if all works as described.

In earlier versions of the stack there was a define where you could decide how the devNonces are created, i dont know if they took this out on 4.5.2 but you should check there, maybe you can carry out the change pretty fast. (just step in the code where the join req is made, at some point you will see something regarding dev nonces and so on)

I mean when you register your device you tell the network server which protocol your device will be running, so I am guessing that is the first kind of filter in this direction. I know of different joining formats for 1.1.x and 1.0.x protocols, now between 1.0.x, no idea, i guess you would have to check the protocols right quick to get more specific info on this.

[bwesen]

Hm yes the terminology of Lora is not my strong side, in our case we just have the Multitech Conduit running a "lora network server" daemon and that's it (then we send/receive packets through it using MQTT from a custom program).

So we don't register any devices individually here, the lora-network-server is setup with a single network key that all the OTAA devices share, so any device that has the key can join the server.

I have realized this is not the 100% recommended way subsequently but we started with this project more than 5 years ago and I just inherited code which works perfectly in the field using a very old Lora stack, up until the 4.5.2 port I'm now trying to do which broke Class C and the NVM stuff :).

The Multitech lora-network-server is not open-source, otherwise I could simply check in that what it does. But as far as I can see in the manual pages, there is no per-device setting for Lora versions there.

But thanks for the brainstorm, I'll definitely check the randomized devNonce on the device side again as a fallback in the "no-NVM" scenario.

[dmeehan1968]

Working on STM32L082CZ based device, which I implemented off the B-L072Z-LRWAN-1 example implementation, I found that the EepromMcuWriteBuffer implementation (located in src/boards/B-L072Z-LRWAN-1/eeprom-board.c) was simplistic and could be further optimised. The principle here might well apply to other board implementations.

Findings:

• Uses single byte writes, looping over the supplied buffer.
• STM32 supports word (4 bytes) writes and its likely (though not explicitly documented) that maximum write cycles apply on a word basis. Writing all four bytes of the same word as implemented in LoRaMac is likely to cause increased wear on the NVM

I found it was fairly trivial to optimise the implementation of EepromMcuWriteBuffer to deal with any non-word aligned bytes supplied either by the source buffer or the target address as single byte writes, but dealing with the rest as word writes. It's likely that the single byte writes would be unnecessary as compilation options would likely word align structure members, but dealing with the edge case was trivial.

Additionally, whilst the STM32L0 optimises the write process to avoid erasing when unnecessary, I found that a simple conditional that checks to see if each word value differs before writing saved about 30% time.

Finally, there appeared to be a glitch in that the STM32L0 would report a Write Protection Error (FLASH_FLAG_WPRERR) prior to writing word values. I found reports of this on the ST Micro forums but without clear explanation of the cause. This didn't seem to be the case when all writes were single byte. To resolve this, I added a call to clear the WPRERR before each write process.

Here are the timings I recorded:

Operation	Simple Write Time (ms)	Comparative Write Time (ms)
Factory Reset	1360	932
Join	580	401
Uplink	125	98

LmnStatus_t EepromMcuWriteBuffer( uint16_t addr, uint8_t *buffer, uint16_t size )
{
    LmnStatus_t status = LMN_STATUS_ERROR;

    assert_param( ( DATA_EEPROM_BASE + addr ) >= DATA_EEPROM_BASE );
    assert_param( buffer != NULL );
    assert_param( size < ( DATA_EEPROM_BANK2_END - DATA_EEPROM_BASE ) );

    __HAL_FLASH_CLEAR_FLAG(FLASH_FLAG_WRPERR);

    if( HAL_FLASHEx_DATAEEPROM_Unlock( ) == HAL_OK )
    {
      CRITICAL_SECTION_BEGIN( );
      size_t offset = 0;
      bool ok = true;
      while (ok && offset < size && ((addr + offset) % sizeof(uint32_t))) {
        ok = HAL_FLASHEx_DATAEEPROM_Program( FLASH_TYPEPROGRAMDATA_BYTE,
                                            ( DATA_EEPROM_BASE + addr + offset ),
                                            buffer[offset] ) == HAL_OK;

        offset += ok ? 1 : 0;
      }
      while(ok && offset < size && (offset % sizeof(uint32_t) == 0)) {
        uint32_t *ptr = (uint32_t *)&buffer[offset];
        if (*ptr != *((uint32_t *)DATA_EEPROM_BASE + addr + offset)) {
          ok = HAL_FLASHEx_DATAEEPROM_Program( FLASH_TYPEPROGRAMDATA_WORD,
                                               ( DATA_EEPROM_BASE + addr + offset ),
                                               *ptr ) == HAL_OK;
        }
        offset += ok ? sizeof(uint32_t) : 0;
      }
      while (ok && offset < size) {
        ok = HAL_FLASHEx_DATAEEPROM_Program( FLASH_TYPEPROGRAMDATA_BYTE,
                                             ( DATA_EEPROM_BASE + addr + offset ),
                                             buffer[offset] ) == HAL_OK;

        offset += ok ? 1 : 0;
      }
      CRITICAL_SECTION_END( );
      status = ok ? LMN_STATUS_OK : LMN_STATUS_ERROR;
    }

    HAL_FLASHEx_DATAEEPROM_Lock( );
    return status;
}

Whether this actually saves any write wear or not is not clear. It certainly saves some processor time which equates to a power saving, important on my battery powered devices. If your device is battery powered, preserves RAM and doesn't reset, then you may not in fact need NVM persistence.

Putting this here as a potential improvement, and would be happy to make a PR if appropriate. The linked issues/discussion suggest that this type of optimisation might be beyond the scope of the project.

[mluis1]

@dmeehan1968 Thanks for the analysis as well as for the proposed improvement.

May I ask if this is your original code or inspired by someone else work. I ask this due to potential copyright or license issues.

[dmeehan1968]

My own work.

[mluis1]

Thanks.

As the proposition applies to more than one target platform we will implement the enhancement.
Once done we will set you as the author of the commit.
Please note that it may take some time to get this done.

[dmeehan1968]

I spotted a mistake in my code, in the middle while loop where it makes a comparison against the existing NVM content, it was adding the address and offset to a uint32 pointer, referencing an address further into NVM than desired. The line should read:

        if (*ptr != *((uint32_t *)(DATA_EEPROM_BASE + addr + offset))) {

[dmeehan1968]

I'd appreciate some clarification on when persistence is actually required.

The application is working to LoRaWAN 1.0.4 (monotonic DevNonce), OTAA, I'm currently testing head of master in anticipation of a 4.7.0 release.

My application is on a battery powered device. Power consumption is such that the battery is likely to last 5+ years and defines the lifetime of the device. Under normal circumstances the device will not be power cycled or reset.

We do have activated/deactivated states controlled by connection of the sensor, and the device only joins the network once activated. Normally, it would never be deactivated, left installed in the same place for its lifetime. If it is moved, sensor removal deactivates and sensor insertion will activate and instigate a new join (as the device might be moved or other network factors changed). The device remains in a low power state when deactivated and therefore DevNonce is stored in RAM.

As such, my understanding is that there is no need for the entire session to be preserved to NVM, as its preserved in RAM. Given that it would be possible to reset the device due to someone interfering with the battery, or the application doing a watchdog reset, its necessary to preserve the DevNonce in NVM. This rare occurrence seems to avoid the risk of excessive join attempts.

It seems therefore that solely the DevNonce can be written (with appropriate CRC validation) to NVM, given that the device will join anew in the event of a reset/power cycle. This will also drastically reduce the number of potential NVM writes, as well as the amount of data written.

The application checks network connectivity periodically (via LinkCheckReq) and in the event of prolonged loss of connectivity will go into a periodic join state that respects the principles of retransmission backoff. This is the only case where one can expect sustained DevNonce updates, once for each join attempt, approximately 1-2 time per hour until connectivity is restored (power loss at the gateway, or network backhaul failure, being the most likely causes).

I appreciate that the provided application examples and the NvmDataMgmt module, cover the lowest common denominator of devices that don't have RAM preservation during low power modes.

It seems that this should be documented clearly, or alternatively that there be a build option provided to either:

1. Adapt the NVM data management to store only DevNonce (useful by example but easy to miss)
2. Adapt the NVM change notifications to only cover DevNonce changes (a documented interface of the API)

Regardless of a change to the API, confirmation/critique of my approach would be appreciated.

[bwesen]

Supporting the EEPROM storage was completely ruled out for us, because that locked up the MCU for seconds every packet sent. And our product can easily reboot or lose power any time, so a strict adherence (never repeating a join nonce) would not be possible.

So we ended up simply disabling everything, there is a compile-time option to ignore the NVM stuff and emulate "NVM" in RAM. We had to enable the option to randomize the Join nonces though or the gateway would complain and delay joining a bit (the MultiTech gateways we use seem to remember the last used nonce only though so as long as you increment or change it randomly every join attempt it would work).

For us who have completely self-contained systems (gateway and nodes) this was an OK compromise and we could just keep going like "in the good old days". I'm sure there are good reasons why the NVM stuff was added in the LoraWAN spec otherwise but if you're not on a WAN...