Zephyr: Alternatives for OpenThread settings

[rlubos]

As we can see that current OpenThread settings storage mechanism, which writes directly to flash memory, is too simple and not felixble enough for certain use-cases (#5943, #12944), we need a more robust replacement. I've spent some time to analyze what options we have and wanted to share my thoughs and hear from others.

OpenThread settings API

Just before describing the actual possibilities, a few words on how OpenThread settings are stored. OpenThread defines an API for settings : https://github.com/openthread/openthread/blob/0b18d8ffad6411ecaf1d818ec8f3f9dce245350e/include/openthread/platform/settings.h.

An individual OpenThread setting consists of a triplet of key, index and value. A key identifies a setting type, a value is self explainatory. So far so good. But there is also a setting index - OpenThread allows to store multiple entries of a setting of the same type (key). The index is not persistant - settings of a single type can be reorderd during write/delete operations. The only requirement is that there are no gaps in index value - they need to be indexed from 0 to setting_count - 1.

Practically most of the OpenThread settings store only one entry with index 0. An exception is information about attached children - here we can have mutliple children stored in the persistent storage.

Possible Zephyr implementations

1. Zephyr settings - an obvious choice would be to use Zephys's settings module. There are some problems though, that in my opinion, make this module unfit for this use case:

• Zephyr settings and OpenThread settings API differ in a design. It took me a while to understand how do the settings in Zephyr work, but now I can see the differences. OpenThread needs a simple read/write/delete mechanism and takes care for applying settings stored/read on it's own. On the other hand, Zephyr settings work in a different way - the settings are readout in a bulk, and then setting-specific callbacks are called to apply them in the respective modules. There even is no API to read a single setting value - the only way to read settings from flash is that bulk settings_load. We could try to bypass this by having a copy of the settings in RAM, but I doubt we want to do this (I don't).
• There is no out of the box support for indexing, but that's not a big issue - the keys are strings, so they could be easily extended. A good idea on how to keep track of the settings stored would be needed though.

1. NVS - this module allows to store entries with a key, value tuple. The module itself seems pretty ok, though I find it difficult to support settings indexing with this approach. We could either:

• @Laczen proposed a solution he used in NVS settings backend, see https://github.com/zephyrproject-rtos/zephyr/issues/13941#issuecomment-468914998 for details
• Store every instance of particualar setting under a single key, and index them manualy during readout. The downside of this approach is, that if we (let's stick to the children example) have 10 children stored, and want do modify/delete a single entry, we'd need to read and store an entire children array. This will have negative impact on the flash wearing and performance.
• Or store each setting entry under unique key (let's say (key << 10 | index). I'm afraid though it'd be hard to keep track of the settings stored with this approach (so that no settings are lost). And it'd require some extra effort when settings are deleted (i. e. the gap would need to be filled with a setting with the last index).

1. A solution proposed in #5943 by @KaSroka - keep using existing OpenThread settings module, but abstract flash API. Instead of writing to flash directly, create an abstract API that would write to a file instead. Should be pretty straightforward to implement, especially that there is a reference implementation for posix already in OpenThread.

2. Use filesystem and store individual settings in individual files - with this approach we could create a hierarchical directory structure to store OpenThread configuration. A top directory, lets say ot would contain a set of directories, which would correspond to specific setting keys. In each directory, we could store files, with unique filenames, that would store the actual setting value. Multiple entries of particular setting would be kept in individual files. We could easily implement settings indexing with this approach, by simply iterating over files in a directory.

Personally I'd go with one of the FS-based solutions, with preference to 4, which I find quite neat. I'd like to hear from others though, to see different opinions or perhaps even more options we can consider.

[rlubos]

FYI @jfischer-phytec-iot @mike-scott @aurel32 @dleach02

[Laczen]

Hi @rlubos, you could use the nvs module and write a little wrapper around it to store string-value pairs:
a. Reserve nvs id's for the strings e.g. everything below 0x8000.
b. Reserve nvs id's for the value's e.g. everything above 0x8000.

The value that belongs to the string at id=0x0001 can be found at id=0x8001. The strings represent the key-index combination.

With some string processing the strings could represent the directory/files of proposal 4.

I have done something similar in PR #12998 to allow support for the nvs backend.

[rlubos]

Hi @Laczen, thank you for your input. I've looked through your PR to see how the NVS settings backend is implemented and it seems that the algorithm you propose in there could find use in OpenThread settings as well. I think that we could even simplify this proposal a little bit for OpenThread and store only the numeric setting key in the bottom part of the NVS id's. As we'd need to crawl through the settings to find desired key anyway, we can handle indexing in a runtime (OpenThread does not require that indexes remain persistent during add/delete operation, so this should be even easier to implement).

The downside I see for this solution, that we need to crawl through the settings stored in order to find specific key. But the fact is, that current OpenThread settings implementation does more or less the same - once specific key is requested, it crawls through the flash area available for settings in order to find it.

I'll add your proposal to the list of possible options, thanks again.

[mtuxpe]

Hi @rlubos , Instead of NVS can you consider nrf RAM retention?
https://www.nordicsemi.com/DocLib/Content/SDK_Doc/nRF5_SDK/v15-2-0/ram_retention_example
NVS is suitable for OpenThread prototype but not for production because expected device lifetime is unknown.

[rlubos]

@mtuxpe I'm not sure if RAM retention is a good candidate for a persistent storage. It's contents are preseved in deep sleep state (SYSTEMOFF in nRF nomenclature), but will be wiped out once the device is powered down.

[rlubos]

I've prepared a proof-of-concept implementation of OpenThread settings based on NFFS filesystem:
https://github.com/rlubos/zephyr/commits/fs-openthread-settings

Unfortunately, all it has proven is that NFFS is not mature enough for real-world usecases. After a few write/delete cycles, NFFS started to misbehave - it was losing the data stored, and eventually ended up in an assertion during initialization.

[Laczen]

Hi @mtuxpe, of course if you can keep the data in ram you should do so. I don't understand why the device lifetime is unknown in case of nvs, nvs does everything to reduce flash wear by not rewriting the same value, only erasing flash pages when required... If you know how much you are going to do updates of the settings you could calculate this quite easily. It is the data that is rewritten that wears out flash. Suppose you have 512 bytes of data that never (or almost never) change (settings names, constant values) and that you have 8 bytes that change frequently then you can calculate the expected lifetime e.g. for a 2 sector system with each sector 1024 bytes big: each write of data takes 16 byte (8 for the data and 8 for nvs: after 512 bytes/16bytes (32) a new sector is started. Since there are 2 sectors you can have 64 writes before each of the sectors is erased. With a lifetime of 10000 erases you can do 64000 writes.

[Laczen]

@rlubos, I have taken a quick look at your proof of concept. I think the idea is good to store the information in a directory/file structure. This should allow you to store the data in whatever filesystem you would like to use. If you would like to use nvs as a "backend" I would suggest that you write a little wrapper that provides functions like fs_opendir, fs_readdir, ... around nvs, where the opendir, readdir,... are in reality just string manipulations of the nvs variable names.

[nwsetec]

@rlubos in your original comment you say regarding Zephyr's settings subsystem:

There even is no API to read a single setting value - the only way to read settings from flash is that bulk settings_load. We could try to bypass this by having a copy of the settings in RAM, but I doubt we want to do this (I don't).

Having a copy of the settings in RAM is how the settings subsystem is used. The Bluetooth settings (bonding keys for up to CONFIG_BT_MAX_PAIRED, device name, maybe more?) are used in this manner.

Often embedded systems have a flash access layer to mirror what is stored in flash in RAM to speed up access. Loading the Openthread settings at power up/reset can be a once only flash read operation then access to the settings is fast via the RAM copy.

The settings subsystem also gives the user the freedom to use FCB or a file as the source/destination of the settings flash storage. Allowing the application writer more flexibility to choose the software components in their system, for example FCB for standard flash storage and nffs and the Simple Management Protocol/MCUMGR for more sophisticated system requirements.

By using the settings subsystem for the Openthread stack a Thread application writer can also use the same settings APIs and storage location as the Openthread stack, Bluetooth stack, etc, minimising specialised requirements and/or extra dependancies for a minimal/optimised Openthread/Zephyr system.

[mike-scott]

@rlubos I agree with @nwsetec here. I have a WIP commit planned for Zephyr v1.15 which introduces persistent storage for the LwM2M subsystem. It uses the settings subsystem: https://github.com/foundriesio/zephyr/commit/a295d863c6581f35502adabefef8f2e0b2929a60

I do still have a few questions regarding my own implementation, but I thought it was relatively painless overall, and it should "play well" with the other subsystems also using storage.

[rlubos]

@nwsetec @mike-scott

Thank you for your feedback.

Havaing a RAM copy in the settings layer is IMO not a good idea for OpenThread due to several reasons:

• It's simply a waste of RAM in case of OpenThread. OT already keeps a copy of specific settings in RAM, in it's internal classes, but it manages it on it's own (again, the design diffrences between OT/Zephyr). So we already have the benefits of RAM access - there's no further benefits of keeping separate copy of them.
• OpenThread settings API allows the settings to be transparent for the underneath storage. With the Zephyr settings module in current form, and a RAM copy, we'd need to track what specific settings OpenThread wants to store and how many of them. Any changes in this matter on OT side, will require changes in the settings layer as well.

If we really want to use Zephyr settings module though, we should rather consider extending it's API to allow readout of a singe key/value pair. There's already an API to store a single entry (settings_save_one), but we miss an option to read one. With such an API, we should be able to implement a layer for OpenThread, that will not bring extra RAM overhead on one side, and remain compatible with other settings users on the other side. @nvlsianpu FYI.

[nwsetec]

@rlubos I agree with you that there should be only one copy of the settings in RAM. Apologies, I hadn't looked through the OpenThread code to see how the settings are being used.

I've read the OpenThread source code a little just now but couldn't confirm, are all the flash reads happening at instance initialisation? If so would it be possible to partially initialise a OpenThread instance before Zephyr's settings_load() function is called to create an equivalent flash read with the settings subsystem callbacks to initialise the instance's fields?

[Laczen]

@rlubos, I agree that using the settings module should be the best method if possible. To me it is also not clear how the flash reads are happening. While looking at your proof of concept code I saw that there are three routines that hint towards a implementation that is close to the settings module:

otError otPlatSettingsBeginChange(otInstance *aInstance)
otError otPlatSettingsCommitChange(otInstance *aInstance)
otError otPlatSettingsAbandonChange(otInstance *aInstance)

Is there a way to apply settings to *aInstance ? So suppose you receive aKey, aIndex, aValue and aValueLength is there a method to apply them to aInstance. If there is such a method you could setup the settings module to apply them for each OT entry found in storage. The saving of settings would just be a settings_save_one().

[nvlsianpu]

@rlubos I have your needs in my mind (and pn the task list as well) - will try to introduce this read on-demand and maybe read to buffer features.

[Laczen]

@rlubos, @nvlsianpu, it is indeed possible to add a settings_read_one() function to the settings module, but...
The way the settings are stored in fcb/file (and probably also other backends) result in: for each settings_read_one() call all stored items (deleted or not) will need to be read and compared if this is the value that is needed. So the loading would probably become very slow...

[rlubos]

@nwsetec I'm not 100% sure either, whether instance initialization is the only place settings are stored. According to you proposal, to initialize OT instance manually with the data read from flash, I'm afraid there's one big obstacle - there're simply no APIs to set all of the data required from the outside. Data like children or parent info can only be updated by the OT stack, or preloaded through OT settings module, there's no way to set them from the outside.

@Laczen No, there is no API to apply specific settings from the outside. That's the design differences I mentioned - OT manages configuration on it's own, requiring only a simple API to store/load data to/from persistant storage.
The unimplemented OT APIs you mention are obsolete and have been removed from OpenThread. I've included them by mistake (they got removed from the header but are still present in OpenThread's settings implementation). But the fact is - they were never used, hence got removed from the API.
And regarding slow performance - that's why I looked for alternatives, because it's another consequence of design mismatch. But if it's desired to use settings module for compability reasons, we need compromises I guess.

[rlubos]

For anyone interested, I've prepared an OpenThread settings API implementation built on top of the Zephyr settings subsystem: https://github.com/rlubos/zephyr/tree/openthread/settings

With this solution, OpenThread settings are identified with keys of the following format: ot/id/instance, where id is assigned by OpenThread stack, and instance is a 32-bit random number, both in hex. The implementation makes use of settings_load_subtree_direct function to iterate over settings instances. This allows for the OpenThread settings layer to be a fully transparent shim layer between OpenThread/Zephyr APIs.

An example settings layout with this approach looks like this:

ot/1/2f00650e
ot/3/1b1ed56e
ot/5/2c2f8d70
ot/5/5f7bbf99

I'm not preparing a PR with this solution yet, as I'm not happy with otPlatSettingsDelete / otPlatSettingsWipe implementation. Currently, what Zephyr provides is only a settings_delete function, which expects an exact setting key value. What I miss here, is a settings_delete_subtree function, that would allow me to delete all instances of a single setting id, or even entire ot tree. For now, I've implemented this functionality with settings_load_subtree_direct (used in this case as a settings iterator), but it's hacky and I'd prefer to add missing functionality into the Zephyr settings first.

[rlubos]

For anyone interested, due to recurring issues with existing OpenThread settings impelmentation (#21780), I've posted a PR with OT settings rework: #21796.

[rlubos]

Resolved by #21796.