Libelektra: Memory Usage Improvements

[markus2330]

I wrote:

I agree there is much to be improved and hope @mpranj will work on this. Currently this is probably more important than CPU optimizations. But there is not much to be decided with one exception: if we should provide both keyName and keyUnescapedName. Did you take a look of how many call-sites of keyName are there? Will it be feasible to get rid of that function and then rename keyUnescapedName to keyName? To get the escaped keyName you then would need something like elektraEncodeKeyName(buffer, keyName(...). This should be okay for printing the keyname? The impact for storage plugins might be not so much, as mainly elektraKeyGetRelativeName would need to be fixed.

@kodebach wrote:

sizeof (Key) IIRC is about 64 bytes. sizeof("system:/elektra/mountpoints/version") == 36 in #3447 the corresponding unescaped name is 30 bytes. That makes 64 bytes for the structure and 66 bytes for the key names. And that is just for "system:/elektra/mountpoints/version", which is not a particularly long key name. In the spec of LCDproc for example every key name starts with spec:/sw/lcdproc/lcdexec/#0/current which is a 35 character prefix (and 31 bytes in unescaped name).

Exactly: is it really 64 bytes? On which architectures? How is the ratio meta-keys to normal keys (with longer names)?

Such questions could be answered with statistics of the mmap files.

Then we can decide if its better to reduce in key names (together with removal of keyUnescapedName which simply passes out the pointer to the user) or in _Key (which also could be reduced by half and would be a much more local change) or somewhere else.

If you remember, I did try to address the significant memory increase in the LCDproc with Elektra version. I didn't really succeed, because most of the increase was key names.

I remember that we tried to reduce the binary sizes which were increased because of the compiled-in spec. And in this case you found out: the key names were the biggest trouble there. In the memory usage evaluation your thesis only shows that Elektra causes a lot of overhead but not why. Btw. I still enjoy to read your thesis, it is very nicely written.

I meant applications that are non-experimental and are in real-world use. Unless I missed some huge news, the default versions of KDE and GNOME still don't use Elektra.

With https://www.libelektra.org/bindings/gsettings and https://github.com/ElektraInitiative/kconfig we can run GNOME&KDE with Elektra and create Elektra's mmap files.

I would also prefer and application that was written with Elektra in mind from the start. Otherwise the application may use Elektra in an unusual way to fulfil some pre-existing constraint or to make the update more seamless. Such setups could mess with the results.

Yes, totally agree! It would be amazing to have an application that used Elektra from the beginning. :rocket: Unfortunately, these applications are at the moment either proprietary or rather small. I doubt they would be useful for memory usage evaluations.

[kodebach]

I remember that we tried to reduce the binary sizes which were increased because of the compiled-in spec. And in this case you found out: the key names were the biggest trouble there. In the memory usage evaluation your thesis only shows that Elektra causes a lot of overhead but not why.

You are right, my mistake. However, there is a good relation between the code sizes of a ksNew (keyNew (...), ...) and the memory usage.

---

In general, I want to say: I just wanted to suggest some ideas. I know we need to do tests and benchmarks to find what actually helps and what doesn't. Please don't take everything I write as a proposal that I want to implement. Often I'm just putting ideas out there.

[markus2330]

As discussed: another idea is to reduce the default size of meta KeySets to 4. I still think, however, we should first analyze the memory consumption before doing such optimizations.

The only "optimization" that does not need necessarily a benchmark is to remove the escaped key name, as this is a huge size difference and effects every key.