Zim-desktop-wiki: Memory leak in spell checker plugin leads to huge memory usage

[Self-Perfection]

Without further ado here is a typical graph of memory usage

RAM usage stays constant when zim is minimized. Apparently new RAM is allocated during interaction with zim, though I was not able to track down specific action that causes leak.

At first I've noticed this behavior with zim 0.65, but considered it a bug specific to my desktop PC. But recently I've upgraded my laptop to Ubuntu 17.04, so it got zim 0.65 too and apparently it has same problem.

Versions 0.66 and 0.67 leaks too.

Maximum memory usage I got since the beginning of tracking zim memory is 3.8 gigabytes. This leak is severe!

Probably my primary notebook size affects leak rate. It is pretty big:

$ find "$NOTEBOOK" -name '*.txt' | wc -l
1568

[jaap-karssenberg]

Just checked the memory footprint on my own system after several days running, but don't recognize this.

Only place that scales with number of pages is the index, but indexing has been re-written from scratch between 0.65 and 0.66, so if issue persist between those releases we can rule out that part.

Could you do a test disabling all plugins you may have in use and see whether that impacts the memory usage?

[Self-Perfection]

@jaap-karssenberg good point! It turns out that the culprit is the spell checker plugin.

It is enough just to switch between different pages to observe the leakage.

[Self-Perfection]

Both my systems use gtkspellcheck as requirement for spell checker plugin.

$ grep 'gtksp\|zim' /var/log/pacman.log | tail -6
[2016-05-09 12:24] [ALPM] upgraded zim (0.65-2 -> 0.65-3)
[2016-09-28 15:26] [ALPM] upgraded zim (0.65-3 -> 0.65-4)
[2017-01-25 20:49] [ALPM] removed python2-gtkspell (2.25.3-13)
[2017-01-25 20:49] [ALPM] installed python2-gtkspellcheck (4.0.5-4)
[2017-05-13 17:39] [ALPM] upgraded zim (0.65-4 -> 0.66-1)
[2017-07-21 13:32] [ALPM] upgraded zim (0.66-1 -> 0.67-1)

I started to notice leak at the end of February - early March. Thus it looks like it switching from gtkspell to gtkspellcheck introduced memory leak but not a new zim version.

So bug might be in gtkspellcheck or in zim somehow incorrectly using gtkspellcheck.

[jaap-karssenberg]

One thing that I see in the code that could use improvement is that we create a new spell checker object every time we load a page. Can imaging this leading to redundant data in memory somehow.

See this method: https://github.com/jaap-karssenberg/zim-desktop-wiki/blob/master/zim/plugins/spell.py#L175

The bug report referred there should be solved in https://github.com/koehlma/pygtkspellcheck/commit/012128940445acf4a0b4ca16d15f9bb9f915d6a7
which is shipped in 4.0.3 and up and according to the log above you have 4.0.5

So probably the method could just read:

def on_new_buffer(self):
    if self._checker:
        self._checker.buffer_initialize()

You can make a copy of the plugin and put it in ~/.local/share/zim/plugins to modify and test. After restarting zim it should use this copy in favor of the default plugin. (Add a print statement if you want to be 100% sure what plugin file is loaded by zim.)

If that doesn't help, we would need more detailed profiling to understand where the memory goes.

[Self-Perfection]

Thank you for quick response!

Preliminary test indicates that your suggestion helped, zim does not leak with suggested change.

Though I'd like to run zim a day or two without restarts to be extra sure that leak is indeed eliminated and there were no negative side effects.

P.S.: It may be just my imagination, but it seems that switching pages became noticeably faster.